A psychometric evaluation of an item bank for an English reading comprehension tool using Rasch analysis

Louis Wai Keung Yim; Che Yee Lye; Poh Wee Koh

doi:10.21831/reid.v10i1.65284

A psychometric evaluation of an item bank for an English reading comprehension tool using Rasch analysis

Louis Wai Keung Yim, Singapore Examinations and Assessment Board, Singapore
Che Yee Lye, Singapore University of Social Sciences, Singapore
Poh Wee Koh, Singapore Examinations and Assessment Board, Singapore

10.21831/reid.v10i1.65284

Abstract

This study reports the psychometric evaluation of an item bank for an Assessment for Learning (AfL) tool to assess primary school students’ reading comprehension skills. A pool of 46 primary 1 to 6 reading passages and their accompanying 522 multiple choice and short answer items were developed based on the Progress in International Reading Literacy Study (PIRLS) assessment framework. They were field-tested at 27 schools in Singapore involving 9834 students aged between 7 and 13. Four main comprehension processes outlined in PIRLS were assessed: focusing on and retrieving explicitly stated information, making straightforward inferences, interpreting and integrating ideas and information, and evaluating and critiquing content and textual elements. Rasch analysis was employed to examine students’ item response patterns for (1) model and item fit; (2) differential item functioning (DIF) about gender and test platform used; (3) local item dependence (LID) within and amongst reading passages; and (4) distractor issues about options within the multiple-choice-type items. Results showed that the data adequately fit the unidimensional Rasch model across all test levels with good internal consistency. Psycho metric issues found amongst items were primarily related to ill-functioning distractors and local dependence on items. Problematic items identified were reviewed and subsequently amended by a panel of assessment professionals for future recalibration. This psychometrically and theoretically sound item bank is envisaged to be valuable to developing comprehensive classroom AfL tools that provide information for the English reading comprehension instructional design in the Singaporean context.

Keywords

Rasch analysis; construct validity; differential item functioning; local item dependence; English reading comprehension

Full Text:

PDF

References

Andrich, D., & Marais, I. (2019). A course in Rasch measurement theory. Springer texts in education. Springer Singapore. https://doi.org/10.1007/978-981-13-7496-8_4

Bond, T. G. (2003). Validity and assessment: A Rasch measurement perspective. Metodologia de las Clecias del Comportamientro, 5(2), 179-194.

Bond, T. G., & Fox. C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge. https://doi.org/10.4324/9781315814698

Cantó-Cerdán, M., Cacho-Martínez, P., Lara-Lacárcel, F., & García-Muñoz, Á. (2021). Rasch analysis for development and reduction of Symptom Questionnaire for Visual Dysfunctions (SQVD). Scientific Reports, 11(1), 14855. https://doi.org/10.1038/s41598-02194166-9

Christensen, K. B., Makransky, G., & Horton, M. (2017). Critical values for Yen’s Q 3: Identification of local dependence in the Rasch model using residual correlations. Applied psychological measurement, 41(3), 178-194. https://doi.org/10.1177/0146621616677520

Fan, J., & Bond, T. (2019). Applying Rasch measurement in language assessment. In V. Aryadoust & M. Raquel (Eds.), Quantitative data analysis for language assessment volume I: Fundamental techniques (pp. 83-102). Routledge. https://www.taylorfrancis.com/chapters/edit/10.4324/9781315187815-5/applying-raschmeasurement-language-assessment-jason-fan-trevor-bond

Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87(6), 1082–1116. https://doi.org/10.3102/0034654317726529

Hagell, P., & Westergren, A. (2016). Sample size and statistical conclusions from tests of fit to the Rasch Model according to the Rasch Unidimensional Measurement Model (RUMM) Program in Health Outcome Measurement. Journal of Applied Measurement, 17(4), 416 – 431.

Hwang, G. J., & Wu, P. H. (2014). Applications, impacts, and trends of mobile technologyenhanced learning: A review of 2008–2012 publications in selected SSCI journals. International Journal of Mobile Learning and Organisation, 8(2), 83-95. https://doi.org/10.1504/IJMLO.2014.062346

Korhonen, J., Linnanmäki, K., & Aunio, P. (2014). Learning difficulties, academic well-being, and educational dropout: A person-centred approach. Learning and Individual Differences, 31, 1-10. https://doi.org/10.1016/j.lindif.2013.12.011

Livingstone, S., Mascheroni, G., & Staksrud, E. (2018). European research on children’s internet use: Assessing the past and anticipating the future. New Media & Society, 20(3), 1103-1122. https://doi.org/10.1177/1461444816685930

Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14(4), 5-8. https://doi.org/10.1111/j.17453992.1995.tb00881.x

Messick, S. (1994). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. ETS Research Report Series, 1994(2), i-28. https://doi.org/10.1002/j.2333-8504.1994.tb01618.x

Ministry of Education of Singapore. (2020). Pedagogy: Teaching and learning English. In English language syllabus – Primary foundation English - Secondary normal (technical) course. Curriculum Planning and Development Division (CPDD), pp. 30–32. https://www.moe.gov.sg//media/files/secondary/syllabuses-nt/eng/felnt_els-2020_syllabus.pdf

Mugnaini, D., Lassi, S., La Malfa, G., & Albertini, G. (2009). Internalizing correlates of dyslexia. World Journal of Pediatrics, 5, 255-264. https://doi.org/10.1007/s12519-009-0049-7

Mullis, I. V. S., & Martin, M. O. (Eds.). (2021). PIRLS 2021 Assessment frameworks. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: https://pirls2021.org/frameworks/wpcontent/uploads/sites/2/2019/04/P21_FW_Ch1_Assessment.pdf

Oakley, G. (2011). The assessment of reading comprehension cognitive strategies: Practices and perceptions of Western Australian teachers. The Australian Journal of Language and Literacy, 34(3), 279-293. https://doi.org/10.1007/BF03651863

OECD. (2016). PISA 2015 results (Volume I): Excellence and equity in education. OECD Publishing. https://doi.org/10.1787/9789264266490-en

Pallant, J. F., & Tennant, A. (2007). An introduction to the Rasch measurement model: An example using the Hospital Anxiety and Depression Scale (HADS). British Journal of Clinical Psychology, 46(1), 1-18. https://doi.org/10.1348/014466506X96931

Pearson, P. D., & Hamm, D. N. (2005). The assessment of reading comprehension: A review of practices—Past, present, and future. In S. G. Paris & S. A. Stahl (Eds.), Children's reading comprehension and assessment, 31-88. https://doi.org/10.4324/9781410612762

Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen Danmarks Paedagogiske Institut.

Ravand, H., & Firoozi, T. (2016). Examining the construct validity of the Master’s UEE using the Rasch model and the six aspects of Messick’s framework. International Journal of Language Testing, 6(1), 1-23. https://www.ijlt.ir/article_114414.html

RAND Reading Study Group (2002). Reading for understanding: Toward an R & D program in reading comprehension. RAND Corporation. https://www.rand.org/pubs/monograph_reports/MR1465.html

Smith, R., Snow, P., Serry, T., & Hammond, L. (2021). The role of background knowledge in reading comprehension: A critical review. Reading Psychology, 42(3), 214-240. https://doi.org/10.1080/02702711.2021.1888348

DOI: https://doi.org/10.21831/reid.v10i1.65284