Psychometric Evaluation of a Mathematical Problem Solving Instrument for Technical and Vocational Contexts
DOI:
https://doi.org/10.21831/elinvo.v11i1.94826Keywords:
Mathematical problem solving, Partial Credit Model, Vocational Education, Psychometric Validation, Polytomous Scoring, Item Response TheoryAbstract
This study presents a comprehensive psychometric evaluation of a mathematical problem solving instrument specifically designed for technical and vocational education contexts. Addressing critical gaps in vocational mathematics assessment, the research employed a robust mixed methods validation approach combining Item Response Theory, specifically the Partial Credit Model, with classical test theory methods. The instrument comprised 24 polytomous items aligned with Polya's problem solving framework and authentic technical scenarios. Data were collected from 382 vocational students across multiple technical specializations in Central Java, Indonesia. Rasch analysis using QUEST software examined item fit statistics, rating scale functionality, person item targeting, and dimensionality. Results revealed strong item level psychometric properties with item reliability of 0.82, excellent model fit (infit/outfit MNSQ ≈ 1.00), ordered rating scale thresholds exceeding 1.0 logits, and acceptable content validity (Aiken's V: 0.67-0.87). However, person reliability of 0.00 and minimal ability variance (SD = 0.05) indicated limited capacity for individual differentiation, suggesting either sample homogeneity or insufficient difficulty spread. The four level scoring rubric demonstrated distinct performance categories validated through item characteristic curves. This research contributes methodologically by extending polytomous Rasch validation to vocational problem solving assessment while highlighting the critical importance of aligning instrument difficulty with target population characteristics for reliable individual level measurement.
References
[1] F. Mutohhari, S. Sutiman, M. Nurtanto, N. Kholifah, and A. Samsudin, “Difficulties in implementing 21st century skills competence in vocational education learning,” International Journal of Evaluation and Research in Education, vol. 10, no. 4, pp. 1229–1236, 2021, doi: 10.11591/ijere.v10i4.22028.
[2] OECD, PISA 2018 Results (Volume I): What Students Know and Can Do. Paris, France: OECD Publishing, 2019. doi: 10.1787/5f07c754-en.
[3] B. Trilling and C. Fadel, 21st century skills: Learning for life in our times. Jossey-Bass, 2009.
[4] World Economic Forum, “The future of jobs report 2020,” World Economic Forum, 2020.
[5] S. Billett, “Apprenticeship as a mode of learning and model of education,” Education + Training, vol. 58, no. 6, pp. 613–628, 2016, doi: 10.1108/ET-01-2016-0001.
[6] P. H. Cuong, N. H. Son, and D. T. T. Huyen, “A bibliometric review of mathematical problem-solving research trends from 2000 to 2022,” International Journal of Learning, Teaching and Educational Research, vol. 24, no. 1, pp. 142–165, 2025, doi: 10.26803/ijlter.24.1.8.
[7] V. Villarroel, S. Bloxham, D. Bruna, C. Bruna, and C. Herrera-Seda, “Authentic assessment: Creating a blueprint for course design,” Assess. Eval. High. Educ., vol. 43, no. 5, pp. 840–854, 2018, doi: 10.1080/02602938.2017.1412396.
[8] D. Ratnawati, “Linking skills development and employability in Indonesian vocational education,” in Comparative vocational education research: Enduring challenges and new ways forward, M. Pilz and J. Li, Eds., Springer, 2020, pp. 173–189. doi: 10.1007/978-3-658-29924-8_11.
[9] V. Geiger, M. Goos, and H. Forgasz, “A rich interpretation of numeracy for the 21st century: A survey of the state of the field,” ZDM Mathematics Education, vol. 47, no. 4, pp. 531–548, 2015, doi: 10.1007/s11858-015-0708-1.
[10] International Journal of Current Science Research and Review, “Trends in vocational mathematics education,” IJCSRR, vol. 7, no. 3, pp. 1856–1867, 2024.
[11] L. D. English and J. Gainsburg, “Problem solving in a 21st-century mathematics curriculum,” in Handbook of international research in mathematics education, 3rd ed., L. D. English and D. Kirshner, Eds., Routledge, 2016, pp. 313–335.
[12] A. H. Schoenfeld, “Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics (Reprint),” Journal of Education, vol. 196, no. 2, pp. 1–38, 2016, doi: 10.1177/002205741619600202.
[13] NCTM, Principles and Standards for School Mathematics. Virginia: National Council of Teachers of Mathematics, 2000.
[14] G. Polya, How to solve it: A new aspect of mathematical method. Princeton, NJ: Princeton University Press, 1945.
[15] A. H. Schoenfeld, Mathematical problem solving. Academic Press, 1985.
[16] C. E. Hmelo-Silver, “International perspectives on problem-based learning: Contexts, cultures, challenges, and adaptations,” Interdisciplinary Journal of Problem-Based Learning, vol. 6, no. 1, pp. 10–15, 2012, doi: 10.7771/1541-5015.1310.
[17] T. Garcia, C. Rodriguez, P. Gonzalez-Castro, D. Alvarez-Garcia, and J. A. Gonzalez-Pienda, “Psychometric properties and factor structure of the Spanish version of the General Self-Efficacy Scale in university students,” Psicothema, vol. 36, no. 1, pp. 63–70, 2024, doi: 10.7334/psicothema2023.296.
[18] D. Boud and N. Falchikov, Rethinking assessment in higher education: Learning for the longer term. Routledge, 2007.
[19] P. Kloosterman, “Beliefs about mathematics and mathematics learning in the secondary school: Measurement and implications for motivation,” in Beliefs: A hidden variable in mathematics education?, G. C. Leder, E. Pehkonen, and G. Törner, Eds., Springer, 2002, pp. 247–269.
[20] R. Ajjawi, J. Tai, T. L. H. Nghia, D. Boud, L. Johnson, and C. J. Patrick, “Aligning assessment with the needs of work-integrated learning: The challenges of authentic assessment in a complex context,” Assess. Eval. High. Educ., vol. 45, no. 2, pp. 304–316, 2020, doi: 10.1080/02602938.2019.1639613.
[21] J. T. Gulikers, T. J. Bastiaens, and P. A. Kirschner, “A five-dimensional framework for authentic assessment,” Educational Technology Research and Development, vol. 52, no. 3, pp. 67–86, 2004, doi: 10.1007/BF02504676.
[22] T. G. Bond and C. M. Fox, Applying the Rasch model: Fundamental measurement in the human sciences, 3rd ed. Routledge, 2015.
[23] S. E. Embretson and S. P. Reise, Item response theory for psychologists. Lawrence Erlbaum Associates, 2000.
[24] W. J. Boone and J. R. Staver, Advances in Rasch analyses in the human sciences. Springer, 2020.
[25] J. Kim and M. Wilson, “Polytomous item explanatory item response theory models,” Educ. Psychol. Meas., vol. 80, no. 4, pp. 726–755, 2020, doi: 10.1177/0013164419892667.
[26] G. Rasch, Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research, 1960.
[27] L. Tesio, A. Caronni, J. Keller, and A. Simone, “Rasch analysis of the modified Barthel Index: Guttman errors and plausible underlying unidimensionality,” Arch. Phys. Med. Rehabil., vol. 104, no. 11, pp. 1820–1829, 2023, doi: 10.1016/j.apmr.2023.04.008.
[28] D. Andrich, “A rating formulation for ordered response categories,” Psychometrika, vol. 43, no. 4, pp. 561–573, 1978, doi: 10.1007/BF02293814.
[29] G. N. Masters, “A Rasch model for partial credit scoring,” Psychometrika, vol. 47, no. 2, pp. 149–174, 1982, doi: 10.1007/BF02296272.
[30] H. Kang, X. Cheng, and Z. Zhang, “Polytomous item response models for multiple ratings: A review and comparison,” Appl. Psychol. Meas., vol. 46, no. 5, pp. 387–406, 2022, doi: 10.1177/01466216221091405.
[31] M. Lopez and S. Ramos, “Psychometric validation of educational assessment instruments: A systematic review,” Educational Measurement and Evaluation Review, vol. 15, no. 2, pp. 112–135, 2024.
[32] M. Gessler and C. Siemer, “Umbrella review: Methodological review of reviews published in peer-reviewed journals with a substantial focus on vocational education and training research,” International Journal for Research in Vocational Education and Training, vol. 7, no. 1, pp. 91–125, 2020, doi: 10.13152/IJRVET.7.1.5.
[33] V. Villarroel, D. Boud, S. Bloxham, D. Bruna, and C. Bruna, “Using principles of authentic assessment to redesign written examinations and tests,” Innovations in Education and Teaching International, vol. 57, no. 1, pp. 38–49, 2020, doi: 10.1080/14703297.2018.1564882.
[34] Z. Misbah, J. Gulikers, S. Dharma, and M. Mulder, “Evaluating competence-based vocational education in Indonesia,” Journal of Vocational Education & Training, vol. 72, no. 4, pp. 488–515, 2020, doi: 10.1080/13636820.2019.1635634.
[35] D. Suryadi, Landasan filosofis penelitian desain didaktis (DDR) [Philosophical foundations of didactical design research]. Gapura Press, 2019.
[36] M. T. Kane, “Validating the interpretations and uses of test scores,” J. Educ. Meas., vol. 50, no. 1, pp. 1–73, 2013, doi: 10.1111/jedm.12000.
[37] S. Messick, “Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning,” American Psychologist, vol. 50, no. 9, pp. 741–749, 1995, doi: 10.1037/0003-066X.50.9.741.
[38] J. W. Creswell and J. D. Creswell, Research design: Qualitative, quantitative, and mixed methods approaches, 5th ed. SAGE, 2018.
[39] I. Etikan, S. A. Musa, and R. S. Alkassim, “Comparison of convenience sampling and purposive sampling,” American Journal of Theoretical and Applied Statistics, vol. 5, no. 1, pp. 1–4, 2016, doi: 10.11648/j.ajtas.20160501.11.
[40] J. M. Linacre, “Sample size and item calibration stability,” Rasch Measurement Transactions, vol. 7, no. 4, p. 328, 1994.
[41] J. C. Nunnally and I. H. Bernstein, Psychometric theory, 3rd ed. McGraw-Hill, 1994.
[42] M. Wilson, Constructing measures: An item response modeling approach. Lawrence Erlbaum Associates, 2005.
[43] L. R. Aiken, “Three coefficients for analyzing the reliability and validity of ratings,” Educ. Psychol. Meas., vol. 45, no. 1, pp. 131–142, 1985, doi: 10.1177/0013164485451012.
[44] R. K. Hambleton and R. W. Jones, “Comparison of classical test theory and item response theory and their applications to test development,” Educational Measurement: Issues and Practice, vol. 12, no. 3, pp. 38–47, 1993, doi: 10.1111/j.1745-3992.1993.tb00543.x.
[45] L. J. Cronbach, “Coefficient alpha and the internal structure of tests,” Psychometrika, vol. 16, no. 3, pp. 297–334, 1951, doi: 10.1007/BF02310555.
[46] R. J. Adams and S. T. Khoo, QUEST: The interactive test analysis system. Australian Council for Educational Research, 1996.
[47] B. D. Wright and J. M. Linacre, “Reasonable mean-square fit values,” Rasch Measurement Transactions, vol. 8, no. 3, p. 370, 1994.
[48] J. M. Linacre, “What do infit and outfit, mean-square and standardized mean?,” Rasch Measurement Transactions, vol. 16, no. 2, p. 878, 2002.
[49] American Educational Research Association, “Code of ethics,” Educational Researcher, vol. 40, no. 3, pp. 145–156, 2011.
[50] J. M. Linacre, “Optimizing rating scale category effectiveness,” J. Appl. Meas., vol. 3, no. 1, pp. 85–106, 2002.
[51] J. F. Hair, W. C. Black, B. J. Babin, and R. E. Anderson, Multivariate data analysis, 8th ed. Cengage Learning, 2019.
[52] A. H. Schoenfeld, “Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics,” in Handbook of research on mathematics teaching and learning, D. A. Grouws, Ed., Macmillan, 1992, pp. 334–370.
[53] M. Wu and R. Adams, “Applying the Rasch model to psycho-social measurement: A practical approach,” Educational Measurement Solutions, vol. 2, no. 1, pp. 1–24, 2007.
[54] A. Gamoran, “Tracking and inequality: New directions for research and practice,” in The Routledge international handbook of the sociology of education, M. W. Apple, S. J. Ball, and L. A. Gandin, Eds., Routledge, 2009, pp. 213–228.
[55] J. Oakes, Keeping track: How schools structure inequality, 2nd ed. Yale University Press, 2005.
[56] B. D. Wright and M. H. Stone, Best Test Design: Rasch Measurement. Chicago, IL, USA: MESA Press, 1979.
[57] C. Nicol and S. Crespo, “Learning to teach with mathematics textbooks: How preservice teachers interpret and use curriculum materials,” Educational Studies in Mathematics, vol. 62, no. 3, pp. 331–355, 2006, doi: 10.1007/s10649-006-5423-y.
[58] P. Black and D. Wiliam, “Developing the theory of formative assessment,” Educ. Assess. Eval. Account., vol. 21, no. 1, pp. 5–31, 2009, doi: 10.1007/s11092-008-9068-5.
[59] W. J. Popham, Transformative assessment. Association for Supervision and Curriculum Development, 2008.
[60] D. Wiliam, “What is assessment for learning?,” Studies in Educational Evaluation, vol. 37, no. 1, pp. 3–14, 2011, doi: 10.1016/j.stueduc.2011.03.001.
[61] B. D. Zumbo, “Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going,” Lang. Assess. Q., vol. 4, no. 2, pp. 223–233, 2007, doi: 10.1080/15434300701375832.
[62] W. R. Shadish, T. D. Cook, and D. T. Campbell, Experimental and quasi-experimental designs. Houghton Mifflin, 2002.
[63] W. J. van der Linden and C. A. W. Glas, Elements of adaptive testing. Springer, 2010. doi: 10.1007/978-0-387-85461-8.
[64] A. H. Schoenfeld, “Reflections on problem solving theory and practice,” The Mathematics Enthusiast, vol. 10, no. 1–2, pp. 9–34, 2013, doi: 10.54870/1551-3440.1258.
[65] H. P. Ginsburg, Entering the child’s mind: The clinical interview in psychological research and practice. Cambridge University Press, 1997.
Downloads
Published
How to Cite
Issue
Section
Citation Check
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
The article published in ELINVO became ELINVO's right in publication.
This work by ELINVO is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
