Construct validity and reliability of a high-stakes mathematics test for junior high school: A case from the Indonesian national examination
DOI:
https://doi.org/10.21831/pep.v30i1.95822Keywords:
construct validity, high-stake mathematics tests, confirmatory factor analysis, test realibilityAbstract
High-stakes tests are used to inform critical decisions concerning students, teachers, schools, and educational policies. Therefore, they must meet rigorous quality-assurance standards. One key approach to ensuring the quality of high-stakes tests is establishing construct validity and reliability. This study aims to prove the construct validity and estimate the construct reliability for a high-stakes mathematics test for junior high school students. This study uses response data from participants taking the national junior high school mathematics exam in Indonesia. A simple random sampling technique was used to select 300 samples. Data were analyzed using second-order confirmatory factor analysis with the help of the R program. The study results revealed four factors that make up the mathematical ability construct: number, algebra, geometry, and statistics. All manifest variables (18 indicators) confirmed a significant contribution. The model fit indicators are met (chi-square = 140.013, df = 131, p = 0.279, RMSEA = 0.015, CFI = 0.994, TLI = 0.993; SRMR = 0.039), so the construct measured in the high-stakes mathematics test is valid for measuring students’ mathematical abilities. Thus, the test construct has a reliability coefficient of 0.98, reflecting that the construct is reliable for measuring mathematical abilities.
References
Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Brooks/Cole Publishing Company.
Apaydin, C. (2020). Second-order factor analysis of teachers’ correct use of their voices scale in classroom management (TCUVS). European Journal of Education Studies, 6(11), 164–179. https://doi.org/10.5281/zenodo.3627322
Bano, H., Khan, M. S., & Khan, S. I. (2021). Development of valid and reliable mathematics achievement test. Ilkogretim Online - Elementary Education Online, 20(2), 1642–1658. https://doi.org/10.17051/ilkonline.2021.02.184
Demir, C. G., & Keleş, Ö. K. (2021). The impact of high-stakes testing on the teaching and learning processes of mathematics. Journal of Pedagogical Research, 5(2), 119–137. https://doi.org/10.33902/jpr.2021269677
DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). Sage.
Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138
Gore, J., Holmes, K., Smith, M., Fray, L., McElduff, P., Weaver, N., & Wallington, C. (2017). Unpacking the career aspirations of Australian school students: towards an evidence base for university equity initiatives in schools. Higher Education Research & Development, 36(7), 1383–1400. https://doi.org/10.1080/07294360.2017.1325847
Gregory, K., & Clarke, M. (2003). High-stakes assessment in England and Singapore. Theory into Practice, 42(1), 66–74. https://doi.org/10.1207/s15430421tip4201_9
Hair, J. F., Anderson, R. E., Tahtam, R. L., & Black, W. C. (1999). Multivariate data analysis (5th ed.). Pearson Education.
Heubert, J. P. (2000). High-stakes testing opportunities and risks for students of color, English-language learners, and students with disabilities. National Center on Accessing the General Curriculum.
Heubert, J. P., & Hauser, R. M. (1999). High stakes: Testing for tracking, promotion and graduation. National Academy of Sciences.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
Kartianom, K., & Mardapi, D. (2017). The utilization of junior high school mathematics national examination data: A conceptual error diagnosis. Research and Evaluation in Education, 3(2), 163–173. https://doi.org/10.21831/reid.v3i2.18120
Kartowagiran, B., Munadi, S., Retnawati, H., & Apino, E. (2018). The equating of battery test packages of mathematics national examination 2013-2016. SHS Web of Conferences. https://doi.org/10.1051/shsconf/20184200022
Khampirat, B. (2019). A second-order confirmatory factor analysis of the career aspirations scale of female Thai students. International Journal of Learning, Teaching and Educational Research, 18(11), 265–282. https://doi.org/10.26803/ijlter.18.11.16
Kline, R. B. (2016). Principles and practice of structural equation modeling (4th ed.). Guilford Press.
Koch, M. J., & DeLuca, C. (2012). Rethinking validation in complex high-stakes assessment contexts. Assessment in Education: Principles, Policy and Practice, 19(1), 99–116. https://doi.org/10.1080/0969594X.2011.604023
Kyriazos, T. A. (2018). Applied psychometrics: Sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in general. Psychology, 09(08), 2207–2230. https://doi.org/10.4236/psych.2018.98126
Lane, S. (2004). Validity of high-stakes assessment: Are students engaged in complex thinking? Educational Measurement: Issues and Practice, 23(3), 6–14. https://doi.org/10.1111/j.1745-3992.2004.tb00160.x
Marchant, G. J. (2004). What is at stake with high stakes testing? A discussion of issues and research. Ohio Journal of Science, 104(2), 2–7.
Minarechová, M. (2012). Negative impacts of high-stakes testing. Journal of Pedagogy, 3(1), 82–100. https://doi.org/10.2478/v10159-012-0004-x
Moss, P. A. (2007). Reconstructing validity. Educational Researcher, 36(8), 470–476. https://doi.org/10.3102/0013189x07311608
Nichols, S. L., Glass, G. V., & Berliner, D. C. (2012). High-stakes testing and student achievement: Updated analyses with NAEP data. Education Policy Analysis Archives, 20(20), 1–35. https://doi.org/10.14507/epaa.v20n20.2012
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
R Core Team. (2022). R: A language and environment for statistical computing. https://www.r-project.org/
Rafi, I., Retnawati, H., Apino, E., Hadiana, D., Lydiati, I., & Rosyada, M. N. (2023). What might be frequently overlooked is actually still beneficial: Learning from post national-standardized school examination. Pedagogical Research, 8(1), em0145. https://doi.org/10.29333/pr/12657
Retnawati, H. (2016). Analisis kuantitatif instrumen penelitian (panduan peneliti, mahasiswa, dan psikometrian). Parama Publishing.
Retnawati, H., Arliani, E., Apino, E., Djidu, H., Kartianom, N., Santoso, A., & Rasmuin, N. (2021). Mapping of difficult mathematics topics in vocational high schools based on national examination data. Indonesian Journal of Educational Assessment, 3(2), 107–127. https://doi.org/10.26499/ijea.v3i2.83
Retnawati, H., Arlinwibowo, J., & Sulistyaningsih, E. (2017). The students’ difficulties in completing geometry items of national examination. International Journal on New Trends in Education and Their Implications, 8(4), 28–41.
Retnawati, H., Hadi, S., Munadi, S., Hadiana, D., Muhardis, M., Apino, E., Djidu, H., Rafi, I., Yusron, E., & Rosyada, M. N. (2019). When national examination no longer determining graduation, will students accomplish it seriously? Indonesian Journal of Educational Assessment, 2(2), 40–49. https://doi.org/10.26499/ijea.v2i2.34
Retnawati, H., Hadi, S., Nugraha, A. C., Arlinwibowo, J., Sulistyaningsih, E., Djidu, H., Apino, E., & Iryanti, H. D. (2017). Implementing the computer-based national examination in Indonesian schools: The challenges and strategies. Problems of Education in the 21st Century, 75(6), 612–633. https://doi.org/10.33225/pec/17.75.612
Retnawati, H., Kartowagiran, B., Arlinwibowo, J., & Sulistyaningsih, E. (2017). Why are the mathematics national examination items difficult and what is teachers’ strategy to overcome it? International Journal of Instruction, 10(3), 257–276. https://doi.org/10.12973/iji.2017.10317
Revelle, W. (2022). Package ‘psych.’ https://cran.r-project.org/web/packages/psych/psych.pdf
Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A., & King, J. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. The Journal of Educational Research, 99(6), 323–338. https://doi.org/10.3200/JOER.99.6.323-338
Smith, M. L., & Fey, P. (2000). Validity and accountability in high-stakes testing. Journal of Teacher Education, 51(5), 334–344. https://doi.org/10.1177/0022487100051005002
Stobart, G. (2009). Determining validity in national curriculum assessments. Educational Research, 51(2), 161–179. https://doi.org/10.1080/00131880902891305
Stobart, G., & Eggen, T. (2012). High-stakes testing - value, fairness and consequences. Assessment in Education: Principles, Policy and Practice, 19(1), 1–6. https://doi.org/10.1080/0969594X.2012.639191
Suciati, S., Munadi, S., & Sugiman, S. (2022). Estimation of test item parameters with polytomous item response using partial credit model (PCM). Proceedings of the 2nd International Conference on Innovation in Education and Pedagogy (ICIEP 2020), 619, 232–237. https://doi.org/10.2991/assehr.k.211219.042
Tabachnick, B., & Fidell, L. (2013). Using multivariate statistics. Pearson Education.
Published
How to Cite
Issue
Section
Citation Check
License
Copyright (c) 2026 Jurnal Penelitian dan Evaluasi Pendidikan
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
The authors submitting a manuscript to this journal agree that, if accepted for publication, copyright publishing of the submission shall be assigned to Jurnal Penelitian dan Evaluasi Pendidikan. However, even though the journal asks for a copyright transfer, the authors retain (or are granted back) significant scholarly rights.
The copyright transfer agreement form can be downloaded here: [JPEP Copyright Transfer Agreement Form]
The copyright form should be signed originally and sent to the Editorial Office through email to jurnalhepi@uny.ac.id
Jurnal Penelitian dan Evaluasi Pendidikan by http://journal.uny.ac.id/index.php/jpep is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
