Analysis of Computerized Adaptive Test to Reveal Misconceptions and Science Literacy in Science Courses
DOI:
https://doi.org/10.21831/jipi.v12i1.94641Keywords:
Science literacy, Misconceptions, Computerized Adaptive Test (CAT), Content validity, ReliabilityAbstract
The development of diagnostic instruments that accurately capable in assessing interdisciplinary science literacy and identifying misconceptions represents a strategic imperative in higher education, particularly within integrated science courses that demand concept transfer across physics, chemistry, biology, and earth sciences. The study advanced both theoretical and methodological frontiers by developing and psychometrically validating a Computerized Adaptive Test (CAT) instrument, which is grounded in a Four-Tier Diagnostic Test framework and designed to profile students’ science literacy and misconception patterns in integrated science contexts. The study employed a quantitative instrument development design with 150 students of science education from three universities in Aceh, Indonesia. The study moved beyond Classical Test Theory by utilizing Item Response Theory (IRT), specifically the Rasch model and two-parameter logistic (2PL) model to evaluate item parameters essential for CAT calibration. The findings demonstrated strong psychometric properties, with Rasch infit and outfit statistics within acceptable ranges confirming unidimensional, while item difficulty parameters spanned from 2.1 to 1.8 logits, providing adequate ability continuum coverage. Critically, domain-specific analysis revealed that items requiring cross-disciplinary concept transfer, particularly those integrating physics and biology, exhibited significantly higher discrimination parameters than items confined to isolated disciplinary content, offering a novel theoretical insight that science literacy is inherently an integrative construct rather than a sum of disciplinary knowledge components. Methodologically, this study advances CAT development by demonstrating that rigorous IRT-based calibration and iterative quality control are essential for ensuring measurement accuracy. The identification of invalid items underscores that item attrition is a necessary feature of responsible test development. Generally, these findings contribute to the broader movement toward adaptive, personalised assessment in higher education, providing a replicable model for researchers and practitioners seeking to leverage CAT technologies to enhance diagnostic precision and support targeted remediation in interdisciplinary science education worldwide.
References
Amelia, R. N., Listiaji, P., Dewi, N. R., Heriyanti, A. P., Atmaja, B. D., Shoba, T. M., & Sajidi, I. (2024). Developing and Validating a Rubric for Measuring Skills in Designing Science Experiments for Prospective Science Teachers. Jurnal Inovasi Pendidikan IPA, 10(1), 32–46. https://doi.org/10.21831/jipi.v10i1.65853
Choi, Y., & McClenen, C. (2020). Development of Adaptive Formative Assessment System Using Computerized Adaptive Testing and Dynamic Bayesian Networks. Applied Sciences, 10(22), 8196. https://doi.org/10.3390/app10228196
Edelsbrunner, P. A., Simonsmeier, B. A., & Schneider, M. (2025). The Cronbach’s Alpha of Domain-Specific Knowledge Tests Before and After Learning: A Meta-Analysis of Published Studies. Educational Psychology Review, 37(1), 4. https://doi.org/10.1007/s10648-024-09982-y
Embretson, S. E., & Reise, S. P. (2013). Item response theory for psychologists. Psychology Press.
Fahmi, F., Chalisah, N., Istyadji, M., Irhasyuarna, Y., & Kusasi, M. (2022). Scientific literacy on the topic of light and optical instruments in the innovation of science teaching materials. Jurnal Inovasi Pendidikan IPA, 8(2), 154–163. https://doi.org/10.21831/jipi.v8i2.41343
Fraenkel, J. R., & Wallen, N. E. (1990). How to design and evaluate research in education. ERIC.
Gurel, D. K., Eryılmaz, A., & McDermott, L. C. (2015). A review and comparison of diagnostic instruments to identify students’ misconceptions in science.
Hartono, A., Djulia, E., Hasruddin, H., & Jayanti, U. N. A. D. (2023). Biology Students’ Science Literacy Level on Genetic Concepts. Jurnal Pendidikan IPA Indonesia, 12(1), 146–152. https://doi.org/10.15294/jpii.v12i1.39941
Ishtiaq Ahmed, & Sundas Ishtiaq. (2021). Reliability and Validity: Importance in medical research. Journal of the Pakistan Medical Association, 71(10), 2401–2406. https://doi.org/10.47391/JPMA.06-861
Isnaini, F., Tiur, H., Silitonga, M., Musa, M., Hidayatullah, S., Sirait, J., & Afrizon, R. (2025). Diagnosing Students’ Problem-Solving Challenges in Rotational Dynamics Using Two-Tier AR Flashcard Tests. Jurnal Inovasi Pendidikan IPA, 11(2), 402–417. https://doi.org/10.21831/jipi.v11i2.80461
Istiyono, E., Dwandaru, W. S. B., Fenditasari, K., Ayub, M. R. S. S. N., & Saepuzaman, D. (2023). The Development of a Four-Tier Diagnostic Test Based on Modern Test Theory in Physics Education. European Journal of Educational Research, volume-12-2023(volume-12-issue-1-january-2023), 371–385. https://doi.org/10.12973/eu-jer.12.1.371
Kaltakci-Gurel, D., Eryilmaz, A., & McDermott, L. C. (2017). Development and application of a four-tier test to assess pre-service physics teachers’ misconceptions about geometrical optics. Research in Science & Technological Education, 35(2), 238–260. https://doi.org/10.1080/02635143.2017.1310094
Lestari, S. (2021). pengaruh model pembelajaran peer led guided inquiry terhadap kompetensi literasi sains ditinjau dari kemampuan akademik. Jurnal Inovasi Pendidikan IPA, 7(1). https://doi.org/10.21831/jipi.v7i1.29845
Maison, M., Lestari, N., & Widaningtyas, A. (2020). Identifikasi Miskonsepsi Siswa Pada Materi Usaha Dan Energi. Jurnal Penelitian Pendidikan IPA, 6(1), 32–39. https://doi.org/10.29303/jppipa.v6i1.314
Nasyidiah, F. I., Siahaan, P., & Sasmita, D. (2020). PENGEMBANGAN INSTRUMEN FOUR-TIER DIAGNOSTIC TEST UNTUK MENDETEKSI MISKONSEPSI SISWA KELAS X PADA MATERI IMPULS. WaPFi (Wahana Pendidikan Fisika), 5(2), 31–40. https://doi.org/10.17509/wapfi.v5i2.27156
Ni’mah, F. (2019). Research trends of scientific literacy in Indonesia: Where are we? Jurnal Inovasi Pendidikan IPA, 5(1), 23–30. https://doi.org/10.21831/jipi.v5i1.20862
Nunnally, J., & Bernstein, I. (1994). Psychometric Theory 3rd edition (MacGraw-Hill, New York).
Nurhidayatulah, N., & Prodjosantoso, A. K. (2018). Miskonsepsi materi larutan penyangga. Jurnal Inovasi Pendidikan IPA, 4(1), 41–51. https://doi.org/10.21831/jipi.v4i1.10029
OECD. (2023). PISA 2022 Assessment and Analytical Framework, PISA (PISA, Tran.). OECD Publishing. https://doi.org/10.1787/dfe0bf9c-en
Oladele, J. I., & Ndlovu, M. (2021). A Review of Standardised Assessment Development Procedure and Algorithms for Computer Adaptive Testing: Applications and Relevance for Fourth Industrial Revolution. International Journal of Learning, Teaching and Educational Research, 20(5), 1–17. https://doi.org/10.26803/ijlter.20.5.1
Önder Çelikkanlı, N., & Kızılcık, H. (2022). A review of studies about four-tier diagnostic tests in physics education. Journal of Turkish Science Education, 19(4).
Rahim, A., Hadi, S., Susilowati, D., Marlina, & Muti’ah. (2023). Developing of Computerized Adaptive Test (CAT) Based on a Learning Management System in Mathematics Final Exam for Junior High School. International Journal of Educational Reform. https://doi.org/10.1177/10567879231211297
Rohmadhani, I. A. N., Susilo, H., & Lestari, U. (2021). Identification misconceptions using Movement and Circulatory System Diagnostic Test (MCSD-Test) in XI class SMA/MA in East Java. Journal of Physics: Conference Series, 1918(5). https://doi.org/10.1088/1742-6596/1918/5/052082
Rusilowati, A., Susanti, R., Sulistyaningsih, T., Asih, T. S. N., Fiona, E., & Aryani, A. (2021). Identify misconception with multiple choice three tier diagnostik test on newton law material. Journal of Physics: Conference Series, 1918(5). https://doi.org/10.1088/1742-6596/1918/5/052058
Suparno, P. (2013). Miskonsepsi dan perubahan konsep dalam pendidikan fisika. Grasindo.
Susongko, P., Abdul Wahab, N. B., Arfiani, Y., & Kusuma, M. (2024). Validation and Implementation of 3-Dimensional Scientific Literacy Test (Lisa3D Test): Measuring Scientific Literacy for Senior High School Students based on Scientific Reasoning, Scientific Inquiry, and Nature of Science. Jurnal Pendidikan IPA Indonesia, 13(3). https://doi.org/10.15294/591rx526
Taqwim, M. A., Sunarno, W., & Ramli, M. (2022). Remediation using SSCS model for reducing misconceptions about work and energy. Jurnal Inovasi Pendidikan IPA, 8(2), 210–223. https://doi.org/10.21831/jipi.v8i2.49343
Ventura-León, J., & Peña-Calero, B. N. (2020). El mundo no debería girar alrededor del alfa de Cronbach ≥ ,70. Adicciones, 33(4), 369–372. https://doi.org/10.20882/adicciones.1576
Wauters, K., Desmet, P., & Van Den Noortgate, W. (2010). Adaptive item‐based learning environments based on the item response theory: Possibilities and challenges. Journal of Computer Assisted Learning, 26(6), 549–562.
Widarti, H. R., Nuriyanti, D., Sari, M. E. F., Wiyarsi, A., Yatimah, S., & Rokhim, D. A. (2024). Identification of learning difficulties and misconceptions of chemical bonding material: A review. Ecletica Quimica, 49. https://doi.org/10.26850/1678-4618.eq.v49.2024.e1508
Downloads
Published
How to Cite
Issue
Section
Citation Check
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
The authors submitting a manuscript to this journal agree that, if accepted for publication, copyright publishing of the submission shall be assigned to Jurnal Inovasi Pendidikan IPA (JIPI). However, even though the journal asks for a copyright transfer, the authors retain (or are granted back) significant scholarly rights.
Jurnal Inovasi Pendidikan IPA by http://journal.uny.ac.id/index.php/jipi/index is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
