Constructing and providing content validity evidence through the Aiken's V index based on the experts' judgments of the instrument to measure mathematical problem-solving skills

Nia Kania; Yaya S. Kusumah; Jarnawi Afgani Dahlan; Elah Nurlaelah; Ferit Gürbüz; Ebenezer Bonyah

doi:10.21831/reid.v10i1.71032

Constructing and providing content validity evidence through the Aiken's V index based on the experts' judgments of the instrument to measure mathematical problem-solving skills

Nia Kania, Universitas Pendidikan Indonesia & Universitas Majalengka, Indonesia
Yaya S. Kusumah, Universitas Pendidikan Indonesia, Indonesia
Jarnawi Afgani Dahlan, Universitas Pendidikan Indonesia, Indonesia
Elah Nurlaelah, Universitas Pendidikan Indonesia, Indonesia
Ferit Gürbüz, Kırklareli University, Turkey
Ebenezer Bonyah, Akenten Appiah Menka University of Skills Training and Entrepreneurial Development, Ghana

10.21831/reid.v10i1.71032

Abstract

Test content-based proof of validity is a type of evidence that supports the validity of a measuring instrument. This research aims to develop a mathematical problem-solving assessment instrument utilizing five experts. This study is classified as developmental research and follows a research design that includes two separate stages: the preliminary design stage and the prototype stage. However, its application is restricted to Prototype 1 and Prototype 2, specifically for expert evaluation. This instrument was designed explicitly for grade VIII students studying mathematics, covering all the topics from the odd semesters. The analysis progressed through three distinct stages—curriculum analysis, content analysis, and context analysis—each contributing to a comprehensive understanding of instructional resources. The study sought to narrow the gap between theoretical knowledge and practical application in mathematics education by incorporating real-world context. Surveys have revealed difficulties in answering mathematical problems, highlighting the need to address gaps in learning to improve competency. The careful and thorough construction of test instruments, considering factors such as validity, established the foundation for creating accurate assessment tools. The content validity assessment by the expert panel, with scores ranging from 0.817 to 0.884 based on the V-Aiken category, confirms that the instrument is vital in assessing students' mathematical problem-solving skills and the implementation of this study yielded many valuable insights for educators and academics. This study helps improve mathematics education resources and evaluations to promote mathematical thinking.

Keywords

assessment practices; classroom assessment; instruction; feedback; validity content

References

Abdullah, A. H., Abidin, N. L. Z., & Ali, M. (2015). Analysis of students’ errors in solving Higher Order Thinking Skills (HOTS) problems for fractions. Asian Social Science, 11(21), 133–142. https://doi.org/10.5539/ass.v11n21p133

Agustan, S., Juniati, D., & Siswono, T. Y. E. (2017). Investigating and analysing prospective teacher’s reflective thinking in solving the mathematical problem: A case study of female-field dependent (FD) prospective teacher. 040018. https://doi.org/10.1063/1.4983956

Apino, E., & Retnawati, H. (2016). International Conference on Recent Trends in Physics 2016 (ICRTP2016). Journal of Physics: Conference Series, 755(1), 011001. https://doi.org/10.1088/1742-6596/755/1/011001

Boonen, A. J. H., Reed, H. C., Schoonenboom, J., & Jolles, J. (2016). It is not a math lesson-we are learning to draw! teachers’ use of visual representations in instructing word problem solving in sixth grade of elementary school. Frontline Learning Research, 4(5), 34–61. https://doi.org/10.14786/flr.v4i5.245

Botelho, A., Baral, S., Erickson, J. A., Benachamardi, P., & Heffernan, N. T. (2023). Leveraging natural language processing to support automated assessment and feedback for student open responses in mathematics. Journal of Computer Assisted Learning, 39(3), 823–840. https://doi.org/10.1111/jcal.12793

Buabeng, İ., Antingane, A. B., & Amoako, İ. (2019). Practices, Challenges and Perceived Influence of Classroom Assessment on Mathematics Instruction. International Journal of Assessment Tools in Education, 6(3), 476–486. https://doi.org/10.21449/ijate.616617

Çekmez, E., & Bülbül, B. Ö. (2018). An example of the use of dynamic mathematics software to create problem-solving environments that serve multiple purposes. Interactive Learning Environments, 26(5), 654–663. https://doi.org/10.1080/10494820.2017.1385029

Danczak, S. M., Thompson, C. D., & Overton, T. L. (2020). Development and validation of an instrument to measure undergraduate chemistry students’ critical thinking skills. Chemistry Education Research and Practice, 21(1), 62–78. https://doi.org/10.1039/C8RP00130H

del Olmo‐Muñoz, J., González‐Calero, J. A., Diago, P. D., Arnau, D., & Arevalillo‐Herráez, M. (2022). Using intra‐task flexibility on an intelligent tutoring system to promote arithmetic problem‐solving proficiency. British Journal of Educational Technology, 53(6), 1976–1992. https://doi.org/10.1111/bjet.13228

Desha, C., Caldera, S., & Hutchinson, D. (2021a). Exploring the development of context appreciation in coursework that targets problem-solving for sustainable development. International Journal of Sustainability in Higher Education, 22(5), 1186–1224. https://doi.org/10.1108/IJSHE-01-2020-0024

Desha, C., Caldera, S., & Hutchinson, D. (2021b). Exploring the development of context appreciation in coursework that targets problem-solving for sustainable development. International Journal of Sustainability in Higher Education, 22(5), 1186–1224. https://doi.org/10.1108/IJSHE-01-2020-0024

Dewi, N. R., & Kusumah, Y. S. (2014). Developing Test of High Order Mathematical Thinking Ability in Integral Calculus Subject. International Journal of Education and Research, 2(12), 101–108.

English, L. D. (2023). Ways of thinking in STEM-based problem solving. ZDM – Mathematics Education, 55(7), 1219–1230. https://doi.org/10.1007/s11858-023-01474-7

Fathiyah, K. N., Alsa, A., & Setiyawati, D. (2019). Psychometric characteristic of positive affect scale within the academic setting. Research and Evaluation in Education, 5(2), 120–129. https://doi.org/10.21831/reid.v5i2.25992

García, T., Boom, J., Kroesbergen, E. H., Núñez, J. C., & Rodríguez, C. (2019). Planning, execution, and revision in mathematics problem solving: Does the order of the phases matter? Studies in Educational Evaluation, 61, 83–93. https://doi.org/10.1016/j.stueduc.2019.03.001

Guzmán, B., Rodríguez, C., Ferreira, R. A., & Hernández-Cabrera, J. A. (2021). Psychometric properties of the revised child mathematics anxiety questionnaire (CMAQ-R) for Spanish speaking children. Psicologia Educativa, 27(2), 115–122. https://doi.org/10.5093/psed2020a17

Haeruddin, Prasetyo, Z. K., & Supahar. (2020). The development of a metacognition instrument for college students to solve physics problems. International Journal of Instruction, 13(1), 767–782. https://doi.org/10.29333/iji.2020.13149a

Hollenstein, L., Thurnheer, S., & Vogt, F. (2022). Problem Solving and Digital Transformation: Acquiring Skills through Pretend Play in Kindergarten. Education Sciences, 12(2). https://doi.org/10.3390/educsci12020092

Hoseana, J. (2024). Using Gardner’s Three-Squares Problem for a Group Project in a Mathematical Problem Solving Module. The Mathematics Enthusiast, 21(1–2), 467–484. https://doi.org/10.54870/1551-3440.1638

Hourigan, M., & Leavy, A. M. (2023). Elementary teachers’ experience of engaging with Teaching Through Problem Solving using Lesson Study. Mathematics Education Research Journal, 35(4), 901–927. https://doi.org/10.1007/s13394-022-00418-w

Imran, S., Yasmeen, R., & Mansoor, M. (2024). Development and validation of self-assessment instrument to measure the digital professionalism of healthcare professionals using social media. BMC Medical Education, 24(1), 243. https://doi.org/10.1186/s12909-024-05142-6

Jiang, R., Liu, R. de, Star, J., Zhen, R., Wang, J., Hong, W., Jiang, S., Sun, Y., & Fu, X. (2021). How mathematics anxiety affects students’ inflexible perseverance in mathematics problem-solving: Examining the mediating role of cognitive reflection. British Journal of Educational Psychology, 91(1), 237–260. https://doi.org/10.1111/bjep.12364

Juniati, D., & Budayasa, I. K. (2022). The Influence of Cognitive and Affective Factors on the Performance of Prospective Mathematics Teachers. European Journal of Educational Research, 11(3), 1379–1391. https://doi.org/10.12973/eu-jer.11.3.1379

Kania, N., Kusumah, Y. S., Dahlan, J. A., & Nurlaelah, E. (2024). Dissertation Draft. Universitas Indonesia.

Kennedy-Clark, S. (2013). Reflection: Research by design: Design-based research and the higher degree research student. Journal of Learning Design, 6(2), 26–32. https://doi.org/10.5204/jld.v8i3.257

Kim, D., Ryoo, D., & Lee, M. (2024). Development of a Scale to Measure the Parental Competency of Science-Gifted Students in South Korea. Journal of Advanced Academics, 35(1), 125–155. https://doi.org/10.1177/1932202X231219277

Lee, C., & Ward-Penny, R. (2022). Agency and fidelity in primary teachers’ efforts to develop mathematical resilience. Teacher Development, 26(1), 75–93. https://doi.org/10.1080/13664530.2021.2006768

Ling, M. K. D., & Loh, S. C. (2023). Relationships between cognitive pattern recognition and specific mathematical domains in mathematics education. International Journal of Mathematical Education in Science and Technology, 54(2), 159–179. https://doi.org/10.1080/0020739X.2021.1949059

Makarova, O., Ldokova, G., & Egorova, R. (2021). Analysis of students’ views of the quality of pedagogical education in Russia. International Journal of Education in Mathematics, Science and Technology, 9(3), 462–481. https://doi.org/10.46328/IJEMST.1624

Malepa-Qhobela, M., & Mosimege, M. (2022). A framework to assist mathematics teachers in integrating problem solving in secondary school classrooms. Issues in Educational Research, 32(4), 1486–1508. https://doi.org/10.3316/informit.806857886364491

Net, W. W. W. P. (2023). The quality of mathematical proficiency in solving geometry problem: difference cognitive independence and motivation. Pegem Journal of Education and Instruction, 13(3), 255–266. https://doi.org/10.47750/pegegog.13.03.27

Öztürk, M., Akkan, Y., & Kaplan, A. (2020). Reading comprehension, Mathematics self-efficacy perception, and Mathematics attitude as correlates of students’ non-routine Mathematics problem-solving skills in Turkey. International Journal of Mathematical Education in Science and Technology, 51(7), 1042–1058. https://doi.org/10.1080/0020739X.2019.1648893

Pambudi, D. S., Budayasa, I. K., & Lukito, A. (2020). The Role of Mathematical Connections in Mathematical Problem Solving. Jurnal Pendidikan Matematika, 14(2), 129–144. https://doi.org/10.22342/jpm.14.2.10985.129-144

Pastore, S. (2023). Italian teachers’ conceptions of large-scale assessment: what are the implications for assessment literacy? Teacher Development, 27(2), 153–171. https://doi.org/10.1080/13664530.2023.2177719

Plessis, J. Du, & Ewing, B. (2017). Reasonable Adjustments in Learning Programs: Teaching Length, Mass and Capacity to Students with Intellectual Disability. Universal Journal of Educational Research, 5(10), 1795–1805. https://doi.org/10.13189/ujer.2017.051018

Plomp, T., & Nieveen, N. (2013). Educational design research: An introduction. Educational design research. 11–50.

Polya, G. (1973). How To Solve It Mathematical Method. In New Jersey: Princeton Univeversity Press. (2nd ed.).

Pongsakdi, N. (2020). What makes mathematical word problem solving challenging? Exploring the roles of word problem characteristics, text comprehension, and arithmetic skills. ZDM - Mathematics Education, 52(1), 33–44. https://doi.org/10.1007/s11858-019-01118-9

Purnomo, E. A., Sukestiyarno, Y. L., Junaedi, I., & Agoestanto, A. (2022). Analysis of Problem Solving Process on HOTS Test for Integral Calculus. Mathematics Teaching-Research Journal, 14(1), 199–214.

Raja, K. S., & Shirley, M. C. (2022). Formative assessment tools for effective classroom. I-Manager’s Journal on School Educational Technology, 17(4), 1. https://doi.org/10.26634/jsch.17.4.18926

Ramdani, R., Syamsuddin, A., & Sirajuddin, S. (2019). Development of Mathematical Module-Problem Solving Approach to Train Student’s Reflective Thinking. Pedagogical Research, 4(4). https://doi.org/10.29333/pr/5861

Rao, N. J., & Banerjee, S. (2023). Classroom Assessment in Higher Education. Higher Education for the Future, 10(1), 11–30. https://doi.org/10.1177/23476311221143231

Retnawati, H. (2014). Membuktikan Validitas Instrumen Penelitian. 31. http://staffnew.uny.ac.id/upload/132255129/pengabdian/7 Validitas Instrumen-alhamdulllah.pdf

Retnawati, H. (2016). Analisis kuantitatif instrumen penelitian (panduan peneliti, mahasiswa, dan psikometrian). Parama publishing.

Retnawati, H. (2018). Teachers’ knowledge about higher-order thinking skills and its learning strategy. Problems of Education in the 21st Century, 76(2), 215–230. https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85045919764&origin=inward

Ryoo, J. H., Park, S., Suh, H., Choi, J., & Kwon, J. (2022). Development of a New Measure of Cognitive Ability Using Automatic Item Generation and Its Psychometric Properties. SAGE Open, 12(2), 215824402210950. https://doi.org/10.1177/21582440221095016

Saadati, F., & Celis, S. (2022). Student Motivation in Learning Mathematics in Technical and Vocational Higher Education: Development of an Instrument. International Journal of Education in Mathematics, Science and Technology, 11(1), 156–178. https://doi.org/10.46328/ijemst.2194

Saadati, F., Chandia, E., Cerda, G., & Felmer, P. (2023). Self-efficacy, practices, and their relationships; the impact of a professional development program for mathematics teachers. Journal of Mathematics Teacher Education, 26(1), 103–124. https://doi.org/10.1007/s10857-021-09523-2

Sachin, D. (2021). THEORETICAL BASIS FOR EVOLVING TAXONOMY OF THE AFFECTIVE DOMAIN. I-Manager’s Journal on Educational Psychology, 15(2), 10. https://doi.org/10.26634/jpsy.15.2.17982

Schöbel, S. M., Janson, A., & Leimeister, J. M. (2023). Gamifying Online Training in Management Education to Support Emotional Engagement and Problem-solving Skills. Journal of Management Education, 47(2), 166–203. https://doi.org/10.1177/10525629221123287

Seepiwsiw, K., & Seehamongkon, Y. (2023). The Development of Mathematical Problem-Solving and Reasoning Abilities of Sixth Graders by Organizing Learning Activities Using Open Approach. Journal of Education and Learning, 12(4), 42. https://doi.org/10.5539/jel.v12n4p42

Spinney, J. E. L., & Kerr, S. E. (2023). Students’ Perceptions of Choice-based Assessment. Journal of the Scholarship of Teaching and Learning, 23(1), 46–58. https://doi.org/10.14434/josotl.v23i1.31471

Sutiarso, S., Rosidin, U., & Sulistiawan, A. (2022). Developing Assessment Instrument Using Polytomous Response in Mathematics. European Journal of Educational Research, 11(3), 1441–1462. https://doi.org/10.12973/eu-jer.11.3.1441

Tahili, M. H., Tolla, I., Ahmad, M. A., Saman, A., & Samad, S. (2021). The Effect of Strategic Collaboration Approach on the National Educational Standards Achievement and Service Quality in Basic Education at Local Government in Indonesia. Research in Social Sciences and Technology, 6(1), 53–82. https://doi.org/10.46303/ressat.2021.4

Tao, Y., Wu, F., Zhang, J., & Yang, X. (2023). Constructing a classroom observation instrument of creative potential for primary school students. Thinking Skills and Creativity, 49, 101317. https://doi.org/10.1016/j.tsc.2023.101317

Turing, A. M. (1954). Solvable and unsolvable problems. Penguin Book.

Ulitzsch, E., He, Q., & Pohl, S. (2022). Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks. Journal of Educational and Behavioral Statistics, 47(1), 3–35. https://doi.org/10.3102/10769986211010467

Verschaffel, L., Schukajlow, S., Star, J., & Van Dooren, W. (2020). Word problems in mathematics education: a survey. ZDM, 52(1), 1–16. https://doi.org/10.1007/s11858-020-01130-4

Vygotsky, L. (1994). The problem of the environment. The vygotsky reader.

Wihardjo, R. S. D., Muktiono, A., Ramadhan, S., Salman, I., & Hadiana, D. (2023). The Analysis of Instrument Quality to Measure Graduate Students’ Higher Order Thinking Skill in Environmental Education Learning. International Journal of Membrane Science and Technology, 10(3), 277–283. https://doi.org/10.15379/ijmst.v10i3.1528

Yigletu, A., Michael, K., & Atnafu, M. (2023). Professional development on assessment for learning and its effect on pre-service teacher’s self-regulated learning. Cogent Education, 10(1). https://doi.org/10.1080/2331186X.2023.2222875

Zeighami, M., Malakoutikhah, A., Shahrbabaki, P. M., Al-Oliamat, K., & Dehghan, M. (2024). Development and validation of the nurses sexual harassment scale in Iran. BMC Nursing, 23(1), 107. https://doi.org/10.1186/s12912-024-01759-6

DOI: https://doi.org/10.21831/reid.v10i1.71032