Usability assessment of a HOTS evaluation tool in Pancasila-based Citizenship Education
Downloads
The goal of this study is to use the System Usability Scale (SUS) method to see how useful a Higher Order Thinking Skills (HOTS) assessment tool would be in Pancasila Education. Thirty fifth-grade Pancasila Education teachers from primary schools took part in a quantitative descriptive method. We used SUS scores to determine how useful the developed tool was, using established usability standards to do so. The results show that the instrument got an average SUS score of 76.5, which puts it in the "practical" and "good" categories. More research shows that teachers thought the tool was easy to use, understood its instructions, and felt sure they could use it in the classroom. These results show how important it is to test the usability of educational tools, especially those that are meant to help people learn 21st-century skills. The study suggests that practicality testing should be a key part of validating educational tools and that more research should be done on how usable they are in digital and long-term learning settings.
Adler-Beléndez, D., Hoppenstedt, E., Husain, M., Chng, E., & Schneider, B. (2020). How are 21st-century skills captured in makerspaces?: A review of the literature. ACM International Conference Proceeding Series, 40–45. https://doi.org/10.1145/3386201.3386214
Afifah, I. R. N., & Retnawati, H. (2019). Is it difficult to teach higher order thinking skills? Journal of Physics: Conference Series, 1320(1). https://doi.org/10.1088/1742-6596/1320/1/012098
Alotaibi, S. K. G. (2024). The role of Islamic education teachers in enhancing higher-order thinking: a systematic review. British Journal of Religious Education. https://doi.org/10.1080/01416200.2024.2436921
Andayani, T., & Madani, F. (2023). Peran penilaian pembelajaran dalam meningkatkan prestasi siswa di pendidikan dasar. Jurnal Educatio FKIP UNMA, 9(2), 924–930. https://doi.org/10.31949/educatio.v9i2.4402
Anderson, L. W., & Krathwohl, D. R. (2011). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Allyn & Bacon.
Anggraini, N. P., Budiyono, & Pratiwi, H. (2019). Analysis of higher order thinking skills students at junior high school in Surakarta. Journal of Physics Conference Series, 1211(1). https://doi.org/10.1088/1742-6596/1211/1/012077
Asok, D., Abirami, A. M., Angeline CV, N., & Lavanya, R. (2017). Active learning environment for achieving higher-order thinking skills in engineering education. Proceedings - 2016 IEEE 4th International Conference on MOOCs, Innovation and Technology in Education, MITE 2016, 47–53. https://doi.org/10.1109/MITE.2016.69
Astuti, A. P., Aziz, A., Sumarti, S. S., & Bharati, D. A. L. (2019). Preparing 21st century teachers: Implementation of 4C character’s pre-service teacher through teaching practice. Journal of Physics: Conference Series, 1233(1). https://doi.org/10.1088/1742-6596/1233/1/012109
Atmaja, F. L., Hidayati, K., & Fineldi, R. J. (2024). Assessment of higher order thinking skills: Perspective of mathematics teacher. AIP Conference Proceedings, 2622(1). https://doi.org/10.1063/5.0133815
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776
Bhardwaj, V., Zhang, S., Tan, Y. Q., & Pandey, V. (2025). Redefining learning: student-centered strategies for academic and personal growth. Frontiers in Education, 10. https://doi.org/10.3389/feduc.2025.1518602
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. https://doi.org/10.1080/0969595980050102
Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In Usability Evaluation In Industry (pp. 207–212). CRC Press. https://doi.org/10.1201/9781498710411-35
Brookhart, S. M. (2010). How to assess higher-order thinking skills in your classroom. ASCD.
Creswell, J. W. (2014). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Pearson.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319. https://doi.org/10.2307/249008
Epinasti, G., Slamet, S. T. Y., & Yamtinah, S. (2021). Teacher’s perception towards assessing of Higher Order Thinking Skills (HOTS) in elementary schools. ACM International Conference Proceeding Series. https://doi.org/10.1145/3516875.3516980
Faizah, R., Taqwa, M. R. A., & Istiyono, E. (2021). Senior high school student’s higher order thinking skills based on gender and grade. Journal of Physics: Conference Series, 1918(2). https://doi.org/10.1088/1742-6596/1918/2/022031
Fullan, M., & Scott, G. (2014). Education plus, new pedagogies for deep learning. Collaborative Impact SPC,(July), 3–9.
Ismail, R., Retnawati, H., & Imawan, O. R. (2024). Construct validity of mathematics high order thinking skills instrument with cultural context: Confirmatory factor analysis. AIP Conference Proceedings, 3150(1). https://doi.org/10.1063/5.0228143
Kocak, O., Coban, M., Aydin, A., & Cakmak, N. (2021). The mediating role of critical thinking and cooperativity in the 21st century skills of higher education students. Thinking Skills and Creativity, 42, 100967. https://doi.org/10.1016/j.tsc.2021.100967
Kosasih, A., Supriyadi, T., Firmansyah, M. I., & Rahminawati, N. (2022). Higher-order thinking skills in primary school: Teachers’ perceptions of Islamic education. Journal of Ethnic and Cultural Studies, 9(1), 56–76. https://doi.org/10.29333/ejecs/994
Kurniawan, T. T., & Utaminingsih, S. (2021). Analysis of 4C-Based HOTS assessment module on critical thinking ability. Journal of Physics: Conference Series, 1823(1). https://doi.org/10.1088/1742-6596/1823/1/012101
Lidiawati, K. R., & Aurelia, T. (2023). Kemampuan berpikir kritis siswa di Indonesia: Rendah atau tinggi. Buletin KPIN, 9(02). https://buletin.k-pin.org/index.php/arsip-artikel/1200-kemampuan-berpikir-kritis-siswa-di-indonesia-rendah-atau-tinggi
Mahmud, M. M., & Wong, S. F. (2022). Digital age: The importance of 21st century skills among the undergraduates. Frontiers in Education, 7. https://doi.org/10.3389/feduc.2022.950553
Nielsen, J. (1993). Usability engineering. Academic Press.
Puspitasari. (2021). Analisis kualitas soal berbasis HOTS pada pembelajaran IPS. Jurnal Pendidikan Sosial Indonesia, 3(1), 42–55.
Rifayanti, Z. E. T., & Mariana, N. (2024). Enhancing critical thinking and problem-solving skills in upper elementary students through game-based learning. Perspektivy Nauki i Obrazovania, 70(4), 396–420. https://doi.org/10.32744/pse.2024.4.25
Sanusi, S., Suraiya, N., Nurasiah, N., Qismullah Yusuf, Y., Ridha, S., Hikmah, N., & Leni, L. (2025). Social knowledge management to optimize adolescents’ social studies learning in Banda Aceh, Indonesia. Cogent Education, 12(1). https://doi.org/10.1080/2331186X.2025.2506062
Sekwena, G. L. (2023). Active learning pedagogy for enriching economics students’ higher order thinking skills. International Journal of Learning, Teaching and Educational Research, 22(3), 241–255. https://doi.org/10.26803/ijlter.22.3.15
Shields, R., & Chugh, R. (2018). Preparing Australian high school learners with 21st century skills. Proceedings of 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering, TALE 2018, 1101–1106. https://doi.org/10.1109/TALE.2018.8615207
Skills, P. for 21st C. (2009). Framework for 21st century learning.
Sucipto, T. L. A., Sajidan, S., Akhyar, M., & Roemintoyo, R. (2025). Mapping teacher’s readiness to improve the government policy in implementing higher-order thinking skills-based learning. Qubahan Academic Journal, 5(1), 764–781. https://doi.org/10.48161/qaj.v5n1a1577
Supriatna, N., Winarti, M., & Wiyanarti, E. (2025). The effectiveness of using flipbooks as an interactive medium in social studies learning based on local wisdom to enhance critical thinking skills. Review of Integrative Business and Economics Research, 14(2), 603–618. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85219026979&partnerID=40&md5=f2f1c6a3fafa55fd715ea543dedb090f
Sutrisno, S., Ponorogo, U. M., & Jumadi, I. J. (2023). Global citizenship education in the perspective of strengthening Pancasila student profiles. Jurnal Civics: Media Kajian Kewarganegaraan, 20(2), 322–332. https://doi.org/10.21831/jc.v20i2.63983
Suyato, S., Hidayah, Y., Arpannudin, I., & Septiningrum, L. (2023). Revitalisasi pendidikan kewarganegaraan abad 21: Analisis keterampilan abad 21. SOCIA: Jurnal Ilmu-Ilmu Sosial, 19(2), 78–84. https://doi.org/10.21831/socia.v19i2.60152
Trilling, B., & Fadel, C. (2009). 21st century skills: Learning for life in our times. Jossey-Bass.
van den Akker, J. (1999). Principles and methods of development research. In Design Approaches and Tools in Education and Training (pp. 1–14). Springer Netherlands. https://doi.org/10.1007/978-94-011-4255-7_1
van Laar, E., van Deursen, A. J. A. M., van Dijk, J. A. G. M., & de Haan, J. (2020). Determinants of 21st-century skills and 21st-century digital skills for workers: A systematic literature review. SAGE Open, 10(1). https://doi.org/10.1177/2158244019900176
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 425–478. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3375136
Vidergor, H. E., & Krupnik-Gottlieb, M. (2015). High order thinking, problem based and project based learning in blended learning environments. In Applied practice for educators of gifted and able learners (pp. 217–232). https://doi.org/10.1007/978-94-6300-004-8_11
Vu, X. H. (2025). Leveraging technology for higher-order thinking development: Instructional design strategies in higher education. International Journal of Technologies in Learning, 32(2), 25–46. https://doi.org/10.18848/2327-0144/CGP/v32i02/25-46
Welda, W., Putra, D. M. D. U., & Dirgayusari, A. M. (2020). Usability testing website dengan menggunakan metode System Usability Scale (SUS)s. International Journal of Natural Science and Engineering, 4(3), 152–161. https://doi.org/10.23887/ijnse.v4i2.28864
Yuliana, I. F., Faizah, F., Fatayah, F., & Priyasmika, R. (2024). Students’ critical thinking skills using HOTS-based assessment on pH calculation. AIP Conference Proceedings, 3106(1). https://doi.org/10.1063/5.0215096
Zana, F. M., Sa’dijah, C., Anwar, L., & Zulnaidi, H. (2024). Curriculum and teacher assessment practices in mathematics learning: Alignment with higher order thinking skills in Indonesian secondary schools. Journal on Mathematics Education, 15(4), 1311–1334. https://doi.org/10.22342/jme.v15i4.pp1311-1334
Copyright (c) 2025 Jurnal Kependidikan: Penelitian Inovasi Pembelajaran
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
The authors submitting a manuscript to this journal agree that, if accepted for publication, copyright publishing of the submission shall be assigned to JK. However, even though the journal asks for a copyright transfer, the authors retain (or are granted back) significant scholarly rights.
The copyright form should be signed originally and sent to the Editorial Office through email to jk@uny.ac.id
Download HERE
Jurnal Kependidikan by http://journal.uny.ac.id/index.php/jk is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
