Alternative item selection strategies for improving test security in computerized adaptive testing of the algorithm

Iwan Suhardi

doi:10.21831/reid.v6i1.30508

Alternative item selection strategies for improving test security in computerized adaptive testing of the algorithm

Iwan Suhardi, Universitas Negeri Makassar, Indonesia

10.21831/reid.v6i1.30508

Abstract

One of the ability estimation methods that is widely applied to the Computerized Adaptive Testing (CAT) algorithm is the maximum likelihood estimation (MLE). However, the maximum likelihood method has the disadvantage of being unable to find a solution to the ability estimation of test-takers when the test takers’ scores do not have a pattern. If there are test takers who get either score of 0 or perfect score, then the abilities of test-takers are usually estimated using the step-size model. However, the step-size model often results in item exposure where certain items will appear more often than other items. This surely threatens the security of the test because items that often appear will be easier to recognize. This study tries to provide an alternative strategy by modifying the step-size model and randomizing the calculation results of the information function obtained. Based on the results of the study, it is found that alternative strategies for item selection can make more varied items appear to improve the security of tests on the CAT.

Keywords

item selection strategy; item exposure; step-size; adaptive testing

Full Text:

PDF

References

Baker, F. B. (1992). Item response theory: Parameter estimation techniques. New York, NY: Marcel Dekker.

Birnbaum, A. (1968). Some latent trait models and their uses in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental rest scores (pp. 397–479). Reading, MA: Addison-Wesley.

Dodd, B. G. (1990). The effect of item selection procedure and stepsize on computerized adaptive attitude measurement using the rating scale model. Applied Psychological Measurement, 14(4), 355–366. https://doi.org/10.1177/014662169001400403

Eignor, D. R., Stocking, M. L., Way, W. D., & Steffen, M. (1993). Case studies in computer adaptive test design through simulation. https://doi.org/10.1002/j.2333-8504.1993.tb01567.x

Grist, S. (1989). Computerized adaptive tests. In ERIC Digest No. 107. Retrieved from https://files.eric.ed.gov/fulltext/ED315425.pdf

Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston, MA: Kluwer Nijhoff.

Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage Publications.

Haryanto, H. (2013). Pengembangan computerized adaptive testing (CAT) dengan algoritma logika Fuzzy. Jurnal Penelitian Dan Evaluasi Pendidikan, 15(1), 47–70. https://doi.org/10.21831/pep.v15i1.1087

Higgins, P. (2009). Candidate measured ability and use of time. Retrieved from https://www.rasch.org/mra/mra-10-09.htm

Lord, Frederic M. (1977). A broad-range tailored test of verbal ability. Applied Psychological Measurement, 1(1), 95–100. https://doi.org/10.1177/014662167700100115

Martinez, L. (2009). Time usage and candidate performance. Retrieved from http://www.rasch.org/mra/mra-06-09.htm

McBride, J. R., & Martin, J. T. (1983). Reliability and validity of adaptive ability tests in a military setting. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 224–236). New York, NY: Academic Press.

Mills, C. N. (1999). Development and introduction of a computer adaptive Graduate Record Examinations General Test. In F. Drasgow & J. B. Olson-Buchanan (Eds.), Innovations in computerized assessment (pp. 117–135). Mahwah, NJ: Lawrence Erlbaum Associates.

Pressman, R. S. (2001). Software engineering: A practitioner’s approach (5th ed.). New York, NY: McGraw-Hill Higher Education.

Rudner, L. M. (1998). An on-line, interactive, computer adaptive testing tutorial. Retrieved from http://edres.org/scripts/cat

Santoso, A. (2010). Pengembangan computerized adaptive testing untuk mengukur hasil belajar mahasiswa Universitas Terbuka. Jurnal Penelitian Dan Evaluasi Pendidikan, 14(1), 62–83. https://doi.org/10.21831/pep.v14i1.1976

Thissen, D. (1990). Reliability and measurement precision. In H. Wainer, N. J. Dorans, R. Flaugher, B. F. Green, R. J. Mislevy, L. Steinberg, & D. Thissen (Eds.), Computerized adaptive testing: A primer (2nd ed., pp. 161–186). Hillsdale, NJ: Erlbaum.

Vispoel, W. P. (1999). Creating computerized adaptive tests of music aptitude: Problems, solutions, and future directions. In F. Drasgow & J. B. Olson-Buchanan (Eds.), Innovations in computerized assessment (pp. 151–176). Mahwah, NJ: Lawrence Erlbaum Associates.

Winarno, W. (2013). Pengembangan computerized adaptive testing (CAT) menggunakan metode pohon segitiga keputusan. Jurnal Penelitian Dan Evaluasi Pendidikan, 16(2), 574–592. https://doi.org/10.21831/pep.v16i2.1132

DOI: https://doi.org/10.21831/reid.v6i1.30508