Applying the Rasch Rating Scale Model (RSM) to investigate the rating scales function in survey research instrument

Jimmy Chong, Siti Eshah Mokshein, Ramlee Mustapha

Abstract


The functionality and quality structure of a rating scale are vital in a survey instrument. This article discusses the underlying assumption of a rating scale and addresses the application of the Rasch Rating Scale Model (RSM) to diagnose the rating scale structure in a survey instrument. The instrument used to demonstrate the process of diagnosis a rating scale functioning in this study was the Vocational Teachers’ Assessment Literacy (VoTAL) instrument. The VoTAL instrument utilized the five-point Likert-type response format and consisted of 88 items representing three assessment literacy constructs. The data were obtained from 224 vocational teachers at five vocational colleges, which were randomly selected from the state of Selangor and the federal territory of Kuala Lumpur in Malaysia. The rating scale diagnosis results showed that the initial five steps of rating scale categories were not properly functioning as intended. It was found that the Andrich threshold of category three was disordered as it was not monotonically advance with categories. Additionally, the result also found that the width of the threshold between category three and category four was too narrow (0.66 logits). The results indicate that the rating scale used in the VoTAL instrument was disordered. Thus, it was suggested that the initial five rating step categories were collapsed down into four categories. In conclusion, this study highlights the importance of assessing the rating scale’s functionality in a survey instrument to reduce measurement error and collect valid and reliable data.

Keywords


rasch measurement model; rating scales; rating scales function; survey instrument

Full Text:

PDF

References


Abd-El-Fattah, S. M. (2015). Rasch Rating Scale Analysis of the Arabic Version of the Physical Activity Self-Efficacy Scale for Adolescents: A Social Cognitive Perspective. Psychology, 06(16), 2161–2180. https://doi.org/10.4236/psych.2015.616213

Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43(4), 561–573. https://doi.org/10.1007/BF02293814

Andrich, D. (1996). Measurement Criteria for Choosing among Models with Graded Responses. In Categorical Variables in Developmental Research. https://doi.org/10.1016/b978-012724965-0/50004-3

Andrich, D. (2011). Rating scales and Rasch measurement. Expert Review of Pharmacoeconomics & Outcomes Research, 11(5), 571–585. https://doi.org/10.1586/erp.11.59

Bond, T., & Fox, C. M. (2015). Applying the Rasch Model: Fundamental Measurement in the Human Sciences, Third Edition. Routledge.

Boone, W. J. (2020). Rasch Basics for the Novice. In M. S. Khine (Ed.), Rasch Measurement (pp. 9–30). Springer Singapore. https://doi.org/10.1007/978-981-15-1800-3_2

Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch analysis in the human sciences. Springer.

Bradley, K. D., Peabody, M. R., Akers, K. S., & Knutson, N. M. (2015). Rating Scales in Survey Research: Using the Rasch model to illustrate the middle category measurement flaw. Survey Practice, 8(1), 1–12. https://doi.org/10.29115/sp-2015-0001

Dimitrov, D. M. (2014). Statistical Methods for Validation of Assessment Scale Data in Counseling and Related Fields. Wiley.

DiStefano, C., & Jiang, N. (2020). Applying the Rasch Rating Scale Method to Questionnaire Data. In M. S. Khine (Ed.), Rasch Measurement (pp. 31–46). Springer Singapore. https://doi.org/10.1007/978-981-15-1800-3_3

Engelhard, G., & Wind, S. A. (2017). Invariant Measurement with Raters and Rating Scales. In Invariant Measurement with Raters and Rating Scales. https://doi.org/10.4324/9781315766829-14

Fink, A. (2003). How to Ask Survey Questions (2nd ed.). SAGE Publications, Inc.

Finn, J. A., Ben-Porath, Y. S., & Tellegen, A. (2015). Dichotomous versus polytomous response options in psychopathology assessment: Method or meaningful variance? Psychological Assessment, 27(1), 184–193. https://doi.org/10.1037/pas0000044

Fowler, F. J. (2013). Survey Research Methods. SAGE Publications.

Hales, S. (1986). Rethinking the Business of Psychology. Journal for the Theory of Social Behaviour, 16(1), 57–76. https://doi.org/10.1111/j.1468-5914.1986.tb00066.x

Hilbert, S., Küchenhoff, H., Sarubin, N., Nakagawa, T. T., & Bühner, M. (2016). The influence of the response format in a personality questionnaire: An analysis of a dichotomous, a likert-type, and a visual analogue scale. TPM - Testing, Psychometrics, Methodology in Applied Psychology, 23(1), 3–24. https://doi.org/10.4473/TPM23.1.1

Lei Chang. (1994). A Psychometric Evaluation of 4-Point and 6-Point Likert-Type Scales in Relation to Reliability and Validity. Applied Psychological Measurement, 18(3), 205–215. https://doi.org/10.1177/014662169401800302

Likert, R. (1932). A Technique for the Measurement of Attitudes (Issue nos. 136-165). publisher not identified.

Linacre, J. M. (2000). Comparing and Choosing between “Partial Credit Models” (PCM) and “Rating Scale Models” (RSM). Rasch Measurement Transactions, 14(3), 768.

Linacre, J. M. (2002). Optimizing Rating Scale Category Effectiveness. Journal of Applied Measurement, 3, 85–106.

Linacre, J. M. (2019). A User’s Guide to WINSTEPS MINITEP, Rasch-Model Computer Programs, Program Manual 4.4.7. winsteps.com. https://doi.org/ISBN 0-941938-03-4

Merbitz, C., Morris, J., & Grip, J. C. (1989). Ordinal scales and foundations of misinference. Archives of Physical Medicine and Rehabilitation, 70(4), 308–312.

Mokshein, S., Ishak, H., & Ahmad, H. (2019). The Use Of Rasch Measurement Model In English Testing. Jurnal Cakrawala Pendidikan, 38(1), 16-32. doi:https://doi.org/10.21831/cp.v38i1.22750

Nardi, P. M. (2015). Doing Survey Research. Taylor & Francis.

Nunnally, J. C. (1967). Psychometric theory. McGraw-Hill.

Oxford Dictionary. (2020). https://www.oxfordlearnersdictionaries.com/definition/english/less_1?q=less

Preston, C. C., & Colman, A. M. (2000). Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104(1), 1–15. https://doi.org/10.1016/S0001-6918(99)00050-5

Rasch, G. (1960). Probabilistic Models for Some Intelligence and Attainment Tests. Danmarks Paedagogiske Institut.

Smith, A. B., Rush, R., Fallowfield, L. J., Velikova, G., & Sharpe, M. (2008). Rasch fit statistics and sample size considerations for polytomous data. BMC Medical Research Methodology, 8(1), 33. https://doi.org/10.1186/1471-2288-8-33

Smith, R. M. (1996). Polytomous mean-square fit statistics. Rasch Measurement Transactions, 10(3), 516–517.

Smith, E. V, Wakely, M. B., De Kruif, R. E. L., & Swartz, C. W. (2003). Optimizing Rating Scales for Self-Efficacy (and Other) Research. Educational and Psychological Measurement, 63(3), 369–391. https://doi.org/10.1177/0013164403251320

Stone, M., & Wright, B. D. (1994). Maximizing rating scale information. Rasch Measurement Transactions, 8(3), 386.

Weng, L. J. (2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64(6), 956–972. https://doi.org/10.1177/0013164404268674

Wright, B. D., & Linacre, J. M. (1989). Observations are always ordinal; measurements, however, must be interval. Archives of Physical Medicine and Rehabilitation, 70(12), 857–860.

Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. MESA press.

Wright, B. D., & Panchapakesan, N. (1969). A Procedure for Sample-Free Item Analysis. Educational and Psychological Measurement, 29(1), 23–48. https://doi.org/10.1177/001316446902900102

Zhu, W., Updyke, W. F., & Lewandowski, C. (1997). Post-hoc Rasch analysis of optimal categorization of an ordered-response scale. Journal of Outcome Measurement, 1(4), 286–304.




DOI: https://doi.org/10.21831/cp.v41i1.39130

Refbacks

  • There are currently no refbacks.




 

Social Media:

     


 

 Creative Commons License
Jurnal Cakrawala Pendidikan by Lembaga Pengembangan dan Penjaminan Mutu Pendidikan UNY is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Based on a work at http://journal.uny.ac.id/index.php/cp/index.

Translator
 
 web
analytics
View Our Stats

Flag Counter