Applying the Rasch Rating Scale Model (RSM) to investigate the rating scales function in survey research instrument

Abstract

The functionality and quality structure of a rating scale are vital in a survey instrument. This article discusses the underlying assumption of a rating scale and addresses the application of the Rasch Rating Scale Model (RSM) to diagnose the rating scale structure in a survey instrument. The instrument used to demonstrate the process of diagnosis a rating scale functioning in this study was the Vocational Teachers’ Assessment Literacy (VoTAL) instrument. The VoTAL instrument utilized the five-point Likert-type response format and consisted of 88 items representing three assessment literacy constructs. The data were obtained from 224 vocational teachers at five vocational colleges, which were randomly selected from the state of Selangor and the federal territory of Kuala Lumpur in Malaysia. The rating scale diagnosis results showed that the initial five steps of rating scale categories were not properly functioning as intended. It was found that the Andrich threshold of category three was disordered as it was not monotonically advance with categories. Additionally, the result also found that the width of the threshold between category three and category four was too narrow (0.66 logits). The results indicate that the rating scale used in the VoTAL instrument was disordered. Thus, it was suggested that the initial five rating step categories were collapsed down into four categories. In conclusion, this study highlights the importance of assessing the rating scale’s functionality in a survey instrument to reduce measurement error and collect valid and reliable data.