Developing an observation instrument for assessing the effectiveness of English teaching at vocational secondary schools

Estu Widodo, Universitas Muhammadiyah Malang, Indonesia
Franciscus Xaverius Sudarsono, Universitas Negeri Yogyakarta, Indonesia


This study was aimed at: (1) developing an observation instrument for assessing English teaching at vocational secondary schools (VSSs), (2) examining the validity of the developed instrument, (3) examining the inter-rater reliability, and (4) assessing the effectiveness of the developed  instrument.  The subjects of the three-phase field-tests were English teachers and students of four VSSs. Observation sheets, questionnaires, and interviewes were employed to collect the data. The instruments were field-tested in three phases. Expert judgment, Kappa Coefficient, and Kane’s (2006) argument-based validation and Bachman & Palmer’s intepretive and use argument were used to determine the instrument validity and reliability. The field-test which involved stakeholders prove that the observation instrument could be implemented effectively to assess the teaching of English at VSSs.


assessment instruments; effective teaching; English; hotel accommodation

Full Text:



Ackerman, D., Gross, B.L., & Vigneron, F. (2009). Peer observation reports and student evaluations of teaching: Who are the experts?. The Alberta Journal of Educational Research, 55, 18-39.

Adamson, L.S. (2008). Development and evaluation of an instrument to assess data informed instructional practice (Doctoral dissertation, John Hopkins University, 2008). Ann Arbor, MI: ProQuest/UMI. (Publication No. AAT 3309798).

Alkin, M.C. (Ed). (1992). Encyclopedia of educational research (6th ed.). New York, NY: Macmillan.

Arthur, W., Tubre, T., Paul, D.S., & Edens, P.S. (2003). Teaching effectiveness: The relationship between reaction and learning evaluation criteria. Educational Psychology, 23, 275-285.

Bachman, L.F. (1990). Fundamental consider-ations in language testing. Oxford: Oxford University Press.

Bachman, L. & Palmer, A. (2010). Language assessment in practice. Oxford: Oxford University Press.

Borich, G.D. (2007). Effective teaching methods (6th ed.). Upper Saddle River, NJ: Prentice Hall.

Borg, W.R. & Gall, M.D. (1989). Educational research: An introduction (4th ed.). New York, NY: Longman.

Brown, H.D. (2007). Principles of language learning and teaching (5th ed.). White Plains, NY: Pearson Education.

Burhanuddin. (2008). Mengeliminasi kesulitan siswa menjawab soal ujian nasional bahasa Inggris melalui pembelajaran. Pendidikan Network. Retrieved from:

Carruth, P.J. & Carruth, A.K. (2003). Evaluation of college faculty: What do accounting students really think?. Journal of College Teaching and Learning, 9, 1-8.

Cashin, W.E. (1990). Student ratings of teaching: Recommendations for use. Center for Faculty Evaluation and Development. Kansas, KS: Kansas State University.

Creemers, B.P.M. & Kyriakides, L. (2008). The dynamics of educational effectiveness: A contribution to policy, practice and theory in contemporary schools. London: Routledge.

Curtis, A. & Cheng, L. (2007). TAPping into teaching effectiveness: A collaborative approach to performance appraisal of ESL teachers in a Canadian context. In Coombee, C., M. Al-Hamly, Davidson, P. et al. (Eds.). Evaluating teacher effectiveness in ESL/EFL contexts. Ann Arbor: The University of Michigan Press, 57-72.

Djojonegoro, W. (1998). Pengembangan sumber daya manusia melalui SMK. Jakarta: Depdikbud.

Elliott, A.C. & Woodward, W.A. (2007). Statistical analysis with spss examples quick reference guidebook. Thousand Oaks, CA: Sage.

Emery, C.R., Kramer, T.R., & Tian, R.G. (2003). Return to academic standards: A critique of student evaluations of teaching effectiveness. Quality Assurance in Education, 11, 37-46.

Field, A. (2009). Discovering statistics using SPSS. (3rd ed.). London: SAGE.

Gower, R., Philips, D., & Walters, S. (1995) Teaching practice handbook (New ed.). Oxford: Macmillan Education.

Kaluge, L. (2003). Inequity progress of pupil groups of upper primary schools. Jurnal Ilmu Pendidikan, 10 (3), 237-253.

Kaluge, L. & Hadiwijaya, A.S. (2007). Good practices among current educational programs for basic education in Indonesia. Jurnal Ilmu Pendidikan, 14 (3), 144-153. Retrieved from

Kane, M. (2006). Validation. In Brennan (Ed.). Educational measurement (4th ed.). Westport, CT: American Council on Education and Praeger, 17-64.

Kane, M.T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1-73.

Kaplan, R.M. & Saccusso, D.P. (1982). Psychological testing: Principles, application, and issues. Monterey, CA: Brooks/Cole.

Kyriakides, L. & Creemers, B.P.M. (2009). The effects of teacher factors on different outcomes: Two studies testing the validity of the dynamic model. Effective Education, 1 (1), 61-85.

Littlewood, W. (2011). Communicative language teaching. In Hinkel, E. (Ed.). Handbook of research in second language teaching and learning volume II (pp.541-557). New York, NY: Routledge.

McCaffrey, D.F., Koretz, D., Lockwood, J.R., & Hamilton, L.S. (2003). Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND.

Ministry of Education and Culture. (2015). Indonesia education strategic plan 2015-2020. Jakarta.

Mitzel, H.E. (Ed). (1982). Encyclopedia of educational research (5th ed., Vol. 4). New York, NY: The Free Press.

Mohanty, G., Gretes, J., Flowers, C., Algozzine, B., & Spooner, F. (2005). Multi-method evaluation of instruction in engineering classes. Journal of Personnel Evaluation in Education, 18, 139-151.

Musthafa, B. (2001). Communicative language teaching in Indonesia: Issues of theoretical assumptions and challenges in the classroom practice. TEFLIN Journal, 12, 184-193.

Nougaret, A.A., Scruggs, T.E., & Mastropieri, M.A. (2005). Does teacher education produce better special education teachers?. Exceptional Children, 71, 3, 217-229.

Oliveri, M.E., Lawless, R., & Young, J.W. (2015). A validity framework for the use and development of exported assessments. Princeton, NJ: ETS.

Ornstein, A.C. (1991). Teacher effectiveness research: Theoretical considerations. In Waxman, H.C. & Walberg, H.J. (Eds.). Effective teaching. Berkeley, CA: McCutchan, 63-80.

Ornstein, A.C. & Lasley II, T.J. (2004). Strategies for effective teaching. (4th ed.). Boston, MA: McGraw-Hill.



  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Find REID (Research and Evaluation in Education) on:


ISSN 2460-6995 (Online)

View REiD Visitor Statistics