Development and psychometric testing of the CIPP evaluation instrument for vocational teacher training programs
Evaluation instruments for vocational teacher upskilling and reskilling programmes are often validated through expert judgement alone, without empirical field testing, and rarely in remote schools. This study examined the preliminary psychometric properties of two CIPP-based instruments developed to evaluate the teacher upskilling and reskilling programme at SMKN 4 Merauke, South Papua. An instrument development design with an exploratory pilot survey was employed. The instruments—a 15-item teacher questionnaire covering the Context, Input, Process and Product dimensions and a 7-item student scale covering Process and Product—were reviewed by expert judges and then administered to all five productive teachers (saturated sampling) and 20 advanced students (purposive sampling) using a five-point Likert scale. Item validity was examined with Pearson item–total correlations and reliability with Cronbach's alpha in R. All seven student items met the critical value (r = 0.587–0.958; r-critical = 0.444), and the student scale showed high internal consistency (α = 0.928; 95% CI 0.854–0.964). For the teacher scale, only five of 15 items exceeded the critical value for N = 5 (r-critical = 0.878); item G5 correlated negatively with the total (r = −0.365), and the correlation matrix was not positive definite, so the alpha estimate (α = 0.884; 95% CI 0.392–0.978) is unstable. The student scale is ready for wider field testing, whereas the teacher scale requires item revision—particularly of double-barrelled items—and re-testing with a substantially larger sample. The study illustrates how small-sample pilot testing can diagnose item-level problems before CIPP instruments are used in remote vocational schools.
Downloads
Astuti, M., Arifin, Z., Nurtanto, M., Mutohhari, F., & Warju, W. (2022). The maturity levels of the digital technology competence in vocational education. International Journal of Evaluation and Research in Education, 11(2), 596–606. https://doi.org/10.11591/ijere.v11i2.22258
Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287–311. https://doi.org/10.1093/poq/nfm006
Blimpo, M. P., & Pugatch, T. (2019). Entrepreneurship education and teacher training in Rwanda. Journal of Development Economics, 140, 186–202. https://doi.org/10.1016/j.jdeveco.2019.05.006
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, Article 149.https://doi.org/10.3389/fpubh.2018.00149
Bonett, D. G., & Wright, T. A. (2015). Cronbach's alpha reliability: Interval estimation, hypothesis testing, and sample size planning. Journal of Organizational Behavior, 36(1), 3–15. https://doi.org/10.1002/job.1960
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061
Brevik, L. M., Gudmundsdottir, G. B., Lund, A., & Strømme, T. A. (2019). Transformative agency in teacher education: Fostering professional digital competence. Teaching and Teacher Education, 86, Article 102875. https://doi.org/10.1016/j.tate.2019.07.005
Chryssolouris, G., Mavrikios, D., & Rentzos, L. (2016). The teaching factory: A manufacturing education paradigm. Procedia CIRP, 57, 44–48. https://doi.org/10.1016/j.procir.2016.11.009
Cizek, G. J. (2012). Defining and distinguishing validity: Interpretations of score meaning and justifications of test use. Psychological Methods, 17(1), 31–43. https://doi.org/10.1037/a0026975
Cronbach, L. J., & Shavelson, R. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64(3), 391–418. https://doi.org/10.1177/0013164404266386
Desimone, L. M. (2009). Improving impact studies of teachers' professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181–199. https://doi.org/10.3102/0013189X08331140
Grollmann, P. (2008). The quality of vocational teachers: Teacher education, institutional roles and professional reality. European Educational Research Journal, 7(4), 535–547. https://doi.org/10.2304/eerj.2008.7.4.535
Guskey, T. R. (2002). Professional development and teacher change. Teachers and Teaching, 8(3), 381–391. https://doi.org/10.1080/135406002100000512
Hertzog, M. A. (2008). Considerations in determining sample size for pilot studies. Research in Nursing & Health, 31(2), 180–191. https://doi.org/10.1002/nur.20247
Hill, H. C., Beisiegel, M., & Jacob, R. (2013). Professional development research: Consensus, crossroads, and challenges. Educational Researcher, 42(9), 476–487. https://doi.org/10.3102/0013189X13512674
Instefjord, E. J., & Munthe, E. (2017). Educating digitally competent teachers: A study of integration of professional digital competence in teacher education. Teaching and Teacher Education, 67, 37–45. https://doi.org/10.1016/j.tate.2017.05.016
Johanson, G. A., & Brooks, G. P. (2010). Initial scale development: Sample size for pilot studies. Educational and Psychological Measurement, 70(3), 394–400. https://doi.org/10.1177/0013164409355692
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000
Karatas, H., & Seval, F. (2011). CIPP evaluation model scale: Development, reliability and validity. Procedia – Social and Behavioral Sciences, 15, 592–599. https://doi.org/10.1016/j.sbspro.2011.03.146
Kennedy, M. M. (2016). How does professional development improve teaching? Review of Educational Research, 86(4), 945–980. https://doi.org/10.3102/0034654315626800
Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45(5), 626–629. https://doi.org/10.1016/j.jpsychires.2010.10.008
McNeish, D. (2018). Thanks coefficient alpha, we'll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/10.1037/met0000144
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
Revelle, W., & Zinbarg, R. E. (2009). Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74(1), 145–154. https://doi.org/10.1007/s11336-008-9102-z
Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74(1), 107–120. https://doi.org/10.1007/s11336-008-9101-0
Sims, S., & Fletcher-Wood, H. (2021). Identifying the characteristics of effective teacher professional development: A critical review. School Effectiveness and School Improvement, 32(1), 47–63. https://doi.org/10.1080/09243453.2020.1772841
Stufflebeam, D. L., & Zhang, G. (2017). The CIPP evaluation model: How to evaluate for improvement and accountability. Guilford Press.
Taber, K. S. (2018). The use of Cronbach's alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach's alpha. International Journal of Medical Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108
Yoto, Y., Marsono, M., Qolik, A., & Romadin, A. (2024). Evaluation of teaching factory using CIPP (Context, Input, Process, Product) model to improve vocational high school students' skills. Jurnal Pendidikan Vokasi, 14(1), 12–28. https://doi.org/10.21831/jpv.v14i1.62573
Copyright (c) 2026 Jurnal Kependidikan: Penelitian Inovasi Pembelajaran
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
