COMPARATIVE ANALYSIS OF AUTOMATED DUBBING AND LOCALIZATION UTILIZING GENERATIVE AI FOR MULTILINGUAL VOICE AND LIP MOVEMENT SYNCHRONIZATION ON AUDIO-VISUAL PODCAST
DOI:
https://doi.org/10.21831/jited.v4i2.4046Keywords:
dubbing, AI generatif, sinkronisasi bibir, kualitas visual, ANOVAAbstract
Penelitian ini bertujuan untuk melakukan analisis komparatif terhadap model dubbing otomatis (Wav2Lip, DiffDub, dan Diff2Lip) dari segi akurasi sinkronisasi bibir dan kualitas visual dengan memanfaatkan teknologi AI generatif. Pendekatan analisis komparatif eksperimental digunakan untuk mengumpulkan data dengan membagi podcast audio-visual ke dalam dua sudut kamera dan mengulang setiap sudut sebanyak sepuluh kali untuk memperoleh hasil rata-rata. Ketiga model dubbing otomatis dievaluasi berdasarkan metrik kualitas visual (PSNR, SSIM, dan LPIPS) dan metrik akurasi sinkronisasi bibir (LSE-D dan LSE-C), kemudian diuji dengan ANOVA satu arah (uji F) untuk menilai perbedaan kinerja antarmodel. Hasil penelitian menunjukkan bahwa: (1) DiffDub merupakan model paling efektif dalam mengatasi tantangan kualitas visual pada dubbing berbasis AI dengan skor PSNR dan SSIM yang tinggi; (2) Diff2Lip menunjukkan akurasi sinkronisasi bibir yang sangat baik dan secara signifikan mengungguli model lain pada LSE-D dan LSE-C; (3) ANOVA menunjukkan perbedaan kinerja yang signifikan antara Wav2Lip, DiffDub, dan Diff2Lip pada semua metrik dengan seluruh nilai p < 0,0001; dan (4) ukuran efek yang dihitung mendukung temuan tersebut, yang menunjukkan bahwa perbedaan yang teramati signifikan secara statistik maupun praktis. Temuan ini menunjukkan potensi teknologi dubbing berbasis AI sebagai alternatif bagi metode dubbing tradisional.
Downloads
References
Beyer, D. (n.d.). Reliable benchmarking: Requirements and solutions.
Bigioi, D., & Corcoran, P. (2023). Multilingual video dubbing—A technology review and current challenges. Frontiers in Signal Processing, 3, 1230755. https://doi.org/10.3389/frsip.2023.1230755
Borenstein, M. (Ed.). (2013). Introduction to meta-analysis (Nachdr.). Wiley.
Brannon, W., Virkar, Y., & Thompson, B. (2023). Dubbing in practice: A large scale study of human localization with insights for automatic dubbing. Transactions of the Association for Computational Linguistics, 11, 419–435. https://doi.org/10.1162/tacl_a_00551
Chung, J. S., & Zisserman, A. (2017). Out of time: Automated lip sync in the wild. In C.-S. Chen, J. Lu, & K.-K. Ma (Eds.), Computer Vision – ACCV 2016 Workshops (Vol. 10117, pp. 251–263). Springer International Publishing. https://doi.org/10.1007/978-3-319-54427-4_19
Díaz-Cintas, J., & Anderman, G. M. (Eds.). (2009). Audiovisual translation: Language transfer on screen. Palgrave Macmillan.
Duquenne, P.-A., Barrault, L., Chung, Y.-A., Meglioli, M. C., Dale, D., Dong, N., Duquenne, P.-A., Elsahar, H., Gong, H., Heffernan, K., Hoffman, J., Klaiber, C., Li, P., Licht, D., Maillard, J., Rakotoarison, A., Sadagopan, K. R., Wenzek, G., Ye, E., … Wang, S. (2023). SeamlessM4T: Massively multilingual & multimodal machine translation (arXiv:2308.11596). arXiv. http://arxiv.org/abs/2308.11596
E-learning dubbing is more than a technology trend. (2024, January 29). https://www.acadecraft.com/blog/the-power-of-e-learning-dubbing-in-creating-an-inclusive-educational-environment/
Education First. (2023). Indeks kecakapan bahasa Inggris EF [Dataset].
Esselink, B. (2000). A practical guide to localization (Rev. ed.). Benjamins.
Esser, F., & Vliegenthart, R. (2017). Comparative research methods. In The International Encyclopedia of Communication Research Methods (pp. 1–22). John Wiley & Sons. https://doi.org/10.1002/9781118901731.iecrm0035
Federico, M., Enyedi, R., Barra-Chicote, R., Giri, R., Isik, U., Krishnaswamy, A., & Sawaf, H. (2020). From speech-to-speech translation to automatic dubbing. Proceedings of the 17th International Conference on Spoken Language Translation, 257–264. https://doi.org/10.18653/v1/2020.iwslt-1.31
GoodStats. (n.d.). Konten video jadi tipe konten favorit Gen Z Indonesia dalam menikmati media sosial. Retrieved August 23, 2024, from https://goodstats.id/article/konten-video-jadi-tipe-konten-favorit-gen-z-indonesia-dalam-menikmati-media-sosial-bgl9m
Google Cloud. (n.d.). Evaluating models | AutoML Translation documentation. Retrieved August 29, 2024, from https://cloud.google.com/translate/automl/docs/evaluate
Gottlieb, H. (1994). Subtitling: Diagonal translation. Perspectives, 2(1), 101–121. https://doi.org/10.1080/0907676X.1994.9961227
Inside the dying art of subtitling. (n.d.). CNET. Retrieved April 14, 2024, from https://www.cnet.com/culture/entertainment/features/inside-the-dying-art-of-subtitling/
Iskandarova, K., & Majidova, R. U. (2023). The significance and crucial role of language in facilitating effective communication. Fan, Jamiyat va Innovatsiyalar, 1(1), 159–162.
Keraf, G. (1997). Komposisi: Sebuah pengantar kemahiran berbahasa. Nusa Indah.
Łańcucki, A. (2021). FastPitch: Parallel text-to-speech with pitch prediction (arXiv:2006.06873). arXiv. http://arxiv.org/abs/2006.06873
Liu, T., Du, C., Fan, S., Chen, F., & Yu, K. (2024). DiffDub: Person-generic visual dubbing using inpainting renderer with diffusion auto-encoder (arXiv:2311.01811). arXiv. http://arxiv.org/abs/2311.01811
Marsh, E. D., & Stoker, G. (n.d.). Theory and methods in political science [9780230366640].
Miggiani, G. S. (2022). Measuring quality in translation for dubbing: A quality assessment model proposal for trainers and stakeholders. XLinguae, 15(2), 85–102. https://doi.org/10.18355/XL.2022.15.02.07
Mukhopadhyay, S., Suri, S., Gadde, R. T., & Shrivastava, A. (2023). Diff2Lip: Audio conditioned diffusion models for lip-synchronization (arXiv:2308.09716). arXiv. http://arxiv.org/abs/2308.09716
National Institute on Deafness and Other Communication Disorders. (2022). Voice, speech, and language: What are they? https://www.nidcd.nih.gov/news/multimedia/how-does-human-body-produce-voice-and-speech
Nendya, M. B., & Mu'min, S. (2016). Auto lip-sync pada karakter virtual 3 dimensi menggunakan blendshape. REKAM: Jurnal Fotografi, Televisi, dan Animasi, 11(2), 137. https://doi.org/10.24821/rekam.v11i2.1299
OECD. (2016). Skills matter: Further results from the Survey of Adult Skills. The Organization for Economic Co-operation and Development.
OECD. (2023). PISA 2022 results (Volume I): The state of learning and equity in education – Country note – Indonesia. OECD. https://doi.org/10.1787/53f23881-en
Paperspace, T. O. (2023, January 27). A review of the image quality metrics used in image generative models. Paperspace Blog. https://blog.paperspace.com/review-metrics-image-synthesis-models/
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V., & Jawahar, C. V. (2020). A lip sync expert is all you need for speech to lip generation in the wild. Proceedings of the 28th ACM International Conference on Multimedia, 484–492. https://doi.org/10.1145/3394171.3413532
Rouder, J. N., Engelhardt, C. R., McCabe, S., & Morey, R. D. (2016). Model comparison in ANOVA. Psychonomic Bulletin & Review, 23(6), 1779–1786. https://doi.org/10.3758/s13423-016-1026-5
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Journal of Information Technology and Education (JITED)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
