Evaluation of Machine Translation Systems: A Literature Review on ChatGPT and Google Translate

Ohod Faisal Ahmed; Ida Kusuma Dewi; Mohammad Yunus Anis

doi:10.24256/ideas.v13i1.6236

Authors

Ohod Faisal Ahmed Linguistics Department, Faculty of Cultural Science, Sebelas Maret University, Indonesia
Ida Kusuma Dewi Translation Studies and Linguistics Department, Faculty of Cultural Science, Sebelas Maret University , Indonesia
Mohammad Yunus Anis Sastra Arab and Translation Department, Faculty of Cultural Science, Sebelas Maret University, Indonesia

DOI:

https://doi.org/10.24256/ideas.v13i1.6236

Keywords:

: Machine Translation, ChatGPT, Google Translate, Comparative Analysis, Fluency

Abstract

Abstract: This is a literature review discussing 15 selected papers about ChatGPT and Google Translate study results based on keyword analysis and publication year. We applied descriptive data analysis technique to analyze the data. We selected Studies on translation performance of natural language processing tools were chosen due to their increasing prominence and diverse applications, ranging from literary to technical translations. The data for this literature review was retrieved from Scopus and Google Scholar. The search was limited to the last five years to ensure the inclusion of recent advancements, particularly those reflecting improvements in ChatGPT’s GPT-4 engine and updates in Google Translate’s neural machine translation capabilities. The results showed that ChatGPT excels in fluency and contextual understanding, particularly in literary and poetic translations, outperforming Google Translate in maintaining stylistic elements and complex language structures. Both systems demonstrated strengths in specialized translations, with ChatGPT showing notable proficiency in medical literature and technical texts. However, challenges remained in low-resource languages and specialized domains, requiring further training and development. Despite technological advancements, human translators are essential for achieving culturally nuanced translations. This study has some implications for future implementing for enhancement contextual understanding, improving accuracy for low-resource languages, and addressing specific error patterns through ongoing research and collaborative efforts between human translators and machine translation tools. These recommendations aim to optimize the performance of ChatGPT and Google Translate, thereby ensuring more accurate and contextually appropriate translations across various fields.

References

Abdulmohsen Alosaimi, B., & Abdulaziz Alawad, N. (2024). Evaluation of the Translation of Separable Phrasal Verbs Generated by ChatGPT. Arab World English Journal, 1(1), 282–291. https://doi.org/10.24093/awej/chatgpt.19

Akula, B., Barrault, L., Gonzalez, G. M., Hansanti, P., & Hoffman, J. (2020). No Language Left Behind: Scaling Human-Centered Machine Translation—Meta Research.

Ali, A., & Pandya, S. (2021). A four-stage framework for the development of a research problem statement in doctoral dissertations. International Journal of Doctoral Studies, 16, 469–485. https://doi.org/10.28945/4839

Almahasees, Z., & Mahmoud, S. (2022). Evaluation of Google Image Translate in Rendering Arabic Signage into English. World Journal of English Language, 12(1), 185–197. https://doi.org/10.5430/wjel.v12n1p185

Bani, M., & Masruddin, M. (2021). Development of an Android-based harmonic oscillation pocket book for senior high school students. JOTSE: Journal of Technology and Science Education, 11(1), 93-103.

Bonyadi, A. (2020). Exploring Linguistic Modifications of Machine-Translated Literary Articles: The Case of Google Translate. Journal of Foreign Language Teaching and Translation Studies, 5(3), 93–106. https://doi.org/10.22034/efl.2020.250576.1057

Cerf, V. G. (2023). Large Language Models. Communications of the ACM, 66(8), 7. https://doi.org/10.1145/3606337

Deb, D., Dey, R., & Balas, V. E. (2019). Literature review and technical reading. Intelligent Systems Reference Library, 153, 9–21. https://doi.org/10.1007/978-981-13-2947-0_2

Gabashvili, I. S. (2023). The impact and applications of ChatGPT: A Systematic Review of Literature Reviews. Aurametrix, April 2023. https://doi.org/10.17605/OSF.IO/87U6Q.Keywords

Gao, R., Lin, Y., Zhao, N., & Cai, Z. G. (2024). Machine translation of Chinese classical poetry: a comparison among ChatGPT, Google Translate, and DeepL Translator. Humanities and Social Sciences Communications, 11(1), 1–10. https://doi.org/10.1057/s41599-024-03363-0

Gao, Y., Wang, R., & Hou, F. (2023). How to Design Translation Prompts for ChatGPT: An Empirical Study.

Garg, A., & Agarwal, M. (2018). Machine Translation: A Literature Review.

GOOD, G. (2015). 済無No Title No Title No Title. Angewandte Chemie International Edition, 6(11), 951–952., 1(April).

Hariri, W. (2023). Unlocking the Potential of ChatGPT: A Comprehensive Exploration of its Applications, Advantages, Limitations, and Future Directions in Natural Language Processing.

Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y. J., Afify, M., & Awadalla, H. H. (2023). How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation.

Ismayanti, D., Said, Y. R., Usman, N., & Nur, M. I. (2024). The Students Ability in Translating Newspaper Headlines into English: A Case Study. IDEAS: Journal on English Language Teaching and Learning, Linguistics and Literature, 12(1), 108-131.

Jiao, W., Huang, J. T., Wang, W., He, Z., Liang, T., Wang, X., Shi, S., & Tu, Z. (2023). ParroT: Translating during Chat using Large Language Models Tuned with Human Translation and Feedback. Findings of the Association for Computational Linguistics: EMNLP 2023, 15009–15020. https://doi.org/10.18653/v1/2023.findings-emnlp.1001

Jiao, W., Wang, W., Huang, J., Wang, X., Shi, S., & Tu, Z. (2023). Is ChatGPT a Good Translator? Yes, with GPT-4 as the engine.

Kadaoui, K., Magdy, S. M., Waheed, A., Khondaker, M. T. I., El-Shangiti, A. O., Nagoudi, E. M. B., & Abdul-Mageed, M. (2023). TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties. ArabicNLP 2023: 1st Arabic Natural Language Processing Conference, Proceedings, ArabicNLP, 52–75. https://doi.org/10.18653/v1/2023.arabicnlp-1.6

Khan, N. A., Osmonaliev, K., & Sarwar, M. Z. (2023). Pushing the Boundaries of Scientific Research with the Use of Artificial Intelligence Tools: Navigating Risks and Unleashing Possibilities. Nepal Journal of Epidemiology, 13(1), 1258–1263. https://doi.org/10.3126/nje.v13i1.53721

KOÇER GÜLDAL, B., & İŞİSAĞ, K. U. (2019). A comparative study on Google Translate: An error analysis of Turkish-to-English translations in terms of the text typology of Katherina Reiss. RumeliDE Dil ve Edebiyat Araştırmaları Dergisi, 5, 367–376. https://doi.org/10.29000/rumelide.606217

Li, H., Graesser, A. C., & Cai, Z. (2014). Comparison of Google translation with human translation. Proceedings of the 27th International Florida Artificial Intelligence Research Society Conference, FLAIRS 2014, 190–195.

Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhao, L., Zhu, D., Li, X., Qiang, N., Shen, D., Liu, T., & Ge, B. (2023). Summary of ChatGPT-related research and perspective towards the future of large language models. Meta-Radiology, 1(2), 100017. https://doi.org/10.1016/j.metrad.2023.100017

Masruddin, M. (2019). Efficacy Of Using Spelling Bee Game In Teaching Vocabulary To Indonesian English As Foreign Language (Efl) Students, The Asian Efl Journal.

Masruddin, Hartina, S., Arifin, M. A., & Langaji, A. (2024). Flipped learning: facilitating student engagement through repeated instruction and direct feedback. Cogent Education, 11(1), 2412500.

Nila, Firda, S., & Susanto, T. (2017). Google Translate impacts students’ translation of economic text: accuracy and acceptability. 6th ELTLT International Conference Proceedings, October, 487–491.

Noviarini, T. (2021). the Translation Results of Google Translate From Indonesian To English. Jurnal Smart, 7(1), 21–26. https://doi.org/10.52657/js.v7i1.1335

Peng, K., Ding, L., Zhong, Q., Shen, L., Liu, X., Zhang, M., Ouyang, Y., & Tao, D. (2023). Towards Making the Most of ChatGPT for Machine Translation. Findings of the Association for Computational Linguistics: EMNLP 2023, 5622–5633. https://doi.org/10.2139/ssrn.4390455

Sepesy Maučec, M., & Donaj, G. (2020). Machine Translation and the Evaluation of Its Quality. Recent Trends in Computational Intelligence, 1–20. https://doi.org/10.5772/intechopen.89063

Siu, S. C. (2023). ChatGPT and GPT-4 for Professional Translators: Exploring the Potential of Large Language Models in Translation. SSRN Electronic Journal, 1–36. https://doi.org/10.2139/ssrn.4448091

Stevanović, I., & Radičević, L. (2020). Comparative Analysis of Machine Translation Systems. International Journal of Computer Applications, 12(2), 5–8.

Temsah, O., Khan, S. A., Chaiah, Y., Senjab, A., Alhasan, K., Jamal, A., Aljamaan, F., Malki, K. H., Halwani, R., Al-Tawfiq, J. A., Temsah, M.-H., & Al-Eyadhy, A. (2023). Overview of Early ChatGPT’s Presence in Medical Literature: Insights From a Hybrid Literature Review by ChatGPT and Human Experts. Cureus, 15(4). https://doi.org/10.7759/cureus.37281

Wang, L., Lyu, C., Ji, T., Zhang, Z., Yu, D., Shi, S., & Tu, Z. (2023). Document-Level Machine Translation with Large Language Models. EMNLP 2023—2023 Conference on Empirical Methods in Natural Language Processing, Proceedings, March, 16646–16661. https://doi.org/10.18653/v1/2023 .emnlp-main.1036

Yahya, A., Husnaini, H., & Putri, N. I. W. (2024). Developing Common Expressions Book in Indonesian Traditional Market in Three Languages (English-Indonesian-Mandarin). Language Circle: Journal of Language and Literature, 18(2), 288-295.