Comparing ChatGPT's and human evaluation of scientific texts translations from English to Portuguese using popular automated translators

Abstract

This paper addresses the challenge researchers face when translating their work. Due to the high cost associated with human translation services, many researchers turn to automatic translation tools as a cost-effective alternative. Therefore, assessing the quality of these translations is crucial. This paper presents a comparative evaluation of translations using both human assessments and ChatGPT. Our study focuses on the translation of a scientific text excerpt from English to Portuguese. We analyze the performance of ChatGPT in two scenarios: comparative evaluations with all translations presented in a single prompt, applied five times to test for consistency, and 20 individual evaluations (five evaluations per translation) in separate chats. In both scenarios, ChatGPT’s assessments exhibit higher consistency in terms of fluency, appropriateness, accuracy, and overall assessment compared to human evaluations. The results also reveal a consensus between human evaluations and ChatGPT assessments regarding the translation with the lowest score, while discrepancies arise in the evaluations of top-performing translations. Finally, the ability to engage in follow-up questions, receive suggestions for improvement, and compare translations using ChatGPT's own recommendations proves a valuable tool for researchers seeking to assess and improve the translation quality of their work.