A machine learning-based evaluation of English-to-Sinhala translation: comparing Google Translate, large language models, and human translators

Abstract

Reliable translation from English to Sinhala is still a great challenge for many sophisticated translation systems using Sinhala as a low-resource language. Although Google Translate is widely used for translationpurposes, recent breakthroughs in large language models such as ChatGPTand DeepSeek provide entirely new opportunities for translation tasks. This study proposes one of the first thorough comparative analyses of English-Sinhala translation systems compared with human translation, both qualitatively and quantitatively. Google Translate, ChatGPT, DeepSeek, and human translations done by native Sinhala speakers were compared for translation quality on a carefully prepared dataset of 150English sentences for general, technical, and academic purposes. Translation quality was compared using BLEU, METEOR, and COMETscores, in addition to human assessment of fluency, grammatical accuracy, and semantic translation quality done by qualified human raters using a prepared rubric with inter-rater reliability tests. Machine learning models were also prepared for predicting translation quality using language-basedpredictors for translation efficiency and translation system identification. The experimental results show that human translations were rated highest on all translation quality measures. Among the automatic translationsystems, LLM-based translation systems performed better on contextual understanding of complex sentences than Google Translate, whichperformed reasonably on simple inputs. Correlation tests showthat COMET correlates better with human translation quality than BLEUandMETEOR. Moreover, the prepared machine learning models were able todetect translation quality trends accurately for translation systempredictions, making these models promising for translation qualityassessment in lowresource language environments.