C anton MT : C antonese- E nglish Neural Machine Translation Looking into Evaluations (2024)

Abstract: Cantonese-English is a low-resource language pair for machine translation (MT) studies, despite the vast amount of English content publicly available online and the large amount of native Cantonese speakers. Based on our previous work on CANTONMT from Hong et al. (2024), where we created the open-source fine-tuned systems for Cantonese-English Neural MT (NMT) using base-models NLLB, OpusMT, and mBART and corpus collections and creation, in this paper, we report our extended experiments on model training and comparisons. In particular, we incorporated human-based evaluations using native Cantonese speakers who are also fluent in the English language. We designed a modified version of the HOPE metric from Gladkoff and Han (2022) for the categorised error analysis and serenity-level statistics (naming HOPES). The models selected for human evaluations are NLLB-mBART fine-tuned and two translators from commercial companies: Bing and GPT4.

## introduction cantonese is a sinitic language spoken in hong kong, macau, and the guangdong region of southern prc, it is the second most spoken sinitic language, after mandarin chinese @xcite . with a substantial 80 million native speakers @xcite , cantonese is still an under-researched area in the spectrum of natural language processing, as demonstrated in acl anthology, where only 47 papers are related to cantonese, compared with 2355 for (mandarin) chinese @xcite . despite having the second most speakers in the family of sinitic languages, most state-of-the-art commercial translators either do not support cantonese or have below-par translation quality when translated to english. this leads to scenarios where individuals seeking cantonese resources face challenges, particularly in casual forums where tones are often very similar to spoken language. we believe that cantonese is a unique language that captures the rich cultural history of hong kong, macau, and the guangdong province of china. two major challenges when dealing with cantonese translations are colloquialism and multilingualism. colloquialism, a linguistic style used for informal and casual conversation, often occurs in cantonese and includes non-standard spelling, slang, and neologisms. as for multilingualism, hong kong was once a british colony and has a rich chinese cultural influence; code-switching 1 happens often in day-to-day conversation; and words can also be loaned from english through phonetic transliteration @xcite . therefore, following the trend of language diversity and inclusion in nlp, we have set out the aim to develop a translation system that can translate texts from cantonese to english and reach comparable results against commercial translators, as reported in our cantonmt1.0 @xcite . as an extended investigation of our first milestone, regarding the evaluation strategy, the models developed are evaluated through a range of metrics, including lexicon-based word surface matching (sacrebleu and hlepor) and those based on embedding spaces (comet and bertscore). following these metrics, the top-performing model is chosen for comparison with the two top-performing commercial translation tools. we designed the hopes (standing for "simplified hope") human evaluation framework, which we modified based on hope, a human-centric post-editing based metric by @xcite . ## human evaluations even with four different automatic metrics, it is still hard to judge the model's performance based on those chosen metrics. therefore, human evaluations are conducted to understand better the comparison with state-of-the-art models and the different types of errors that the trained models or deployed translators tend to make. ## discussion and conclusion in this paper, we further investigated the system performances from cantonmt, an open-sourced platform for cantonese⇔english translation. we designed hopes metric for human evaluation purposes, which is a simplified version of the hope framework by @xcite . the simplified hopes metric has only four error types including mistranslation (mis), term errors (term), style (style), and grammatical errors (gram), while keeping the original error severity features from hope. the human evaluation result shows that nllp-mbart fine-tuned model has average error score 12.58, vesus 8.3475 and 2.3575 from bing and gpt4-ft. regarding error severity levels, nllb-mbart has fewer minor-errors than bing, though more major-errors at this stage. as we mentioned in cantonmt @xcite , in terms of concerns of data privacy such as handling of sensitive data (e.g., in clinical applications related to health analytics of patient data @xcite ), cantonmt can be fully controlled by users without interference from any third parties. we believe the performance of cantonmt models can be continuously improved with more high-quality real and synthetic data integrated for fine-tuning.