How effective is machine translation on low-resource code-switching? A case study comparing human and automatic metrics (2023)

Abstract: This paper presents an investigation into the differences between processing monolingual input and code-switching (CSW) input in the context of machine translation (MT). Specifically, we compare the performance of three MT systems (Google, mBART-50 and M2M-100-big) in terms of their ability to translate monolingual Vietnamese, a low-resource language, and Vietnamese-English CSW respectively. To our knowledge, this is the first study to systematically analyse what might happen when multilingual MT systems are exposed to CSW data using both automatic and human metrics. We find that state-of-the-art neural translation systems not only achieve higher scores on automatic metrics when processing CSW input (compared to monolingual input), but also produce translations that are consistently rated as more semantically faithful by humans. We further suggest that automatic evaluation alone is insufficient for evaluating the translation of CSW input. Our findings establish a new benchmark that offers insights into the relationship between MT and CSW.

## introduction code-switching (csw) is the linguistic phenomenon where two or more languages are mixed within a discourse or utterance. this is illustrated in the following example which mixes english and vietnamese. in this paper, we explore the limits of three off-the-shelf state-of-the-art machine translation systems in terms of their ability to translate vietnamese/english csw data, using both automatic and human evaluation metrics. as far as we are aware, this is the first study to investigate the efficacy of machine translation on csw data involving a low-resource language which is also structurally vastly different from english. in fact, existing work has mainly focused on comparatively better resourced and/or typologically similar languages, such as spanish/english @xcite , french/english @xcite or hindi/english @xcite . vietnamese/english, or vietnamese in particular, remains severely under-represented in nlp. we conduct our analysis using a variety of both automatic and human metrics in order to i) better understand the strengths and weaknesses of different systems, and ii) gain some insight into the relationship between automatic and human metrics with respect to csw input. we find that systems not only achieve higher scores on csw input (compared to monolingual input) according to automatic metrics, but also produce translations that are considered more semantically faithful by humans. automatic metrics furthermore fail to correlate with human judgements, which suggests that automatic evaluation alone is not enough for evaluating mt output of csw input. we release our annotations to facilitate future research. ## experiments we evaluated our three chosen mt systems in two settings: code-switching to english (csw-en) 13 and monolingual vietnamese to english (vi-en). recall that the sentences in the vi-en setting are the same as the csw-en setting except all english words and phrases were manually translated to vietnamese by a human translator (section 2.1). this enabled us to directly compare the effect of csw against a highly controlled baseline. altogether, we obtained 200 translations from each system (100 clauses x 2 settings) and 600 translations in total (3 systems). we then asked our bilingual annotators to manually assign binary judgements to each translation based on the three human metrics (1800 judgements). specifically, after training, foot_13 we asked the l1 english annotator to assign judgements for fluency and grammaticality, and the l1 vietnamese annotator to assign judgements for semantic faithfulness. we believe judgements for fluency and grammaticality require native assessment of the english translation irrespective of the source, while semantic faithfulness also requires native assessment of the vietnamese source. in all cases, a positive judgement was only awarded if the translation fully met the criteria of the given metric; this conservative approach ensured greater confidence that positive judgements truly reflected a more competent translation. ## results and discussion results from all experiments are shown in google outperforms mbart-50 and m2m-100 big on all metrics except for comet in csw-en; this suggests that google is the best of the three mt systems on our csw/monolingual test sets. it is furthermore noteworthy that performance on cswen translation for all three mt systems is consistently and significantly higher than monolingual vien translation. in fact, a do-nothing csw baseline seems to outperform mbart-50 and m2m-100 big at monolingual vi-en translation in terms of bleu. we hypothesise that this is because the translation might be considered 'easier' when csw fragments only need to be copied to the output. for example, among the three human metrics, we also observe that the scores for semantic faithfulness were consistently higher given csw input compared to monolingual input. while this is again likely due to the constraining nature of csw input, this result potentially suggests a specific aspect of mt where csw input can contribute to enhancing system output. we direct readers to appendix b for some detailed examples. ultimately, we consider this finding worthy of further investigation, especially in relation to the development of models involving the understanding and/or generation of code-switching texts. ## conclusion in this work, we compared the performance of three state-of-the-art mt systems on csw input, using both automatic and human metrics. we found that systems not only achieved higher scores on automatic metrics when processing csw input (com-pared to monolingual input), but also produced translations that were consistently rated as more semantically faithful by humans. we furthermore observed that automatic and human metrics do not agree, which again highlights the need for more sophisticated, robust metrics, especially in nonmonolingual tasks. our findings establish a new benchmark in the relationship between mt and csw, and motivate further research into how csw might be used to improve future systems.