A social media NMT engine for a low-resource language combination (2023)

Abstract: The aim of this article is to present a new Neural Machine Translation (NMT) from Spanish into Galician for the social media domain that was trained with a Twitter corpus. Our main goal is to outline the methods used to build the corpus and the steps taken to train the engine in a low-resource language context. We have evalu-ated the engine performance both with regular automatic metrics and with a new methodology based on the non-inferiority process and contrasted this information with an error classification human evalua-tion conducted by professional linguists. We will present the steps carried out fol-lowing the conclusions of a previous pilot study, describe the new process followed, analyze the new engine and present the final conclusions.

## introduction in recent years, the low-resource languages domain has received some attention from our research community. many papers covered different strategies to overcome the need for data to train engines for low-resource languages. @xcite gave a complete overview of the main techniques and solutions employed in this field: data augmentation techniques, such as word or phrase replacement, back-translation, parallel data mining; unsupervised nmt; semi-supervised nmt; multilingual nmt; transfer learning in nmt; and zero-shot nmt. a considerable amount of work has been also done in social media research, mainly in sentiment analysis and translation of user-generated content fields. the majority of these papers are focused on phrase-based machine translation (pbmt) engines. only @xcite attempted to compare the machine translations of tweets using phrase-based and neural mt and the usage of different amounts and types of training corpora for each of the two approaches. the results of their research showed that using a tiny twitter corpus is useless for nmt training, although the system improved when using backtranslation and out-of-domain corpora. this particular procedure is the one used in our nmt training and adapted to a low-resource language combination, from spanish to galician. ## background this contribution presents the most significant findings from a doctoral thesis on low-resource languages and nmt as a means of promoting and using a minority language in the context of social media 1 . it is carried out in the despitemt project framework, dedicated to researching the uses of mt applied to the media 2 . this study is based on a previous pilot completed in 2022, which focused on creating a spanish into galician nmt engine for social media and proposing a new methodology for evaluating this type of nmt engine @xcite . for the pilot study, we created an nmt engine based on joey through the online platform mut-nmt 3 (a minimalist nmt toolkit for novices, https://aclanthology.org/d19-3019/ ). the corpus used to train the engine was a mix of two corpora. we used the paracrawl corpus spanish -galician (1,879,651 sentences and 44,626,394 words) as a generic base corpus. to build a social media corpus, ## retraining process of the nmt engine for social media in the second training of the nmt engine, we changed two settings of the first nmt engine setup: the amount of data of the specific corpus and the nmt technology. we kept the paracrawl corpus as a base generic corpus but decided to expand our twitter corpus. our first twitter corpus contained only nearly 70000 unique sentences. to build a larger twitter corpus, we chose more institutional galician accounts (see regarding the nmt technology, we used a transformer-big configuration for marian and sentencepiece @xcite . we decided to change the nmt technology used to have better control of the training parameters as the joey mutnmt platform does not allow this. the bleu score obtained was 85% against the test corpus extracted only from the twitter corpus, which was higher than the bleu score achieved in the pilot study. ## nmt engine evaluation we carried out two different types of evaluation of our nmt engine. first, we replicated the noninferiority evaluation presented in our first study with some adjustments and obtained better results @xcite . second, we conducted an error evaluation using the dqf-mqm framework @xcite @xcite , which was carried out by three professional linguists with over ten years of experience using the online platform contenquo. contentquo is an online platform, specially dedicated to translation quality evaluation with specific workflows and predefined quality templates, such as the one used, dqf. the goal of doing two different evaluations was to find a link between errors and a negative attitude toward the machinetranslated tweets, similar to the methodology applied by @xcite to carry out the error-classification evaluation based on the dqf-mqm framework, we selected three galician linguists with more than 10 years of experience in the spanish-galician language direction and with experience in machine translation through proz.com. the professional evaluation was remunerated and conducted in the app contentquo foot_1 (see figure 3 ). we asked proofreaders to review 30 tweets that were translated from spanish into galician using our nmt engine for social media (890 words). linguists were asked to assess the raw mt output. those tweets were the same used in the non-inferiority evaluation survey. a brief explanation before the evaluation was given to contextualize the task and explain the objective of the assignment. they dedicated one to two hours to this task. the mean overall quality score obtained was 94.55%. although this is a good score taking into account that the tweets were not postedited, we were more interested in the type of errors annotated by the professional linguists and in the severity of the errors (see regarding the severity, no critical errors were found and the majority of the errors were minor. only a few errors were classified as major. the dqf-mqm template also allowed the classification of the errors as neutral without affecting the quality score. in an intriguing finding was that major errors were mostly distributed in threads and short paragraphs, which could explain the survey's low acceptance (do campo et al., forthcoming). furthermore, while minor errors were distributed indiscriminately in all types of tweets, it appears that this severity of errors has no effect on users' perceptions of naturalness. we also asked the linguists for a general comment about the quality of the raw machine-translated tweets. generally speaking, they agreed on the good quality of the engine. they highlighted that some segments did not need any change, while others need a few changes to be correct with respect to the spanish text. ## conclusions the purpose of this article was to describe the process of developing an nmt engine for social media in a low-resource language combination using twitter data and back-translation as primary strategies. we have shown that increasing the in-domain twitter corpus and using back-translation improved the engine's performance in terms of both automatic and human evaluation. we also want to emphasize that the size of the in-domain twitter corpus will be determined by the proximity of the languages used. as spanish and galician are very close languages, we saw promising results with a small in-domain corpus. as shown by us and other authors, twitter is a good source of monolingual data crawling. in this article, we have shown that it can be used for more than common uses such as sentiment analysis. furthermore, professional linguists concluded that the raw machine-translated tweets evaluated could benefit from minor post-editing. the double evaluation conducted -non-inferiority (do campo et al., forthcoming) and human evaluation-demonstrated that our engine is capable of translating social media content. finally, we want to contextualize the importance of conducting nmt research on low-resource languages in order to promote their use. both our training process and evaluation methodology can be replicated in other language combinations that are similar to ours, particularly if they want to promote the low-resource language on social media and the internet. with our research, we hope to help galician to reach the younger population through social media and reduce the loss of speakers in the last decades.