Noisy UGC Translation at the Character Level: Revisiting Open-Vocabulary Capabilities and Robustness of Char-Based Models (2021)

Abstract: This work explores the capacities of character-based Neural Machine Translation to translate noisy User-Generated Content (UGC) with a strong focus on exploring the limits of such approaches to handle productive UGC phenomena, which almost by definition, cannot be seen at training time. Within a strict zero-shot scenario, we first study the detrimental impact on translation performance of various user-generated content phenomena on a small annotated dataset we developed and then show that such models are indeed incapable of handling unknown letters, which leads to catastrophic translation failure once such characters are encountered. We further confirm this behavior with a simple, yet insightful, copy task experiment and highlight the importance of reducing the vocabulary size hyper-parameter to increase the robustness of character-based models for machine translation.

## introduction neural machine translation (nmt) models fall far short from being able to translate noisy user-generated content (ugc): the quality of their translation is often even worse than that of a traditional phrase-based system @xcite . in addition to ambiguous grammatical constructs and profusion of ellipsis, the main difficulty encountered when translating ugc is the high number of out-of-vocabulary tokens (oovs) resulting from misspelled words, emoticons, hashtags, mentions, and all specific constructs used in online forums and social medias @xcite @xcite . some of those phenomena can be perceived as noise while the others are typical markers of language variation among speakers. moreover, a certain amount of those same phenomena operate at the lexical level (either at the character, subword or word levels) @xcite . this is why, focusing more on the noise axis, char-based models appear to offer a natural solution to this problem @xcite : indeed these open vocabulary models are designed specifically to address the oov problem. in this work, we explore the ability of out-of-thebox character-based nmt models @xcite to address the challenges raised by ugc translation. while character-based models may seem promising for such task, to the best of our knowledge, they have only been tested either on data sets in which noise has been artificially added through sampling an edited word error data set @xcite and on canonical data set, in which they prove to be very effective for translating morphologically-rich languages with a high number of oovs @xcite . however, our starting-points experiments show that character-based systems are outperformed by bpe models even when translating noisy ugc. to understand this counter-intuitive result, we conduct several experiments and analyses. in particular, we manually annotated 400 sentences at the token level using a fine-grained typology, to perform our analyses. these sentences correspond to the worst and the best translated utterances of two mt systems (a char-based and a transformer-based model). our results highlight the extreme sensibility of characterbased models to the vocabulary size, a parameter often overlooked in the literature. using a simple set of experiments, we thus show that these models are unable to perform an easy copy task due to their poor handling of unknown and rare characters. by adjusting the vocabulary size parameter, we drastically improve the robustness of our character-based model without causing a large drop in in-domain performance. our contributions are as follows: ## character-level nmt in this work, we explore two character-based translation models. the first model we consider, charcnn is a classic encoder-decoder in which the encoder uses character-based embeddings in combination with convolutional and highway layers to replace the standard look-up based word representations. the model considers, as input, a stream of words (i.e. it assumes the input has been tokenized beforehand) and tries to learn a word representation that is more robust to noise by unveiling regularities at the character level. this architecture was initially proposed by @xcite for language modeling; costa-jussà and fonollosa (2016) shows how it can be used in an nmt system and report improvements up to 3 bleu @xcite points when translating from a morphologicallyrich language, german, to english. the second model we consider does not rely on an explicit segmentation into words: @xcite introduce the char2char model that directly maps a source characters sequence to a target characters sequence without any segmentation thanks to a character-level convolutional network with max-pooling at the encoder. it can be considered as an open-vocabulary model: it can generate any word made of any of the most frequent characters of the train set (where is a model hyper-parameter) and only outputs @xmath0. @xcite show that this model outperforms subword-level (i.e. bpe-based) translation models on two wmt'15 tasks (de-en and cs-en) and gives comparable performance on two tasks (fi-en and ru-en). @xcite additionally report that in a multilingual setting, the character-level encoder significantly outperforms the subword-level encoder on all the language pairs. these two models have been originally tested on wmt or iwslt tasks that consider texts that mostly qualify as canonical with very few spelling or grammatical errors. the impact of noise on charcnn and char2char has been evaluated by @xcite and @xcite . by adding noise to canonical texts (the tedtalk dataset). their results show that the different character levels models fail to translate even moderately noisy texts when trained on 'clean' data and that it is necessary to train a model on noisy data to make it robust. note that, as explained in section 3, there is no ugc parallel corpus large enough to train a nmt model and we must rely on the models' ability to learn, from canonical data only, noise-and error-resistant representations of their input that are robust to the noise and errors found in ugcs. this is why, in this work, we are interested in studying mt performance in a zero-shot scenario when translating noisy ugc. to the best of our knowledge, no work has studied the performance of the character-based nmt architectures on an actual ugc scenario with realworld gathered and fine-grained annotated noisy sentences. this sets the main motivation for the present work. ## datasets training sets due to the lack of a large parallel corpus of noisy sentences, we train our systems with 'standard' parallel datasets, namely the corpora used in the wmt campaign @xcite and the opensubtitles corpus @xcite . the former contains canonical texts (2.2m sentences), while the latter (9.2m sentences) is made of written dialogues from popular tv series. the reason for using opensubtitles is the assumed greatest similarity, compared to wmt, between its content and the ugc conversational nature that predominates most social media. however, ugc differs significantly from subtitles in many aspects: in ugc, emotions are often denoted with repetitions, there are many typographical and spelling errors, and sentences may contain emojis that can even replace some words (e.g. ♥ can stands for the verb 'love' in sentences such as 'i ♥ you'). ugc productivity limits the pertinence of domain adaptation methods such as fine-tuning, as there will always be new forms that will not have been seen during training @xcite . this is why we focus here on zero-shot scenarios, as we believe they provide a clearer experimental protocol when it comes to study the impact of ugc specificities on mt models. ## nmt models character-based models in our experiments, we compare the two character-level nmt systems introduced in section 2, namely charcnn and char2char. both models were trained as outof-the box systems using the implementations provided by @xcite for charcnn, foot_2 and by @xcite for char2char. 4 it must be noted that the charcnn extracts character n-grams for each input word and predicts a word contained in the target vocabulary or a special token, <unk>, otherwise, whereas the char2char is capable of open-vocabulary translation and does not @xmath1, unless an out-of-vocabulary character (char-oov) is present in the input. ## experiments and results we train and evaluate the mt models presented in the previous section using the data train and test configurations in section 3. we first compare performances for our clean test sets (whether in-domain or our-of-domain, accordingly) and noisy ugc when translated using bpe-and character-based models before exploring into detail our results. 6 the sources and references of the mtnt corpus are less similar (bleu score: 5) since all user mentions and url are removed in the distributed version of the corpus. 7 the alignments are computed by concatenating the test set to the training set and using fastalign @xcite . as expected, all models perform better when they are trained on the opensubs corpus than when they are trained on wmt, as the former is intuitively more similar to ugc data than wmt is. more surprisingly, it appears that character-based models are largely outperformed by bpe-based models for most train-test combinations and, therefore, that their capacity to learn word representations that are robust to noise can be questioned. another interesting observation is that, while the transformer achieves the best results on all test sets when trained on opensubtitles, it is outperformed by seq2seq when the wmt training set is considered. the situation is similar for characterbased models. this observation suggests that these ## impact of the char-vocabulary size on translation quality to validate in a more realistic setting the conclusion we draw from our copy tasks experiments, we present in figure 2 , the impact of unknown characters on translation quality of the pfsmb test set for the transformer and char2char models. while the translation quality is almost the same for both models when there is no char-oovs in the input, the occurrence of an unknown character hurts the translation quality of the char2char model beyond the mistranslation of a single character. motivated by these observations, we carried out experiments on our char2char models, trained on opensubtitles, with different character vocabulary size. results in for vocabulary sizes larger than 90 characters, the char2char system achieves the same bleu scores as with 90 characters. note that in our first experiments (reported in ## discussion our work follows many studies on the robustness of character-based models, most notably from belinkov and bisk (2018) who compared different translation unit levels for nmt and showed that, overall, mt systems trained on clean data only presented considerable performance decrease when processing synthetic character-level noise and wordlevel noise. here, we also perform comparisons with other mainstream strong bpe-based baselines and because of our annotated ugc data set are able to provide a more precise view on the phenomena at stake when processing natural noisy-input. similarly to @xcite what concluded for morpho-syntactic classification tasks, our results show that a more fined-grained granularity of learning representations, characters over bpes, provides a higher robustness to certain types of noise. contrary to the both aforementioned works, our study is performed using annotated real-world noisy ugc, which proved crucial for our study. in line to @xcite findings, where finegrained nmt granularity provided robustness advantages when processing misspelling, our results show that the best and worst translations' specificities distribution point to a better performance of char2char for the missing diacritics category, giving insights on more specific types of misspellings that affect performance. continuing this research track, we broaden the spectrum of studying ugc specificities, exploring the effects of the vocabulary training size and show that tuning it can achieve better results when translating noisy ugc. this simple hyper-parameter choice proved effective, providing an alternative to using fine-tuning methods @xcite or back-translation @xcite . ## conclusion we showed that in zero-shot scenarios, char-based models are not robust to ugc idiosyncrasies. we presented several experiments that explained this counter-intuitive result by an over-sensibility of these models to the vocabulary size. we demonstrated that drastically lowering this parameter increased the robustness of our char-based models when facing noisy ugc while maintaining almost the same level of performance for in-domain and out-of-domain canonical datasets. interestingly, we noticed a lack in the literature regarding the importance of vocabulary considerations when training character-based nmt models. however, our results show that a path for improvement exists, leading to more generic and stable models and making us believe that further research on the matter can bring promising alternatives to domain-adaptation via target domain data addition. our data set is available at https://github.com/josecar25/ char_based_nmt-noisy_ugc .