Sesame Street to Mount Sinai: BERT -constrained character-level M oses models for multilingual lexical normalization (2021)

Abstract: This paper describes the HEL-LJU submissions to the MultiLexNorm shared task on multilingual lexical normalization. Our system is based on a BERT token classification preprocessing step, where for each token the type of the necessary transformation is predicted (none, uppercase, lowercase, capitalize, modify), and a character-level SMT step where the text is translated from original to normalized given the BERT-predicted transformation constraints. For some languages, depending on the results on development data, the training data was extended by back-translating OpenSubtitles data. In the final ordering of the ten participating teams, the HEL-LJU team has taken the second place, scoring better than the previous state-of-the-art.

## introduction in this paper, we describe the hel-lju submission to the multilexnorm shared task on multilingual lexical normalization. lexical normalization is a task of transforming non-standard input tokens into output tokens that follow a specific linguistic standard. in this shared task, lexical normalization is defined even narrower as "the task of transforming an utterance into its standard form, word by word, where one-to-many (1-n) and many-to-one (n-1) replacements are included", disregarding thereby word deletions and insertions @xcite . the main motivation behind lexical normalization is to minimize the variability of the linguistic signal, either for computational usage or human consumption. accordingly, the shared task submissions are evaluated both intrinsically and extrinsically (on a dependency parsing task). the need for lexical normalization for computational usage is diminishing these days, given the end-to-end methodology that is becoming more and more popular, where the systems are robust enough to accept non-standard input without any explicit normalization. however, for human consumption the need for lexical normalization is still very much present. the three types of data that still frequently require normalization are user-generated content (i.e., "internet language" @xcite ), historical data (e.g., 18th century slovenian @xcite , which is frequently not understood even by native speakers of slovenian) and dialectal data (very frequently not understood by non-native speakers of a language, or even by the speaker of the same language, as is the case with swiss dialects of german @xcite ). ## related work over the last decade, character-level statistical machine translation (csmt) has shown very strong results on varying types of non-standard data, such as user-generated content @xcite , historical data @xcite and dialectal data @xcite . even more, the csmt approach has shown to behave very similarly in a controlled comparison on various types of non-standard data, such as with slovenian user-generated content and historical texts @xcite . it has also shown to be the preferred way of adapting language technologies to non-standard data if the availability of human supervision is low @xcite . while neural approaches have almost entirely replaced statistical ones in "standard" translation settings (translating between distinct languages), recent studies have shown that smt-based approaches remain competitive for normalization tasks @xcite . normalization systems not based on translation architectures have also been proposed. for example, monoise (van der goot, 2019) generates a list of normalization candidates for each token and then ranks them using a variety of features derived from the original text and the proposed normalization candidates. the data in the multilexnorm shared task all come from the user-generated-content domain, and comprise mostly of twitter data. an overview of the 12 datasets is given in for some of the languages, the data was split into a training and a development set by the organizers. for the other languages, we split the data randomly (90% training, 10% development) and kept the split constant across our experiments. ## character-level mt architectures for normalization following our earlier experience, we cast normalization as a character-level machine translation problem. in order to enable contextual dependencies, we train and test on entire tweets. we preprocessed the data by replacing urls by a placeholder and token boundaries by a reserved character. these changes were reverted during postprocessing. we evaluated an smt model 1 , an rnn-based 1 the translation model is monotonous, i.e. without any distortion component. the language model is a character 10gram model trained with kenlm on the provided training data. the model weights are tuned on the development set with mert (minimum error rate training) using character csmt c-rnn c-trf lg.-specific 92.1% 85.0% 66.7% lg.-independent 90.4% 86.4% 89.4% nmt model foot_0 , and a transformer-based nmt model foot_1 . the smt models are based on moses @xcite , the nmt models are based on opennmt-py @xcite . the results are reported in ## adding constraints to the normalization process the sequence labeling models are based on pre-trained language-specific bert or electra models, which are fine-tuned on the task using derived training data. 5 sentential context. as expected, they were consistently outperformed by the models using entire tweets. 5 we use the ner model class of the simpletransformers li- ## including synthetic training data from back-translation the results of the language-independent models of section 4 suggest that the provided training data is of insufficient size to train reliable translation models, especially neural ones. a well-known strategy to augment the training data in mt is backtranslation, where target language data is translated to the source language by an auxiliary model @xcite . the resulting parallel data (a standard target side, and a noisy source side) is then included in the training data of the main model. in the normalization setting, this amounts to finding "clean" data and running it through a model that produces a noisy version of it. to this end, we used filtered subsets of the monolingual opensubtitles corpora from opus 7 @xcite as input data for producing back-translations. we filtered the opus data using the opusfilter package @xcite ) and the following filters: the resulting dataset is then "unnormalized" using a backward csmt model trained on the provided training data in the opposite direction, with a beam of 200. lines whose translation candidates are all identical to the input are rejected. we run the csmt model for 72 hours per language. depending on the initial data size, the filters and the speed of the csmt model, this results in 16k to 127k additional sentences per language (see the resulting back-translations massively outnumber the original training data for most languages, which may affect the final model negatively. therefore, we also provide random samples of back-translations that contain at most 20 times as many sentence pairs as the given training data (see bottom row of bt concatenating the original training data with the back-translated data for phrase table extraction. in addition to the model trained only on the provided data, we therefore obtain four csmt models, two with the full augmented data and two with the subsampled augmented data. ## submissions the experiments reported in the previous sections have shown us that for our data and our setup: for our first submission, we choose the csmt model setup that has led to the best development accuracy for each language (i.e., the setups highlighted in italics in for the second submission, we re-create the phrase table and the language model by including the previously held-out development set. we copy the model weights obtained by mert from the first submission. note that for those languages where the full back-translations are used, the added development instances amount to a tiny fraction of the overall data. in this case, we expect the results for the two submissions to be very similar. the results of the intrinsic evaluation are summarized in the same ranking is seen in the extrinsic evaluation, although it only concerns german, english and italian. one should note however that the las scores of all systems are very close, showing again that normalization does not provide a substantial advantage for recent downstream-task systems. ## conclusion in this paper we have described our submission to the multilexnorm shared task on multilingual lexical normalization. we compare character-level smt, rnn and transformer models, showing that in our case, where training data is very limited, smt still outperforms the two neural options. increasing the amount of training data by merging data from all languages, or by means of backtranslation of opensubtitles data, does help the neural approaches, but they still do not perform better than smt. we further investigate the possibility of predicting via bert-like models if and how a token should be modified, and show that giving this information to the smt process improves the final results. maybe the path to mount sinai passes through sesame street, after all. . .