ÚFAL at M ulti L ex N orm 2021: Improving Multilingual Lexical Normalization by Fine-tuning B y T 5 (2021)

Abstract: We present the winning entry to the Multilingual Lexical Normalization (MultiLexNorm) shared task at W-NUT 2021 (van der Goot et al., 2021a), which evaluates lexical-normalization systems on 12 social media datasets in 11 languages. We base our solution on a pre-trained byte-level language model, ByT5 (Xue et al., 2021a), which we further pre-train on synthetic data and then fine-tune on authentic normalization data. Our system achieves the best performance by a wide margin in intrinsic evaluation, and also the best performance in extrinsic evaluation through dependency parsing. The source code is released at https://github.com/ufal/multilexnorm2021 and the fine-tuned models at https://huggingface.co/ufal .

## introduction people produce text in natural language every minute of every day. however, in many cases, for example on social media like twitter, such texts are not conforming to a formal style. instead, they are in colloquial form, which is perfectly understandable to other people, but challenging for automatic natural language processing. to make the processing of such texts more viable, the task of lexical normalization can be used to replace the input forms with their canonical (more formal, lexically normalized) variants. the aim of the w-nut 2021: multilingual lexical normalization (multilexnorm) shared task @xcite is to evaluate participant lexical-normalization systems on 12 social media datasets in 11 languages, including two codeswitching datasets. both intrinsic and extrinsic evaluation is performed, where the latter is measured through dependency parsing performed on the normalized data. recently, large pre-trained models like bert @xcite or t5 @xcite have demonstrated superior performance in many nlp tasks when trained in a transfer learning setting. in line with that, we approach the lexical normalization shared task as a fine-tuning of a large pre-trained model, specifically the multilingual byte-level generative language model byt5 @xcite . our system achieves the best performance in the shared task, both in intrinsic and extrinsic evaluations. in the intrinsic evaluation, our system reaches 67.3% error reduction compared to leaveas-is baseline; the second-best system has 53.6%. the source code is released at https://github. com/ufal/multilexnorm2021 and the fine-tuned models are available in huggingface transformers @xcite ) at https://huggingface. co/ufal . ## related work lexical normalization can be considered a simplified variant of a well-studied problem of grammar error correction (gec). @xcite approach gec as a neural machine translation task using the transformer architecture @xcite , which is pre-trained using a vast amount of synthetic data generated by character-level and word-level edits. recently, @xcite presented a gec system based on multilingual mt5 @xcite , reaching state-of-the-art results on several datasets with the gigantic xxl model size with 13b parameters. while the mentioned gec systems are autoregressive, lexical normalization can be easily solved in a non-autoregressive way, because the normalizations of different words are relatively independent. besides, successful non-autoregressive models have been recently proposed for general gec @xcite . although fine-tuned language models have been successfully used in the state-of-the-art gec systems, this has not been the case in the field of lexi- s o c i a l p e o p l e a r e g r e a t ## model our model is based on a large pre-trained multilingual model. in accordance with @xcite , we call such models (like bert or t5) the foundation models. specifically, we utilize the byt5 @xcite foundation model. it is a byte-level generative sequence-to-sequence model, which processes a sequence of bytes of utf-8 encoding on input and produces a sequence of utf-8 encoding bytes on output. byt5 models were proposed as an alternative to multilingual mt5 foundation models @xcite and have been shown to perform remarkably well on noisy text data (compared to subword-based mt5), like tweetqa @xcite . ## experiments the multilexnorm shared task consists of 12 social media datasets in 11 languages, including two code-switching datasets. all these datasets are based on twitter data, with the danish and the dutch ones also including data from other sources. the datasets, their original sources and some of their properties are listed in we train an independent model for every dataset. the pre-training starts from the small variant of byt5, which contains 300m parameters. we keep all hyperparameters of the model unchanged, including a dropout rate of 10%. the training employs batch size 128 and the adafactor optimizer @xcite to decrease memory usage. the pre-training is performed using the synthetic data described in section 3.2, for at least 100 000 optimizer steps. 5 we utilize inverse square root decay with peak learning rate of 5 • 10 -4 and 4 000 warm-up steps. the fine-tuning phase is carried out with a constant learning rate 1 • 10 -4 for 50 epochs. we consider two training data configurations: the multilexnorm participants were allowed to submit two runs. the first run of our team úfal is a single model, while the second run is an ensemble of 4 models. 7 we perform ensembling by considering for each word and each model 16 replacements including their probabilities (using a beam-search decoder), and producing the replacement with the highest average probability. apart from the participant systems, several baselines are also evaluated: lai (leave-as-is), mfr (most frequent replacement based on the training data) and the monoise tool (van der goot, 2019). ## conclusion we presented the winning system of the w-nut 2021: multilingual lexical normalization (multi-lexnorm) shared task, which is based on a byt5 foundation model. in intrinsic evaluation, the system performance is superior by a very wide margin, and our system also delivers the best performance in extrinsic evaluation. we release both the source code at https://github.com/ufal/ multilexnorm2021 and the fine-tuned models at https://huggingface.co/ufal . in future work, we would like to change the model architecture to encode input sentences only once, either by decoding the whole sentence, or by separating all input words with different sentinel tokens and then decoding individual words by initializing the decoder with the corresponding sentinel tokens.