Multilingual Sequence Labeling Approach to solve Lexical Normalization (2021)

Abstract: The task of converting a nonstandard text to a standard and readable text is known as lexical normalization. Almost all the Natural Language Processing (NLP) applications require the text data in normalized form to build quality task-specific models. Hence, lexical normalization has been proven to improve the performance of numerous natural language processing tasks on social media. This study aims to solve the problem of Lexical Normalization by formulating the Lexical Normalization task as a Sequence Labeling problem. This paper proposes a sequence labeling approach to solve the problem of Lexical Normalization in combination with the word-alignment technique. The goal is to use a single model to normalize text in various languages namely Croatian, Danish, Dutch, English, Indonesian-English, German, Italian, Serbian, Slovenian, Spanish, Turkish, and Turkish-German. This is a shared task in “2021 The 7th Workshop on Noisy User-generated Text (W-NUT)” in which the participants are expected to create a system/model that performs lexical normalization, which is the translation of non-canonical texts into their canonical equivalents, comprising data from over 12 languages. The proposed single multilingual model achieves an overall ERR score of 43.75 on intrinsic evaluation and an overall Labeled Attachment Score (LAS) score of 63.12 on extrinsic evaluation. Further, the proposed method achieves the highest Error Reduction Rate (ERR) score of 61.33 among the participants in the shared task. This study highlights the effects of using additional training data to get better results as well as using a pre-trained Language model trained on multiple languages rather than only on one language.

## introduction information can be found in abundance on social media. in fact, social media is a significant data resource for many natural language processing (nlp) tasks. such text is characterized by 'noise', or non-standard language filled with lexical variants, acronyms, hashtags, and mentions. hence, the performance of conventional nlp tools on social media text, is poor as the data resources they used were standard texts. although data extracted from social media contains a high degree of noise, the data can be extracted in huge quantities because of fast speed and casual style. existing natural language processing techniques face numerous challenges as a result of the innovative/creative/slang language usage present on social media. slang, acronyms, and abbreviated words are all part of this so-called new "online" language. it is demonstrated by @xcite that the performance of nlp tools often drops dramatically when used on social media data. the challenge can be solved by adapting the input text to a more conventional format, a process known as lexical normalisation. this paper focuses to solve the problem of lexical normalization on a word-by-word or token-by-token basis. figure 1 shows examples of un-normalized and the corresponding normalized sentences. the following languages are to be lexically normalized in this task: croatian (hr), danish (da), dutch (nl), english (en), indonesian-english (iden), german (de), italian (it), serbian (sr), slovenian (sl), spanish (es), turkish (tr), and turkish-german (trde). the proposed model is based on a sequence labelingbased technique in which the input tokens are in an un-normalized form and the target tokens are in normalized forms. to reduce the target labels and to make predictions faster, only those tokens for which normalization is required are added. for the tokens which do need not be normalized, their target label fixed to be a single target token (for instance 'o'). this sequence labeling model is finetuned on a pre-trained multilingual model to encompass the information on all the possible languages. further, a post-processing layer for word-alignment is applied which further help to improve the performance in terms of error rate reduction (errword-level accuracy normalized). this paper elucidates four essential points to improve lexical normalization systems.: it was later discovered by @xcite der goot, 2019) that the detection step can be completely avoided by treating each original word as a candidate for normalization. at present, the stateof-the-art model for almost all the languages is monoise (van der goot, 2019) which is a two-step approach. in this paper, there are different techniques for generating candidates. to get the best or correct candidate, monoise uses random forest as a classifier that takes input features generated in the first step (candidate-generation step). there are methods using sequence-to-sequence (seq2seq) based architectures (lourentzou et al., 2019) and contextual words and phrasal embeddings (muller et al., 2019) which achieved a performance similar to that of monoise, especially in the english language. most of the work in lexical normalization is performed for the english language (han and baldwin, 2011; xu et al., 2015). nonetheless, research has been conducted on languages other than english by (van der goot, 2019). ## multilingual sequence labeling approach the proposed system architecture is depicted in figure 2 . the main components of multilingual sequence labeling approach to solve lexical normalization problem system architecture are as follows: ## results and discussions this paper used error reduction rate (err) (van der goot et al.) as an evaluation metric for intrinsic evaluation. err represents • tp: annotators normalized and the proposed system normalized correctly. for the extrinsic evaluation, dependency parsing is considered. here, the impact of lexical normalization on the quality of the parsed trees is computed using the las, i.e., the percentage of words which have both the correct syntactic head and the correct dependency label. ## conclusion and future scope in this paper, we propose a multilingual sequence labeling based approach to solve the problem of lexical normalization. we show that a single model is capable of performing well on all languages. we finetuned multilingual bert on a downstream task of lexical normalization along with finetuning of some of its hyperparameters. our proposed system, trinkaai, scored an average err score of 43.75 in the intrinsic evaluation and an average las score of 63.31 in the extrinsic evaluation. further, trinkaai achieved the highest err score of 61.33 for the spanish language. this denotes that the performance of the system can be improved considerably by incorporating additional data during training. trinkaai also achieved the highest las score of 60.31 for the en-tweetbank2 test data. the performance of our proposed system can be further improved by training the model with 'large' configurations having a high number of parameters. the accuracy can be improved by additional data and by using language-sensitive embeddings.