M ulti L ex N orm: A Shared Task on Multilingual Lexical Normalization (2021)

Abstract: Lexical normalization is the task of transforming an utterance into its standardized form. This task is beneficial for downstream analysis, as it provides a way to harmonize (often spontaneous) linguistic variation. Such variation is typical for social media on which information is shared in a multitude of ways, including diverse languages and code-switching. Since the seminal work of Han and Baldwin (2011) a decade ago, lexical normalization has attracted attention in English and multiple other languages. However, there exists a lack of a common benchmark for comparison of systems across languages with a homogeneous data and evaluation setup. The MultiLexNorm shared task sets out to fill this gap. We provide the largest publicly available multilingual lexical normalization benchmark including 13 language variants. We propose a homogenized evaluation setup with both intrinsic and extrinsic evaluation. As extrinsic evaluation, we use dependency parsing and part-of-speech tagging with adapted evaluation metrics (a-LAS, a-UAS, and a-POS) to account for alignment discrepancies. The shared task hosted at W-NUT 2021 attracted 9 participants and 18 submissions. The results show that neural normalization systems outperform the previous state-of-the-art system by a large margin. Downstream parsing and part-of-speech tagging performance is positively affected but to varying degrees, with improvements of up to 1.72 a-LAS, 0.85 a-UAS, and 1.54 a-POS for the winning system.

## introduction the rise of social media has led to a tremendous increase in the amount of data shared over the internet. but because of its spontaneous nature, the data naturally abounds with numerous language variations, both intended (e.g., slang, abbreviations, non-standard capitalization) and unintended ones (e.g., typos). this, in turn, poses considerable problems for existing natural language processing (nlp) tools (e.g., @xcite , most of which were originally designed to process canonical texts. one way to improve the performance of such systems is to normalize text and thus make it more similar to the data the nlp systems were initially designed for (and trained on). at this point, to avoid confusion with other existing notions of text normalization (cf. @xcite , we should state that, throughout this paper, we will only deal with lexical normalization-a task which @xcite define as "a mapping from 'ill-formed' outof-vocabulary (oov) lexical items to their standard lexical forms." we focus only on social media data, as opposed to historical data @xcite or medical data @xcite , and extend the scope of this task further to the cases where wrong in-vocabulary (iv) tokens can be normalized to (i.e., replaced with) their invocabulary counterparts, arriving at the following formulation: ## data our selection of languages is purely based on dataset availability. we are aware that the benchmark contains mostly indo-european languages, and encourage additions to this benchmark in the future to increase language variety. 2 we kindly request future work to cite the original data sources, and provide the bib-files on our website. lexical normalization is a subjective task, as in many cases multiple interpretations and annotations are plausible. furthermore, annotators may disagree on what is "normal", and whether normalization is necessary for certain words. we have summarized all results on studies on inter-annotator agreement that we are aware of in two types of agreement are reported in the literature: (1) cohen's/fleiss' kappa score on the choice of whether a word is in need of normalization; and (2) "same candidate", which reports the percentage of times annotators agreed on the normalization replacement for words normalized by multiple annotators. results in besides converting all data to the same format, we have attempted to converge annotation styles whenever possible. in particular, we applied the same normalization annotation in these cases: 2 please contact the first author of this paper if you are interested in adding a language. if more languages are added, future versions of multilexnorm will be released via the repository. with regard to data availability and composition, we note that some of the datasets were published before the shared task was held. 3 all datasets contain data from the twitter platform; the dutch corpus also includes forum and sms data, and the danish dataset includes texts from arto, a danish social media network. more details about the data collection for each dataset can be found in the dataset statement (appendix a). an overview of our datasets is shown in ## extrinsic evaluation we perform a main extrinsic evaluation of the impact of normalization on dependency parsing using universal dependency annotations @xcite . see section 5.2 for test set details. we used version 2.8 of all treebanks, and syntacticallysplit multiword tokens and empty nodes (ellipsis) are undone with ud-conversion-tools. 6 we trained machamp (van der goot et al., 2021) with default settings and xlm-r embeddings @xcite . we use the largest canonical treebank of each of the respective languages as the source domain, and attempt to improve performance on the target domain by normalizing data first. in other words, we take the input text, use a normalization system to get the normalized version, and pass this to the parser as input. the parsers are not trained on social media data, to evaluate the impact of normalization on parsing in a domain-shift scenario. because the normalized version should be closer to the canonical training data of the parser, performance is expected to improve compared to using the input directly. the training treebanks are ud_english-ewt @xcite , ud_german-gsd @xcite , ud_italian-isdt @xcite , and ud_turkish-imst @xcite . for the extrinsic evaluation, we report the average over three runs with different random seeds (only the parser is retrained, not the normalization models). ## conclusions with multilexnorm, we have developed a multilingual benchmark for lexical normalization consisting of previously-created datasets spanning 12 language variants. we proposed a standard evaluation metric, and both intrinsic and extrinsic evaluation via dependency parsing and pos tagging. we hosted a shared task with this new benchmark, which enabled comparison of performance of 21 models (18 submissions by participants, and 3 inhouse baselines). the results of the shared task show that the previous state of the art on lexical normalization is outperformed by a large margin. the extrinsic evaluation on dependency parsing and pos tagging shows that lexical normalization is beneficial (with improvements in a-las and a-uas of up to +1.72 and +0.85, and improvements in a-pos of up to +1.54, respectively), but there is still a performance gap compared to the performance levels observed on canonical data. we hope that the proposed benchmark will lead to more research in multilingual normalization, and more transparent and fairer comparisons. all submissions, evaluation scripts, and baseline models are available in the shared task repository.