Domain Adaptation in Multilingual and Multi-Domain Monolingual Settings for Complex Word Identification (2022)

Abstract: Complex word identification (CWI) is a cornerstone process towards proper text simplification. CWI is highly dependent on context, whereas its difficulty is augmented by the scarcity of available datasets which vary greatly in terms of domains and languages. As such, it becomes increasingly more difficult to develop a robust model that generalizes across a wide array of input examples. In this paper, we propose a novel training technique for the CWI task based on domain adaptation to improve the target character and context representations. This technique addresses the problem of working with multiple domains, inasmuch as it creates a way of smoothing the differences between the explored datasets. Moreover, we also propose a similar auxiliary task, namely text simplification, that can be used to complement lexical complexity prediction. Our model obtains a boost of up to 2.42% in terms of Pearson Correlation Coefficients in contrast to vanilla training techniques, when considering the CompLex from the Lexical Complexity Prediction 2021 dataset. At the same time, we obtain an increase of 3% in Pearson scores, while considering a cross-lingual setup relying on the Complex Word Identification 2018 dataset. In addition, our model yields state-of-the-art results in terms of Mean Absolute Error.

## introduction the overarching goal of the complex word identification (cwi) task is to find words that can be simplified in a given text @xcite . evaluating word difficulty represents one step towards achieving simplified, which in return facilitates access to knowledge to a wider audience texts @xcite . however, complex word identification is a highly contextualized task, far from being trivial. the datasets are scarce and, most of the time, the input entries are limited or cover different domains/areas of expertise. there- ## discussions the domain adaptation technique supports our model to learn general cross-domain or crosslanguage features, while achieving higher performance. moreover, jointly training on two different tasks (i.e., lexical complexity prediction and text simplification), coupled with domain adaptation to generalize the features from the two tasks, can lead to improved results. however, there are entries for which our models were unable to properly predict the complexity score, namely: a) entries with a different level of complexity (i.e. biomedical), and b) entries part of a language that was not present in the training dataset (i.e., french). for the former, scientific terms (e.g., "sitosterolemia"), abbreviations (e.g., "es"), or complex elements (e.g., "h3-2mek9") impose a series of difficulties for our feature extractors, considering the absence of these tokens from the transformer vocabulary. the latter category of problematic entries creates new challenges in the sense that it represents a completely new language on which the architecture is tested. however, as seen in the results section, the cross-lingual domain adaptation technique offers good improvements, helping the model achieve better performance on french, even though the initial architecture was not exposed to any french example. ## conclusions and future work this work proposes a series of training techniques, including domain adaptation, as well as multi-task adversarial learning, that can be used for improving the overall performance of the models for cwi. domain adaptation improves results by encouraging the models to extract more general features, that can be further used for the lexical complexity prediction task. moreover, by jointly training the model on the cwi tasks and an auxiliary similar task (i.e., text simplification), the overall performance is improved. the task discriminator also ensures the extraction of general features, thus making the model more robust on the cwi dataset. for future work, we intend to experiment with meta-learning @xcite alongside domain adaptation @xcite , considering the scope of the previously applied training techniques. this would enable us to initialize the model's weights in the best manner, thus ensuring optimal results during the training phase.