P oly U - CBS at TSAR -2022 Shared Task: A Simple, Rank-Based Method for Complex Word Substitution in Two Steps (2022)

Abstract: In this paper, we describe the system we presented at the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022) regarding the shared task on Lexical Simplification for English, Portuguese, and Spanish. We proposed an unsupervised approach in two steps: First, we used a masked language model with word masking for each language to extract possible candidates for the replacement of a difficult word; second, we ranked the candidates according to three different Transformer-based metrics. Finally, we determined our list of candidates based on the lowest average rank across different metrics.

## introduction the notion of linguistic complexity has been widely debated in both theoretical and computational linguistics, and has been interpreted very differently depending on the discipline. specifically, in the field of natural language processing (nlp), complexity has often been associated with the difficulties that language users encounter while processing concrete linguistic productions (e.g., sentences, utterances, etc.) @xcite @xcite @xcite , with research focusing on applications that aim to simplify challenging texts and to make them more easily readable for a wider variety of users @xcite . previously, nlp shared tasks focused on the problem of identifying a complex word in a sentence, or assigning a difficulty score to it @xcite . the tsar-2022 shared task @xcite instead focused on the next step; that is, how to find simpler words as replacement candidates for a given target word in a multilingual setting. consequently, the task can be seen as similar to lexical substitution in context @xcite . in this paper, we describe our contribution to the tsar-2022 shared task, which is a system for english, portuguese, and spanish that i) generates replacement candidates for a given word via masked language modeling, and ii) assigns scores to the candidates by averaging the ranks assigned by different transformer-based metrics. ## related work the goal of the previous shared tasks regarding lexical complexity was to identify complex words in a sentence context, and complexity was defined as a binary variable @xcite . however, these tasks were oversimplified because there is no clear-cut choice in many contexts, and human annotators prefer to assign a score based on a continuous scale of difficulty. @xcite introduced the complex corpus, a gold-standard benchmark for lexical complexity in english, in which words and multiword expressions are extracted from different text genres (legal, religious, and biomedical genres) and are annotated with continuous scores that reflect their difficulty in the sentence context. the same corpus was then used as the source material for the semeval-2021 shared task regarding lexical complexity in context @xcite . the estimation of lexical complexity is only one component in the lexical simplification pipeline, which also involves generating candidates for substitution, ranking them, and assessing their degree of fitness in the given sentence context. datasets focusing on the latter parts of the pipeline have been published for english @xcite @xcite , japanese @xcite , portuguese (hartmann and aluísio, 2020; @xcite , french @xcite con, 2021; @xcite , and chinese @xcite . the current state-of-the-art system for english, lsbert, was introduced by @xcite . the system first generates a list of possible replacement candidates via the masked language modeling function of bert @xcite by being fed the original sentence concatenated with a copy of the sentence in which the original word has been masked. the system then performs a re-ranking using different features, e.g. frequency, vectorbased semantic similarity, and/or language model probability. studies using lsbert @xcite have shown that the approach could easily be adapted to other languages and still achieve state-of-the-art results. ## results and conclusion the results for english, portuguese and spanish can be seen, respectively, in a possible reason is that the initial selection of the candidates is already based on a transformer language model, so it could be the case that the information coming from the language model-based rankings is redundant, or tend to suggest the same subset of candidates. on the other hand, the cosine metric between the contextualized embeddings is assessing a paradigmatic type of similarity between the target and the candidate word: this is not necessarily taken into account by the other metrics, which are more focused on the syntagmatic axis. our method relying on base + cossim, which was submitted as polyu-cbs3, was the one reporting the best scores on all the three datasets (15th overall on english, 5th on spanish, 3rd on portuguese). it is noticeable that our methods always outperform tuner on the metrics of potential and map. the lsbert is the best performing method on english and portuguese datasets, although our base + cossim is a close match to the latter. finally, base + cossim outperforms both tuner and lsbert on spanish. we take the results as a preliminary evidence that our method, based on two simple steps and ranking, can be highly effective for the task of lexical simplification. a possible way to further improve the methodology will be to introduce different methods of extracting candidate words. in our preliminary experiments, we found that a similarity ranking based on traditional, static embedding model alone can lead to improvements of the performance on english. however, for languages with a richer morphology like the romance ones, a morphological adapter would be needed to generate the form that best fits the target sentence. another possible direction could be using a generative model treating the task as a text-to-text problem @xcite , which could be finetuned on supervised lexical substitution data and combined with a frequency filter to ensure that the proposed replacement is actually a simpler word.