Context-aware S wedish Lexical Simplification (2023)

Abstract: We present results from the development and evaluation of context-aware Lexical simplification (LS) systems for the Swedish language. Three versions of LS models, LäsBERT, LäsBERT-baseline, and LäsGPT, were created and evaluated on a newly constructed Swedish LS evaluation dataset. The LS systems demonstrated promising potential in aiding audiences with reading difficulties by providing context-aware word replacements. While there were areas for improvement, particularly in complex word identification, the systems showed agreement with human annotators on word replacements.

## introduction lexical simplification (ls) is the task of replacing complex words with easier ones. the approaches to this task usually involve replacing words with simpler synonyms found in a linguistic database @xcite @xcite , implementing rules to "translate" linguistic units into easier ones @xcite , or using word embeddings to generate similar substitution candidates @xcite . as mentioned in @xcite these methods usually fail to take the context of the target word into account, resulting in nonsensical substitutions. recently, with the introduction of large-scale pre-trained transformer language models such as bert @xcite and gpt-3 @xcite a new chapter in the field of natural language processing (nlp) has begun. gpt-3 and bert perform well on a broad set of downstream nlp tasks @xcite . bert has already been implemented in ls systems for english @xcite , portuguese @xcite , spanish, and german @xcite . the benefit of using these models, trained on vast amounts of text, over conventional methods is that these models can generate more contextually appropriate substitutions for complex words, which is reflected in the high performance of these systems @xcite @xcite . the tsar-2022 shared task @xcite demonstrated that bert-based ls systems were surpassed only by a gpt-3 based method @xcite , highlighting the effectiveness of large-scale pre-trained transformers in the realm of lexical simplification. in this paper, three versions of a swedish ls system are presented: two versions of an ls system (called läsbert) inspired by the approach by @xcite using two swedish bert models for substitution generation. one version uses a bert model fine-tuned on easy-to-read-text and one uses an out-of-the-box model to investigate how fine-tuning the bert model affects the endto-end performance of the ls system. furthermore, a gpt-3 based ls system (called läsgpt) was developed which uses openai:s gpt-3 for generating substitutes. these three systems are evaluated on a newly collected evaluation dataset. ## data used in our studies various data sources were used to train the random forest classifier (rfc) for complex word identification, fine-tune bert to generate easier substitutes, and construct the first evaluation dataset for the swedish ls systems. ## creating the evaluation dataset the evaluation dataset was collected automatically and evaluated manually. the collection process began with retrieving all c1 and c2 level words in the kelly swedish list. these words represent words that are used by proficient users and were therefore assumed to be complex words. the corpus frequency of these words in the bloggmix corpus was retrieved. following this, all available synonyms to the retrieved words were saved from the synlex dictionary. the corpus frequencies of these synonyms were also saved. the final step was to find an example sentence in saldo where the complex word occurred, resulting in 185 quadruples consisting of a complex word, its corpus frequency, a dictionary of suitable synonyms, and an example sentence. after a manual annotation process, nonsensical quadruples were removed, leaving a total of 150 quadruples. three native swedish student annotators were enlisted to evaluate the dataset. the annotators assessed the quality, coverage, and complexity of the dataset. quality refers to if the alternatives were synonymous with the complex word in the context of the example sentence. coverage refers to if all possible synonyms were listed in the dataset. complexity refers to the perceived complexity of the complex word. student annotators from linköping university's cognitive science bachelors program were recruited. two online versions of the swedish academic aptitude test, högskoleprovet, (universitets och högskolerådet, 2023) were used to assess their word knowledge. the combined maximum score was 40 and the annotators scored 37, 33, and 35 respectively, indicating their strong lexical proficiency. each annotator got 50 separate quadruples to evaluate to ensure that all of the 150 quadruples in the dataset were human-annotated once. the annotators answered three questions regarding the quality, coverage, and complexity with "true" or "false" for each quadruple. in the dataset fit in the context of the example sentence (86.6% of the quadruples). for 72% of the quadruples, the annotators thought that there were no omitted synonyms that could replace the complex word in the sentence. however, as discussed by @xcite , humans generally don't recall all possible substitutions for a given word when working from memory. the perceived coverage of the dataset is therefore probably higher than the actual coverage. this has the possible effect of artificially limiting the score that a lexical simplification system can achieve on the dataset since valid substitutions could be missing in the set of correct alternatives. the annotators did generally not think that the words sourced from the kelly swedish list were complex, with only 28,6% of quadruples being annotated as complex. however, since the annotators were native swedish speakers with a university education, the perception of what constitutes a complex word might not generalise well to audiences with reading difficulties. the dataset is freely available at https://github. com/emilgraichen/swedishlsdataset . ## method in this section, we will describe the implementation and evaluation of three ls systems, each varying only in the substitution generation subtask. the developed systems are two bert-based versions of an ls system called läsbert, and one version of a gpt-3 based ls system called läsgpt. the structure of this section is based on the general pipeline of other ls systems described in section 2. ## results the performance of the random forest classifier is presented in accuracy is the proportion of all correctly classified classes in the dataset, which in the case of this classifier was 0.55. läsbert (baseline) läsbert (fine-tuned) läsgpt recall 53/150 (35.3%) 53/150 (35.3%) 49/150 (32.7%) synonymous replacements läsbert (baseline) läsbert (finetuned) läsgpt total complex words 14/150 (9.33%) 12/150 (8%) 16/150 (10.6%) replaced complex words 14/53 (26.4%) 12/53 (22.6%) 16/49 (32.7%) replacements läsbert (baseline) läsbert (finetuned) läsgpt total complex words replaced with a synonymous and more frequent word 13/150 (8.7%) 11/150 (7.33%) 15/150 (10%) synonymous that resulted in a more frequent word 13/14 (92.9%) 11/12 (91.7%) 15/16 (93.8%) läsbert (baseline) läsbert (finetuned) läsgpt true positive (annotated as complex and replaced) 26/43 (60.5%) 26/43 (60.5%) 22/43 (51.2%) true negative (annotated as non-complex and not replaced) 80/107 (74.7%) 79/107 (73.4%) 80/107 (74.7%) total agreement 106/150 (70.1%) 105/150 (70%) 102/150 (68%) the results in both läsbert versions replaced 60.5% of the words that were annotated as complex by the humans. läsgpt scored lower and replaced 51.2% of the words annotated by humans as complex. läsgpt and the baseline version of läsbert both agreed with the annotators on 74.4% of the words that were annotated as non-complex. the baseline version of läsbert had the highest overall agreement with the human annotators with 70.1% of the words being aligned with the human annotators. ## discussion the results revealed that both the läsbert and läsgpt systems had relatively low recall rates, replacing only about one-third of the complex words in the evaluation dataset. this indicates the need for improvement in the systems' ability to identify and replace complex words accurately. the cwi component of the ls pipeline was highlighted as an area for future improvement. regarding system-annotator agreement, the ls systems showed agreement with human annotators between 68% (läsgpt) and 70.1% (läsbert baseline) of the time. the läsbert versions performed slightly better, with an agreement of 60.5% for true positives, indicating that the systems and human annotators generally agreed on which words needed to be replaced. when it comes to synonymous replacements, läsgpt performed the best, with a rate of 10.6% of complex words replaced by synonyms. however, when considering only the replaced complex words, the synonymous replacement rate improved to 32.7% for läsgpt. the läsbert models demonstrated lower percentages of synonymous substitutions. furthermore, almost all synonymous replacements resulted in words with higher corpus frequencies, indicating a simplification effect. läsgpt had a slightly bigger impact on text simplification, with 10% of the words resulting in a word with higher corpus frequency. while there is still potential for improvement, the relatively low perceived complexity of the complex words in the dataset and the more promising system-annotator agreement suggests that some issues are attributable to the dataset itself rather than to the ls systems. the effects of fine-tuning the language model for substitution generation did not affect the number of words replaced by the model on this evaluation dataset. both versions performed similarly, identifying and replacing 35.3% of complex words and agreeing with human annotators 70.1% and 70% of the time, respectively. this lack of difference is assumed to be attributed to the small size of the evaluation dataset, limiting the expression of subtle effects. the evaluation also revealed that both versions had a very similar number of synonymous replacements, with next to all of these replacements also resulting in words with higher corpus frequency. interestingly, the baseline version tended to make more synonymous and simpler replacements than the fine-tuned version. this indicates that it's not worth the effort to fine-tune the language model since it seems to have a detrimental rather than beneficial effect on the end-to-end performance. the reason behind the reduced performance of the fine-tuned version remains unclear. a possible explanation is that fine-tuning had an adverse impact on the model's overall language comprehension. ## conclusion the lexical simplifiers presented in this paper do not differ substantially in their performance from each other. the läsbert versions have a slightly higher recall, whilst läsgpt performs slightly more synonymous replacements that also have a higher corpus frequency. the absolute percentage of the number of substitutions is not very high with around just a third of the complex words in the dataset being replaced by the ls systems. however, the agreement between the systems and annotators on which words should be substituted is relatively high (68% to 70.1%). there is room for improvement of the evaluation dataset. a higher proportion of perceived complex words is needed to more accurately reflect which words need to be simplified. the fine-tuning process did not have a noteworthy impact on the number of words replaced by the model. both the fine-tuned and non-fine-tuned versions identified and replaced approximately 35.3% of complex words and had a similar agreement with human annotators. however, the evaluation revealed that the baseline version tended to make slightly more synonymous and simpler replacements compared to the fine-tuned version. this suggests that fine-tuning the model may not be beneficial and could potentially have a detrimental effect on the system's performance. the exact reason for the reduced performance of the fine-tuned version remains unclear, but it may be that the finetuning process have negatively affected the model's overall language comprehension.