The BEA 2024 Shared Task on the Multilingual Lexical Simplification Pipeline (2024)

Abstract: We report the findings of the 2024 Multilingual Lexical Simplification Pipeline shared task. We released a new dataset comprising 5,927 instances of lexical complexity prediction and lexical simplification on common contexts across 10 languages, split into trial (300) and test (5,627). 10 teams participated across 2 tracks and 10 languages with 233 runs evaluated across all systems. Five teams participated in all languages for the lexical complexity prediction task and 4 teams participated in all languages for the lexical simplification task. Teams employed a range of strategies, making use of open and closed source large language models for lexical simplification, as well as feature-based approaches for lexical complexity prediction. The highest scoring team on the combined multilingual data was able to obtain a Pearson’s correlation of 0.6241 and an ACC@1@Top1 of 0.3772, both demonstrating that there is still room for improvement on two difficult sub-tasks of the lexical simplification pipeline.

## introduction the lexical simplification pipeline is a family of systems designed to automatically identify and replace complex vocabulary with simpler alternatives @xcite . the lexical simplification pipeline provides a more targeted approach to simplification than automated text simplification @xcite @xcite which directly rewrites entire sentences. the two core 1 https://github.com/mlsp2024/mlsp_data/ operations included in the lexical simplification pipeline are (1) lexical complexity prediction (lcp) and (2) the replacement of complex words with simple synonyms. lcp @xcite @xcite , a form of complex word identification (cwi) @xcite , involves assigning continuous values (0-1) to given tokens in context, representing the difficulty that an intended reader population may associate with that target word. the second task, often referred to just as lexical simplification (ls) @xcite involves generating simple substitutions for target words in context. this task has been explored for single words and multi-word expressions, and is related to the identification of simple paraphrases @xcite . we previously identified two shortcomings of current work on the lexical simplification pipeline @xcite as follows: we introduce the multilingual lexical simplification pipeline (mlsp) shared task, which provides a newly annotated dataset across 10 languages for lcp and ls. the annotations for both these tasks are provided on common targets in common contexts allowing further exploration of the interplay between the two tasks and evaluation of the full pipeline on common datasets. we release data in english, spanish, french, portuguese, sinhala, filipino, japanese, italian, german and catalan. of these languages, there were previously no available lcp resources for portuguese, sinhala, filipino, italian or catalan and no ls resources available for sinhala, filipino, italian, german or catalan. in the remainder of this findings paper, we overview previous related shared tasks (section 2); give a description of the task (section 3); overview the preparation of our shared task dataset (section 4); the participating systems (section 5) and the results (section 6). we conclude with a discussion of wider factors affecting our task (section 7). cwi 2016 at semeval: at semeval 2016, the cwi task @xcite requested participants predict which words in a given sentence would be considered complex by a nonnative english speaker. a new dataset composed of 9,200 instances was created. the task attracted 21 teams which produced a total of 42 systems. a post-completion analysis @xcite highlighted the difficulty of the shared-task. the authors claimed that a disproportionate train/test split with over 40 times more test data, together with low inter-annotator agreement, was to blame for poor system performances. cwi 2018 at bea: the bea 2018 cwi shared task @xcite proposed to tackle cwi in english, german, and spanish (training and test data were provided), together with a multilingual task with french as a target language without training data. teams were asked to classify words as either complex or simple (binary) and/or provide a probability for the complexity of each word. the shared task attracted eleven teams. alexs 2020 at iberlef: additionally, the iberlef 2020 forum proposed a shared task on spanish cwi(ortiz-zambranoa and montejo-ráezb, 2020). this workshop attracted seven teams, of which three submitted to the final task. the teams competed on the newly annotated vytedu-cw corpus which provided binary complexity judgments over educational texts. lcp 2021 at semeval: the semeval 2021 shared task on lcp @xcite also provided a new dataset for complexity detection for single words and multi-word expressions in english attracting 55 teams. annotations were provided as continuous complexity judgements as opposed to binary complexity values. teams made use of deep learning based approaches to predict lexical complexity values across the corpus. simpletext 2021 at clef: the simpletext workshop @xcite has been running at clef since 2021. this workshop aims to provide benchmarks and datasets for the improvement of the accessibility of scientific information. the workshop provides datasets that participants can compete on each year in the areas of: (1) passage selection for the creation of simplified extractive summaries; (2) identification of difficult concepts and (3) query-based simplified rewriting of scientific abstracts. tsar 2022 shared task on ls: the tsar-2022 shared task @xcite provided annotations for ls in english, spanish and portuguese. participants were required to predict up to 10 simple substitutions for a complex word in each language. participants were free to contribute to one, two or all three languages. 14 teams submitted 60 runs across the three languages. successful systems made use of prompt engineering @xcite with large language models, as well as incorporating feature-based approaches @xcite . ## task description our dataset consists of instances of marked words in context, where participants are required to develop systems that first identify the complexity level of the marked word and then provide suggestions for appropriate simplifications. this unites the two previous tasks of lcp and ls into a single task, executed on common data. we have provided test data in 10 languages (catalan, english, filipino, french, german, japanese, italian, portuguese, sinhala, spanish) with our final dataset totalling 300 trial instances and 5627 test instances. participants were free to choose which tasks and language tracks they participated in. ## data and resources we initially provided participants with labelled trial data only (30 instances across 10 contexts per language, designed to indicate the format of the task). we did not provide training data, but instead pointed participants to existing resources for lcp and complex word identification arising from previous shared tasks. we have provided a simplified example of the task presented to participants below: ## results the full results for lcp and ls are displayed in appendix b and appendix c respectively. each team was permitted to submit up to 3 runs per language track, with teams permitted to submit to both the combined track and the individual language tracks. the id field indicates the run id of the participants systems. where teams submitted a separate system to the combined track, the results for each individual language were also separately processed and included in the results tables for the individual language tracks, these are indicated by a run id preceding with 'a'. all team outputs can be found via github 9 . whilst all systems provide interesting insights into the nature of the lexical simplification pipeline, we have chosen to highlight a small number of systems below. the full descriptions of each system are available in the proceedings. the results demonstrate that the gpt-4 based approach of the tmu-hit team performed well across both tasks and all language tracks. this system consistently outperforms the baseline and is consistently the first or second highest ranked system. prompt-based strategies have previously proved to be effective for ls, but not for the lcp task. the archaeology submission based on machine translation performs well for lcp, ranking as the second team in the combined track. this system uses a feature-based regression, demonstrating that this is still a competitive approach. the system does not perform as well on the ls task, and this is likely due to the challenge of correctly identifying targets after back-translation. the retuyt-inco submission attains second place in lcp for catalan, filipino, sinhala and spanish. this submission made use of bespoke resources, including synthetic data for low-resource languages. the competitive performance of this submission on these tracks indicates that this approach may be appropriate for future low-resource languages that cannot be handled through a conventional prompt-based approach. the gmu team attained first place for the en-lcp task, setting a new hard to beat baseline for this dataset. their approach also attains strong ls results for all languages, consistently attaining the 2nd or third ranked team in each language and ranking as the second team on the combined track. finally, the isep team chose to only compete in a reduced set of languages for the ls task. this focus allowed them to submit a competitive system for catalan (1st place), portuguese (1st place), french (2nd team) as well as english (4th team) and german (4th team), outperforming the baseline in all cases. we provided a simple baseline for lcp based on word-frequency and for ls based on a simple llm-prompting strategy following prior work. the baseline is included in all results tables as 'baseline', except for the combined results table, where we have not included a baseline result. we have sorted each results table, including the baselines, according to the pearson's correlation for lcp and acc@1@top1 for ls and we refer to systems 'above the baseline' in this context. for lcp our baseline system was generally competitive, expect for sinhala. the system was based on word frequencies and the frequencies we had available for sinhala were not suitable for the task. our baseline received a negative correlation to the gold labels for sinhala (as did several participant systems). for other systems, our baseline performs strongly (ranking between the 2nd and 4th system for all languages except for english and sinhala) confirming our hypothesis that word frequency would be a strong indicator of lexical difficulty. for english, the baseline system attains a strong correlation of 0.7480, but is outperformed by 9 other systems. the english lcp track was more subscribed than any other. for ls, our baseline system received mixed results, generally attaining a mid-table ranking. our approach was to reuse the prompt from the previous ls shared-task winner, which is a similar strategy to many of the submitted systems which also further improved on this same approach. our system performs particularly poorly for filipino and for sinhala, and this is likely the result of the base language model lacking training data for these languages. although we have ranked our systems according to pearson's correlation for lcp, it is also interesting to observe the r 2 metric of each system as compared to the baseline. the r 2 metric describes the proportion of variance captured by the system's results, i.e., how well do the lcp values returned by the system describe the lcp values in the gold labels. a negative r 2 indicates that the returned values are a poor fit to the gold values, whereas a positive r 2 indicates a good fit. our baseline attains a positive r 2 for all systems, except for catalan and sinhala. notably, for english the baseline system attains the highest r 2 of any system. this is also true for filipino (all other systems have negative r 2 ), german and portuguese. this indicates that although systems are able to provide correlative lcp judgements, additional factors are still required to fully represent the underlying data distributions. ## discussion we provided 10 languages for the evaluation of lcp and ls. unsurprisingly, the most subscribed language track was english, with the most prior work and existing resources in nlp concentrated on english. we hope to address the imbalance in lcp/ls research by providing equal amounts of data for all languages that we have included. the english submissions attained the highest scores overall for lcp and ls, demonstrating that the english task is better resourced. further developments in multilingual nlp and in bespoke resources for individual target languages will help to improve the performance of other systems on the tasks in our dataset. our dataset covers widespread global languages such as english, spanish and french. there are a disproportionate number of languages in our dataset that are influenced from the romance family (spanish, catalan, french, italian, portuguese). we hope to extend the dataset in further iterations to include other widespread languages such as mandarin chinese, hindi, modern standard arabic and bengali. in addition to focussing future development on widespread languages, our work has also shown that lcp and ls can be effectively applied to low-resource languages. future work to develop lcp/ls resources using the multils framework @xcite which we have followed will be incorporated into our dataset to enable the ls task for wider digital communities. whereas previous approaches to lcp have focussed on regression studies, e.g., using a language model with a regression head, it is interesting to note that many of the systems were able to use a prompting strategy to get good results for the lcp task. the tmu-hit system relies on prompting to generate n judgements, effectively forcing the lm to undertake the annotation task. this proves effective across many languages. the use of language models to replicate the annotators is an interesting area of future exploration which may have significant repercussions across other similar lexical semantics tasks such as hate speech or sentiment analysis. nonetheless, feature based systems such as the frequency baseline and the feature-based regression of the archaeology team still performed competitively, demonstrating that this can be an effective method for lcp, especially when large language models are not available for the target language. the principal strategy for the ls task employed by our participants was through prompt engineering. it is worth noting that several of the top-ranked submissions on this task used gpt4/gpt3.5, both of which are closed-source proprietary models. whilst differing prompt engineering strategies were employed throughout the task, it is very difficult to separate the differences in performances that can be attributed to (a) the prompting strategies used and (b) the language models that they have been applied to. a possible future strategy to prevent model-variance may be to provide all teams access to some common model and enforce its use in a task. ## conclusion we present the findings of the 2024 multilingual lexical simplification pipeline shared task hosted at the 19th workshop on innovative use of nlp for building educational applications. we provided the first multilingual dataset for lcp and ls on common targets, spanning ten languages and nearly 6,000 instances. ten teams participated in our task employing a range of llmbased strategies at the forefront of modern nlp. seven teams submitted system description papers. our shared task has progressed the forefront of lexical simplification research and the organisers look forward to seeing future multilingual lexical simplification research born of these efforts. all datasets, baselines and participant submissions are available through the mlsp2024 github organisation foot_8 . the japanese section of the dataset was created as a part a joint research project of nara institute of science and technology and nikkei, inc. horacio saggion, stefan bott and akio hayakawa acknowledge funding from the european uniont's horizon europe research and innovation program under the grant agreement no. 101132431 (idem project) -views and opinions expressed are however those of the author(s) only and do necessarily reflect those of the european union. horacio saggion, stefan bott and akio hayakawa also thank the support from departament de recerca i universitats de la generalitat de catalunya (ajuts sgr-cat 2021). the english section of the dataset was annotated thanks to the manchester metropolitan university rise programme. the organisers would like to extend their thanks to the organisers of the rise programme and to the student annotators who participated in the annotation process for their timely completion and attendance at annotation meetings. the organisers would like to thank all the annotators that participated across the ten languages we have provided, both paid and unpaid. this work would not have been possible without the kind support of friends and professional connections within the organiser's networks. we would also like to thank the ten teams, consisting of 37 colleagues at various levels of their academic journeys, for their participation in the shared task. particular thanks go to those that provided reviews for other participants shared task submissions. finally, we would like to thank the organisers of the 19th workshop on innovative use of nlp for building educational applications (bea 2024), who kindly hosted our shared task. particular thanks go to victoria yaneva for serving as liaison to the bea organisers and also to andrea horbach for her help in facilitating the softconf track.