Towards Content Accessibility Through Lexical Simplification for M altese as a Low-Resource Language (2024)

Abstract: Natural Language Processing techniques have been developed to assist in simplifying online content while preserving meaning. However, for low-resource languages, like Maltese, there are still numerous challenges and limitations. Lexical Simplification (LS) is a core technique typically adopted to improve content accessibility, and has been widely studied for high-resource languages such as English and French. Motivated by the need to improve access to Maltese content and the limitations in this context, this work set out to develop and evaluate an LS system for Maltese text. An LS pipeline was developed consisting of (1) potential complex word identification, (2) substitute generation, (3) substitute selection, and (4) substitute ranking. An evaluation data set was developed to assess the performance of each step. Results are encouraging and will lead to numerous future work. Finally, a single-blind study was carried out with over 200 participants, where the system’s perceived quality in text simplification was evaluated. Results suggest that meaning is retained about 50% of the time, and when meaning is retained, about 70% of system-generated sentences are either perceived as simpler or of equal simplicity to the original. Challenges remain, and this study proposes a number of areas that may benefit from further research.

## introduction lexical simplification (ls) is a technique through which complex words are replaced with simpler alternatives while aiming to retain meaning and contextual validity. although this has been the subject of various studies @xcite @xcite , it is more often adopted within high-resource languages, such as english or french @xcite . on the other hand, low-resource languages, such as maltese, lack sufficient highquality resources @xcite required for robust natural language processing (nlp). reading difficulties are widely acknowledged as a barrier to information access @xcite . for this reason, ls techniques can be adopted at the core of language-based assistive technologies (ats) to enhance content accessibility @xcite . such ats could benefit different groups of people, from non-native speakers to persons with low literacy levels and individuals with learning difficulties, among others @xcite . unfortunately, limited research exists for low-resource languages, particularly for maltese. to the best of our knowledge, the state-of-theart with respect to automated text simplification for maltese is a 2014 study based on unsupervised lexical substitution @xcite . however, @xcite argues that due to a lack of resource availability as well as choice of techniques, especially at the time, his work produces unsatisfactory results. with this in mind, the primary objective of this work is to leverage existing nlp techniques as well as arising linguistic resources for maltese to develop an effective ls system that is capable of simplifying complex words in maltese news articles, which is an easy domain for collecting high quality data. this study makes several contributions, including (1) evidence-based insights on the various ls pipeline steps and on the system overall (including perceived quality), (2) an annotated data set based on content derived from online maltese news portals in collaboration with a maltese language expert, as well as (3) a framework for implementing an ls system for low-resource languages. an at in the form of a browser extension was also developed as a reference implementation based on the arising framework; however, this is considered out of scope for this paper. this paper is based on the primary author's postgraduate research at the faculty of ict, university of malta. this research is in conformity with the university of malta's research code of practice and research ethics review procedures. the rest of the paper is organised as follows. section 2 presents a summary of related work, followed by a discussion of the data set generated for this study in section 3. section 4 describes the developed framework and evaluation of the pipeline, while results are then presented in section 5. section 6 presents concluding remarks and limitations. ## related work several ls systems exist for various high resource languages, including english @xcite , turkish (uluslu, 2022), french @xcite , spanish @xcite , and chinese @xcite , among others. to our knowledge, the only prior work that performs ls for maltese is by @xcite , a system that makes use of n-grams and bag of words vectors to determine which words can substitute a target word. the system has some issues such as a small data set that was developed exclusively by the author and a poorly performing system that only produces acceptable substitutes 5% of the time. since then, new resources became available that would allow us to make a much better system. our work is mostly inspired by lsbert @xcite . this system is adequate for low resource languages as it does not require a training set, only a pre-trained masked language model like bert @xcite . it makes use of bert predictions, token vector similarity, and word frequency to determine which words can substitute a target word. it achieves state-of-the-art results for substitute generation and outperforms baseline systems with commonly-used data sets, attaining the highest accuracy. we adapt this system for the maltese language and improve upon it in order to achieve better results. ## data set to evaluate and tune the individual modules and the ls system as a whole, a data set 1 was manually 1 https://osf.io/kx5yd/?@xmath0. to create this data set, we scraped sentences (with permission) from four popular maltese news portals foot_0 relating to articles of different news categories. in this way, the data set contains sentences typically viewed by target users of the ls system. this text-scraping approach is commonly used when compiling such a data set @xcite . the sentences were stratified by news category and number of (word level) tokens. the categories were determined by extracting the top-level category from the news web page and manually determining which category names across different websites were equivalent. only categories that were common across all websites were used, which gave us five categories across four websites: commerce, sports, lifestyle, politics, and general. four sentences per category per website were extracted, resulting in 20 sentences per website, or 80 sentences in total. this is suitable for system evaluation, as it has the same size as nnseval foot_1 , and is a manageable workload for manual annotation. the chosen sentences also had to meet sentence length requirements to avoid unusually long or short sentences. the lengths of all the sentences in the news websites formed a unimodal distribution with a peak centred between lengths 10 and 25. the sentences sampled from the categories had to fit within this range to ensure that they have a typical sentence length for news articles. once the 80 sentences were sampled, they were evenly split into two (stratified by website and category): the dev and test set. the former is used to determine the optimal system hyperparameters and the latter to evaluate the tuned system and report results. the target words that were used in the data set were selected automatically as described in section 4 (content words that are not entity names or english words). since the target words could be either complex (and thus could be simplifiable) or already simple, we refer to them as potentially complex words. this allows the data set to include instances where (i) a target word has substitutes, some of which are simpler, (ii) a target word has substitutes but none of them are simpler, or (iii) a target word does not have any viable substitutes. this is preferred over other data sets such as nn-seval which only presents complex words and their simpler substitutes since it is more representative of what the system will encounter in practice. two annotators worked to manually annotate candidate words for each target word by using a maltese thesaurus @xcite and a maltese word2vec model (w2v_cc_300d) foot_2 to assist in finding candidates. using a word2vec model was favoured over bert-based models since the latter would produce words that our system would produce which would be a bias in our favour. annotators were allowed to include candidates that are not suggested by these resources or to not use any candidates at all if necessary (in which case, the system should not substitute the target word). we recruited a professional proofreader for maltese to review and edit the manually annotated substitutes. this allows us to be more confident in the accuracy and correctness of the substitutes. the proofreader was asked to only review the substitutes in terms of meaning and context, and not simplicity, as the simplicity component is evaluated in a subsequent stage. moreover, unlike for the frenlys data set @xcite , hypernyms or hyponyms were not considered as correct candidate substitutes for most cases since these would result in changing the original sentence's meaning. moreover, the proofreader was instructed to disregard the pro-clitic foot_3 preceding the target words when checking candidates. pro-clitics change according to the word they are attached to (e.g. 'the sun', 'the sand' and 'the boy' in maltese become @xcite and so need to be fixed if the latter is substituted. the next step was selecting which candidates were simpler than the target word. this was a more subjective task, so all annotators were tasked with annotating all the sentences in order to aggregate their annotations and be able to calculate an interannotator score. the number of annotators typically recruited varies across studies: some recruit 5 per 50 sentences @xcite and some recruit 50 per sentence @xcite . we recruited 4 native maltese speakers as annotators of varying backgrounds. we deem the task as a binary annotation task rather than a scoring task, such that annotators had to only mark which candi-dates they deemed simpler than the target word (or none at all if none are simpler). we then needed to aggregate the annotations to handle disagreements. some researchers used pairwise agreement @xcite while others used the kappa index @xcite . we adopted a simpler approach: every annotation was tallied and normalised by the number of annotators (dividing by 4), generating scores for the candidates and target word. the target word would also get a score according to how many annotators considered none of the candidates to be simpler. candidates with higher scores than the target word were deemed simpler substitutes. if the highest-scoring candidate and the target word had identical scores, both are listed as simpler substitutes. an analysis of the dev set revealed that out of 280 target words: 228 included the target word among the simpler substitutes and 52 did not (one of the sentences didn't have a single target word and was ignored). since the dev set would be used to tune the system, it was important to balance these two cases to avoid biasing the model. the test set doesn't need to be balanced as it is meant to be representative of news content. to balance the dev set, we under-sampled the majority class by randomly sampling 52 target words and discarding the annotation of the rest of them (on the dev set, the system does not attempt to identify target words automatically but only works with what is annotated). apart from the maltese data set, we also wanted to test our system on an english data set that has been used to evaluate other systems in order to compare our performance. for substitute generation, we selected nnseval for this purpose due to its similar size to our data set and also due to it also being split with a 50:50 ratio, ensuring comparable results. we did not find any lexical simplification systems or data sets that are compatible with the way we select simple substitutes (as a binary classification task that includes the target word itself), so we were not able to evaluate our system on an english data set. ## results and evaluation we automatically evaluated each step in the pipeline after tuning the hyperparameters. we also conducted a human evaluation of the full system through a single-blind study. the automatic evaluation metrics are just different ways of comparing the generated substitutes with the correct substitutes (where correct substitutes are either the set of substitutable words or the set of simple words). the precision metric is the percentage of correctly generated substitutes out of all generated substitutes. the recall metric is the percentage of correctly generated substitutes out of all correct substitutes. the accuracy metric is the percentage of generated substitutes that are correct. the precision@1 metric is the percentage of target words with a correct highest-scoring generated substitute. finally, the f-score metric is the harmonic mean of precision and recall. different subsets of these metrics are used to evaluate different modules. when evaluating the sg and ss modules, we selected the evaluation metrics precision, recall, and f-score. these are the most widely used for sg @xcite @xcite . hyperparameter tuning the sg module on the maltese dev set revealed that it performs best (fscore 0.169) with pro-clitic consideration, right-toleft mask filling, use of csr, 1 mask, and a beam size of 5. the fact that 1 mask was better is surprising given the complex morphology of maltese and the small number of generated substitutes (1 mask × beam size 5 = 5 candidates). this is evidence in favour of bertu's performance which is suggesting good substitutes with just one token. on english using the english dev set, hyperparameter tuning revealed that the best performing parameters (f-score 0.196) were the same as for maltese, but using up to 3 masks instead of 1, and using bert-large-uncased. hyperparameter tuning the ss module on the maltese dev set revealed that it performs best (precision 0.188) with cosine similarity filtering, without scaling, using a similarity threshold of 0.85 and no pos tag filtering. surprisingly, pos tag filtering actually lowered both precision and recall, which probably means that the maltese pos tagger used could be improved. we opted to just reuse the hyperparameters for english as well rather than performing tuning again since there were no language specific hyperparameters like in the sg module. the results on the maltese and english test sets are shown in when evaluating the sr module, we selected the evaluation metrics accuracy, precision, recall, f-score, and precision@1. hyperparameter tuning the sr module on the maltese dev set revealed that it performs best with sg as a source for candidate words, cosine similarity for similarity scoring, no normalisation, and lightgbm as an ml model. the model gives importance to all features, but mostly to the similarity and pseudo log-likelihood scores, as shown in we compared the results of the sr module when using the candidates provided by the sg module with the results of the sr module when using the annotated candidates in the data set. this is to see how the performance of the sr module would change if the sg module was perfect. the results on the maltese test sets are shown in ## conclusion we developed and evaluated an ls pipeline for maltese, together with the compilation of a maltese ls data set that was used throughout the process. the various pipeline steps were individually evaluated, with promising results. our approach also produced significant improvements over the results obtained in the unsupervised lexical substitution system developed for maltese @xcite . the overall system was also evaluated through a single-blind study with 207 individuals. this was done to determine the overall perceived quality of the system-generated simplified text, and encouraging results were obtained as outlined in section 5.1. furthermore, participants generally concurred when presented with sentences that the system determined as already in their simplest form. arising from this work, a browser extension was also developed (malteasy), acting as a reference implementation of an ls-based at for maltese online content accessibility. although not presented in this paper, malteasy provides the team with a first-cut design that motivates the need for further framework improvements and user studies, clearly informing the future of this work.