“Geen makkie”: Interpretable Classification and Simplification of D utch Text Complexity (2023)

Abstract: An inclusive society needs to facilitate access to information for all of its members, including citizens with low literacy and with non-native language skills. We present an approach to assess Dutch text complexity on the sentence level and conduct an interpretability analysis to explore the link between neural models and linguistic complexity features. Building on these findings, we develop the first contextual lexical simplification model for Dutch and publish a pilot dataset for evaluation. We go beyondprevious work which primarily targeted lexical substitution and propose strategies for adjusting the model’s linguistic register to generate simpler candidates. Our results indicate that continual pre-training and multi-task learning with conceptually related tasks are promising directions for ensuring the simplicity of the generated substitutions.

## introduction reading is a foundational skill for acquiring new information. many sources of information are only available in written form, including educational material, newspaper articles, and letters from municipalities. although many people learn how to read as a child, not everyone becomes equally skilled at it. in the netherlands alone, more than 2.5 out of 14 million people over 16 years old are low-literate, meaning that they experience challenges with reading or writing. 3 as a result, they face obstacles in achieving academic success, seeking employment opportunities, and keeping up-to-date with current events. one way to address this problem is to reduce text complexity. texts that contain many infrequent words and complex sentence structures are difficult to read, especially for readers with low literacy and language learners. automated natural language processing tools for text complexity assessment can help both in assisting editors in the selection of adequate texts and by signaling potential comprehension problems to copywriters. by estimating text complexity, we can select texts that are sufficiently easy for a particular target audience or simplify texts that are too difficult. recent neural models for text complexity assessment have obtained good results in classifying texts into discrete categories of complexity @xcite . the global classification label can be a first indicator but it does not point to specific parts of the input that are complex, leaving it to the human editor to identify the necessary simplifications. in this work, we first explore dutch complexity prediction on the sentence level (as opposed to full-text classification in previous work) and then zoom in even further. the complexity of a text is affected by an interplay of various factors, including its structural characteristics, domain, and layout. a crucial component is the choice of the lexical units and their complexity. a system for lexical simplification can support humans in detecting lexical complexity and suggest simpler alternatives. in the sentence children bear the future, and our resolution to support them determines the world they inherit, a lexical simplification model could propose to substitute bear with simpler words such as carry, hold, or shape. these suggestions can assist human writers in revising and simplifying their text. previous approaches to dutch lexical simplification generated substitution candidates by naively substituting words according to a static alignment of synonyms without considering the context of the sentence. this approach does not account for ambiguous words and synonyms that only maintain semantic coherence in a subset of contexts. in the example above, resolution can be interpreted as intention, but in the context of tv screens, it refers to sharpness. in order to ensure meaning preservation, lexical simplification needs to be context-sensitive. contributions we fine-tune bertje @xcite , a dutch pre-trained transformer model, to predict sentence-level complexity and use interpretability methods to show that it captures relevant linguistic cues. we visualize the local attribution values of the model's predictions in a demo to point end users to complex parts of the sentence. in order to facilitate the simplification process, we introduce lsbertje, the first contextual model for lexical simplification in dutch. we explore three approaches to adapt the linguistic register of the model, to re-enforce a preference for simplicity in the generated substitutions. ## related work we discuss complexity assessment and lexical simplification as separate consecutive stages in line with related work. ## complexity classification we train a neural classifier for determining binary labels of dutch sentence complexity and compare its performance to several feature-based classifiers. we then analyze if the neural model captures relevant complexity cues. ## context-aware simplification in the second step of the simplification pipeline, we generate context-aware simplifications for dutch. lsbertje we present lsbertje, the first model for contextualized lexical simplification in dutch. we base lsbertje on lsbert @xcite by altering its language-specific components to dutch. we replace the language model that generates simplifications with the dutch bert model, bertje. we also replace the stemmer used in filtering with the snowball stemmer. foot_5 ## register adaptation techniques lsbertje relies on a base model that was pretrained for masked language modeling and captures aspects of text complexity only as an incidental byproduct. it uses a masked language modeling mechanism that induces semantic preservation by repeating the input sentence. the goal of generating simpler substitutions is only implicitly targeted by restricting the generation to tokens consisting of a single subtoken. this effectively prevents the model from generating infrequent or morphologically more complex words, but the model is not explicitly optimized for capturing different levels of text complexity. we explore three strategies to adapt the linguistic register of the model so that it generates simpler substitutions: conceptual finetuning, continual pre-training, and multi-task learning. ## conclusion in this work, we have introduced two state-of-theart components for complexity prediction and simplification in dutch. it can support teachers and text editors in making texts more accessible for people who face reading challenges. we developed a demo that predicts binary complexity labels for dutch sentences and highlights words that contributed positively or negatively to the prediction. additionally, the demo interface provides scales for different aspects of sentencelevel complexity to enable a more fine-grained interpretation by the user. we introduced lsbertje, which is the first model for contextualized lexical simplification in dutch (to the best of our knowledge). we show that the model can generate adequate simplifications without additional fine-tuning. this base setup can serve as a reasonable starting scenario for contextaware simplification generation for resource-poor languages. we developed a pilot evaluation dataset for dutch that allowed us to perform initial comparisons. for a more elaborate analysis, a larger dutch dataset needs to be curated in future work. we explored strategies to adapt the linguistic register of the model to ensure the simplicity of the generated substitutions and find that both multi-task learning and continual pre-training show considerable potential. we further analyzed the model's robustness and discovered a strong sensitivity to task-specific hyperparameters but little variation across random seeds.