Reading Between the Lines: A dataset and a study on why some texts are tougher than others (2025)

Abstract: Our research aims at better understanding what makes a text difficult to read for specific audiences with intellectual disabilities, more specifically, people who have limitations in cognitive functioning, such as reading and understanding skills, an IQ below 70, and challenges in conceptual domains. We introduce a scheme for the annotation of difficulties which is based on empirical research in psychology as well as on research in translation studies. The paper describes the annotated dataset, primarily derived from the parallel texts (standard English and Easy to Read English translations) made available online. we fine-tuned four different pre-trained transformer models to perform the task of multiclass classification to predict the strategies required for simplification. We also investigate the possibility to interpret the decisions of this language model when it is aimed at predicting the difficulty of sentences in this dataset.

## introduction the universal declaration of human rights, in its article 19, affirms everyone's right to seek and receive information. similarly, article 21 of the un convention on the rights of persons with disabilities underscores the need for accessible formats, ensuring that individuals with disabilities can access public information without additional cost. for people with intellectual disabilities-those with limitations in cognitive functioning, including difficulties in reading and understanding, an iq below 70, and challenges in conceptual domains (american association on intellectual and developmental disabilities (aaidd), n.d.)-language simplification is crucial for ensuring accessibility and equality, making it essential for them to fully enjoy their human rights. text simplification (ts) research aims to make text easier to read while preserving its meaning and key information @xcite . earlier studies involved lexical, syntactic and semantic modifications, while modern research benefits from the use of large language models (llms), with still unclear cost-to-performance benefits, as they do not outperform smaller pre-trained language models (plms), such as bert, on text classification tasks @xcite . computational studies often overlook insights from translation studies, particularly the various strategies proposed @xcite @xcite @xcite , focusing on the systematic processes involved in translating a source text into a target text across languages. translation studies provide a complementary lens by examining strategies used in intralingual translation, where a source text is converted into a target text in the same language. @xcite argue that such shifts often achieve full correspondence between source and target texts. of particular relevance are two types of intralingual translation. diamesic translation involves shifting communication modes (e.g., spoken to written) while retaining the same language @xcite . diastratic translation, on the other hand, involves register shifts within the same language, such as from standard english (se) to easy to read (e2r) english, i.e. the variation of language that is easy to read and understand for people with reading difficulties, including people with intellectual disabilities, people with little command of the language, people with poor literacy and so forth @xcite . compared to standard language e2r language is a simplified version for the sake of readability for specific audiences @xcite . as a result, it forms the foundation of diverse and adaptable translation strategies designed to make information accessible to people with intellectual disabilities. previous studies in text simplification have pri-marily focused on lexical simplification, where individual words or phrases are simplified without considering the broader sentence structure or context. for instance, @xcite developed datasets and tools specifically tailored for lexical simplification tasks, emphasising wordlevel transformations. while this approach has proven effective for specific applications, it often overlooks the interplay between lexical and syntactic features within a sentence. other notable resources, such as the asset corpus (alva-manchego et al., 2020), have focused on sentence simplification but rely on predefined, fine-grained operations at the word or phrase level. similarly, corpora like wikilarge @xcite offer paired datasets for simplification but lack explicit annotations for the strategies applied during simplification. these resources are invaluable for training machine learning models but are limited in their ability to capture a comprehensive view of the simplification process. in contrast to the resources mentioned above, our dataset adopts a holistic approach to sentence simplification, focusing on sentence-level transformations that encompass lexical, syntactic, and semantic changes, while focusing on the reason to make these changes. unlike lexical simplification datasets, which isolate individual words or phrases, our dataset explicitly annotates entire sentences with six predefined categories representing diverse simplification strategies. this allows for better understanding of the simplification process, capturing how different strategies interact within a sentence to enhance its readability and accessibility. furthermore, by annotating se and e2r sentence pairs, our dataset provides a unique resource for exploring context-sensitive simplification strategies. this makes it particularly valuable for tasks that require an integrated understanding of sentence-level transformations. this study explores strategies to make information more accessible through text simplification. our contributions concern: (1) the development of an extended taxonomy of translation strategies that integrates insights from text simplification research, (2) the annotation of a parallel corpus of complex and simplified texts sourced from diverse public services in scotland (see section 2), (3) the investigation of setting to train transformer-based models to predict the application of specific simplification strategies, and (4) an investigation into interpretability of their predictions using explain-able ai (xai) techniques to explain the model's decision-making process. while large language models (llms) demonstrate impressive performance, their "black-box" nature often makes it challenging to understand their predictions. to address this, we employ integrated gradients @xcite , an xai method grounded in axiomatic attribution principles. ig identifies the most influential words in the input by analysing gradient variation. by aligning these attributions with human judgments, we enhance the interpretability of the model and build trust in its application. ## dataset the original corpus consists of over 76 parallel texts, primarily sourced from the scottish care service, political manifestos for the 2024 uk general election, and newsletters from the national charity disability equality scotland. these texts span a diverse range of topics, including health care services, environmental policies, the legal system, waste management, disability advocacy, and linguistic accessibility. in the field of translation studies, many taxonomies have been developed to identify the strategies professional translators apply when producing a target text. most of these strategies have been developed in the field of interlingual translation, first from a written text into another written text @xcite @xcite , and then from a spoken text into a written text @xcite @xcite @xcite . the study of intralingual translation strategies is relatively more recent and mainly focuses on diamesic translation @xcite @xcite @xcite . rarer is the number of authors who have tried to define strategies for the translation of written texts within the same language @xcite . to our knowledge, only silvia hansen-schirra and sommer (2020) and @xcite have addressed intralingual translation practices into e2r. however, none of these taxonomies completely satisfy our need to account for all the simplification strategies we identified in our corpus, as too little detail was provided. the opposite happens in the completely different field of automatic text simplification (ats), where details are, instead, provided. here, the focus of typologies is on lin-guistic descriptions and string edits. a significant contribution in ats has been provided by @xcite , whose typology essentially focuses on operations that mainly deal with adding, deleting, replacing, and moving words. however, texts translated in e2r language clearly show that professionals in the field apply many more operations that pertain to the field of pragmatics and semiotics, focused on how concepts are distributed and or explained to help the user understand them. it is in this context that this section will try to illustrate the annotation framework that we have developed and used in this study. because the form of translation we are focussing on in this paper is diastratic (from se to e2r), we used inclusion europe's pioneering guidelines inclusion europe (2009) as a basis for our annotation framework, which was then used to identify the strategies used in our corpus. the principle of inclusion europe's guidelines is language simplification, further subdivided into three levels: lexical, syntactical, and semantic. the lexical level mainly focuses on the use of nouns, verb tenses, adjectives, and adverbs. in particular, the guidelines require to only use basic vocabulary words. for the english language, the basic vocabuwords -contains 850 commonly used word roots, like thing, do, good, or very. the syntactical level mainly focuses on the use of the order of words and clauses in a sentence, and that of sentences in the text. in particular, the guidelines require to only use a (chrono-)logically linear word, clause, and sentence order. the semantic level mainly focuses on the distribution of concepts in the text. in particular, the guidelines require one concept per sentence. information for all also add other pieces of information, like the use of pictograms to reinforce the information provided in the text. however, these will not be considered in the present study. based on these principles, and a qualitative analysis of the illustrated corpus, we came up with the following nine macro-strategies, that easily adapt to our heterogenous corpus. macro-strategies are further subdivided into strategies and micro-strategies. the macro-strategies have been thought as points in a continuum between two poles: those resulting in most addition of text (explanation) to those resulting in the most deduction of text (omission), the middle being constituted by transcription, with no addition or deduction of text (figure 1 ). examples are taken from our corpus. ## classification model: multiclass text classification with transformers this experiment investigates the application of pretrained transformer-based models for multiclass text classification, focusing on the prediction of simplification strategies need to simplify the respective se sentences. for this experiment, seven categories were manually annotated for a selection of 155 complex sentences and their 161 corresponding simplified sentences, randomly selected from various texts see the annotation process consisted of a first analysis of the parallel texts, and a review of the existing typologies used to illustrate translation operations, both in the field of computational linguistics and translation studies. thanks to these contributions, we came to the definition of the typology provided in the training dataset consists of standard english sentences paired with their simplified counterparts. each simplified counterpart was designed to include precisely one simplification strategy, where a single complexity was restored to its original form. this design ensures that the relationship between a sentence and its simplified version highlights specific simplification strategies, allowing the model to associate each sentence with different parts of the complexity being resolved. to streamline classification, these fine-grained simplification strategies were mapped to broader macro-categories based on a predefined hierarchical structure, simplifying the labels while preserving their semantic distinctions. ## interpretability of predictions we have trained a classifier for predicting the difficulty of sentences by means of collecting simple and difficult sentences from wikipedia and finetuning mbert @xcite . by means of the implementation of the integrated gradients in the captum library @xcite , we can: we utilised the integrated gradients (ig) method to identify the tokens in a sentence that contributed most significantly to the model's predictions. ig achieves this by calculating the gradients of the model's output with respect to its input, thereby highlighting the importance of individual features. for example: consider the following sentence from our dataset: the integrated gradients approach offered actionable insights by attributing importance scores to specific words, revealing their influence on the model's predictions. for this sentence, the prediction probabilities are: simple: 0.02, and complex: 0.98. by applying the ig method, we identified a total of 1303 complex words from the original sentences. these words were then compared against their corresponding simplified, e2r versions to determine which complex words were removed during simplification. this comparison yielded 877 removed words, representing 67.31% of the total complex words identified. the removed words are indicative of tokens that were deemed complex by both the model and human editors, as their removal from the e2r versions suggests that they were perceived as difficult or unnecessary for simplified comprehension. this alignment between the model-predicted complex words and those removed in human-curated simplifications demonstrates the model's effectiveness in predicting words that are likely to be complex and corroborates the utility of the ig method for interpretability in text simplification tasks. as shown in figure 4 , the most frequently removed complex words included meaningful content terms such as "care," "organisations," and "consistent." ## findings and contributions the findings demonstrate that transformer-based models are capable of handling the complexities of typology classification, especially when supported by preprocessing techniques and loss weighting strategies. the model exhibits moderate success in identifying phenomena that require simplification. however, it encounters notable challenges with underrepresented classes and specific simplification strategies, such as "grammatical adjustments" and "omission." in summary, while transformer-based models hold considerable potential for simplifying texts to improve accessibility, addressing class imbalance through the use of comprehensive, balanced datasets is crucial. leveraging the complete dataset further enhances the model's reliability and enables it to generalise effectively across all simplification categories. one of the critical findings of this study is the utility of the ig framework for interpretability. ig provides insights that align closely with human annotations regarding complexity. for example, ig effectively identifies tokens contributing to diffi-culty, such as "sustainable" or "stability", while assigning minimal importance to semantically neutral words like "and" or "to." this alignment bridges the gap between machine predictions and human reasoning, enabling iterative improvements in model development. the alignment of the model's predictions with the removal of complex words by human editors demonstrates its capability to predict readability effectively. in particular, 67.31% of the complex words identified by ig were removed in the humansimplified versions, highlighting the model's predictive accuracy in real-world applications. moreover, the study shows the close connection between linguistic complexity and simplification practices. frequent removal of meaningful content words, such as "care," "organisations," and "consistent," highlights the importance of meaning and context in making texts easier to understand for different audiences. ## conclusions building on the annotation framework, several key insights emerge regarding the challenges and strategies involved in translating texts into e2r english. first, intralingual translation facilitates a more straightforward comparison between source and target texts due to the inherent isomorphism between the source and target languages. second, the choice of translation strategies must be tailored to the specific type of intralingual translation, ensuring that the target text aligns with its intended function. for example, in diastratic translation-specifically the transformation of standard english into e2r english-the focus lies on simplifying vocabulary, syntax, and semantic structures while maintaining fidelity to the source text and accessibility for the target audience. moreover, the proposed taxonomy, encompassing 9 macro-strategies, 33 strategies, and 15 microstrategies, illustrates the cognitive complexity of intralingual translation. these challenges underscore the limitations of current automation tools, as computational analyses reveal the nuanced skills required for transcription and modification strategies. even in the era of generative artificial intelligence, text simplification remains a non-trivial task due to its intricate linguistic demands. the novelty of our approach lies not only in the dataset itself but also in the methodology, which bridges translation studies and text simplification by categorizing transformations into well-defined categories. this integration offers new insights into the strategies employed in simplification and provides a robust framework for developing models that can generalise across multiple types of linguistic transformations. the results highlight the significant progress achieved with our approach, as the fine-tuned mbert model outperformed the baseline majorityclass strategy, which achieved an accuracy of 24.5% and a weighted f1-score of 9.6%. in contrast, mbert achieved 70% accuracy, a weighted f1-score of 72%, and a macro f1-score of 65%, demonstrating its ability to generalise across majority and minority classes. employing integrated gradients (ig) enhances the interpretability of model predictions, ensuring closer alignment with human annotations. ig offers a clearer understanding of the input data elements the model prioritises, thereby elucidating its decision-making processes. our primary results align with the identification of complex words that were either modified or removed in the simplified versions. in particular, 67.31% of the complex words identified by ig were removed in the humansimplified versions, highlighting the model's accuracy in applications. this transparency is critical for identifying strengths and weaknesses, guiding iterative improvements, and fostering trust in machine-generated outputs. additionally, ig serves as a tool to validate the predictions of the llm model against expert judgments, ensuring reliability and consistency in its reasoning, and ensuring that it makes the right predictions for the right reasons @xcite . future research should prioritise addressing class imbalance through advanced techniques such as hierarchical annotations, domain-specific embeddings, or data augmentation. incorporating multiple annotators would also enable the calculation of agreement metrics, improving the evaluation of annotation reliability. expanding the interpretability framework to cross-linguistic simplifications presents another promising avenue. leveraging the full scottish government dataset and employing advanced machine learning techniques could further enhance performance across all linguistic categories. this work ultimately contributes to the broader goal of creating accessible, inclusive texts while promoting trust and transparency in ai-driven systems.