Evaluating LLM s for Targeted Concept Simplification for Domain-Specific Texts (2024)

Abstract: One useful application of NLP models is to support people in reading complex text from unfamiliar domains (e.g., scientific articles). Simplifying the entire text makes it understandable but sometimes removes important details. On the contrary, helping adult readers understand difficult concepts in context can enhance their vocabulary and knowledge. In a preliminary human study, we first identify that lack of context and unfamiliarity with difficult concepts is a major reason for adult readers’ difficulty with domain-specific text. We then introduce targeted concept simplification, a simplification task for rewriting text to help readers comprehend text containing unfamiliar concepts. We also introduce WikiDomains, a new dataset of 22k definitions from 13 academic domains paired with a difficult concept within each definition. We benchmark the performance of open-source and commercial LLMs and a simple dictionary baseline on this task across human judgments of ease of understanding and meaning preservation. Interestingly, our human judges preferred explanations about the difficult concept more than simplifications of the concept phrase. Further, no single model achieved superior performance across all quality dimensions, and automated metrics also show low correlations with human evaluations of concept simplification (~0.2), opening up rich avenues for research on personalized human reading comprehension support.

## introduction text simplification helps lay audiences understand challenging text by simplifying difficult terms, syntax, or discourse @xcite or by adding content to elaborate on the text @xcite . with advances in neural models, especially ## background cognitive support and human reading comprehension successful reading comprehension is key to integrating new knowledge and fostering learning from text (lorch jr and van den @xcite . cognitive theories suggest that comprehension is a multi-stage process that primarily involves 1) constructing a local meaning representation of text such as concepts, facts, and their relations @xcite , and 2) forming a schema and filling in gaps using background knowledge to create a "mental picture" of what the text is about @xcite . adult readers lacking domain knowledge can be supported by explicit cues, such as examples and explanations, to help them construct better mental representations of ideas from the text @xcite . text simplification reducing reading-level complexity and syntax @xcite in text simplification benefits specific audiences like students, second language learners, and individuals with dyslexia @xcite , but may not enhance comprehension for general adult readers @xcite . contextual explanations can enhance comprehension but findings from studies of elaborating events in news domains @xcite may not be the same as difficulty with concepts in academic texts. while wikipedia and news corpora @xcite @xcite have advanced text simplification, they focus more on syntax and discourse difficulties than on academic concepts. similarly, lexicons are limited to medicine @xcite and science concepts @xcite , highlighting the need for a multi-domain corpus to advance personalized simplification for a general audience. complex terms and jargon lexical simplification systems @xcite have been shown to benefit children, people with language impairments or medical jargon simplification @xcite . figure 2 : results of annotator study: we asked annotators to read complex text for (1) what made the text difficult for them to understand and (2) how they would want a tutor to edit the text to help their understanding. however, beyond lab studies, it is challenging to specify reader knowledge in large-scale evaluations. proxies for audience knowledge include specialized lexicons @xcite , coarse indicators such as reading grade level @xcite , or binary indicators to denote science knowledge audience @xcite . @xcite highlighted the challenge of specifying audience knowledge at a finer level, suggesting the use of domain as a proxy for concept familiarity. building on this, we provide a multi-domain corpus of challenging definitions to specify fine-grained audience levels. unlike prior work on generating definitions @xcite or simplifying all difficult concepts (fatima and strube, 2023), we focus on rewriting definitions to address specific concept difficulties, enabling readers to leverage their background knowledge and improve comprehension @xcite . while previous tools explored simple strategies like adding definitions for complex terms @xcite , we evaluate llms that can provide contextual explanations (srikanth and li, 2021). ## what ai assistance can benefit reading domain specific text to better motivate the scope of this task, we investigate adult readers' difficulties with domainspecific text and what types of help they would want from an ai-tutor. we randomly selected a set of 900 text examples from wikipedia-derived definitions spanning 13 domains (see section 5 for details about definitions and domain selection). for each example, we ask a human annotator to re-spond in free-text to: 1) the reasons for difficulty (if any) when reading the definitions, and 2) what they would ask a tutor to change in the definition if they faced a difficulty. in figure 2 , we report recurring themes from annotator responses to both questions for cases where annotators had difficulty reading from these examples (categories were agreed on by the authors, see details in appendix section a). these results suggest that specific difficult concepts used in the definitions were one of the most frequent reasons for reading difficulty (52% of definitions had such difficulty), and annotators frequently asked for help from a tutor with these concepts (28%). this indicates that our proposed task, targeted concept simplification, is an important subtask for simplification aimed to resolve a key challenge for lay adult readers. when asking a tutor for help, annotators also explicitly asked for more details on the difficult concept (rather than less). this suggests that contextual elaborations (srikanth and li, 2021) for these difficult concepts are a better alternative over lexical simplifications to support their comprehension and knowledge. annotators also asked for examples/analogies (17%), visual/audio aids (8%), or identified general issues with the writing complexity (23%, e.g., an issue with general reading level or syntactic complexity), though to a lesser degree. the majority of our annotators had an above high-school level educational qualification (see ## task definition we present targeted concept simplification: a text simplification task focused on specific words or phrases that readers find difficult to understand. this setup allows for personalized and controlled rewriting of difficult concepts. our initial user study (section 3) shows that unfamiliar words or phrases often hinder comprehension. the task of targeted concept simplification is to rewrite an input definition containing a concept c to make it understandable to someone unfamiliar with the concept. for example, figure 1 shows the definition of the term "arbitrary precision arithmetic." the task is to rewrite the definition to help someone unfamiliar with the difficult concept "digits of precision." possible approaches could involve replacing "digits of precision" with a simpler phrase like "as many digits as needed," explaining it within the definition, or perhaps even adding examples, analogies, or illustrations. the usefulness of each strategy will depend on its ability to complement the reader's existing knowledge about the topic @xcite . unlike other text simplification tasks, our task targets simplifying concepts difficult for the reader rather than simplifying the entire text. ## the wikidomains dataset to support research on targeted concept simplification, we introduce a dataset of 22k definitions from 13 academic domains foot_0 , where each definition is a 1-2 sentence explanation of a term foot_1 . within each definition, we select a difficult concept-a word or phrase that could impede the reader's ability to comprehend the definition as whole. we take inspiration from @xcite who collected definitions from wikipedia science glossaries; however, instead of glossaries, we directly collect definitions from wikipedia articles of concepts spanning 13 domains (see to collect definitions, we start with johnson (2021)'s dataset that contains all english wikipedia articles with probabilities of belonging to highlevel domains. these domains are broad academic topics (e.g., physics, economics) that wikipedia editors identified through consensus @xcite . we refer to each wikipedia article title as a term and take the first sentence of its lead section as its definition @xcite . for every domain, we first select articles with domain assignment probabilities greater than a threshold δ domain . 4 to filter out low-importance articles that could be named entities, unimportant places or things, we also excluded articles having a pagerank percentile score less than δ pr . 5 finally, we also excluded articles that were additionally members of domains related to named entities, events, or things (e.g., biography). we select a training, development, and test split of 15,873/3,384/3,304 examples (see ## experiments we explore the performance of existing nlp tools on targeted concept simplification and possible avenues for future improvement. more concretely, we investigate the following research questions: rq1: what is the performance of out-of-thebox nlp tools in this task? rq2: which types of simplification strategies improve human understanding of difficult concepts and the definitions that they appear in? rq3: for targeted concept simplification, how do human evaluations compare to automatic metrics commonly used in text simplification? we perform experiments on the wikidomains test data created in section 5. as an additional evaluation set, we also use the scientific definitions dataset from @xcite (scidef) that contains definitions of science terms extracted from wikipedia glossaries and medquad @xcite . we perform the same post-processing on scidef as with wikidomains to select a difficult concept within each definition. ## discussion we close our paper by discussing the research questions we posed in our experiments and how they may relate to future improvements on this task. ## conclusion to support comprehension of domain-specific text for adult readers, we introduced the task of targeted concept simplification to study fine-grained simplification of difficult concepts in context. our human annotation study highlights the importance of aiding users' understanding of these concepts in domain-specific texts. we also introduced wikidomains, a dataset of 22k definitions across 13 academic domains, to support this task. our findings show a preference for strategies that add explanatory details over simplifying difficult concepts. human evaluations of llm rewrites indicate considerable room for improvement, especially for personalized help with difficult concepts.