Automated main concept generation for narrative discourse assessment in aphasia (2025)

Abstract: We present an interesting application of narrative understanding in the clinical assessment of aphasia, where story retelling tasks are used to evaluate a patient’s communication abilities. This clinical setting provides a framework to help operationalize narrative discourse analysis and an application-focused evaluation method for narrative understanding systems. In particular, we highlight the use of main concepts (MCs)—a list of statements that capture a story’s gist—for aphasic discourse analysis. We then propose automatically generating MCs from novel stories, which experts can edit manually, thus enabling wider adaptation of current assessment tools. We further develop a prompt ensemble method using large language models (LLMs) to automatically generate MCs for a novel story. We evaluate our method on an existing narrative summarization dataset to establish its intrinsic validity. We further apply it to a set of stories that have been annotated with MCs through extensive analysis of retells from non-aphasic and aphasic participants (Kurland et al., 2021, 2025). Our results show that our proposed method can generate most of the gold-standard MCs for stories from this dataset. Finally, we release this dataset of stories with annotated MCs to spur more research in this area.

## introduction narratives are an essential tool for communication and understanding. several advances have been made towards developing theoretical @xcite ) and computational methods @xcite for understanding narratives. however, evaluating the effectiveness of the obtained narratives has remained challenging owing to the subjective nature of the task and the difficulty of validating them in real-world applications where they serve specific communication purposes @xcite . for instance, while prior work 1 https://github.com/slanglab/aphasia time now for storycorps, @xcite . this story starts one night in 1995. phil donney and his younger sister laura could hear their parents arguing. the fight ended when their father murdered their mother. phil was seven, his sister four, and they went to live with their mom's sisters. phil is now 23 he recently came to storycorps with abby liebman, one of the aunts who raised him. phil asks, "what was it like becoming a parent to my sister and i overnight?" abby answers, "well, at the time i was living in a two bedroom condo and felt a little panicky, to be honest. i really didn't know how to parent. i knew how to be an aunt. [...] but now that i had to be the parent, i realized that things had really changed. [...] then phil asks, "so where do you feel we are now?" and abby answers, "you know, when you first came to live with me, there was no doubt in my mind m2: phil's mother was murdered by his father when he was young. m3: phil and his sister were raised by their mother's sister abby. m4: abby was comfortable with the aunt role. m5: abby wasn't sure how to be a parent. m6: over time, their new roles became natural. m7: creating a loving environment was paramount to abby. ## main concept generation task given the input text of a story (t), which can be either an audio transcript or obtained from a written source, the task involves generating a list of main concepts @xmath0, m 2 ....m n }. each concept m i is a statement that is relevant to the story and consists of one main verb and its subject, object, modifiers, and subordinate clauses if appropriate. this definition of an mc follows the precedent established in the aphasia and speech/language disorders literature. the same definition was provided to human annotators when creating the original gold-standard mcs used in clinical practice @xcite @xcite . prior clinical research has demonstrated that mcs defined in this manner provide meaningful and reliable measures for clinical assessment applications @xcite . the number of main concepts varies across stories, depending on their content. the set of main concepts together conveys the gist of the story. foot_0 figure 1 shows an example drawn from bats, showing an audio story's transcript and its annotated main concepts. the story captures a conversation between a nephew and his aunt, reflecting on how the aunt became a mother figure to him and his sister after a tragedy, exploring their evolving family bonds and shared journey. this story has seven annotated main concepts that capture the summary of this conversation. ## prompt ensemble method for mc generation step 1: generating mcs. we prompt an lm to generate main concepts in a zero-shot setting, without providing any examples. since lm outputs can vary based on prompt wording, we experiment with multiple prompts. prior work demonstrates that using names of pedagogically popular theories can be an effective way to design prompts @xcite . thus, we prompt lm with names of theories used for understanding story grammar @xcite or five-finger story-retelling strategies which are often used to teach students how to analyze and summarize narratives effectively @xcite . we also prompt with 5w's and 5w's1h framework often used by journalists to understand key story aspects @xcite . in each prompt, we provide the theory/concept definition and instruct the lm to generate mcs as per the theory/concept. additionally, as baselines, we prompt lm with simpler prompts wherein we directly ask the llm to help understand the story or explain the plot points of the story. 3 detailed prompts are provided in appendix a.1. the raw lm response generated by the lm can often contain lengthy sentences combining multiple concepts. since our objective is to generate mcs that are amenable to manual review, we further prompt the llm to decompose its outputs into simpler de-contextualized statements. this step also allows us to generate mcs that structurally match the gold-standard mcs, further facilitating clustering in later steps. we use the few-shot prompt by @xcite for this step, asking the lm to decompose each sentence in the raw lm response into simpler statements. 4 step 2: union of mcs generated by different approaches. prior work often selects the bestperforming prompt, but we found this insufficient as different prompts and samples obtained via the same prompt can capture different story aspects and hence generate different main concepts ( §5.4). previous studies have explored using prompt sensitivity to improve downstream task performance by aggregating word-level llm outputs @xcite . building on these efforts, we perform a set union of decomposed and de-contextualized statements obtained from the previous step. in particular, we take a union of all statements obtained from the three bestperforming prompts and their samples to improve coverage of gold-standard mcs. step 3: semantic deduplication. while a naive union of outputs from different prompts can help improve the coverage of gold-standard mcs, it can create a long list with many redundant mcs that are similar in meaning but expressed using different words. this results in a very high yield-total number of words in the mc list-making manual review cumbersome. one approach to address this issue would be to use an llm for deduplication, by giving it the union list of mcs and prompting it for a deduplicated mc list. however, this approach faces several limitations. first, it is constrained by the length of the context-window of an llm; for long stories with a large number of mcs, the input list of mcs to deduplicate across different prompts can quickly become large and may not fit in the context window. secondly, even as llms continue to support longer context windows, they still struggle to actually utilize the entire context. for instance, the recent needle in a haystack analysis @xcite , which tests the in-context retrieval ability of long context llms, suggests that both open and closed-sourced llms struggle with utilizing long context. furthermore, llms have also been shown to perform poorly on tasks requiring long context understanding, such as book-length narrative summarization @xcite . for deduplication, such capabilities are all the more necessary. instead, we use a clustering-based approach that groups mcs that are similar in meaning and selects one representative mc from each cluster. we first convert each mc into an embedding using sentence-bert @xcite . we then l2 normalize the embedding vectors, followed by clustering. we cluster with the dp-means algorithm (kulis and jordan, 2012), foot_3 which hanley et al. (2025) found useful for embedding-based semantic clustering. in dp-means, the granularity of clusters is controlled by a hyperparameter δ ≥ 0, the maximum squared distance between a data point and its cluster's centroid; the model chooses the total number of clusters in response to that constraint. a small δ gives a large number of spatially small clusters, each with high internal semantic similarity; and conversely for large δ. we report results in terms of δ, which is roughly twice of the familiar cosine distance. 6 finally, to obtain one representative mc for each cluster, we select the embedding closest to the centroid and use its text as the representative mc. the representative mcs form our final mc list. ## conclusion we present an application of narrative understanding in the clinical assessment of aphasia using story retelling, which we hope can benefit current research in developing and evaluating narrative understanding systems. we further propose the task of main concept generation to enable the adaptation of current assessment tools to novel and patientcentric stories. we also present an llm-based prompt ensemble method for automatically generating mcs from stories. our experimental results on both the bats clinical dataset and the narrasum dataset demonstrate that our proposed method can successfully generate a concise list of mcs that match manually curated gold-standard mcs. more broadly, our main concept generation approach can also benefit broader applications such as assessing functional communication across diverse clinical populations and in educational contexts like intelligent tutoring systems.