Extracting Behaviors from G erman Clinical Interviews in Support of Autism Spectrum Diagnosis (2025)

Abstract: Accurate identification of behaviors is essential for diagnosing developmental disorders such as Autism Spectrum Disorder (ASD). We frame the extraction of behaviors from text as a specialized form of event extraction grounded in the TimeML framework and evaluate two approaches: a pipeline model and an end-to-end model that directly extracts behavior spans from raw text. We introduce two novel datasets: a new clinical annotation of an existing Reddit corpus of parent-authored posts in English and a clinically annotated corpus of German ASD diagnostic interviews. On the English dataset, the end-to-end BERT model achieved an F1 score of 73.4% in behavior classification, outperforming the pipeline models (F1: 66.8% and 53.65%). On the German clinical dataset, the end-to-end model reached an even higher F1 score of 80.1%, again outperforming the pipeline (F1: 78.7%) and approaching the gold-annotated upper bound (F1: 92.9%). These results demonstrate that behavior classification benefits from direct extraction, and that our method generalizes across domains and languages.

## introduction accurate identification of behaviors and symptoms is essential for diagnosing developmental disorders such as autism spectrum disorder (asd), where behavior markers like repetitive movement, avoidant behaviors or absence of socially important behaviors are key diagnostic criteria @xcite . however, existing tools for behavior and symptom identification rely heavily on qualitative analysis techniques (e.g., interviews, observations) that are time-consuming and subject to interpretation and whereby valuable information can be overlooked @xcite . although event extraction (ee) methods in nlp have been used in clinical contexts, such as extraction of symptoms and treatment decision in clinical records of medical disorders @xcite @xcite , general ee is ill-suited for asd due to its heterogeneity and the nuanced presentation of behaviors and symptoms. standard ee approaches identify general events, but not all events represent human behavior, particularly those that lack agentive or embodied action @xcite . to support asd diagnosis with behavior extraction, it is thus necessary to specify which events count as behaviors in the clinical sense. in this work, we frame binary behavior extraction as a specialized form of ee, and apply it to the analysis of actual clinical interviews in a novel corpus. our contributions are threefold: (1) we define a binary behavior classification scheme that embeds behavioral definitions within the timeml framework. in section 2 we explore previous work on event extraction in clinical applications, in particular for behavior classification in asd. we then describe the methods applied to both the english pilot study and the german clinical data, including data preprocessing, the pipeline, and the direct classification approach for behavior analysis. in the subsequent section 5 about the english pilot study, we detail datasets, considered models, training and results for ee, and behavior classification. we conclude section 5 with learned lessons from the pilot study. next, we focus on our experiments with german data in section 6, which is structured in the same way as section 5 and details the new dataset, models, training, results and discussion. we present our conclusions in section 7 and the limitations of our work in section 8. ## defining behaviors in terms of events behaviors and events share key properties: they are observable, unfold over time, involve agents, and can be causally linked to outcomes. however, not all events describe behaviors. for behavior extraction in text, we require a more specific definition grounded in linguistic and psychological theory. the timeml annotation framework @xcite categorizes events into types such as occurrence (actions that happen), perception (sensory experiences), reporting (communication acts), aspectual (beginning, ending, or continuing another event), i_action/i_state (intentions or mental states), and state (persistent conditions). these categories offer a rich base for distinguishing between behaviors and other event types. our behavior annotation follows a two-step process: (1) identify timeml-style events in text; and this filtered annotation is used to train both the pipeline and end-to-end behavior classification models. by grounding our behavior definition in the timeml schema, we bridge the gap between generic event detection and clinically meaningful behavior identification. ## methods we experiment with two datasets: a publicly available english reddit dataset newly annotated for events and behaviors, and a novel german clinical interview dataset. the first enables comparison with existing methods, while the second provides real-world clinical insights. we discuss here the common aspects of the methods used in the two experiments. ## pilots with existing (english) data while developing the new german clinical dataset, we piloted our approach on an existing english dataset of non-clinical texts, which was newly annotated for behaviors and events by our clinical collaborators. this allowed us to evaluate different methods in comparison to existing event extraction approaches. we report these preliminary experiments in this section. ## experiments on german clinical data our main experiment involves the same steps as the pilot with english data, but this time applied to real clinical data in german. ## conclusions we introduce a novel approach for identifying behaviors in text to support asd diagnosis, by formulating behavior classification as a refinement of ee. our analysis focuses on asd behaviors which are described in third person by caretakers. our approach was tested both on a newly created german dataset of clinical interviews with caretakers of potential asd patients-to our knowledge, the first clinical german dataset with event and behavior annotations-as well on an existing, publicly available english dataset, which we also newly annotated using the same scheme. both of the new behavior classification datasets created for this work, and annotated by psychologists with extensive training in timeml annotation, will be released. our results on both datasets show that the endto-end model outperforms pipeline models that use an ee model followed by a behavior classifier, primarily due to error accumulation in the ee step. however, with optimal annotations in the ee step, a pipeline approach can outperform the end-to-end model. these results are similar in both english and german, suggesting that our approach is rooted in semantics of events and behaviors. we can also infer from the results that the application of llms in the ee field is still challenging. overall, phi3 outperformed t5 with a slight margin, but different prompts for phi3 had a notable impact on the performance indicating that prompt engineering needs to be further improved. on behavior classification, phi3 performed notably worse than on ee, possibly because of the smaller dataset and an non-optimized prompt. these results show that the extraction of behaviors conceptualized in terms of ee especially coupled with token classification has promise for the further development of this technology as well as implications for the development of clinical tool for disorders with idiosyncratic descriptions of behaviors. for example, we presented a prototype visual platform at healtac2025, where clinicians can upload texts such as session transcripts, and our models extract and highlight events and behaviors in the submitted text.