Overview of # SMM 4 H 2024 – Task 2: Cross-Lingual Few-Shot Relation Extraction for Pharmacovigilance in F rench, G erman, and J apanese (2024)

Abstract: This paper provides an overview of Task 2 from the Social Media Mining for Health 2024 shared task (#SMM4H 2024), which focused on Named Entity Recognition (NER, Subtask 2a) and the joint task of NER and Relation Extraction (RE, Subtask 2b) for detecting adverse drug reactions (ADRs) in German, Japanese, and French texts written by patients. Participants were challenged with a few-shot learning scenario, necessitating models that can effectively generalize from limited annotated examples. Despite the diverse strategies employed by the participants, the overall performance across submissions from three teams highlighted significant challenges. The results underscored the complexity of extracting entities and relations in multi-lingual contexts, especially from the noisy and informal nature of user-generated content. Further research is required to develop robust systems capable of accurately identifying and associating ADR-related information in low-resource and multilingual settings.

## introduction an adverse drug reaction (adr) is defined as a "harmful or unpleasant reaction, resulting from an intervention related to the use of a medicinal product" (edwards and @xcite . adrs pose a significant challenge in pharmacovigilance. no medication is devoid of side effects, and despite clinical trials for each drug, the trial populations often fail to represent the entirety of real-world patients in terms of age, gender, health status, or ethnicity @xcite . moreover, post-release surveillance efforts may miss patients experiencing issues with the medication @xcite , emphasizing the need for continuous monitoring of medication usage and effects. natural language processing can support this process by extracting potentially novel adrs from text sources. clinical texts and scientific literature are valuable resources containing information about adrs. still, they are either difficult to access for researchers outside a hospital or are published multiple weeks or even months after the occurrence/detection of an adverse effect. social media, such as x (formerly known as twitter) or patient forums, in which patients share and discuss their sorrows, concerns, and potential adrs, instead became an alternative and up-to-date text source @xcite @xcite . not all information from social media is necessarily reliable from a medical point of view. still, it directly reflects the patient's perspective and at a much faster speed than wellcurated scientific text. to produce robust enough systems to process the vast amount of online data automatically, various datasets have been introduced in this context @xcite @xcite , shared tasks have been conducted @xcite @xcite , and models have been published @xcite . however, like other text processing domains, most datasets exist only for english. to raise interest in this critical topic -particularly for non-english languages, which are not wellrepresented in this domain @xcite we provide a shared task at #smm4h 2024 @xcite : cross-lingual few-shot relation extraction for pharmacovigilance in french, german, and japanese. this paper describes the results and findings of this task. it targets joint cross-lingual named entity recognition and relation extraction in a multi-lingual setting. the data consists of french, german, and japanese texts written by patients on social media such as x and patient fora, and is a subset of the keepha corpus @xcite . the output is a text file with predictions in brat format and shows an identifier (e.g., t1), a label (e.g., drug), the offsets of the entity, and the actual string (e.g., famenita) in the case of ner. for re, the annotations/predictions are extended by relations (identified with, e.g., r1), the relation type (e.g., caused), and the head (arg1) and tail (arg2) arguments, referring to entity identifiers. ## shared task in this section, we present the task details and schedule, the data used for the challenge, our baseline system, and the participants' approaches. ## results in the following, we briefly describe the results for the two subtasks. ## analysis we provide a brief analysis of the achieved results and the challenges the participants and their systems faced. this section is meant for further versions of the shared task and showcases common pitfalls. ## discussion & conclusion in task 2 of #smm4h 2024, the participants had to tackle a difficult task. starting with a small, multilingual, and layperson dataset and only a few examples for french plus no english data support, their systems had to distinguish three medical entities and two different relations, which are determined by temporal order and medical knowledge: the order of drug mentions, and disorder/function mentions decides if the relation between the expressions is a "cause" or a "treatment" relation. based on our baseline development using an llm and the participants' submissions, a lot of post-processing seems necessary to be applied to the llm output. it cannot be taken as is since the desired output format might not be consistently returned. we also noticed that it matters in which language the prompts are given to an llm and that sometimes, for example, with an english prompt, the returned entities are correct but in english and, therefore, do not match the gold entities. additionally, even if the llm approach worked better than a transformer-based approach, the results were still unsatisfactory, especially for relation extraction. of course, combining different languages in different scripts and colloquial texts on which the models were not trained is somewhat tricky. however, given the success of llms in other domains or other genres of text, e.g., scientific documents, we were surprised that none of the teams beat the baseline. we, therefore, think that there is still a lot of work to be done in the medical domain concerning patient-generated texts, especially for non-english speaking patients, and that even llms seem to be only a part of a potential solution. it is worth looking into the details of the sin-gle systems' benefits in future work. despite the low number of participating teams, there were still many different approaches to task 2 (rule-, transformer-, and llm-based). inspecting the detected entities and relation of each system might yield further insights and lead to a more successful system.