NCUEE - NLP at S em E val-2023 Task 8: Identifying Medical Causal Claims and Extracting PIO Frames Using the Transformer Models (2023)

Abstract: This study describes the model design of the NCUEE-NLP system for the SemEval-2023 Task 8. We use the pre-trained transformer models and fine-tune the task datasets to identify medical causal claims and extract population, intervention, and outcome elements in a Reddit post when a claim is given. Our best system submission for the causal claim identification subtask achieved a F1-score of 70.15%. Our best submission for the PIO frame extraction subtask achieved F1-scores of 37.78% for Population class, 43.58% for Intervention class, and 30.67% for Outcome class, resulting in a macro-averaging F1-score of 37.34%. Our system evaluation results ranked second position among all participating teams.

## introduction a medical causal claim is an assertion that invokes causal relationships between variables such as "a drug has certain effects on a disease" @xcite . a manually-annotated corpus for causal statements in pubmed publications has been used to train prediction models for identifying "correlational," "conditional causal," "direct causal," and "no relationship" statements in research conclusion sections @xcite . biobert @xcite has been shown to outperform bert @xcite and linear svm in identifying exaggerated causal claims made in health press releases @xcite . empirical analyses have found that biomedical tweets are densely populated with claims @xcite . entitybased claim representations have been proposed to extract condensed claims from medical tweets @xcite . the pico framework is widely used to identify sentences in a given medical text belonging to four elements: 1) population (p): this usually describes the characteristics of populations involved; 2) intervention (i): this addresses the primary intervention along with any risk factors; 3) comparison (c): this compares the efficacy of any new interventions with the primary intervention; and 4) outcome (o): this measures the results of the intervention including improvements or side effects. the pico elements have been used as a knowledge representation framework to analyze clinical questions @xcite . a corpus of clinical trial publications has been annotated with pico elements @xcite . pico elements in medical texts have been detected based on long short-term memory networks @xcite . the contextualized bert embedding has been used to enhance pio element detection performance @xcite . the semeval-2023 task 8 (khetan, et al., 2023) organized a challenge to identify medical causal claims and extract related pio frames. this task consists of two subtasks. 1) subtask 1: for the provided snippet of text, participants should identify the span of text that belongs to either four defined categories, including "claim," "experience," "experience based claim," and "question". 2) subtask 2: for the given text snippet and its identified claim, participants should extract the related "population", "intervention," and "outcome" frames. this paper describes the ncuee-nlp (national central university, dept. of electrical ## the ncuee-nlp system figure 1 shows the system workflow. our model mainly depends on the deberta transformer @xcite . decoding-enhanced bert with disentangled attention (deberta) improves the bert @xcite and roberta @xcite models using two novel techniques: 1) a disentangled attention mechanism, and 2) an enhanced mask decoder. for disentangled attention computation, each word is represented using content and position vectors, and then attention weights among words are computed by applying disentangled matrices to their contents and relative positions. in the enhanced mask decoder architecture, absolute positions in the decoding layer are used to predict the masked tokens in model pre-training. debertav3 @xcite further improves the original deberta model by replacing the masked language learning (mlm) pre-training task with an electra-style replaced token detection (rtd) task @xcite . in addition, a new gradient-disentangled embedding sharing method is proposed to avoid the tug-of-war dynamics for improving both the training efficiency and the quality of the original deberta pre-trained model. for subtask 1, we fine-tune the pre-trained deberta transformer as a sentence-level classification task. first, we segment each post's content into several sentences using trankit (nguyen et al., 2021), a light-weight transformerbased toolkit for multilingual nlp tasks. each sentence is then assigned to one of the following defined categories: 1) claim: communicating a causal interaction between an intervention and an outcome; 2) experience: relating a specific outcome to an intervention or population based on someone's experience; 3) experience based claim: a claim based on someone's experience; 4) question: posing a question; and 5) others: sentences which do not belong to the above categories. a sentence may be composed of several tokens with different annotated categories based on the datasets provided by task organizers. we determine a sentence category using the majority voting mechanism from those constituent tokens. during the evaluation phase, we then convert an obtained sentence prediction result into token-level labels for performance evaluation. no separate fine-tuning of classification at the token level is needed. each sentence label will be assigned to all of its tokens. for subtask 2, we fine-tune the pre-trained deberta transformer with a weak supervised mechanism @xcite as a sequence labeling task. the weak supervised signal "@" is used to emphasize the starting and ending positions in a given claim. a sentence represented in terms of a token sequence will be aligned with population (noted as pop-tag), intervention (int-tag) and outcome (out-tag) for frame extraction. a null-tag is used to indicate that a token belongs to no extracted pio frames. for performance evaluation, we transform the sequence labeling output to obtain the starting and ending positions of extracted spans corresponding to each pio frame. ## conclusions this study describes the ncuee-nlp system in the semeval-2023 task 8, including system design, implementation and evaluation. we used different versions of pre-trained transformer models and fine-tuned two subtasks using reddit posts with annotations provided by the organizers. our best submissions ranked second among all participating teams for all subtasks and evaluation metrics.