HEVS - TUW at S em E val-2023 Task 8: Ensemble of Language Models and Rule-based Classifiers for Claims Identification and PICO Extraction (2023)

Abstract: This paper describes the HEVS-TUW team submission to the SemEval-2023 Task 8: Causal Claims. We participated in two subtasks: (1) causal claims detection and (2) PIO identification. For subtask 1, we experimented with an ensemble of weakly supervised question detection and fine-tuned Transformer-based models. For subtask 2 of PIO frame extraction, we used a combination of deep representation learning and a rule-based approach. Our best model for subtask 1 ranks fourth with an F1-score of 65.77%. It shows moderate benefit from ensembling models pre-trained on independent categories. The results for subtask 2 warrant further investigation for improvement.

## introduction identification and verification of causal claims from unstructured text data is essential for various decision-making processes, particularly in healthcare. the semeval-2023 task 8 @xcite aims to advance the state-of-the-art in this area by focusing on two subtasks: identification of causal claims and extraction of population, intervention, and outcome (pio) entities. the first subtask involves identifying the span of text that contains one of the four entities: a causal claim, a personal experience, a personal experience based on a claim or a question. this can be done at the sentence level, but only a part of a sentence may be annotated with one of these categories. the second subtask involves extracting the pio frame related to the identified causal claim in a text snippet. the model utilizes both word-level, including contextual information and character-level features capturing different aspects of the data. ## background causal claims identification in the open domain is widely researched, but the healthcare domain has only garnered attention recently @xcite @xcite . in the healthcare domain, large amounts of medical notes, social media posts, research articles, and patient forums are generated daily. manually extracting causal claims and pio frames from such data is time-consuming and errorprone. for a decade, pio extraction was limited to sentence-level information extraction due to the unavailability of frame-annotated datasets @xcite . after the release of the ebm-pico corpus, the extraction efforts moved to span and frame extraction @xcite . nonetheless, previous studies on pio frame extraction primarily concentrated on extracting them from well-written, peer-reviewed literature @xcite @xcite . the semeval-2023 task 8 overtakes the challenge of extracting these frames from noisy social media data. the task organizers provide 597 english-language pio-labelled reddit posts. we approach pio frame extraction as binary sequence labelling and use a combination of deep learning and a rule-based approach that captures multiple feature representations from the data, as the dataset is relatively small and noisy. the semeval-2023 task 8 provides an opportunity for researchers to develop novel methods for causal claim identification and pio frame extraction from noisy social media data and to benchmark their performance against state-of-the-art methods. we hope that the shared task will lead to the development of more effective and accurate methods for identifying and extracting causal claims and pio frames from unstructured text data. ## system overview we participated in both subtasks of semeval-2023 task 8. in this section we describe our approach. ## experimental setup this section describes our experimental setup for subtasks (1) and (2), both of which were evaluated using a macro f1-score measure. ## conclusion we participated in both subtasks of semeval-2023 task 8. our submissions are mainly based on fine-tuning transformers-based models and creating an ensemble of these models. results show a positive impact of using independent binary classification models for each entity type in subtask 1.