CAISA at S em E val-2023 Task 8: Counterfactual Data Augmentation for Mitigating Class Imbalance in Causal Claim Identification (2023)

Abstract: Class imbalance problem can cause machine learning models to produce an undesirable performance on the minority class as well as the whole dataset. Using data augmentation techniques to increase the number of samples is one way to tackle this problem. We introduce a novel counterfactual data augmentation by verb replacement for the identification of medical claims. In addition, we investigate the impact of this method and compare it with 3 other data augmentation techniques, showing that the proposed method can result in significant (relative) improvement on the minority class.

## introduction automatic identification of medical claims @xcite is a task with various real-life applications in industries such as healthcare @xcite and insurance @xcite as well as content moderation @xcite . however, it can be a difficult task due to the lack of data for all or some categories. one solution for such an issue is increasing the number of data points in each category, especially the one that has significantly fewer samples. we can do this using data augmentation techniques @xcite , which modify certain characteristics of an input sequence (or its representation in the embedding space) in order to create different versions of it. one example is entity replacement @xcite , where entities in one sequence can be swapped with equivalent ones from another sequence. the advantage of this type of augmentation is that it provides more real context to the target entities. given that the task at hand is claim detection, we hypothesize that the verb in a sentence can be determinant in its category. therefore, we address the problem of class imbalance using a novel data augmentation technique where we replace a verb in a sentence with other verbs from the training data. our experiments show that verb replacement can improve the performance of a model on the target category. in addition, for more comparison, we experiment with several other data augmentation techniques, namely noise insertion @xcite , entity replacement @xcite , augmentation with youchat foot_0 , and augmentation in the embedding space @xcite . ## system overview the annotated data gives us the span of each category. the spans can be complete sentences or part of a sentence. one approach to address the task is to frame it as a token classification task, similar to named entity recognition tasks. however, named entities seem to be easier to spot because of their locality. on the contrary, given that a longer sequence of words could belong to a category, we take a broader look to recognize them. as a result, we formulate the problem as sentence classification. the dataset statistics indicate that in more than 87 percent of the resulting sentences, all the words green shows the best performer, blue is the second best, and red is the worst. ## baseline models we compare the performance of augmentation methods with two baseline models, namely a conditional random fields @xcite model and the distilbert pre-trained language model for subtask 1 @xcite , and the biobert model @xcite for subtask 2. conditional random fields (crf). this model is particularly suited for sequence labeling tasks. it considers a set of manually defined feature functions to predict the label of a token. in our case, we only consider some simple features such as the word itself, the word endings, whether the word is uppercase or lowercase, and if it is a number or not. with the same features, we also consider bigrams. distilbert. this model is a lighter and more robust version of the bert model @xcite . we also consider the performance of this model without any augmentation as one of the baselines. biobert. this is another variant of the bert model that has been trained on medical texts in addition to the general text used for training bert. ## results and analysis we perform augmentation on the claims class for subtask 1 and the whole dataset for subtask 2. ## conclusion we proposed verb replacement as a novel counterfactual data augmentation technique to increase the number of samples in the minority class for causal claim identification. then, we showed that this method can significantly improve the performance of the machine learning model in the minority class. comparing it with three other augmentation methods, we also found out that the proposed method can outperform them in some cases.