Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques (2024)

Abstract: Recent research on sequence labelling has been exploring different strategies to mitigate the lack of manually annotated data for the large majority of the world languages. Among others, the most successful approaches have been based on (i) the crosslingual transfer capabilities of multilingual pre-trained language models (model-transfer), (ii) data translation and label projection (data-transfer) and (iii), prompt-based learning by reusing the mask objective to exploit the few-shot capabilities of pre-trained language models (few-shot). Previous work seems to conclude that model-transfer outperform data-transfer methods and that few-shot techniques based on prompting are superior to updating the model’s weights via fine-tuning. In this paper we empirically demonstrate that, for Argument Mining, a sequence labelling task which requires the detection of long and complex discourse structures, previous insights on crosslingual transfer or few-shot learning do not apply. Contrary to previous work, we show that for Argument Mining data-transfer obtains better results than model-transfer and that fine-tuning outperforms few-shot methods. Regarding the former, the domain of the dataset used for data-transfer seems to be a deciding factor, while, for few-shot, the type of task (length and complexity of the sequence spans) and sampling method proves to be crucial.

## introduction transfer learning and pre-trained language models are closely related as the knowledge learned for one or more tasks in one specific language can be applied to other tasks or languages @xcite . in this paper, we analyze how this feature can be applied in scenarios where not much data is accessible as it is the case of argument mining in the clinical domain. in data-transfer approaches, data can be translated and the required annotations projected to train supervised models. model-transfer methods avoid the long process of generating the training data by applying multilingual pre-trained language models to learn the annotations in one language and generate the predictions in a different one @xcite @xcite . alternatively, by fewshot prompting there is a possibility to reach comparable results by providing a few examples from the problem at hand to pre-trained language models @xcite . in sequence labelling tasks, these methods have shown to be effective with a minimal loss in performance based on a very few annotated examples. these few-shot methods have widely been tested on popular benchmark datasets, such as in those for named entity recognition (ner) @xcite , ontonotes 5.0 @xcite , mit-movie @xcite ) concluding that modeltransfer outperform data-transfer methods and that few-shot techniques based on prompting are superior to updating the model's weights via fine-tuning. however, such conclusions have been based on results obtained on sequence labelling tasks for which the sequence spans are commonly short and quite homogeneous in terms of the structure and content of the label words. in this paper we explore whether these conclusions still hold for argument mining, a task in natural language processing (nlp) aimed at extracting long and complex discourse structures from text. argument mining usually involves two distinct subproblems: (1) argument component detection, focusing on locating the spans of arguments and identifying their types (e.g., claims and premises), and (2) classification of argument relations, which involves classifying the relationship between two argument components as supporting or attacking. in order to do so, we use abstrct @xcite a corpus of medical abstracts annotated for the detection of argument components. the original corpus is published in english and has been extended it into a parallel multilingual corpus of medical arguments in spanish, italian, and french foot_0 by translating with state-of-the-art language models and projecting the annotations to the target languages using the technique of @xcite . summarizing, we investigate the following two research questions to address data scarcity in argument mining: in this paper we empirically demonstrate that, for argument mining (am), a sequence labelling task that requires the detection of long and complex discourse structures, previous insights on crosslingual transfer or few-shot learning do not apply. contrary to previous work, we show that for argument mining data-transfer obtains better results than model-transfer and that fine-tuning outperforms few-shot methods. regarding the former, the domain of the dataset used for datatransfer seems to be a deciding factor, while, for few-shot, the type of task (length and complexity of the sequence spans) and the sampling method proves to be crucial. data and code for the experiments described in this paper are publicly available in: https://github.com/anaryegen/few_ shot_argument_mining . ## related work in this section, we review the closest work to the paper's main topics, namely, argument mining, cross-lingual transfer and few-shot learning. ## data the starting point for experimentation on argument mining in data scarce settings is abstrct, a dataset of randomized controlled trials (rct) manually annotated with argument components and relations @xcite . the original abstrct consists of abstracts of clinical trials in english collected from the medline database and manually annotated with two types of argument components: claims and premises. a 'claim' is a concluding statement about the outcome of the study. in the medical domain it typically refers to a judgement regarding a possible diagnosis or a treatment. a 'premise' corresponds to an observation or measurement in the study (ground truth), which supports or attacks another argument component, usually a claim. it is important to stress that premises are observed facts, therefore, credible without further evidence. the training set consists of 350 abstracts that cover the neoplasm disease, 50 more abstracts about neoplasm are used for development, while the three evaluation sets are composed of: 100 abstracts about neoplasm, 100 abstracts about glaucoma and finally a mixed set of 100 abstracts with 20 abstracts for each of the diseases in the ab-srct dataset (i.e. neoplasm, glaucoma, hypertension, hepatitis and diabetes). the number of the sequences with premise and claim argument components in these sets is shown in we also generated a post-processed version by programatically correcting systematic errors performed during the automatic projection of the annotations. this post-processed version fixed relatively simple but repetitive issues such as omitting the labelling of articles as argument types. as a result, we obtained three versions of the projected data: auto projected, post-processed and manually corrected. the full training data is used for multilingual and cross-lingual experiments. to perform fewshot experiments the data is randomly sampled following different sampling approaches. ## experimental setup an important feature of am with respect to other sequence labelling tasks is that arguments are considerably long and composed by a variety of word types. the experiments are based on the three different techniques that we will be comparing to establish which one is the optimal one for am in data-scarce settings: (i) data-transfer, (ii) model-transfer and (iii), few-shot learning for sequence labelling. results are reported using f1 macro-averaged score calculated at sequence level, namely, the f1score is computed for each argument component following the usual method for sequence labelling tasks as formulated for named entity recognition @xcite . ## results following the completion of the experiments outlined in section 4, this section reports the obtained results using mbert foot_2 . ## error analysis in general, fine-tuning the model on the complete dataset often results in misclassifications with a tendency to assign claim labels in place of premise. additionally, dealing with long sequences poses challenges in accurately identifying both boundaries and classes for the system. this pattern persists in zero-shot results, and it can be attributed to an inherent imbalance in the data, particularly in terms of the disparity between the number of claim and premise labels and the length of arguments in the sequences. each sequence predominantly corresponds to a single argument type, and instances where a sequence contains compound arguments, or when the argument span is only a proportion of the input, are less frequent. consequently, in such examples, the most prevalent error involves misidentifying claim as premise and recognizing only one argument component in sequences with multiple components. these errors tend to occur more systematically in classifications under the zero-shot setting. in k-shot scenarios, the model consistently struggles to accurately identify both the correct spans and class labels. furthermore, as the number of k decreases, there is an increase in randomness in the assigned classes for each token, meaning that each token in a sequence may be classified differently. in particular, it is notable in the 5-and 10-shot. under k-shot the model struggles to predict b-tokens. whereas in the k-percent the opposite occurs, namely, the model learns to predict the beginning of the sequence and fails to predict o sequences correctly. nevertheless, it is observed that as the amount of data increases, the quality of the predicted outcomes improves. the described errors persist consistently in the case of entlm. additionally, when dealing with smaller training sets, the trained model tends to assign a single argument type to all examples in a document. as the value of k increases, the randomness in predictions also grows proportionally. in other words, a larger amount of data leads to more unpredictable assigning of labels by the model on the token level. a potential explanation for such behavior may be the selection of the label words. the concept involves computing label-specific words to later substitute them for few-shot learning. given that the length of an argument is usually long enough, one selected label word may not represent the argument type correctly. ## concluding remarks in this paper, we address the argument component identification task in the clinical domain in a scenario of lack of manually annotated data for languages other than english. we address the problem by applying cross-lingual transfer and promptbased learning strategies in the abstrct corpus. experimentation was facilitated by the generation of multilingual dataset by machine-translating and projecting the annotations of the original english abstrct into french, italian, and spanish. the results of our experiments show that for long and structurally complex sequence labelling, as it is the case of component identification in argument mining, data-transfer is a better strategy than model-transfer (rs1). thus, fine-tuning mbert in monolingual and multilingual settings showed results on an average of around 60 f1-scores for three test sets, outperforming any other approach, be that model-transfer or few-shot learning. furthermore, we have addressed the question of how much data is required to obtain similar results to those using the full data for training (rs2) by performing experiments in a few-shot learning approach. thus, corpus splits of different granularity (5, 10, 20, and 50 shot or percentage) were used in the experimentation with entlm and mbert. the models in general perform better when trained with data sampled using the k-percent method (in comparison to k-shot) and by fine-tuning a pretrained language model (instead of using a prompting method such as entlm). finally, empirical results indicate that by fine-tuning the multilingual model mbert with 20% of the data performance is competitive with data-and model-transfer approaches.