Multilingual Argument Mining: Datasets and Analysis (2020)

Abstract: The growing interest in argument mining and computational argumentation brings with it a plethora of Natural Language Understanding (NLU) tasks and corresponding datasets. However, as with many other NLU tasks, the dominant language is English, with resources in other languages being few and far between. In this work, we explore the potential of transfer learning using the multilingual BERT model to address argument mining tasks in non-English languages, based on English datasets and the use of machine translation. We show that such methods are well suited for classifying the stance of arguments and detecting evidence, but less so for assessing the quality of arguments, presumably because quality is harder to preserve under translation. In addition, focusing on the translate-train approach, we show how the choice of languages for translation, and the relations among them, affect the accuracy of the resultant model. Finally, to facilitate evaluation of transfer learning on argument mining tasks, we provide a human-generated dataset with more than 10k arguments in multiple languages, as well as machine translation of the English datasets.

## introduction argument mining has received much attention in recent years, with research mainly focused on english and, to some extent, german texts. recent advancements in natural language understanding suggest that in order to train appropriate models for argument mining tasks in other languages, we do not need to manually label text in these languages, but rather employ transfer learning from the english-based models @xcite . in this work we examine three argument mining tasks: (1) stance classification: given a topic and an argument that supports or contests the topic, determine the argument's stance towards the topic; to facilitate transfer learning from english datasets for these tasks, we employ multilingual bert (mbert) released by @xcite , a pre-trained language model that supports 104 languages, and use it mainly in a translate-train approach. namely, the english dataset is automatically translated into the desired language(s) using machine translation (mt); an augmented dataset composed of the original english text and all the translated copies is created; the mbert model is fine-tuned on a subset of the dataset; and the resultant model is then used to solve the relevant downstream task in the desired language. previous works have suggested that translating the original dataset to as large a number of languages as possible is beneficial @xcite . in this work, we show a more nuanced picture, where often selecting a subset of related languages is preferable. in addition, we also examine the translate-test approach, in which one creates a classification model only with english data. at prediction time, the non-english text is automatically translated into english, and then analyzed by the model. this approach is less appealing since the initial translation step increases prediction run-time, and on our data also tends to perform worse. we examine two text sources for performance evaluation on non-english texts. the first one is a "pseudo test" set -an automatic translation of an english evaluation set for a task. while such texts can be easily generated, it is not clear how well they represent "real" texts, authored by humans. hence, we also examine human-authored texts, in several non-english languages, collected via crowdsourcing specifically for this work. both datasets are released as part of this work. 1 when translating the evaluation set, either automatically or by a human translator, one would like to assume that the initial label of the english text is maintained after translation. while this is often the case, we show that this assumption becomes more dubious as the argument mining task becomes more complex and subjective, as well as when the original labels are not clearly agreed upon. in summary, the main contributions of this paper are: (1) a comparative analysis of the translatetrain approach on three central argument mining tasks using different subsets of languages, showing that training on more data helps, but that, in some cases, training on related languages is sufficient; (2) multilingual benchmark datasets for the three tasks; stance detection (or classification) is often contrasted with sentiment analysis, in that the task is not simply to classify the sentiment of a text, but rather its stance w.r.t. some given target. early work on this task includes @xcite ; @xcite , while in the context of argument mining, the task has probably been introduced by @xcite . as with many other nlu tasks, earlier works developed classifiers based on various features (e.g., @xcite ), while more modern approaches rely on deep learning. see @xcite for a recent benchmarking report on such methods. research on stance detection in a multilingual setting is rather recent. @xcite explore stance detection in twitter for catalan and spanish; @xcite do this for political debates in social media in these two languages as well as french and italian; @xcite analyze the stance of comments in the context of 1 https://www.research.ibm.com/ haifa/dept/vst/debating_data.shtml# multilingualargumentmining the switzerland election in german, french and italian. stance detection is reminiscent of the natural language inference (nli) problem, where one is given two sentences, and the objective is to determine whether one entails the other, contradicts it or is neutral. this task had been researched extensively, with @xcite providing a 15-language benchmark for the multilingual setting. another earlier work on a related task is addressing the support/attack relation prediction of two arguments in italian @xcite . evidence detection is the task of determining, given some text and a topic, whether the text can serve as evidence in the context of the topic @xcite . we follow @xcite , in defining evidence as a single sentence that clearly supports or contests the topic, yet is not merely a belief or a claim. rather, it provides an indication for whether a relevant belief or a claim is true, and, since we use their datasets, we restrict our analysis to evidence of type study and expert. in a multilingual setting, a similar task, that of premise detection, was considered in @xcite for german, french, spanish, and chinese; in @xcite for russian; in @xcite for french and german; and in @xcite for chinese. argument quality prediction is the task of evaluating the quality of an argument, either on an objective scale -the input is an argument and the output is a quality score; or in a relative mannerthe input is a pair of arguments and the output is which of them is of higher quality. while there are many, arguably independent, dimensions for quality @xcite , it seems that people -and, consequently, algorithms -can usually perform this task in a consistent manner @xcite @xcite . to the best of our knowledge, this task was not previously considered in a multilingual setting. in contrast with previous multilingual research on argument mining, in this work we address three different problems, of varying complexity, over a relatively large number of languages. this allows us to draw more holistic conclusions on the efficacy -and pitfalls -of transfer learning in the argument mining domain. it is interesting to compare these conclusions with other wide-scope multilingual nlu research, such as the xtreme @xcite and xglue @xcite . the second dataset is a corpus of 35,211 sentences from wikipedia on 321 controversial topics, annotated for their stance towards the topic and the extent to which they can serve as an evidence for the topic @xcite . this dataset (referred as evien) is used for the stance classification and evidence detection tasks. example 1 shows an argument and an evidence for one topic. example 1 (argument and evidence) topic: we should legalize cannabis argument: cannabis can provide relief for a number of ailments without side effects. evidence: in 1999, a study by the division of neuroscience and behavioral health found no evidence of a link between cannabis use and the subsequent abuse of other illicit drugs. a third dataset was used to augment the training data for evidence detection and stance classification -the so-called vld dataset of @xcite , which includes around 200k sentences from newspaper articles, pertaining to 337 topics. 3 data selection the argsen dataset was filtered for stance classification by selecting arguments with a clear stance (@xmath0.75) for training and evaluation. for argument quality, arguments with a clear agreement on their quality were selected -quality score above 0.9 or below 0.4. a positive label for evidence detection was assigned for evidence from evien with a score above 0.7, and a negative label to those with score below 0.3 (those with in-between scores were not used). for stance classification on the evidence data, all translation we used the watson language translator foot_2 to translate the selected english data into 5 languages: german (de), dutch (nl), spanish (es), french (fr), and italian (it). the translation is a one-time process, which can be applied to any target language (tl) that the mt engine in use supports. the labels for the mt data were projected from the english data. following the data splits provided in the official release of each english corpus (into a training, development, and test sets), the translations of the training data were used for fine-tuning mbert, and the translations of the test data were used for evaluation. the translations of the test data of argsen and evien into the 5 non-english languages -the pseudo test sets -are herein referred to as argsmt and evimt. the statistics of the translated english data are summarized in ## experimental setup our experiments are aimed at providing a comparative analysis of the translate-train approach when trained on different subsets of languages, and identifying when that approach is beneficial on the three argumentation tasks. we begin by describing the setup used in all experiments. ## results the results below for arguments (for stance classification on that data and argument quality) are averages over 5 evaluation runs of randomly initialized models that were trained in the same manner. for evidence sentences (stance classification on that data and evidence detection), the results are from a single evaluation run. ## analysis the performance of a translate-train model may be affected, among other factors, by the translation quality, the extent in which a task-specific label is preserved across that translation, and, for our data, the discussed topic. these are analyzed below. ## conclusions we have examined the translate-train paradigm for three multilingual argument mining tasks: stance classification, evidence detection, and argument quality, evaluating a wide range of multilingual models on machine-translated and human-authored data. these tasks differ in their complexity, as reflected in the agreement of annotators on the correct label, the extent to which this label is preserved across translation, and, ultimately, in the accuracy of the models. accordingly, our results show that the translatetrain approach is well suited for stance classification, as performance improves when augmenting the english training data with automatic translations from other languages. for evidence detection, 7 in contrast with the above two tasks, the results on argument quality show that training only on english is at least as good, if not better, than any translate-train model. this is reflected by the clearly opposite trends observed in figure 3 vs. those observed in figure 1a . taken together, our results confirm the validity of the common translate-train paradigm for argument mining tasks such as stance and evidence detection, for which the label is relatively well preserved under translation. however, for the more subtle argument-quality task, where the label -as might be expected -is far less preserved, a new approach might be needed. future work might wish to explore how translation can preserve not only the semantics of texts, but also finer aspects that contribute to its quality.