C o FE : A New Dataset of Intra-Multilingual Multi-target Stance Classification from an Online E uropean Participatory Democracy Platform (2022)

Abstract: Stance Recognition over proposals is the task of automatically detecting whether a comment on a specific proposal is in favor of this proposal, against this proposal or that neither inference is likely. The dataset that we propose to use is an online debating platform inaugurated in 2021, where users can submit proposals and comment over proposals or over other comments. It contains 4.2k proposals and 20k comments focused on various topics. Every comment and proposal can come written in another language, with more than 40% of the proposal/comment pairs containing at least two languages, creating a unique intra-multilingual setting. A portion of the data (more than 7k comment/proposal pairs, in 26 languages) was annotated by the writers with a self-tag assessing whether they are in favor or against the proposal. Another part of the data (without self-tag) has been manually annotated: 1206 comments in 6 morphologically different languages (fr, de, en, el, it, hu) were tagged, leading to a Krippendorff’s α of 0.69. This setting allows defining an intra-multilingual and multi-target stance classification task over online debates.

## introduction and related works stance recognition is a relevant tool for many real-life applications, from misinformation detection @xcite or poll verification @xcite to large-scale citizen consultation project @xcite . some recent work focused on tweets either in a non-interactional manner, like the semeval-2016 task @xcite , or by including the interactions between the users and applying stance detection over the whole thread @xcite . when working on online debates, authors employed linguistics-based methods inside debates using pre-defined opposed targets such as "iphone vs blackberry" @xcite , classifying ideological debates @xcite and on social justice subjects such as "abortion" or "gay rights". they then used hybrid models, i.e. machine learning models employing linguistic cues as features @xcite . they were followed by more complex probabilistic graphical systems @xcite @xcite , allowing to model the dynamics of the debate and the disagreements between speech turns, and finally deep neural methods @xcite . @xcite studied the dynamics of the stances on eight controversial topics in online debates. on multilingual stance analysis over tweets, @xcite present a model using mainly highlevel linguistic features like stylistic, structural, affective or contextual knowledge, but no dense contextual vectors. @xcite proposed a few-shot cross-lingual neural model, by aggregating different language datasets altogether. stance annotated datasets are often restricted to a few targets of concepts @xcite . in @xcite , the authors propose the x-stance dataset, containing 67k comments over 150 political issues in 3 languages. their approach was to reformulate the target in a natural question in order to easily train one multilingual multi-target model on the entire dataset. similarly, in the procon dataset, containing 6,019 comments over 419 controversial issues, each target was also reformulated as a question @xcite . however, none of these datasets contains interactional data. on contrary, @xcite presented the debating europe (de) dataset, a multi-target, multi-lingual stance classification over online debates, integrating the interactional context inside a model. in all the presented works, the language of the comments and propositions are the same, which can be seen as intra-monolingual. positioning and motivation stance recognition is generally restricted to tasks targeting a few defined entities or concepts @xcite figure 1 : number of posts and comments per language, using iso 3166-1 alpha-2 country codes. @xcite . in the proposed dataset, the targets are proposals that can be written in any language, making the task more difficult due to the high variability in terms of topics and in terms of languages. the work the most similar to ours is the one of vamvas and sennrich (2020), where they proposed a somewhat similar framework with the xstance dataset. but in their case, the data they release is restricted to 3 languages and one (small) country only. another similar work is the one of @xcite , with the debating europe (de) dataset, which contains only 2 languages with intramonolingual discussions, and annotations just for english only. we differ principally from related works by the multilingual aspect: in our dataset the comments and the propositions in the same discussion can be written in different languages (see examples the first motivation of this work relates to the lack of an appropriate intra-multilingual multitarget stance-annotated debate dataset. in the context of a citizen consultation project, various questions are asked and contributors can either answer these questions or express their stance by commenting on prior comments made by other users, in a discussion. we created such a corpus, together with the appropriate annotation schema and guidelines. it is also important to note that restricting a dataset to one language could induce nationality or cultural bias. contributions the contributions of this paper are the following. firstly, we propose a new dataset of stance in intra-multilingual online debates, containing binary self-annotations from the users in 34% of the cases. secondly, we annotate more than 1200 comments in 6 different languages, and obtained a high inter-annotator agreement of 0.69 using krippendorff's α. in the proposed dataset, we want to address the issue of classifying whether a comment is pro, against or neutral towards the proposal it is commenting on. the novelty of this proposed dataset remains in the use of intra-multilingual data and highly variable target. firstly, the structure of the platform makes it possible for users speaking different languages to interact on the same proposal page, hence the comments and the proposal are not necessarily written in the same language. secondly, there are many proposals on the cofe platform, hence the target of the comment (i.e. the proposal) is highly varying in terms of topic and vocabulary. ## conclusion we presented a new dataset for stance recognition in online debates on contemporary issues related to the future of the european union, containing 20k comments for 4.2k propositions in 26 languages. this dataset is rich in intra-multilingual interactions between participants, meaning that users can interact with each others using different languages. 46% of the threads have at least two languages. on top of the 7k binary pro/against self-annotations in 25 languages contained in the dataset, a set of 1206 comments from morphologically different languages has been labeled in a 3-class fashion by external annotators. finally, a few baselines have been tested over the externally annotated dataset cf e . future work could embrace using targetbased data-augmentation @xcite over our dataset which has a very versatile target space, or integrating the available metadata present in the release, like the number of dis/likes per comment and the number of endorsements per proposal.