Do Dependency Relations Help in the Task of Stance Detection? (2022)

Abstract: In this paper we present a set of multilingual experiments tackling the task of Stance Detection in five different languages: English, Spanish, Catalan, French and Italian. Furthermore, we study the phenomenon of stance with respect to six different targets – one per language, and two different for Italian – employing a variety of machine learning algorithms that primarily exploit morphological and syntactic knowledge as features, represented throughout the format of Universal Dependencies. Results seem to suggest that the methodology employed is not beneficial per se, but might be useful to exploit the same features with a different methodology.

## introduction and related work the task of monitoring people's opinion towards particular targets in political topics or public life debates has grown in the last decade, thus leading to the creation of a specific area of investigation in nlp named stance detection (sd). research on this topic, indeed, might have an impact on different aspects of everyone's life such as public administration, policy-making, advertisement, marketing strategies and security. in fact, through the constant monitoring of people's opinion, desires, complaints and beliefs on political agenda or public services, administrators could better meet population's needs @xcite . sd, as a task, shares various similarities with sentiment analysis (sa), and, exactly like sentiment analysis, also sd has been applied in several domains. for instance, to discover the reputation of an enterprise, what is the general public thought regarding a political reform, if costumers of a fashion brand are happy about the customer service, etc... nevertheless, whereas the aim of sa is categorizing texts according to a notion of polarity (positive, negative or neutral), the aim of sd consists in classifying texts according to the attitude they express towards a given target of interest @xcite . the first shared task entirely dedicated to sd was held for english at semeval in 2016, i.e., task 6 "detecting stance in tweets" @xcite . in the following years, many more shared tasks followed tackling the same issue in different languages and regarding different targets: chinese @xcite , spanish and catalan @xcite , @xcite , and lastly spanish and basque @xcite . provided that several approaches based on different types of linguistic knowledge have been applied to address the sd task, in this paper we investigate the contribution of syntactic information and in particular that provided by dependency relations. therefore, we exploit some of the datasets made available in the above-mentioned evaluation campaigns in different languages. in particular, we aimed at answering the following research question: rq: do features derived from morphology and syntax help automatic systems to address the task of stance detection? indeed, some research already explored different kinds of syntactic features and their interaction in several nlp tasks, showing their effectiveness. for example, @xcite exploited syntactic dependency-based n-grams for generalpurpose classification tasks, @xcite investigated sentiment and syntax with to the development of a sentiment treebank, and kanayama and iwamoto (2020) showed a pipeline method that makes the most of syntactic structures based on universal dependencies (ud foot_0 ), achieving high precision in sentiment analysis for 17 languages. morphology and syntax have also been proved useful in a number of other tasks, such as rumour detection @xcite , authorship attribution @xcite 2014) and humor recognition @xcite . to the best of our knowledge there is no prior work exploiting dependency-based syntactic features for addressing the task of stance detection. ## methodology the main goal of the experiments presented in this work consists in evaluating the contribution of syntax-based linguistic features extracted from the datasets described above to the task of sd. therefore, we performed a set of experiments where several models were implemented exploiting classical machine learning algorithms and state-of-the-art language models implemented with the python libraries scikit-learn and keras. the methodology we propose here, in which a multilingual setting is proposed and neural models are evaluated together with dependency-based features, recalls the idea that dependency based syntax might be useful in a variety of language scenarios for the task of sd and with a manifold of algorithms and architectures. ## experiments and results we propose two different experimental settings. the first one aims at exploring the dependencybased features listed above paired with classical machine learning (ml) algorithms, in order to perform a feature selection and discover the best combination. in the second setting, we experiment with the multilingual bidirectional encoder representations from transformers (m-bert) and different additions of the features explored in the first setting. ## discussion and insights the outcomes obtained in the investigation are slightly disappointing, but they do not come as a total surprise. when we were formulating the research question regarding sd, we had anticipated that there were no linguistic theories nor research work pointing towards the fact that morphosyntax might prove useful in this task. furthermore, a clear explanation could be seen by observing how two simple sentences having opposite stance, present identical syntactic structure: ex.1 i love the sardines movement. ex.2 i hate the sardines movement. with the experience matured with this research, we can state that -even if we are not obtaining the new state-of-the-art results -the outcomes lead in the direction of further investigation, pointing mainly towards a better understanding of features' behaviour when stacked in a pre-trained language model such as m-bert. finally, even though the results obtained with m-bert turned out to be not statistically significant, this research was oriented in studying whether some features derived from morphology and syntax could help automatic systems to address the task i love the sardines movement . of stance detection. it would be unfair not mentioning the fact that in the first experimental setting, that was mainly dedicated to the selection of the best features to be later fed as linguistic input into m-bert, we actually obtained better results with respect to state-of-the-art models in four languages out of six and in the remaining two we obtained close results that are definitely comparable (see the macro f1-score of the best ml systems in ## conclusion the lesson learned form this work suggests that morphosyntactic cues combine well as features in classical machine learning algorithms, but they do not seem to provide an increment in terms of performance in the neural architecture of m-bert in the case study of multilingual stance detection. indeed, as shown in linguistics, the expression of one's stance is frequently a phenomenon that seems to depend more often on semantics rather than on syntactic patterns or constructions. if you are reading this right now you are not blind ... lol sconj pron aux verb pron adv adv pron aux part adj punct intj