Multilingual Irony Detection with Dependency Syntax and Neural Models (2020)

Abstract: This paper presents an in-depth investigation of the effectiveness of dependency-based syntactic features on the irony detection task in a multilingual perspective (English, Spanish, French and Italian). It focuses on the contribution from syntactic knowledge, exploiting linguistic resources where syntax is annotated according to the Universal Dependencies scheme. Three distinct experimental settings are provided. In the first, a variety of syntactic dependency-based features combined with classical machine learning classifiers are explored. In the second scenario, two well-known types of word embeddings are trained on parsed data and tested against gold standard datasets. In the third setting, dependency-based syntactic features are combined into the Multilingual BERT architecture. The results suggest that fine-grained dependency-based syntactic information is informative for the detection of irony.

## introduction irony is a pragmatic phenomenon that can be defined as a sort of incongruity between the literal meaning of an utterance and its intended meaning @xcite @xcite . in text processing, the challenging nature of irony is mainly due to the inherent features of the phenomenon itself, that may assume a large variety of merged forms and facets. several semantic and syntactic devices can be used to express irony (e.g., analogy, oxymoron, paradox, euphemism and rhetorical questions), and a variety of different linguistic elements can cause the incongruity, determine the clash and play the role of irony triggers within a text. most computational research in irony detection is applied on social media texts and focuses primarily on content-based processing of the linguistic information using semantic or pragmatic devices, neglecting syntax. for example, @xcite propose a method for sarcasm detection in tweets as the task of detecting incongruity between an observed and an expected word, while @xcite rely on external common knowledge to the utterance shared by author and reader. however, in addition to these pragmatic devices, some hints about the role of syntax can be found in linguistic literature where the deviation from syntactic norms has been reported as a possible trigger of the phenomenon. for instance, @xcite explore the usage of split interrogatives and of topic-comment utterances in english, arguing they both have a double function (they express two distinct communicative acts) and that one of those is precisely that of conveying an ironic meaning (e.g. "who are you, willy wonka?"). 1 the interpretation of an ironic utterance involves not only the knowledge of the usage of the language, inventory and rules including grammar, lexis, syntax, structure and style, but also the perception of the violation of these rules. irony tactics handle the syntagmatic patterns of discourse, the way in which items precede, follow and are related to each other. as @xcite shows referring to arabic, syntactic norms, which normally see adverbs placed after verbal phrases, can be violated to trigger irony. however, when the commonly used ironic device of topic shift occurs, lexical and syntactic signals are this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . some research already explored different kinds of syntactic features and their interaction in several nlp tasks, showing their effectiveness. for example, @xcite exploited syntactic dependency-based n-grams for general-purpose classification tasks, @xcite investigated sentiment and syntax with the development of a sentiment treebank, and @xcite showed a pipeline method that makes the most of syntactic structures based on universal dependencies, achieving high precision in sentiment detection for 17 languages. morphology and syntax have also been proved useful in a number of other tasks, such as rumor detection @xcite , authorship attribution @xcite and humor detection @xcite . to the best of our knowledge, very few studies use syntactic information specifically for irony detection. among them we cite @xcite , who employed a svc combined with shallow features based on morphology and dependency syntax outperforming strong baselines in the irosva 2019 irony detection shared task in spanish variants @xcite . this paper aims to go one step further focusing for the first time on the development of syntax-aware irony detection systems in a multilingual perspective (english, spanish, french and italian), providing an in-depth investigation of the impact of different sources of syntactic information when used on top of several machine learning models, including recurrent neural networks and transformers. it is important to note that our aim is not to outperform the current state of the art on monolingual irony detection, but to investigate whether irony detection based on syntax alone can achieve comparable results with existing systems when evaluated in multiple languages. we believe this is an important first step before moving to complex scenarios where both syntax and pragmatic knowledge are incorporated into deep learning models which could lead to explicitly model the syntax-pragmatic interface of irony. to this end, we propose to address the following research questions: finally, in the third setting, dependency-based syntactic features are combined into the multilingual bert architecture. our results show that our models outperform syntax-agnostic ones, which is an important first step towards syntactically-informed irony detection. the paper is organized as follows. in the next section, related work regarding irony detection is surveyed, while section 3 describes the data and experimental settings we followed. in section 4, we provide a description of the experiments we performed, and, in the last section, we discuss the experiments and provide some final remarks through an error analysis. we conclude the paper with some insights about the impact of morpho-syntax information and suggest possible directions for future work. ## related work the identification of irony and the description of pragmatic and linguistic devices that trigger it has always been a controversial topic @xcite @xcite , a challenge for both humans and automatic tools. this motivates the interest for irony of the nlp community, the organization of tasks regarding its processing and the subsequent creation of benchmarks. in the last decade several shared tasks have been organized in order to advance with the techniques regarding the automatic processing of figurative language @xcite , and more recently irony and sarcasm detection have been gaining greater attention. being irony a pragmatic device inherently related to culture and language, the task captured the attention of various researchers, dealing with different languages and mostly focusing on social media texts. the presence of irony for italian has been investigated -among other things-in the sentipolc task of the evalita 2014 @xcite and 2016 @xcite and at ironita @xcite . it was addressed for french within the context of deft 2017 @xcite , for english at semeval 2018 (task 3: irony detection in english tweets) @xcite as well as for spanish at the irosva @xcite . as a side effect, several annotated corpora were released, among which also those that allowed us the composition of the experimental settings provided in this paper (see § 4). similar to other nlp tasks, neural approaches in irony detection have been widely used, and proved successful even in the mentioned shared tasks. for instance, the best-scoring team for task 3 at semeval 2018 used a densely connected lstm based on pre-trained word embeddings, sentiment and pos tag features @xcite ; they built a multi-task model to predict the missing irony hashtag, whether a tweet is ironic or not and the fine-grained type of irony. a similar multi-task learning approach was adopted in @xcite at ironita 2018, which introduced a 2-layer bilstm that exploits additional information, such as automatically-generated sentiment polarity lexica, word-embedding lexica and pos tags. the irony detection system proposed in @xcite at irosva 2019 was based on the transformer encoder model. at deft 2017 the winning team of the shared task for french @xcite proposed an approach based on varying the size of the layers of a cnn combined with three different sentiment-based word embeddings. finally, a multilingual scenario is also proposed in @xcite , that describe both feature-based models (with random forest showing the best results) and a cnn architecture with multilingual embeddings, and test their approach on arabic, english and french (for the latter, the deft 2017 dataset was used, as in the present work). as mentioned in section 1, to the best of our knowledge, very few studies use syntactic information specifically for irony detection. while in this work as well we tested our models in a multilingual setting, the novelty of the approach provided in this paper consists in using features which represent syntactic knowledge that can be extracted from text by applying morphological and syntactic analysis. universal dependencies played a key role in this respect: in recent years they have become a popular and widely-used annotation format @xcite , providing a set of broadly attested universal grammatical relations, in which existing dependency schemes for different languages can be mapped. the proliferation of ud treebanks in a short time span highlights the need for a consistent representation of linguistic phenomena, not only for typical contrastive studies, but also when using such resources for downstream applications. ## data and experimental setting in order to address the research questions, we follow a binary classification irony detection task, testing an automatic system on four different languages: english, spanish, french and italian. this will allow us to investigate whether syntactic structures, independently of the target language, are informative to understand whether a message is ironic or not. in the multilingual experimental setting we took advantage of four datasets that have been made available during the last few years within evaluation campaigns, and all featured by a binary annotation for irony. in provided that the availability of morphological and syntactic knowledge is crucial for performing the experiments described in the rest of the paper, we needed to obtain a representation of all the datasets in ud format. with the exception of twittirò @xcite , a subset of the ironita dataset already available as a ud treebank (see the current release in the ud official repository foot_1 ), we obtained the dependency-based annotation for the other corpora by applying the udpipe foot_2 pipeline (for tokenization, pos-tagging and parsing). figure 1 provides an example drawn from twittirò. considering that all the datasets used in this work consist of twitter data, whenever possible, we used resources where this genre, or at least user-generated content of some kind, was included as training data for parsing. more precisely, the model for english has been trained on the ewt treebank @xcite , that for spanish on both gsd-spanish @xcite and ancora corpora @xcite . the model for italian -for the remaining part of ironita foot_3 -was trained on postwita @xcite and isdt treebanks @xcite , while that for french on the gsd-french corpus @xcite . we are aware that there actually exists a twitter treebank for english, i.e. tweebank v2 @xcite ), but it is not fully compliant with the ud format specifications (e.g. it violates the single root constraint posed in ud format). we thus opted for the ewt in order to preserve annotation consistency among resources. the different amount of data used for training udpipe, the variety of text genres among the datasets, and the fact that the ud annotation of only one dataset has undergone a manual correction -i.e. twit-tirò, also mentioned above -can make the quality of the ud data used in this study not entirely homogeneous. in spite of such disparity in the annotation reliability, and bearing in mind that a higher accuracy in this regard can be crucial, we considered the output provided by the parser for all the languages reasonably satisfactory for the purposes of our study. ## methodology the main aim of the experiments presented in this section consists in evaluating the contribution to irony detection made by the linguistic information provided in the datasets described above. the task we propose is, therefore, a straightforward binary classification task on irony detection, that is, the task for which literature offers baselines and fair-sized annotated datasets for a variety of languages. for addressing the task, we performed a set of experiments where several models were implemented exploiting classical machine learning algorithms, deep learning architectures and state-of-the-art language models implemented with the python libraries scikit-learn foot_4 and keras foot_5 . we tested different sets of pre-trained word embeddings to initialize the neural models, namely fasttext foot_6 and a dependency-based word2vec proposed by @xcite . the latter was trained on the concatenation of all the treebanks available in the ud repository for each considered language. in order to combine these methods with syntactic features inspired by @xcite , we used data where not only a binary annotation for irony is applied, but also a morphological and syntactic analysis is available (see § 4). ## experiments and results we performed different sets of experiments training the models on the training sets and evaluating them against gold test sets made available in the shared tasks referred to in ## conclusion and future work in the present paper, we have investigated different scenarios regarding irony detection exploiting dependency-based syntactic features in a multilingual perspective. our current findings provide a meaningful support to the hypothesis that morpho-syntactic knowledge extracted from treebanks can be usefully exploited for addressing the irony detection task. in particular, they pave the way for a further investigation where the combination of a dependency-based syntactic approach and state-of-the-art neural models can be explored. in the future, in order to validate our findings, we will propose a novel and wider experimental setting, where more languages are included, e.g. testing our approach on other languages for which both ironyannotated datasets and ud resources are available, such as arabic @xcite or german foot_19 @xcite .