MAD - TSC : A Multilingual Aligned News Dataset for Target-dependent Sentiment Classification (2023)

Abstract: Target-dependent sentiment classification (TSC) enables a fine-grained automatic analysis of sentiments expressed in texts. Sentiment expression varies depending on the domain, and it is necessary to create domain-specific datasets. While socially important, TSC in the news domain remains relatively understudied. We introduce MAD-TSC, a new dataset which differs substantially from existing resources. First, it includes aligned examples in eight languages to facilitate a comparison of performance for individual languages, and a direct comparison of human and machine translation. Second, the dataset is sampled from a diversified parallel news corpus, and is diversified in terms of news sources and geographic spread of entities. Finally, MAD-TSC is more challenging than existing datasets because its examples are more complex. We exemplify the use of MAD-TSC with comprehensive monolingual and multilingual experiments. The latter show that machine translations can successfully replace manual ones, and that performance for all included languages can match that of English by automatically translating test examples.

## introduction text analysis needs to address both objective aspects, such as topic extraction, and subjective aspects, such as sentiment and opinion classification. in spite of recent progress brought by the introduction of large language models @xcite @xcite , sentiment classification remains a challenging task. expression of sentiments varies according to the data sources, languages and domain of the texts. these challenges are particularly important in targetdependent sentiment classification (tsc), which focuses on determining the sentiment expressed toward a given entity in a given context. the bulk of tsc-related research efforts are focused on major languages. this focus is an effect of the availability of generic and of task-specific resources in these languages @xcite . a majority of datasets are monolingual and, when they are multilingual @xcite @xcite @xcite , the examples are not aligned across languages. equally, a majority of existing datasets and methods are devised for texts such as tweets, reviews or comments @xcite @xcite in which sentiment is most often expressed explicitly. somewhat surprisingly, less attention is given to tsc in news, despite the usefulness of an automatic analysis of their content for the understanding of societally impactful processes such as disinformation or polarization @xcite . our main contribution is the introduction of mad-tsc, the first large multilingual aligned dataset for tsc in news articles. it includes 5,110 annotated entity mentions from 4,714 unique sentences. each sentence has professionally-translated and aligned versions in eight languages @xcite . sentences originate from 286 news sources published in over 30 countries. these characteristics differentiate the proposed dataset from existing resources, and in particular from newsmtsc @xcite , a monolingual dataset focused on american politics which is the closest to mad-tsc. we first present the dataset creation process, and provide a qualitative analysis of its content. then, we propose a thorough evaluation of state-of-the-art tsc methods on mad-tsc in monolingual and multilingual settings, with particular focus on the usability of machine translation in tsc. the main findings are the following: • the proposed dataset is more challenging since it includes more complex examples compared to existing resources, as detailed in subsection 3.5. the last two findings are particularly interesting since they show that if a domain-specific tsc dataset is available, it can be effectively used for multiple languages. overall, the introduction of mad-tsc will facilitate progress in multilingual tsc. the dataset and the full evaluation protocol are available online foot_0 to encourage further research, and to ensure reproducibility. ## related work target-dependent sentiment classification (also named aspect-based sentiment analysis @xcite or classification @xcite ) is a complex task due to the numerous factors which influence the way sentiments are expressed in texts @xcite , such as the language, the domain or the personal biases of the author/reader. tsc is often evaluated on short texts such as tweets @xcite , reviews @xcite or comments @xcite . a common characteristic of these resources is that sentiment is often expressed in an explicit way. news are more challenging texts because sentiments are expressed implicitly or indirectly @xcite , often include multiple targets in a single sentence @xcite , and both negative and positive arguments about a target entity are combined due to the fact that journalists are supposed to be objective @xcite @xcite . multilinguality is important in order to be able to analyze texts in different languages. multilingual datasets are proposed for tweets @xcite , reviews @xcite and institutional texts @xcite . while interesting, these datasets differ from mad-tsc because they do not focus on news. equally important, they are only aligned at a domain level, but not at an example level. this limits their utility in terms of comparative evaluation in monolingual and multilingual settings. multilingual approaches were also explored for news representation. for instance, bilingual word embeddings were used to compensate data scarcity in under-resourced languages @xcite or to transfer models between source and target languages in zero-shot settings @xcite . classical tsc methods rely on engineered features based on lexicons and syntactic analysis @xcite @xcite @xcite . strong progress in tsc was made possible by the introduction of deep language models, such as bert @xcite . improvements are obtained when pretraining includes a larger proportion of news (hamborg and donnay, 2021) -this is the case for english roberta @xcite or xlnet @xcite -or when including an intermediate tuning on domain-related data @xcite . naturally, further improvements are obtained by introducing tsc-specific architectures @xcite @xcite . we follow this trend and use pretrained language models in our experiments. to our knowledge, there are only four datasets dedicated to tsc in news. the first one @xcite has 1,592 examples, and includes only quotes from political news. quotes are interesting because they include a lot of sentiment expressions, but they are also easier since sentiment is often expressed explicitly @xcite . the second one @xcite has 1,274 examples. the size of these datasets makes them difficult to use with deep-learningbased tsc methods. the third dataset @xcite includes 3,002 examples. however, as noted later by its authors @xcite , the dataset is imbalanced and its sentiment expressions are predominantly explicit. the dataset which is closest to ours is newsmtsc @xcite . their common characteris-tics include: a focus on political news, an identical definition of the task, and a similar annotation process. importantly, we follow @xcite and instruct annotators to think from the author's perspective in a holistic way. they are instructed to consider the "what" of the sentence (events, facts) but also the "how" (choice of words, author's attitude). this choice contrasts with previous works @xcite , which distinguish author-and reader-levels in tsc, and is important in order to minimize the influence of personal biases. the main differences between mad-tsc and newsmtsc relate to multilingualism, complexity of examples, political topics and geography of examples, while maintaining the same order of magnitude in the number of samples, with 5,110 for mad-tsc , and 11,361 for newsmtsc. these differences are detailed in subsection 3.5, and they make mad-tsc appropriate for a thorough evaluation of tsc in multilingual settings. ## dataset construction we build on previous works devoted to the creation of sentiment classification datasets @xcite , and particularly of tsc ones @xcite @xcite . we describe the main steps of the dataset creation process and analyze the resulting dataset. ## conclusion we introduce mad-tsc, a dataset for multilingual target-dependent sentiment classification. compared to existing resources, the proposed dataset is aligned across languages, includes examples about geographically diversified entities. examples are longer and more complex because sentiment is often expressed in an implicit way. given its aligned character, mad-tsc dataset enables a comparison of sentiment classification between languages. performance varies significantly, and this variation is to a large extent explained by the quality of pretrained models available for each language. importantly, the mt experiments show that human translations can be replaced by automatic ones. the automatic translation of test sets from target languages to english is particularly interesting since it brings target-dependent sentiment classification in different languages to the same quality level as that of english. this allows tsc to be scaled for the languages included in this study without the need to develop language-specific training sets. the only condition is to have a labeled domain-related dataset in one language, english or other. future work will focus on limitations of the dataset: (1) the handling of example complexity, (2) the coverage of the dataset in terms of domains and entity types, (3) the number of included languages, and (4) the quality of pretrained language models. these limitations are discussed in more details in the dedicated section below.