A Dataset for Explainable Sentiment Analysis in the G erman Automotive Industry (2023)

Abstract: While deep learning models have greatly improved the performance of many tasks related to sentiment analysis and classification, they are often criticized for being untrustworthy due to their black-box nature. As a result, numerous explainability techniques have been proposed to better understand the model predictions and to improve the deep learning models. In this work, we introduce InfoBarometer, the first benchmark for examining interpretable methods related to sentiment analysis in the German automotive sector based on online news. Each news article in our dataset is annotated w.r.t. overall sentiment (i.e., positive, negative and neutral), the target of the sentiment (focusing on innovation-related topics such as e.g. electromobility) and the rationales, i.e., textual explanations for the sentiment label that can be leveraged during both training and evaluation. For this research, we compare different state-of-the-art approaches to perform sentiment analysis and observe that even models that perform very well in classification do not score high on explainability metrics like model plausibility and faithfulness. We calculated the polarity scores for the best method BERT and got an F-score of 73.6. Moreover, we evaluated different interpretability algorithms (LIME, SHAP, Integrated Gradients, Saliency) based on explicitly marked rationales by human annotators quantitatively and qualitatively. Our experiments demonstrate that the textual explanations often do not agree with human interpretations, and rarely help to justify the models decision. However, local and global features provide useful insights to help uncover spurious features in the model and biases within the dataset. We intend to make our dataset public for other researchers

## introduction there is a wealth of economic (online) news so that even specialists find it challenging to analyze all information in a timely manner. therefore, methods that seek to automatically identify the sentiment in textual data for calculating economic indices or trends have become prominent in recent years @xcite @xcite . in particular, deep learning based approaches have yielded high performance @xcite @xcite , and their results even seem to correlate with surveybased indicators @xcite . however, sentiment analysis foot_0 is challenging due to the domain-specific language and unavailability of labeled datasets needed for training or finetuning neural models @xcite . it is also a complex task, as a document may contain multiple targets whose sentiments may be opposite. for example, in the sentence um gute produkte und leistungen zu liefern, hat die firma hohe investitionen getätigt. (in order to deliver good products and services, the company has made high investments.), the author expresses a positive sentiment towards products and services using the opinion words gute (good) and a negative sentiment towards investitionen (investments) using hohe (high). apart from mixed sentiment, another challenge is that depending on the context, the sentiment may also shift to the opposite, e.g., hoher komfort/hoher verlust (high comfort vs. high loss). since model interpretability is a critical requirement for many downstream applications, recently explainable artificial intelligence (xai) techniques that make models explainable have been proposed @xcite , and also been adopted for sentiment analysis in general @xcite or with respect to financial news @xcite . in line with @xcite , we consider that an ai system is explainable, if the task model is intrinsically interpretable or if it is complemented with an interpretable and faithful explanation. in this work, we focus on transparent inference through the generation of a post-hoc explanation for the final output prediction @xcite . it is well known that machine learning techniques suffer from an inadequate generalization capacity on real-world datasets, which are often scarce and imbalanced. while transfer learning and regularization techniques have been proposed to solve this issue @xcite , overfitting the training data is still a common problem, especially for non-english texts. as has been stated by various authors, explainable modelling can contribute to trustworthy ai systems, that go beyond quantitative performance metrics such as accuracy @xcite . to this aim, we build up a corpus of german news articles that report recent innovations in the automotive field, which includes class labels for topics such as e-mobility, autonomous driving, diesel, hydrogen, synthetic fuel, and misc, sentiment labels i.e., positive, neutral, negative, and human rationales for studying trustworthiness, inspired by recent work in this field @xcite @xcite . we conduct experiments building on state-of-the art machine learning models for sentiment analysis, as well as interpretation methods @xcite using established evaluation metrics for interpretability @xcite . human explanation for sentiment classification is essential for high-impact settings such as financial analysis where human rationales are required to support specialists' decisions. basic research questions are • rq1 how well can we automatically classify news articles as expressing a positive, negative or neutral opinion at the document-level and which approach gives the best performance. how well is human iaa for our corpus. our main contributions are as follows: ## infobarometer dataset in this section, we provide the dataset collection strategy, the annotation procedure and statistics on our dataset. ## methodology due to the small size of our dataset, we think that there is high need for explainability, regarding model interpretability. through our experiments, we like to investigate if explainability techniques can uncover biases within the dataset and check the trustworthiness of the ml models trained on the infobarometer dataset. ## experimental setup we execute experiments on topic classification and sentiment analysis for four different ml approaches. the model predictions as well as the xai models' explanations are compared to ground truth data that has been annotated by domain experts. for an overview of the proposed approach see figure 4 . ## results -performance comparison for xai methods we compare lime, saliency, inputxgradient and shap with respect to the evaluation criteria plausibility (human agreement), faithfulness and runtime complexity. ## final discussion in this paper, we introduced a new benchmark corpus compiled from online news articles, annotated by 3 domain experts for document-level sentiment. moreover, it contains multiple rationales that provide evidence for the annotators choice of the overall sentiment. since many articles have a mixed sentiment, including borderline cases that are difficult to classify, highlighting positive as well as negative aspects mentioned in one single article, yields increased transparency. we used the corpus as a benchmark for the german language, where resources for studying explainability are scarce. we investigated several ml architectures for the task, in combination with different post-hoc explainability methods. since there is no single solution that is best suited to every use case, our analysis allows identifying the strengths and limitations of each method. our findings indicate that bert yields the best performance in terms of sentiment classification accuracy. in combination with shap, it offers a global view of feature importance, which helps detecting spurious features and bias. we think that end users will profit from xai methods which allow to get an aggregated view of feature importance for a particular topic category, or based on a specific time frame. however, due to the high dimension of our data, local explanations are overall not very plausible, regardless of the underlying ml model and explainability method. moreover, the bert model is less faithful than cnn and lstm, due to high complexity of the model. for our use case, the computational time for generating explanations with lime, ig or saliency would be acceptable in a real-time application, except for shap which suffers from a long computational time. in future work, we seek to identify the training data points responsible for model misclassifications and find training instances that show bias through influence functions @xcite , and investigate the impact of the pretrained embeddings and model.