Target Two Birds With One ST o N e: Entity-Level Sentiment and Tone Analysis in C roatian News Headlines (2023)

Abstract: Sentiment analysis is often used to examine how different actors are portrayed in the media, and analysis of news headlines is of particular interest due to their attention-grabbing role. We address the task of entity-level sentiment analysis from Croatian news headlines. We frame the task as targeted sentiment analysis (TSA), explicitly differentiating between sentiment toward a named entity and the overall tone of the headline. We describe SToNe, a new dataset for this task with sentiment and tone labels. We implement several neural benchmark models, utilizing single- and multi-task training, and show that TSA can benefit from tone information. Finally, we gauge the difficulty of this task by leveraging dataset cartography.

## introduction sentiment analysis (sa) is a common method in media and communication studies used to examine how different topics, events, or actors are portrayed in the media. it has been used to address media bias, framing, agenda setting, priming, and negativity in the news; e.g., @xcite @xcite . while sa often involves entire news reports, news headlines emerge as a relevant unit of analysis as they play a central role in grabbing the attention of audiences in the digital environment @xcite , optimize the relevance of the story for the headline scanning audience @xcite , and also act as a strong framing mechanism @xcite . although sentiment is often used interchangeably with tone, valence, affect, or polarity (cf. @xcite ; @xcite ), here we consider sentiment and tone as distinct concepts. sentiment is operationalized as a category that is † equal contribution. always determined in relation to a particular entity, whereas tone is more general and captures the overall mood and polarity of the entire news story or another unit of analysis. this conceptualization draws on the distinction made by @xcite between the "actor-related dimension of negativity" and the "frame-related dimension of negativity". the former dimension corresponds to sentiment, while the latter corresponds to tone. sa also has a long history in natural language processing (nlp) @xcite . the typical applications range from large-scale market research, product review analysis, and customer satisfaction estimation to voter profiling in political campaigns and media analysis. typically, sa aims to determine the overall sentiment expressed in the text, thus corresponding to the "frame-related dimension of negativity". often, this boils down to determining the sentiment polarity as either negative, neutral, or positive. in contrast, targeted sentiment analysis (tsa; @xcite al. (2019) ) focuses on sentiment expressed toward specific targets. specifically, entity-level sentiment analysis may be operationalized as tsa with the pre-extracted named entity (ne) mentions as targets, thus corresponding to the "actor-related dimension of negativity". this paper addresses the tsa task for the croatian language, more precisely, entity-level sentiment analysis from croatian news headlines. to this end, we first propose a novel dataset for this task called stone (sentiment and tone from news), with manually annotated entity-level sentiment and tone for news headlines. we investigate the relationship between targeted sentiment and tone, showing that there is a statistical dependence between the two. we then introduce and evaluate several neural benchmark models for this task. building on our finding that targeted sentiment depends on the tone of a headline, we examine whether multitask modeling of tsa and tone can improve tsa prediction performance. finally, to gauge the difficulty of the tsa on croatian news headlines, we diagnose the dataset using the cartography technique of @xcite , examining the relationship between annotator agreement and model correctness. the results show that, while tsa on croatian news headlines is challenging for humans and state-of-the-art models, our benchmark models considerably outperform the baseline. we also show that using the tone signal in a multi-task setup can improve tsa performance further. our contribution is threefold: (1) a novel dataset for the task of entity-level sentiment and tone analysis in croatian news headlines, which we make publicly available, 1 (2) neural benchmark models for this task in single-and multi-task setups, and (3) dataset diagnostics by means of dataset cartography. our work brings valuable insights for tsa in the slavic languages niche. ## related work the explicit emphasis on the target entity in tsa requires modeling the relationship between targets and their surrounding context. previous work captured this target-context interaction by isolating the target entity with bio tags and recurrent models @xcite or utilizing the attention mechanism @xcite . another approach leveraged transformerbased models such as bert for tsa tasks on user reviews @xcite and twitter data (mutlu and özgür, 2022) by focusing on target tokens for sentiment classification. in our work, we employ the target entity extraction method for the bert-based model, but we simplify the extraction by using only target embeddings. similar bert-based approaches were adapted for the targeted sentiment analysis in the news domain on a sentence @xcite and headline @xcite level for english and spanish language, respectively. in the realm of nlp for slavic languages, our work is similar to that of @xcite , who annotated news articles for slovene and croatian language and performed sentiment classification of news articles on three levels of granularity (document, paragraph, and sentence level). however, they did not analyze sentiment toward specific named entities. our work is most similar to @xcite , who annotated and analyzed the dataset of news headlines for sentiment analysis toward target entities in english and polish. we adopt a similar dataset design for annotating sentiment toward the target entity in news headlines for the croatian language, but we also consider the general tone of the headline. ## stone dataset our main contribution is stone, a dataset containing headlines from croatian news outlets labeled with sentiment towards nes and the general tone of the headline. to compile the dataset, we first sampled the headlines from a database of news articles acquired by takelab retriever, foot_0 a tool for analyzing croatian online news media. to identify the nes in the headlines, we ran the bertić model fine-tuned for the task of ne recognition, which achieves an average f1-score of 89.21 on croatian news hr500k dataset @xcite . we retained only the headlines that contained at least one ne. if a headline contained several nes, we randomly picked one as the target. consequently, a headline may appear in our dataset several times with a different target. we relied on a simple ternary annotation scheme, using the negative (neg), neutral (ntr), and positive (pos) labels for both targeted sentiment and tone. while we considered more fine-grained schemes, such as the one proposed by @xcite , we decided to use the ternary one as it proved to be sufficient in preliminary annotation rounds. this aligns with our intuition that, as one of the primary purposes of news headlines is to draw attention and generate interest, sentiment and tone labels should capture the reader's immediate first impression rather than the result of a conscious and deliberate evaluation process. the annotation was carried out by ten annotators using the alanno tool for annotation management @xcite . all annotators were native speakers of the croatian language. the annotators worked independently, with six annotators per instance to account for the highly subjective nature of the task. the data annotation process was completed within 14 person-hours. the text of the articles was not made available to the annotators, only the headline. for each instance, the annotators labeled both the tone and targeted sentiment but were advised first to determine the tone of the headline and then the targeted sentiment, assuming this order -going from general to more specificwould make annotating easier. the annotators were instructed to judge the tone and the sentiment based on their immediate impression of the headline. the guidelines further instructed them to consider the presence of epithets portraying the target entity or entire headline in a certain light and the context providing information about the nature of the event described in the headline. if none of these features were relevant, the annotators were told to rely on background knowledge of the subject in question. the annotators were also instructed to report erroneously identified nes, and these headlines were discarded. the final dataset contains 2855 headlines. for targeted sentiment, 1486 headlines were labeled as negative, 653 as neutral, and 716 as positive. regarding the tone, 1262 headlines were labeled as negative, 666 as neutral, and 927 as positive. interannotator agreement with the fleiss-kappa metric is κ = 0.416 and κ = 0.493 for targeted sentiment and tone, respectively, which is considered a moderate agreement (landis and @xcite . a moderate level of agreement is expected, considering the highly subjective nature of these tasks. ## benchmark models the backbone of all our experiments was the bertić model @xcite , based on electra @xcite . pre-trained bertić achieves state-of-the-art performance on many nlp tasks in slavic languages, including croatian. we use bertić to produce contextualized representations of nes in the headline for tsa. 3 gold dataset. we compiled the gold dataset for evaluating benchmark models by aggregating the labels of the ten annotators for both sentiment and tone using a majority vote. to sidestep the problem of adjudicating labels in cases with no majority agreement, we removed all instances where there are ties in either sentiment or tone annotations. we leave alternatives, including adjudication steps, more fine-grained schemes, or label distribution prediction for future work. the so-obtained gold set contains 2307 instances, of which 508 are negative, 1151 are neutral, and 648 are positive sentiment instances. for tone, there are 428 negative, 1060 neutral, and 819 positive instances. we randomly split the gold set into training, validation, and testing in a 70:10:20 ratio. single-task setup. we implemented one rudimentary baseline and four deep-learning benchmarks in the single-task setup for the tsa task. we use the univariate bayes as a baseline, with class likelihood parameters estimated by computing the labels' distribution for lemmatized nes appearing in the training set. entities were lemmatized using classla @xcite . the intuition was that the sentiment of some nes will be predictable regardless of the context they appear in. for out-of-vocabulary nes, the prediction was made by sampling a label from the training set distribution conditioned on the ne type. we implemented four benchmark models. in the first model (target), the entire headline is fed to the model, followed by the extraction of only the target ne embeddings span. the second model (masked) is fed with a headline where the target entity is replaced by a special [mask] token. this tests the assumption that the targeted sentiment depends only on the context independently of the concrete entity. we experimented with including the ne type as a feature concatenated to the averaged embedding of an ne span before feeding it to the classification layer for both target (target+type) and masked (masked+type) models. we fed the averaged contextualized embeddings of the ne span to the classification layer in each model. multi-task setup. annotation analysis in section 3 revealed that there is a statistical dependency between tone and sentiment labels. we hypothesize that this dependency can be leveraged to obtain more accurate tsa predictions. to investigate this, we combined tsa and tone classification tasks into a multi-task training design @xcite . the multi-task setup was implemented on top of bertić with one linear classification head for tone classification and the other for targeted sentiment classification (target single-task model). we implemented three multi-task setups. the first, alternate batch setup mimics the suggested procedure for dataset annotation, where, for each instance, the tone is annotated first, and the entity-level sentiment is annotated second. within each mini-batch, we first present the model with tone instances, calculate the loss, and update the parameters based on the derivatives of the tone loss gradients. we then do the same for same-batch instances but this time for sentiment labels. we alternate between the two tasks during each epoch. this is in contrast to the second setup we considered, alternate epoch, where we first update model parameters depending on all tone training instances and then update the parameters based on all sentiment training instances using the appropriate classification head. task-wise updates are conducted in a mini-batch fashion. the third setup is average batch, where we calculate the loss on a mini-batch level for both tasks and then update the model parameters based on the derivatives of the averaged batch loss. ## dataset diagnostics the dataset cartography method @xcite ) makes it possible to analyze the characteristics of a dataset in relation to model performance. the method uses three metrics -confidence, variability, and correctness -to measure the model's performance on the individual instances over training epochs. instances may then be grouped into three regions reflecting their difficulty: easy-to-learn instances are of high confidence and low variability, hard-to-learn instances are of both low confidence and low variability, while other instances are considered ambiguous. figure 2a shows the cartography of the stone dataset for the average batch model. the dataset exhibits patterns already observed for other nlp datasets. this visualization is especially useful for identifying hard-to-learn instances with critically low correctness. however, as noted by @xcite , poor model perfor- category title gold label model prediction majority localities 1 kaos u portugalu: policajci su pucali na nogometnoj utakmici neg ntr .67 (chaos in portugal: policemen fired shots at a football game) 2 italija bilježi stalni porast broja zaraženih, ali ministar zdravstva očekuje početak pada potkraj proljeća neg ntr .67 (italy is recording a constant rise of infected, but the minister of health is expecting a decline by the start of spring) quotes 3 bandić: informatika se u škole uvodi da bi se izbacio vjeronauk neg ntr .83 (bandić: informatics is being introduced in schools to cancel religious studies) 4 michael phelps: sad mi je tek jasno da sam bio jednaki šupak od čovjeka kao jordan neg ntr .50 (michael phelps: it is only now clear to me that i was just as big of an asshole as jordan) inference 5 posrtali su tamo gdje nisu smjeli! ovo su utakmice koje su hajduk koštale osvajanja naslova prvaka neg ntr .67 (they failed where they shouldn't have! these are the matches which cost hajduk the championship title) 6 kraljica u seksi kombinezonu: u ovom se vojvotkinja catherine nikad ne bi pojavila neg ntr/poz .83 (queen in a sexy overall: dutchess catherine would never be seen wearing this) instead of looking at human consensus, to gain an insight into the phenomena the model seems to struggle with, we looked into instances with low correctness (≤ .1). ## conclusion we introduced a dataset for entity-level sentiment and tone analysis in croatian news headlines. we tested neural benchmark models in a single-and multi-task setup, achieving the best results with representations of named entities and multi-task training. dataset cartography identified several problematic cases for the model, which could be addressed in future work. future work may also consider different framings of the tsa task.