EMONA : Event-level Moral Opinions in News Articles (2024)

Abstract: Most previous research on moral frames has focused on social media short texts, little work has explored moral sentiment within news articles. In news articles, authors often express their opinions or political stance through moral judgment towards events, specifically whether the event is right or wrong according to social moral rules. This paper initiates a new task to understand moral opinions towards events in news articles. We have created a new dataset, EMONA, and annotated event-level moral opinions in news articles. This dataset consists of 400 news articles containing over 10k sentences and 45k events, among which 9,613 events received moral foundation labels. Extracting event morality is a challenging task, as moral judgment towards events can be very implicit. Baseline models were built for event moral identification and classification. In addition, we also conduct extrinsic evaluations to integrate event-level moral opinions into three downstream tasks. The statistical analysis and experiments show that moral opinions of events can serve as informative features for identifying ideological bias or subjective events.

## introduction morality refers to a set of social moral principles to distinguish between right and wrong @xcite . moral judgment plays a crucial role in expressing public opinions, driving social movements, and shaping policy decisions. @xcite @xcite . moral foundations theory @xcite , provides a theoretical framework to categorize social moral principles into five dimensions, each associated with a positive and negative judgment: care/harm, fairness/cheating, loyalty/betrayal, authority/subversion, and purity/degradation (figure 1 provides detailed values). extracting moral 1 https://github.com/yuanyuanlei-nlp/emona_dataset framing from text demands a combination of sociological moral knowledge and contextual semantic understanding @xcite @xcite . in many studies, the moral foundations lexicon @xcite has been widely used to match words and identify moral foundations in text. realizing the limitations of this lexicon match approach, researchers have started creating moral foundation annotations in text and training moral foundation detectors using annotations. however, such resource creation efforts have mainly been devoted to social media short text analysis and moral frames were usually annotated for an entire social media post @xcite @xcite , in contrast, little work has explored moral sentiment within news articles at a more finegrained and nuanced level yet. in this paper, we propose a new task to understand moral opinions towards events in news articles. the concept of event refers to an occurrence or action, and is the basic element in story telling @xcite . in news media, the authors often express their stance through moral judgment towards events, so as to shape public opinions @xcite @xcite . to facilitate a profound study towards morality aspect of events, we create a new dataset, emona, annotated with event-level moral opinions in news articles. while we believe the created dataset emona can be widely useful for studying moral opinion injection in context, this effort is initially motivated by the potential significant role of moral opinions for media bias analysis. in addition, recognizing the difficulty of identifying subjective events in news articles, we also aim to understand whether event moral opinions enables uncovering implicit subjective events that are otherwise hard to detect. to address these research questions, we have carefully chosen 400 documents from three sources for annotation, including 180 documents from all-sides that spans over 12 domains and indicates article-level ideology bias @xcite ; half (150 documents) of the basil dataset that has sentence-level media bias annotated @xcite ; and the entire mpqa 3.0 dataset (70 documents) that has fine-grained opinions toward entities and events annotated @xcite . the annotation process went through two passes: the first pass annotates event mentions, and the second pass annotates moral opinions for individual event mentions with respect to the context of the news article. annotating event-level moral opinions within a news article turned out to be a challenging and demanding task for human annotators, we recruited five annotators and steadily improved pairwise inter-annotator agreements for both tasks to a satisfactory level. in total, the dataset contains over 10k sentences and 45k event mentions, among which, 9,613 event mentions were annotated as bearing moral opinions. the challenge of event morality analysis indeed lies in the implicit nature of moral opinions toward events. in some cases, event-level moral opinions can be too implicit to be identified as opinions. take the example 1 in figure 2 as an instance, which are annotated as non-opinionated content by previous work @xcite , the semantic implies cheating criticism to-wards empty talk, and infers praise towards patriotism event. the implicit nature of event-level moral opinions requires the annotators to take both local sentential context and global broader context into consideration during annotations. the numerical and visualization analysis of the dataset show that annotated event-level moral opinions can effectively reflect article-level ideology, and designate sentence-level political bias. take the example 2 in figure 2 as an illustration, liberal media focus on fairness judgment by framing the story as texas gov. contrasts christianity with homosexuality, while conservative media emphasizes on purity moral by praising texas gov. defends religious faith. while not evident to readers, the journalists often subtly influence public opinions through moral value implication @xcite . the annotated moral opinions towards events can uncover and provide fine-grained explanations for such ideological bias. we build baseline models for event moral identification and classification, and we further conduct extrinsic evaluations on three downstream tasks: article-level ideology classification, sentence-level media bias identification, and event-level opinions identification. the experiments demonstrate the usefulness of detecting event-level moral opinions on all the three extrinsic evaluation tasks, with f1 score improved by 3.35% to 4.71%, and also vali-date the value of our new dataset. ## related work moral foundation theory is a theoretical framework developed by sociologists @xcite @xcite which categorizes moral values into five primary dimensions. it has been extensively employed to examine the influence of moral principles on human behaviors, judgments, decision-making, and political stances @xcite @xcite @xcite . @xcite was widely employed in many applicational studies @xcite @xcite for identifying moral foundations. however, this lexicon match approach suffers from both over-labeling commonly used indicator words and overlooking context-dependent expressions of moral foundations. researchers @xcite have considered leveraging distributional word embeddings to mitigate the drawbacks of using the lexicon only. ## dataset analysis this section provides a numerical and visual analysis (figure 3 , 7) of the correlations between eventlevel moral opinions and article-level ideology, sentence-level media bias, and event-level general opinions. ## intrinsic evaluation in this section, we conduct intrinsic evaluations for the newly created dataset emona. specifically, we propose the following tasks and build evaluation models for them: (1) event moral identification: identify whether an event contains moral judgment or not (2) event moral classification: classify moral labels for events, including ten moral labels and non-moral label (3) end-to-end system: predict label for every words in a plain article, including ten moral labels, non-moral label, and non-event label. the former two tasks target towards events, while the end-to-end system generates labels for every words. the intrinsic evaluation models we built can serve as baselines for future work. ## extrinsic evaluation to validate the potential applications of this dataset, we further conduct extrinsic evaluations. motivated by the analysis that confirms positive correlations between moral opinions of events and ideological bias or general opinions, we propose to incorporate event-level moral opinions into three extrinsic evaluation tasks: (1) article-level ideology classification (2) sentence-level media bias identification (3) event-level opinion identification. ## conclusion this paper defines a new task that understands moral opinions towards events in news articles. we have created a new dataset emona, and provide the first annotations of event-level moral opinions in news articles. the dataset analysis, intrinsic evaluation, and extrinsic evaluation on three downstream applications are conducted for this new dataset, which showcase that event-level moral opinions can effectively reflect article-level ideology, designate sentence-level political bias, and uncover event-level implicit opinions. for future work, we will continue to improve the performance of event level moral opinion identification and classification, in addition, we are interested to explore other applications of this task, such as stance detection and news summarization.