The Moral Foundations W eibo Corpus (2024)

Abstract: Moral sentiments expressed in natural language significantly influence both online and offline environments, shaping behavioral styles and interaction patterns, including social media self-presentation, cyberbullying, adherence to social norms, and ethical decision-making. To effectively measure moral sentiments in natural language processing texts, it is crucial to utilize large, annotated datasets that provide nuanced understanding for accurate analysis and model training. However, existing corpora, while valuable, often face linguistic limitations. To address this gap in the Chinese language domain, we introduce the Moral Foundation Weibo Corpus. This corpus consists of 25,671 Chinese comments on Weibo, encompassing six diverse topic areas. Each comment is manually annotated by at least three systematically trained annotators based on ten moral categories derived from a grounded theory of morality. To assess annotator reliability, we present the kappa test results, a gold standard for measuring consistency. Additionally, we apply several the latest large language models to supplement the manual annotations, conducting analytical experiments to compare their performance and report baseline results for moral sentiment classification.

## introduction moral foundations, delineated as intrinsic, universally applicable, and emotionally grounded psychological systems, stand as fundamental pillars of human morality @xcite . the exponential surge in global social media usage over the last decade has sparked inquiries into the intricate interplay between human psychology and online behavior @xcite . online behaviors, predominantly manifested through digital speech, serve as direct indicators of individuals' * these authors contributed equally to this work. previous resources have predominantly catered to english users @xcite , neglecting the vast chinese online community. with approximately 1.09 billion users, china constitutes one of the largest social media populations globally @xcite . in addition to this, there is a big difference between chinese and english regarding the expression of moral values and sentiments @xcite . in english, sentiments are usually expressed more directly, such as using explicit words to describe joy, anger, and sadness. in contrast, chinese culture is more concerned with maintaining face and social harmony and thus may be more subtle and euphemistic in its expression of sentiments. in other words, only english corpus is insufficient for analyzing the sentiments and moral foundations of chinese. together, the development of a chinese corpus assumes paramount importance in advancing moral natural language processing. in this context, our study focuses on constructing a moral foundations chinese corpus, leveraging weibo, china's foremost social media platform, as the primary data source. the moral foundations weibo corpus (mfwc) consists of 25,761 posts. adhering to the ten moral categories delineated in moral foundation theory @xcite : care/harm, fairness/cheating, loyalty/betrayal, authority/subversion, and purity/degradation, six prevalent topics among weibo users were selected for discussion: animal protection, real estate, people's livelihoods, volunteers, volunteer army, and the san francisco meeting be-tween the chinese and u.s. heads of state. analyzing moral sentiments within these thematic realms facilitates a nuanced understanding of prevalent moral sentiments on weibo. weibo is a public social media platform, current statistics reveal that weibo boasts 598 million monthly active users @xcite , it provides a rich and diverse dataset for analyzing moral sentiments within the chinese online community. the choice of weibo as the corpus foundation first stems from its extensive user base, drawn from diverse backgrounds, ensuring a rich and comprehensive dataset for analysis. mfwc has thus expanded the coverage of the existing corpus to include a significant portion of the global online population. furthermore, weibo's all-chinese system allows mfwc to bridge a crucial gap in existing corpora by catering to non-english-speaking users, thus fostering greater inclusivity and cross-cultural understanding in moral natural language processing research. we focused our corpus compilation effort on weibo for a number of reasons. variations in moral language across different social media platforms can indeed be attributed to disparities in language and social contexts @xcite . the distinctive features of weibo, china's predominant social media platform, elevate the significance and uniqueness of the moral foundations weibo corpus. unlike its western social media platform, weibo boasts a content moderation system characterized by heightened stringency. so it fosters a digital environment where expressions of moral sentiments are notably more moderated within the mfwc. for instance, when it comes to sensitive topics or political issues, you're likely to see relatively few angry comments or heated expressions of sentiments in mfwc, which may be more common on western social media platform. sentiments that may be more common on weibo are discreet, restrained, or indirectly expressed sentiments as well as innuendos to avoid touching on content that may trigger censorship. the stringent content moderation mechanisms on weibo contribute to the distinctiveness of the mfwc. by providing a dataset characterized by more moderate expressions of moral sentiments, the mfwc offers a valuable contrast to other corpus. this contrast not only enriches the diversity of available datasets but also underscores the importance of considering cultural and contextual factors in moral natural language processing research. second, the existing english annotation corpus cannot well adapt to the characteristics and needs of chinese moral expressions. in order to gain a deeper understanding of moral issues in chinese culture and society, we need to construct a chinese annotated corpus that is consistent with the latest theoretical revisions. besides, the complex language characteristics and contextual understanding involved in determining moral issues in the chinese context. the meaning of basic human communication can also become difficult to understand due to the complexity of language @xcite , it goes beyond the scope of determining simple keywords. therefore, sentencebased research is the main direction for studying the morality of chinese texts @xcite . compared to english, chinese has a unique grammar structure and vocabulary expression, which requires a chinese annotated corpus to deeply understand moral issues in chinese. we selected a large number of samples based on weibo entries in different moral and sentimental fields. in the selection of data sources, the selected samples ensure that they are representative and cover a variety of topics and perspectives. in order to ascertain the efficacy and comparative performance of distinct language models in discerning moral sentiments and to establish a baseline, we subjected some the latest large language models, namely gpt-4 @xcite , llama 3 @xcite and qwen 2 @xcite , to testing. besides, we also report baseline results for multiple computational approaches to measuring moral sentiment in text. these baselines can serve as a foundation for the classification models in moral sentiments detection tasks and provide a reference for future research, which can be instructive in paving the way for improved performance of more sophisticated models in the future. in view of the lack of chinese corpus resources, the main contributions of this study include the following three parts: (1) we have established a moral foundations weibo corpus, providing important basic data for future related research. (2) we provide richer and unique data interpretation using measurement baselines, which can provide inspiration for future research. ## corpus overview as mentioned above, mfwc contains 25,671 blog posts spanning seven distinct thematic domains. the selection of these thematic areas was guided by several considerations: firstly, we sought to include topics featuring prominent moral and sentimental expressions (e.g., animal protection) in our expectations to facilitate effective analysis of the dataset. secondly, the popularity and discussion intensity of the topic are also taken into consideration (e.g., real estate). the inclusion of widely discussed topics with substantial participant engagement enriches the dataset, fostering a diverse range of sentimental expressions and moral concerns. thirdly, we cover some sensitive elements, such as political and historical events (e.g., volunteer army, china-u.s. prime minister meets in san francisco). sensitive topics often encapsulate pressing societal issues and profound moral dilemmas, thereby enhancing the corpus's analytical depth and relevance. additionally, we considered the relevance and significance of each topic in contemporary culture and society, encompassing a diverse array of subjects and aiming to capture the multifaceted nature of digital discussions, thus providing a balanced portrayal of the online landscape. through the selection of these themes, we aim to enrich the diversity of the corpus and to differentiate the expressions of moral sentiments in the annotated corpus. in the domains we have chosen, these themes focus on different social events and the moral sentiments expressed within are susceptible to the discourse context and therefore moral sentiments are expressed differently. for example, the moral sentiments contained in the animal protection corpus are substantively distinct from those expressed in the volunteer army corpus, as these two topics focus on very different objects, with the former containing mainly care for animals and the latter placing the bulk of them on trolling the film. extra-domain predictions appear difficult due to this heterogeneity, and it is difficult for outsiders to generalise data from different domains through the expression of moral sentiments in one domain. based on this point, we provide moral sentiment annotations for tweets from different contexts to address this issue. ## baseline classification language models of moral sentiments while human annotation remains the most accurate method for measuring moral sentiment in text, due to the diversity of the chinese language and the large sample sizes typically required to study textbased moral sentiments, it is often necessary to supplement human annotations with classification models. our goal here is to establish baselines that can help us better predict moral sentiments. next, we report a baseline for moral sentiment classification using a range of models. to accomplish this task, we selected a number of models from a range of widely used models. these models include chatgpt-4 @xcite , llama 3 @xcite , and qwen 2 @xcite . each post in the mfwc dataset is assigned a specific label by three annotators, i.e., the six different moral sentiment categories discussed in section 3. this is a multi-label categorization task, meaning the categories of moral sentiments are not independent of each other, but are related. here, we provide both single-label and multi-label categorization results. gpt-4 @xcite in the first approach, we used gpt-4. gpt-4, a state-of-the-art language model developed by openai, has been fine-tuned to understand and generate human-like text across various tasks, making it highly suitable for complex sentiment analysis tasks. llama 3 @xcite in the second approach, we used llama 3. llama 3 is a versatile language model that has been designed for a range of natural language processing tasks. while it may qwen 2 @xcite ] lastly, we used qwen 2. qwen 2 is an advanced language model with particular strengths in precision tasks. it has been optimized for handling nuanced language variations, which makes it a strong contender in the moral sentiment classification task. to compare models sets, we rely on three performance metrics: precision, recall, and f1. precision, the number of true positives divided by the number of predicted positives, represents the proportion of predicted positive cases that actually are positive cases. in contrast, recall, the number of true positives divided by the number of true positives and false negatives, represents the proportion of positive cases that the classifier correctly identifies. finally, the f1 score is the harmonic mean of precision and recall (the product of twice the precision and recall divided by the sum of precision and recall), and is used to evaluate the performance of a classification models in a comprehensive way. the f1 score takes into account the balance between precision and recall, and is a good metric to use when there is a need to strike a balance between the two. ## results the results of the baseline models are provided in tables 3 to 13 . as expected, performance varied substantially across models, discourse domains, and prediction tasks. further, our results suggest that in the context of different domains and prediction tasks, each model showed different strengths and weaknesses. for instance, we found that gpt-4 performs best in balancing precision and recall, so its f1 value is usually the highest and suitable for most tasks that require balancing the two. llama 3 excels in recall and is better suited for tasks requiring high recall, but is slightly weaker in precision than the other models. qwen 2, on the other hand, excels in precision excels and is suitable for tasks requiring highly accurate predictions, but is weaker in recall, which affects its f1 value. lastly, performance differences, again, depend on the discourse domain and the moral foundation being analyzed. this variability in performance emphasises the need to take full account of the applicability of models when selecting and applying them, especially when dealing with complex and variable natural language data. our classification results generally demonstrate the feasibility of using multiple methods to measure moral sentiments in natural language. however, these results also highlight the complexity and challenge of this task. regardless of the model used, we observed significant variability in performance across different discussion domains and moral bases. this variability suggests that there are still shortcomings in the adaptability of the current approach to different contexts. in future research, it is necessary to delve deeper into the root causes of these performance variations and develop methods that can reduce them. in particular, researchers should aim to identify and understand the specific factors that lead to performance fluctuations, such as the semantic complexity of the text, the nuances of moral sentiments. in this way, we can not only improve the performance of current models, but also provide a more solid theoretical and technical foundation for the field of moral sentiment analysis. ## discussion natural language processing provides the fundamental tools for processing and understanding human language, which is essential for automated sentiment analysis. consequently, the integration of natural language processing and sentiment analysis represents an optimal convergence between linguistic theory and computational technology @xcite . as computational power increases, the quality and quantity of text mining and natural language processing techniques continue to improve, and the field of ethics through natural language research is becoming more and more widespread @xcite ). in the field of moral sentiment analysis, the most crucial element is the availability of theory-driven text data, which is necessary for the accurate quantification of moral sentiments @xcite . moral text data encompasses information pertaining to a multitude of dimensions within the domain of morality. by identifying the salient moral elements in a text, it will help to make moral judgments @xcite . to address this need, we developed the mfwc using posts and comments from weibo. the mfwc comprises 25,671 weibo posts from seven distinct topic areas, each of which has been annotated with one of ten types of moral sentiments based on the moral foundations theory. furthermore, the mfwc has been employed to establish a series of models classification baselines for measuring moral sentiments in texts. these baselines provide a foundation for further research and development in the field, enabling researchers to benchmark their models and improve their understanding of moral language dynamics. the construction of our chinese corpus addresses a significant gap in existing moral corpora. historically, the majority of moral language resources and datasets have been concentrated on western languages, particularly english. the construction of a comprehensive chinese moral corpus enables the reflection of the linguistic and cultural nuances of moral discourse in chinese @xcite . this corpus is of particular benefit to researchers seeking to develop linguistically supervised models tailored to chinese. the corpus offers a robust dataset for natural language processing applications, enhancing the capacity to analyse and interpret moral reasoning, sentiment, and values in chinese texts. the creation of such a corpus enables more accurate sentiment analysis, moral judgement, and ethical reasoning within the context of chinese culture and language. furthermore, this corpus can facilitate cross-cultural studies by providing a basis for comparing moral language across different languages and cultures @xcite . researchers can utilise this resource to develop models that are not only linguistically appropriate but also culturally sensitive, thereby ensuring that moral judgments and sentiments are understood within the correct cultural framework. by expanding the moral corpus to include chinese, we contribute to the creation of a more inclusive and representative dataset that can support a range of natural language processing applications. these include automated ethical decision-making systems and sentiment analysis in social media. in mfwc, we also present research on a new chinese moral-sentimental computation. our results demonstrate that the logistic regression models outperforms the other two on the classification task. however, these performance differences do not appear to be consistent across different discourse domains. with mfwc, researchers can gain a deeper understanding of the reasons for this, enabling them to comprehend the dynamics of sentiment in online language, capture trends in popular opinion, and identify subtle changes in social media. it is our hope that mfwc and this report will assist researchers by providing a unique data set and facilitate new contributions to the fields of natural language processing and social sciences. however, due to the vastness of chinese culture, our corpus may not be able to contain all chinese corpora. as more and more researchers utilise mfwc, we anticipate that the resources we provide here will be further expanded to better generalise to chinese studies.