Annotating and Training for Population Subjective Views (2023)

Abstract: In this paper, we present a dataset of subjective views (beliefs and attitudes) held by individuals or groups. We analyze the usefulness of the dataset by training a neural classifier that identifies belief-containing sentences that are relevant for our broader project of interest—scientific modeling of complex systems. We also explore and discuss difficulties related to annotation of subjective views and propose ways of addressing them.

## introduction collecting annotated data for training natural language processing (nlp) models is a difficult and expensive task, involving selection of data to annotate, preparing guidelines, training annotators, and more. with best prepared annotation efforts, one has to deal with disagreement among annotators, also known as human label variation @xcite , and find ways to mitigate or embrace it. the issue is even more prominent when it comes to annotation of tasks that deal with subjectivitywhen an annotation assignment is not guaranteed to have one correct answer, but is open to interpretation. an objective task, e.g., determining whether a word is a noun under an annotation schema informed by a certain linguistic framework, would be less complicated than a subjective task of determining whether or not a tweet is sarcastic. other examples of highly subjective tasks are emotion (e.g., @xcite , humour (e.g., @xcite , and, to an extent, fake news detection (e.g., @xcite . in this paper, we present work on another task that has a high level of subjectivity: identifying subjective views of populations. we describe the task, the associated annotation effort, model-training experiments with the resulting dataset, and initial work on using the outputs of the trained model. this work and its goals, as well as the definitions of subjective views and related terms, stem from work on using computational models to understand complex systems, e.g., agricultural value chains (avc), food supply chains, or pandemics. philosophers such as heidegger argue that our "being in the world" means that all our decisions are subjective and depend on our current operating context @xcite . with people being active participants and decision makers in the systems that modeling experts are trying to understand, these systems have to be, to an extent, driven by subjective beliefs of the human participants. thus, in order to have a complete mechanistic understanding of a complex process, it is crucial for modelers to access subjective views of the populations involved. with the abundance of information available online making it difficult for modelers to identify relevant subjective views, our goal is to identify them automatically. with this paper, we make the following contributions: we define beliefs as people's views on how the world works, or in other words, their mental models or parts of it. for instance, the following example shows people's understanding of the relation between price and quality, which can impact their purchasing behaviors, and, in turn, impact the food supply chain: consumers generally recognize that cheaper prices correspond with lower quality and tend to remain loyal to their preferences when prices increase. we define attitudes as subjective views that indicate people's feelings towards objects and events. the example below shows how people's attitude (wanting to secure more food) led to their behavior (cultivating crops twice within a season): however, the members cultivated rice twice in 2009/10 [...] because they did not plan to cultivate rice in 2010/11 and wanted to secure a whole year's worth of rice for their own consumption. what unites these two types of subjective views is that they both have a potential to impact human behavior, which can in turn impact complex systems that need to be modeled. for simplicity, we refer to both of them as beliefs in this paper. for a comparison between our definition of beliefs and that in other datasets and related tasks (e.g., opinion mining and stance detection), see section 6. in this effort, sentences are annotated with respect to a trigger word-a word that can potentially indicate a belief, e.g., think, feel, hope, and want. the list of trigger words used (further referred to as known triggers) was created by modeling domain experts and augmented by the authors during initial data analysis. in cases where a trigger or the sentence can have multiple possible meanings, annotators are encouraged to use the paragraph context of the sentence for disambiguation. ## application we are in early stages of using the the belief identifier for the modeling project. we rank beliefs based on similarity to topics and provide them to modeling experts. while automatically identified beliefs, expressed in natural text, cannot be directly fed into models, they can inform modelers' decisions on what parameters to include in models and how to weigh them. so far, the work on belief identification has been met with enthusiasm since, at a minimum, we can save modelers time by surfacing the information that they would normally need to manually search for. we are working on ways to improve the quality of the belief identifier as well as to make the information regarding population beliefs that we provide more systematic. our approach of using quality control is more similar to that of chen et al.: with a rather limited number of data points available for training (about a thousand) and between two and three annotations per data point, modeling uncertainty did not seem feasible. additionally, we had reasons to believe that some annotation variation came from annotator-and guidelines-related issues (see section 2.5.3) and not from the inherent subjectivity of the task, in which case uncertainty would need to be modeled. ## limitations and future work while our dataset shows promise based on the models we train with it, at about 1000 annotated examples in the training partition, it is relatively small. before working on increasing the size of the dataset, we need to work on improving the guidelines provided to mechanical turk workers and finding more robust ways of excluding bad faith annotators. the dataset also currently misses some information that could be useful, e.g., polarity, beliefs involving out-of-sentence coreference resolution, as well as believer and belief span annotations. we plan to address all of these in future work. ## conclusion in this paper, we create a dataset of subjective views of populations and test it by training and experimenting with a belief identifier model. we discuss the issues related to annotation and human label variation that we encountered during the annotation exercise such as the difficulty of creating guidelines for a subjective task and lack of certainty in annotators adhering to the annotation guidelines for various reasons (e.g., annotators focusing on different aspects of the guidelines or not annotating in good faith). we also compare two ways of managing human label variation-annotation quality control vs. majority voting with different thresholds-by evaluating a model performance under the two conditions. finally, we provide support for the idea that human language variation should be embraced by doing an error analysis of the model predictions, which shows how language ambiguity as well as human factor and guidelines-related issues make it impossible to rely strictly on majority voting without qualitative analysis while evaluating systems working on subjective tasks. figure 3 : lime analysis of a false negative example in which the model incorrectly judges a sentence with the first person pronoun we as non-belief. this could be happening because the dataset the model was trained on (quality controlled mturk) focuses on reported (non-author) beliefs, so the word we does not get associated with beliefs.