Analyzing Subjectivity Using a Transformer-Based Regressor Trained on Naïve Speakers’ Judgements (2023)

Abstract: The problem of subjectivity detection is often approached as a preparatory binary task for sentiment analysis, despite the fact that theoretically subjectivity is often defined as a matter of degree. In this work, we approach subjectivity analysis as a regression task and test the efficiency of a transformer RoBERTa model in annotating subjectivity of online news, including news from social media, based on a small subset of human-labeled training data. The results of experiments comparing our model to an existing rule-based subjectivity regressor and a state-of-the-art binary classifier reveal that: 1) our model highly correlates with the human subjectivity ratings and outperforms the widely used rule-based “pattern” subjectivity regressor (De Smedt and Daelemans, 2012); 2) our model performs well as a binary classifier and generalizes to the benchmark subjectivity dataset (Pang and Lee, 2004); 3) in contrast, state-of-the-art classifiers trained on the benchmark dataset show catastrophic performance on our human-labeled data. The results bring to light the issues of the gold standard subjectivity dataset, and the models trained on it, which seem to distinguish between the origin/style of the texts rather than subjectivity as perceived by human English speakers.

## introduction the task of subjectivity detection refers to identifying opinions, attitudes, beliefs and private states in a given text. subjectivity detection as a task has received a lot of attention over the past decades, resulting in an abundance of methods and tools for subjectivity analysis. while in the earlier works, subjectivity was detected using rule-based approaches employing subjectivity lexicons @xcite , in the more recent studies, subjectivity detection is often approached with machine learning classifiers trained on existing gold standard datasets annotated for subjectivity @xcite . despite a relatively large body of literature on the topic, subjectivity detection has often been perceived as a preparatory step for sentiment analysis, that is, detection of positive or negative polarity of texts @xcite . accurate sentiment analysis relies primarily on subjective fragments of the text. for this reason, subjectivity detection has been mostly viewed as a binary classification task. however, given the complex nature of the interplay of viewpoints in texts and numerous ways of expressing oneself with varying intensity, subjectivity can also be considered a gradual measure. to the best of our knowledge, the most widely used tool for subjectivity analysis that offers a gradual subjectivity estimate is pattern regressor @xcite , which is rule-based, and therefore, less accurate compared to state-of-the-art systems. in the present paper, we approach the problem of identifying subjectivity as a regression task and use a semi-supervised approach to train a taskagnostic transformer model (roberta) to produce sentence-level subjectivity scores based on a small subset of human annotations. the data that we use for training are a dataset of news articles and social media news posts produced by major uk news sources, with a small subset of it labeled by native english speakers. we describe the training procedure and compare the results of the model to the average native speaker's judgements, and to the widely used rule-based pattern regressor @xcite to demonstrate that our model converges with native speaker intuitions and outperforms the rule-based regressor. the model is further evaluated as a binary classifier on our dataset and on the benchmark subjectivity dataset @xcite , showing good performance and generalizability to other discourse types. finally, we show that our model generalizes better to other discourse types than the current state-ofthe-art systems trained on the gold standard subjectivity dataset @xcite . these results highlight the importance of relying on actual human annotations rather than automatic labeling for compilation of subjectivity datasets and open further discussion about the nature of subjectivity models trained on the gold standard dataset. ## related work the problem of sentence-level subjectivity detection in the previous literature has been approached in two ways. the traditional approach is rule-based and uses subjectivity lexicons and linguistic pattern extraction to define subjective and objective text fragments @xcite @xcite . a more recent state-ofthe-art approach uses machine learning based text classification algorithms to detect subjectivity (e.g., @xcite @xcite @xcite . although training deep neural models can give impressive results, they require large annotated datasets and substantial computational resources, which are not always available. to overcome this issue, recent studies on subjectivity detection started employing pretrained language representation transformer models, such as bert @xcite , and fine-tuning them for subjectivity classification task, which showed very promising results @xcite @xcite . for instance, @xcite repot an impressive accuracy of 98.3% in subjectivity detection in the benchmark subjectivity dataset @xcite using a method based on electra-large transformer @xcite . in the present work, we adopt the approach of using a pretrained task-agnostic language model and fine-tuning it on subjectivity detection task for our own news dataset. the dataset consists of facebook news posts and online news articles produced by four major uk news sources, with the total size of 7,751 sentences. subjectivity of utterances is known to be a gradual factor in the cognitive theoretical accounts of subjectivity @xcite ; some utterances are perceived as more subjective than others. for this reason, in contrast to many previous studies on automatic subjectivity analysis, we approached the problem of subjectivity detection as a regression rather than a binary classification problem. to do so, we trained our model on a subset of our data that have been annotated for the degree of subjectivity by human raters using a 7-point scale. in the previous literature, benchmark datasets used for training subjectivity detection models were often obtained via annotations based on certain guidelines. for example, one of the earliest resources, namely mpqa opinion corpus @xcite , was compiled following a precise event-and entity-level annotation scheme for what is considered subjective and objective. in languages other than english, e.g., in italian corpus subjectivita @xcite , sentence-level subjectivity annotations were also obtained by asking annotators to follow specific guidelines on what should or should not be considered subjective. although the guidelines are often theory-based, it is difficult to estimate how they relate to the actual native speakers' intuitions. for example, telling annotators to label third person attitudes and beliefs as objective reflects a certain theoretical choice but may not reflect language users' perceptions (e.g., "according to the guests, the show was extremely unprofessional"). even more difficult to relate to human judgements are automatically collected subjectivity datasets, such as the benchmark subj dataset @xcite , which is a widely used dataset for model training and evaluation. this dataset contains 5,000 movie review snippets that are automatically labeled as subjective and 5,000 sentences from plot summaries that are automatically labeled as objective. however, a closer look at this dataset reveals many cases where objectivity of the sentences taken from the movie plot summaries is questionable (e.g., "what better place for a writer to pick up a girl?" is considered as being objective). in the present work, we train our model on subjectivity annotations by native speakers who were not asked to follow any guidelines except for brief definitions of subjective ("expressing opinions, attitudes and beliefs") and objective ("stating factual information"), which means that our model results represent how subjectivity would be perceived by naïve language users. similar approach to obtaining annotations with only definitions of subjective and objective as guidelines was used in the compilation of a czech subjectivity dataset @xcite . ## conclusion in this paper, we approached the problem of subjectivity analysis as a regression task and tested the efficiency of transformer language representation models, such as roberta, in annotating subjectivity using a paradigm of semi-supervised approach based on a small subset of human-labeled data. our model showed a very high correlation with the average human rater and significantly outperformed a widely used rule-based pattern subjectivity regressor. the model also performed well as a binary classifier, both on our news dataset and on the benchmark subjectivity dataset exemplifying different discourse types. in contrast, we found that the state-of-the-art classifiers with best performance on the benchmark dataset and trained on that dataset show catastrophic performance on our human-labeled dataset, which is not very different from the baseline. this means that our model generalizes across domains much better than the current best systems. moreover, this brings to light the issues of the subjectivity dataset that is considered the gold standard for subjectivity detection task. rather than labelling for actual subjectivity, the gold standard dataset uses the origin of the texts (movie review snippets vs movie plot descriptions) as a proxy for subjectivity. although the origin of the text undoubtedly correlates with subjectivity, these distinctions are not the same. as a result, state-of-the-art subjectivity classifiers trained on this dataset might be learning how to distinguish the language of movie review snippets from that of movie plot descriptions, rather than classifying subjectivity, as perceived by native speakers. future work could further analyze how the performance of state-of-the-art classifiers trained on the benchmark subjectivity dataset compares to humanlabeled subjectivity ratings in order to shed light on what exactly these systems are learning. our work highlights the importance of using human annotations in such complex tasks as subjectivity detection. future work can also be done in further comparing the performance of systems that are trained on the datasets labeled following explicit theoretical instructions to those trained on naïve human judgements about subjectivity. in addition, future studies on automatic subjectivity detection systems could investigate the origins of the differences in subjectivity perception across native speakers.