Mixed Feelings: Cross-Domain Sentiment Classification of Patient Feedback (2025)

Abstract: Sentiment analysis of patient feedback from the public health domain can aid decision makers in evaluating the provided services. The current paper focuses on free-text comments in patient surveys about general practitioners and psychiatric healthcare, annotated with four sentence-level polarity classes - positive, negative, mixed and neutral - while also attempting to alleviate data scarcity by leveraging general-domain sources in the form of reviews. For several different architectures, we compare in-domain and out-of-domain effects, as well as the effects of training joint multi-domain models.

## introduction sentiment analysis (sa), the computational analysis of opinions and emotions expressed in text, is one of the applications of natural language processing (nlp) that have found the most widespread use across many different areas, including medical domains @xcite . as the task is mostly approached as one of supervised learning, access to sufficient amounts of labeled data is the main driver of performance. however, as manual annotation is costly, labeled data also represents a main bottleneck. for this reason it is typically desirable to be able to reuse existing resources when developing sa tools for a new area of application. unfortunately, domain-sensitivity is a well-known effect across many different nlp tasks. models trained on data from one domain (or genre or text-type) often underperform when applied to another due to variations in language use, terminology, and contextual nuances @xcite . * the authors contributed equally. this paper investigates cross-domain effects in polarity classification of public health data, more specifically free-text comments from patient surveys for general practitioners and psychiatric healthcare providers. we here investigate the usefulness of data from a different domain and genre, i.e. professionally authored reviews collected from norwegian news publishers. the datasets are annotated at the sentence level with the same four-class polarity labels; positive, negative, mixed, and neutral. in the following, we compare non-neural and neural architectures in both in-domain and cross-domain settings with the goal of providing high-quality sentiment analysis for norwegian patient comments. ## datasets we here briefly describe the two annotated sa datasets that form the basis of our experiments, also discussing some of their key differences. norpac for the health domain we will be using a dataset introduced by @xcite , comprising free-text comments from surveys conducted by the norwegian institute of public health (niph), as part of their so-called patient-reported experience measures (prems). the dataset is dubbed norpac -short for norwegian patient comment corpus -and comprises two related subdomains, corresponding to feedback on general practitioners (gps) and special mental healthcare (smh), with a total of 7693 sentences (4002 from gp and 3691 from smh) annotated for polarity. the norpac dataset is a valuable accession to norwegian corpora, as it gives valuable insights to the national public health system. the texts are written by patients after encounters with the healthcare system, and gives rise to language with an everyday character, such as sentences with a conversational tone or even incomplete sentences and spelling mistakes. example 1 shows a positive patient feedback sentence that is written solely in capital letters, in addition to containing a typing mistake in the personal pronoun jeg, 'i'. example 2 shows a negative review with a colloquial tone, containing three exclamation marks at the end of the utterance. norec the norwegian review corpus (norec; @xcite comprises full-text reviews collected from major norwegian news sources, covering a range of different domains (movies, music, literature, restaurants, various consumer products, etc.). we here use a version dubbed norec f ine @xcite , a subset of roughly 11,000 sentences across more than 400 reviews with fine-grained sentiment annotations, here aggregated to the sentence-level @xcite using the above-mentioned label set of four classes. 1 in contrast to norpac, the reviews are written by professional authors, meaning more creative writing but with sentences that are typically complete and grammatically correct. it er is en a pølse sausage i in salatens salad's.the tid, time, en a slags kind.of mumlemanisk mumblemaninc manns-modernitets-manifestasjon man-modernity-manifestation 'it is a sausage in the age of the salad, a kind of mumble-manic male-modernity-manifestation' example 3 shows one of many creative sentences in norec. en pølse i salatens tid, 'a sausage in the age of the salad', is a figurative way to emphasize the fact that this movie is not among the trendy, i.e. 'the salad', but rather acts like 'a sausage'. further, the author describes the movie as a mumlemanisk manns-modernitetsmanifestasjon, 'mumble-manic male-modernitymanifestation'. this exemplifies the complexity of many of the texts in the norec dataset where authors may construct new and creative expressions. genre and text type the two datasets can be said to be found at opposite ends in terms of language and writing style. in contrast to the professionally authored reviews in norec, containing grammatically correct texts with higher complexity and creativity, the norpac patient comments consist of more colloquial language. it also comes with many of the other hallmarks of usergenerated content, such as more frequent spelling mistakes and incomplete sentences, as well as unorthodox use of case and punctuation. while such properties will generally contribute to increasing the vocabulary size, norec still contains almost three times as many unique lemmas as norpac, due to the fact that it contains more creative and varied language (with a higher degree of figurative expressions, etc.), as mentioned above, in addition to covering multiple domains. class distribution for the norec reviews, we see that we have many more examples for the positive than the negative category. for the norpac patient feedback, in contrast, the negative category is notably larger, although the number of positive and negative examples are more balanced than in norec. another striking difference is the much higher ratio of neutral sentences in norec compared to norpac; 47% vs. 12%, respectively. this is not surprising if we consider the genre differences; professional reviews need to provide a lot of nonsentiment bearing background and descriptions of the object under review. the ratio of sentences with mixed polarity, however, is similar across the datasets, and is also the smallest sentiment class. related to the class distribution, we also observe some interesting differences with respect to the average token length of sentences. while the length is the same across the positive and negative sentences in the norec reviews, the length of negative sentences in the norpac patient comments tend to be substantially longer than the positive ones. however, for both datasets we see that neutral sentences tend to be shorter, while the mixed class displays substantially longer average length, which intuitively makes sense given that they per definition must express at least two opposing sentiments. ## experimental results below we report experimental results for a range of different models and architectures on the datasets described above. we start by providing details about the models and the experimental set-up, before discussing the results for both indomain and cross-domain classification. ## summary this paper has reported experimental results for polarity classification of sentences in a norwegian dataset dubbed norpac, comprising freetext comments from patient surveys collected as part of evaluating public healthcare services. in addition to assessing cross-domain effects between two healthcare sub-domains -feedback on general practitioners and psychiatric healthcarewe have also assessed the effect of leveraging general-domain sentiment annotations, based on the norec review data. rather than just annotating the simple binary classification of positive/negative sentences, our datasets additionally indicate both neutral and mixed sentences. we show how several of our tested model configurations surpass 85% weighted f 1 for this fourclass set-up. we also show how including out-ofdomain data improves model performance when in-domain data is limited, but that better performance can be achieved with in-domain data alone once the the amount of annotated data crosses a critical threshold. our analyses give new insights into both the norpac and norec datasets, including the differences and similarities between them. (rcn), namely 'sentiment analysis for norwegian text' (sant), funded by an iktpluss grant from rcn (project no. 270908), and 'strengthening the patient voice in health service evaluation: machine learning on free-text comments from surveys and online sources', funded by a helsevel grant from rcn (project no. 331770). moreover, the computations were performed on resources provided through sigma2 -the national research infrastructure provider for high-performance computing and large-scale data storage in norway.