From Stance to Concern: Adaptation of Propositional Analysis to New Tasks and Domains (2022)

Abstract: We present a generalized paradigm for adaptation of propositional analysis (predicate-argument pairs) to new tasks and domains. We leverage an analogy between stances (belief-driven sentiment) and concerns (topical issues with moral dimensions/endorsements) to produce an explanatory representation. A key contribution is the combination of semi-automatic resource building for extraction of domain-dependent concern types (with 2-4 hours of human labor per domain) and an entirely automatic procedure for extraction of domain-independent moral dimensions and endorsement values. Prudent (automatic) selection of terms from propositional structures for lexical expansion (via semantic similarity) produces new moral dimension lexicons at three levels of granularity beyond a strong baseline lexicon. We develop a ground truth (GT) based on expert annotators and compare our concern detection output to GT, to yield 231% improvement in recall over baseline, with only a 10% loss in precision. F1 yields 66% improvement over baseline and 97.8% of human performance. Our lexically based approach yields large savings over approaches that employ costly human labor and model building. We provide to the community a newly expanded moral dimension/value lexicon, annotation guidelines, and GT.

## introduction this paper presents a generalized paradigm for adaptation of tasks involving predicate-argument pairs, i.e., combinations of actions and their participants, to new tasks and domains. predicateargument analysis has been a longstanding area of research for many tasks: event detection @xcite , opinion extraction @xcite , textual entailment @xcite , and coreference @xcite . we refer to the induction of such representations as propositional analysis. we induce a proposition predicate(x 1 ,x 2 ,...) to represent sentences such as john wears a mask: wear(john,mask). we pivot off this explanatory representation to answer questions such as what is john's stance towards mask wearing? or what concerns does john have about mask wearing? stance detection has recently (re-)emerged as a very active research area, yet many approaches generally equate stance to raw (bag-of-word) sentiment and often employ machine-learning based models requiring large amounts of (labeled or unlabeled) training data. within such approaches, the notion of stance varies, but generally falls into one of a handful of "sentiment-like" categories for stance holder x regarding topic y, i.e., x agrees/disagrees with y @xcite , x favors/disfavors y @xcite , x is pro/anti y @xcite , x has a positive/negative opinion about y (aldayel and magdy, 2021), or x is in favor/against/neither y @xcite . we adopt the stance definition of @xcite , originally formulated for covid-19. there, a stance is a belief-driven sentiment, derived via propositional analysis (i.e., i believe masks do not help [and if that belief were true, i would be antimask]), instead of a bag-of-words lexical matching or embedding approach that produces a basic pro/anti label. this variant of stance detection uses a proposition to identify a domain-relevant belief in the covid-19 domain; the belief is leveraged to compute a belief-driven sentiment and attitude toward a topic in that domain (e.g., masks). for example, i believe masks do not protect me is rendered as a belief type protect with an underlying proposition protect(masks) and a negative overall stance toward the propositional content: masks. we implement and evaluate an analogous propositional framework for a new task, concern detection. we define a concern to be a topical type (e.g., economic) coupled with a set of values: (1) moral dimensions from moral foundations theory (mft) @xcite @xcite , represented as vice/virtue pairings (authority/subversion, care/harm, fairness/cheating, loyalty/betrayal, and purity/degradation); and (2) corresponding endorsement values, where "vice" is between 1 to 5 and "virtue" @xmath0. dimensions and endorsements shown in propositional representation is the centerpiece of both stance detection and concern detection, distinguishing our lexical-based approach from modelbased approaches trained on word-level annotations. predicate-argument structure captures relationships between multi-word constituents that need not be contiguous, thus inducing explainability. that is, jean-luc melenchon is not adjacent to economy, yet these terms are crucially related via the intermediate term ruin. this enables answers to questions such as what does the author believe jean-luc melenchon did to the economy? moreover, domain adaptation is streamlined through predicate-argument annotation, reducing effort needed for human annotation. annotation at the level of predicate-argument pairs factors out commonalities, reducing redundancy in the resource building process. during resource building each verb is visited only once rather than the multiple times required for word-level corpora annotation (see §3). for example, annotation of the word lead is done all in one shot with a handful of automatically presented de-duplicated cases, whereas corpus-based annotation requires repeated annotations of lead, substantially increasing human labor. we demonstrate that the key to reduced adaptation time is the coupling of semi-automatic resource building for concern types with automated expansion of domain-independent concern values using semantic similarity. we develop a ground truth (gt) based on expert annotators and compare stance proposition: ruin(jean-luc melenchon, economy) belief type belief/sent values destroy belief strength: 2.5, sentiment strength: -1 concern proposition: ruin(jean-luc melenchon, economy) concern type moral dims/values economic care: 1.4, purity: 2.75 ## background and motivation this section provides background and motivation for task and domain adaptability applied to concern detection in the 2017 french election domain ( §2.1, §2.2), including concern values induced from moral foundations theory (mft) ( §2.3). ## task and domain adaptation adaptation of stance detection to concern detection gives rise to a new framework for rapid development of a task-adapted system that retains domain adaptability and uses relatively low amounts of data, with only 2-4 hours of human categorization. ## annotation for ground truth we conduct an annotation task to develop ground truth (gt), against which to compare our concern detection system variants, based on 50 heldout tweets from the held-out development portion of the english subset of the kaggle twitter dataset on 2017 french elections @xcite . ground truth was produced for concern types and vice/virtue pairs for any of five moral dimensions, in accordance with guidelines in appendix c. annotation was completed by two non-algorithm developers (one with expertise in linguistics, the other with expertise in psychosocial moral indicators). for concern types the inter-rater reliability (irr) is calculated through macroaveraging of kappa scores @xcite which produces a 66% agreement, considered strong according to @xcite . by contrast, the macroaveraged irr for moral dimensions is low (16%), which is considered weak. for the task is required to achieve a reliable gt. we note that "ground truth" is inherently problematic with moral indicators given the complex way in which dimensions vary with socio-cultural factors. thus, for moral dimensions, system output is compared only against the single annotator with expertise in psychosocial moral indicators. appendix d presents the ground truth resulting from these annotations. kaggle data @xcite are open and publicly available, intended for research on text analytics. the data carry no privacy or copyright restrictions. furthermore, irb designates our work as non human subject research. annotators spent two hours apiece on the task. ## sample runs we have implemented/validated a system to detect concerns based on induced lexical resources, using spacy and srl. we derive a proposition coupled with a concern type and values (moral dimensions/endorsements). systems @xmath1,vice/@xmath2, with an average runtime of 0.6s per tweet on mac, with no gpus required. representative outputs on a 1k-tweet held-out portion of the development dataset from kag-gle 2017 french elections are shown in in example 3, electoral_process_ voting_laws (triggered by voters) is coupled with moral dimension/endorsement values, care (care), loyalty (country), authority (care, country), and degradation (care, voters). this example overgenerates, assigning degradation based on the terms care and voters. a further lexicon enhancement is needed (as alluded to in §7) for elimination of spurious lexical entries that lead to false positives (i.e., a reduction in precision). ## results and analysis we explore the performance of each of our four moral dimension lexicons by @xmath3,vice/@xmath4. concerns types are evaluated for an exact match against the gt concern (e.g., crimi-nal_justice) and moral values are evaluated for an exact match against the binary choice in the gt (vice or virtue). concern 1 represents the lexiconbased chooser described in §3.2. our (strong) baseline for moral values is "moral 1" @xcite . subsequent variants (moral 2-4) are the expanded moral dimension lexicons using the techniques described in §3.3. we also compare against random choosers for concern types (concern 0) and values (moral 0). system performance is measured by weighted macro-averaged precision (p), recall (r), and f1 scores. domain-dependent concern type is not affected by moral lexicon variants and independently has its own p/r/f1 scores. concern values (i.e., moral dimensions) are impacted by lexicon variants and therefore have a row corresponding to each variant. we note the importance of applying a weighted macro average to these scores due an imbalance in the distribution of classes (delgado and tibau, 2019), where the probability of one class can be substantially higher or lower from others. for example, we observed that the num- in contrast, moral 0 achieves 79.65% of the performance of the moral 1 baseline, with an f1 score of 19.65-not too far off from the 24.67 baseline f1 score. moreover, the f1 scores for moral 2-4 surpass this baseline, with statistically significant improvements indicated between all system pairs at the 3.5% level or better, according to the mcnemar statistical test (mcnemar, 1947). 7 that is, all lexicon expansion improvements are statistically significant. notably, a 231% improvement in recall is achieved for moral 4 over the strong baseline (moral 1): 65.15 vs. 19.69. this is achieved with only a 10% loss of precision, ultimately yielding an f1 score of 40.85 which is a 66% improvement. we observe that the precision-recall gap increases as the notion of similarity is loosened: (1) lead is similar to strip in moral 4, but not in moral 3 (reducing moral 4 precision); (2) price is similar to value in moral 4 but not in moral 3 (increasing moral 4 recall). we further assess system performance by comparing concern 1 performance to human performance. average f1 score for the two annotators is 78.88, and concern 1 performance is 77.17 f1. concern detection thus yields 97.8% of human performance on concern type detection. error analysis of fp/fn's for concern types reveals that concern detection fails to assign any concern type (fn) to infighting among left-wing could hand front national victory, which is annotated as electoral_process_voting_laws, because terms like front and left-wing are not present in the concern type lexicon. for concern values, the annotator does not assign a moral dimension to macron is center right, yet moral 4 inaccurately detects (fp) care, authority, and betrayal due to the existence of the word center and also detects purity from the word right. for this same sentence moral 1 also inaccurately detects fairness from the word right. many cases similar to these impact precision values for each moral 1-4, potentially requiring lexicon tuning (see §7). we conduct further analysis to determine whether performance is impacted by potential overfitting of concern value detection to the domain of interest during development. we compare araque's original concern value detection tools @xcite , a unigram model trained on hurricane sandy data from the moral foundations twitter corpus (mftc) @xcite , to our proposition-based concern value detection that uses semantically expanded lexicons based on the kaggle french elections data. we level the playing field by applying both approaches to both datasets, measuring each against their respective gts. testing our moral lexicon expansion for stability retention across domains, we level the playing field by relaxing the constraint that only a propositional sub-portion of the text is considered as input. we apply our algorithm to the entire text (full tweet) to induce a fairer comparison with araque et al's unigram model. more importantly, our expanded moral 4 lexicon demonstrates stability across domains: 42.61 for 2017 french elections and 42.03 for hurricane sandy (vs. 26.14 and 34.13, respectively, for araque et al's moral 1 lexicon). this illustrates the potential for proposition-based domain adaptability but highlights the need for hybridized detection to ensure both task adaptability and explainability (discussed further below). ## conclusions, limitations, future work we implement and validate a generalized paradigm for adaptation of tasks involving propositions to a new task (concern detection) and domain (french elections). our primary contribution is the provi-sion of a framework for rapid adaptability, based on: (a) semi-automatic domain-dependent extraction of types from propositional analysis; and (b) fully automatic domain-independent induction of values from propositional arguments and semantic resources. we demonstrate that the coupling of (a) and (b) leads to rapid ramp-up resource construction for a new task and domain (2-4 hours). we demonstrate that a deeper propositional analysis is key to generalizing domain-adaptable resource-building for new tasks. we develop an automatic procedure for expanding moral dimensions that incorporates propositional analysis, semanticrole labeling, and in-depth wordnet (wn) expansion, to produce three increasingly expanded moral dimension/endorsement lexicons. we develop a ground truth (gt) based on expert annotators and compare our concern detection output to gt, to yield 231% improvement in recall over baseline, with only a 10% loss in precision. f1 yields 66% improvement over baseline and 97.8% of human performance. our approach yields large savings over those that employ costly human labor and model building. work produced herein provides the community with a newly expanded moral dimension/value lexicon, annotation guidelines, and gt for 50 tweets, intended for research purposes. we show that our proposition-based moral lexicon expansion provides stability across domains. however, the results of concern value detection using the full text of an input (a tweet) highlight the importance of adopting a hybrid approach that captures fine-grained distinctions and produces an explainable representation that is not otherwise available (e.g., in the unigram language model of @xcite ). an avenue of future research is to explore a hybridized approach that makes fine-grained concern value distinctions for domain adaptability, while leveraging an explainable propositional representation for task adaptability. as a first step, we will apply our propositional analysis iteratively on full-text inputs to detect answers to questions not otherwise extractable from raw textual strings (see related discussion in §1). our results do not require any tuning of the lexicons to remove terms that result in a high number of false positives (fp) and false negatives (fn). future work will explore fine tuning of the lexicons to address cases seen in §6 with an eye toward improving the precision without a large drop in recall, to yield an even higher f1-score. a current limitation (for future study) is the omission of additional moral dimensions, e.g., liberty/oppression @xcite . this reflects political equality related to dislike of oppression and concern for victims, not a desire for reciprocity. in political discourse, this is apparent in antiauthoritarianism and anti-government anger, which makes it an important dimension for the topic of the french election @xcite and widens the range of potentially relevant information that could indicate influence attempts. future work also includes exploration of other cultural frameworks, in addition to or in place of moral foundations theory, e.g., inglehart's cultural map model @xcite and schwartz value theory @xcite . cultural models that allow more room for non-western values (e.g. survival needs) are important for reducing bias, and a feasible avenue for improving the performance and applicability of concern detection. another limitation is that gt development for a given cultural context is difficult, especially with diverse annotators. @xcite show that systematic disagreements between annotators with differing socio-cultural backgrounds are obfuscated through aggregating crowd-sourced annotations. future work will explore vector-based approaches to assign weights based on their representativeness in a given culture. cultural values are also reflected in language (e.g., gendered vs. non-gendered languages, culturally intrinsic concepts). accordingly, our future work involves processing concerns for other languages through adaption of srl @xcite to multilingual input, starting with french, employing multilingual preprocessing via spacy @xcite and eurowordnet and related multilingual wordnets @xcite , to expand moral dimensions to other languages.