QA -Adj: Adding Adjectives to QA -based Semantics (2023)

Abstract: Identifying all predicate-argument relations in a sentence has been a fundamental research target in NLP. While traditionally these relations were modeled via formal schemata, the recent QA-SRL paradigm (and its extensions) present appealing advantages of capturing such relations through intuitive natural language question-answer (QA) pairs. In this paper, we extend the QA-based semantics framework to cover adjectival predicates, which carry important information in many downstream settings yet have been scarcely addressed in NLP research. Firstly, based on some prior literature and empirical assessment, we propose capturing four types of core adjectival arguments, through corresponding question types. Notably, our coverage goes beyond prior annotations of adjectival arguments, while also explicating valuable implicit arguments. Next, we develop an extensive data annotation methodology, involving controlled crowdsourcing and targeted expert review. Following, we create a high-quality dataset, consisting of 9K adjective mentions with 12K predicate-argument instances (QAs). Finally, we present and analyze baseline models based on text-to-text language modeling, indicating challenges for future research, particularly regarding the scarce argument types. Overall, we suggest that our contributions can provide the basis for research on contemporary modeling of adjectival information.

## introduction a main challenge addressed by natural language processing research is designing useful semantic representations, capturing and explicating important aspects of the meaning of a text. numerous recent works illustrate how even in the era of strong end-to-end neural models, leveraging explicit semantic representations facilitates downstream processing of challenging tasks @xcite @xcite . numerous semantic representations have been proposed and pursued @xcite . traditionally, semantic representations rely on pre-defined schemata of linguistic classes, e.g. semantic roles or relations. thus, mapping natural language onto its representations becomes a complex annotation task that requires significant linguistic expertise, causing challenges in data collection and utility in new domains and languages. recently, many researchers and practitioners seek to benefit from an explicit representation of text meaning while alleviating the reliance on hardto-scale structured formalisms. for instance, open information extraction @xcite has gained popularity as a light-weight, nl-based alternative to semantic role labeling (srl) formalisms like propbank @xcite or framenet @xcite . more recently, several works proposed using question-answer pairs (qas) as an intermediate structure, e.g. in order to assess information alignment between texts for evaluating summarization quality @xcite @xcite and faithfulness @xcite . while these latter works utilize "generalpurpose" question-answering datasets and models for generating the qas, @xcite put forward a systematically targeted qa-based semantic framework dubbed qasem. pioneered by addressing verbal predicate-argument relations in qa-srl @xcite , this framework integrates three systematic qa-driven representations, jointly covering semantic role labeling for verbs @xcite @xcite , nominalizations @xcite and informational discourse relations @xcite . however, current qa-based approaches lack principled coverage of adjectival information. in natural language text, adjectives carry vital information about the properties of entities, essential for many downstream nlp applications. for example, in benchmarks of the widely-used sentiment analysis task @xcite , adjectives comprise 75% of the annotated "sentiment triggers". in this work, we extend the qasem paradigm by capturing and explicating the fundamental aspects of adjectival information using natural-language question-answer pairs. our representation, termed qa-adj, consists of four adjective-related rolesobject, comparison, domain, and extent. as we will see later, these roles provide a fairly complete representation of the core arguments of adjectives. roles are annotated using question templates while arguments are captured as answers, as illustrated in the main contributions of this paper are as follows: (1) we formulate a qa-based representation for capturing adjectival arguments, grouping them into four semantic categories; (2) we present a method for collecting low-cost, high-quality qa-adj data through controlled crowdsourcing; (3) we create a qa-adj dataset, comprising over 5k sentences and 12k qa pairs, assess its quality, and compare it to propbank annotations for adjectival predicates @xcite ; (4) we finetune a baseline qa-adj parser and evaluate its performance, providing a foundation for future model development. overall, our work provides an intuitive qabased representation for explicitly capturing the semantics of adjectives, as well as a dataset and a parser for future research. ## dataset construction preprocessing and annotation interface in this section, we describe the dataset creation process and in section 5 analyze its quality. we annotated over 5k sentences with 9k adjective mentions, across two domains: wikinews and wikipedia. we select sentences that are also covered by previous annotated qasem datasets @xcite @xcite . in each sentence, we identify the target adjectives using spacy's pos-tagger. if an adjective is preceded by one of the words 'more', 'less', 'most', or 'least', then it is considered part of the target adjective. we developed a graphical user interface (gui) (see appendix, figure 1 ) deployed at amazon's mechanical turk crowdsourcing platform. the worker, presented with a sentence with a marked adjective as a target, should generate question-answer pairs pertaining to this adjective. questions are generated by filling templated slots using drop-down lists, whereas answers are selected by highlighting spans from the sentence, and manually corrected if needs be. the gui also includes a short overview of the task and instructions, along with 5 annotation examples. annotator selection and training we adapted the controlled crowdsourcing process used by @xcite for qa-srl. after establishing the task formulation and interface, the first two authors jointly annotated 60 instances as a seed gold set, for evaluating and guiding worker qualification. we then release a preliminary crowd-wide annotation round and contact workers who exhibit reasonable performance. they are asked to review our short guidelines, which highlight a few subtle aspects, and then annotate four qualification rounds, of 15-30 target adjectives each. each round is followed by extensive feedback via email, pointing at errors and missed arguments, which are identified by automatic comparison to expert annotation. in total, this worker training process lasted approximately 8 weeks, and cost 240@xmath0 and 9 hours of expert verification, while the training set costs 1456@xmath1 and 28 hours of expert review time for the entire dataset. this approach allowed us to efficiently collect a high-quality dataset for our qa-based representation of adjectival semantics. ## dataset analysis and quality in this section, we report several analyses to quantify and establish the quality and coverage of the qa-adj dataset. additional information about the joint distribution of different roles is reported in appendix a.4. ## baseline models we devise an initial qa-adj parser to serve as a baseline for future work on this task. we first apply the same preprocessing steps for identifying target adjectives as in our data collection procedure ( §4). then, following a prior qa-driven semantic parser @xcite , we fine-tune the text-to-text transformer model (t5; @xcite , which unifies multiple text modeling tasks, and achieves state-of-the-art results in various nlp benchmarks. we use huggingface @xcite for finetuning the t5 model. a special token is marking the target adjective within the input sentence, while the output is formatted as question: <@xmath2: <a>. in case the semantic role is empty, the parser is to generate the special token [no-qa]. in preliminary experiments, training a single model to generate all four qa pairs in one go has yielded poor results. we hypothesize this is due to the sparsity of the domain, comparison and extent question types, which appear in 8%, 5%, and 5% of the training examples, respectively. therefore, to set up baseline results, we finetune an independent t5 model on each question type separately. the train set per question type consists of all instances which have the specific question type answered, along with random negative samples, i.e. empty qa instances. the ratio of negative samples is treated as a hyper-parameter of the model and is optimized on the development set. since holding a separate fine-tuned t5 model for every qa type is memory-consuming, we also fine-tune a single t5 model using the union of the training sets of each question type, using a different prefix for each qa type. ## conclusion in this work, we propose and realize a new approach to representing the semantics of adjectives using natural language question-answer pairs, focusing on four generic, core semantic dimensions. this intuitive representation enables high-quality yet scalable annotation through controlled crowdsourcing along with minimal expert verification. our annotations explicate the fundamental aspects of an adjective's meaning in context, substantially overlapping with an expertly annotated srl resource while adding previously uncovered implicit arguments. we advocate utilizing qa-adj downstream as an alternative for syntactical or semantic representations. as an example, recent works on aspect-based sentiment analysis use syntactic or semantic dependencies as scaffolds for enhancing domain transfer @xcite @xcite . explicating relations between adjectives (sentiment/opinion terms) and their semantic objects (aspect terms) directly, qa-adj is a worthwhile alternative to dependency representation. future works should explore methods for improving the baseline models presented in this work, such as prompt tuning @xcite or multi-task learning with related qa-semantic tasks @xcite . in addition, since the annotations are based on natural language and layman workers, it is appealing to transfer the scheme into various languages, possibly utilizing both machine translation and/or crowd annotations.