QAS em Parsing: Text-to-text Modeling of QA -based Semantics (2022)

Abstract: Various works suggest the appeals of incorporating explicit semantic representations when addressing challenging realistic NLP scenarios. Common approaches offer either comprehensive linguistically-based formalisms, like AMR, or alternatively Open-IE, which provides a shallow and partial representation. More recently, an appealing trend introduces semi-structured natural-language structures as an intermediate meaning-capturing representation, often in the form of questions and answers.In this work, we further promote this line of research by considering three prior QA-based semantic representations. These cover verbal, nominalized and discourse-based predications, regarded as jointly providing a comprehensive representation of textual information — termed QASem. To facilitate this perspective, we investigate how to best utilize pre-trained sequence-to-sequence language models, which seem particularly promising for generating representations that consist of natural language expressions (questions and answers). In particular, we examine and analyze input and output linearization strategies, as well as data augmentation and multitask learning for a scarce training data setup. Consequently, we release the first unified QASem parsing tool, easily applicable for downstream tasks that can benefit from an explicit semi-structured account of information units in text.

## introduction a traditional line of research in nlp has been devoted to designing various kinds of semantic representations, that aim to explicate textual meaning with a formal, consistent annotation schema. representations such as semantic role labeling (srl; e.g. @xcite , discourse representation theory @xcite and others @xcite @xcite @xcite provide applications with an explicit account of semantic relations in a text. numerous recent works illustrate how leveraging explicit representations facilitate downstream processing of challenging tasks @xcite @xcite @xcite . while traditional representations rely on pre-defined schemata or lexica of linguistic classes (e.g. semantic roles), the popular approach of open information extraction (openie; @xcite aims for more loosely-structured, easily attainable representations, comprised of tuples of natural language fragments. these light-weight structures, however, come with a cost of lacking consistency and comprehensive coverage, and do not capture deeper semantic information like semantic roles. in a recent trend, which can be seen as an emerging mid-point between full-fledged semantic formalisms and bare-bone textual fragments, researches leverage question-answer pairs (qas) as a representation of textual information @xcite . for example, several works proposed using qas as an intermediate structure for assessing information alignment between texts, e.g. for evaluating summarization quality @xcite @xcite and faithfulness @xcite , using a question-generation plus questionanswering (qg-qa) approach. nevertheless, such question generation and answering models were not trained to provide a coherent representation of text meaning. in this work, we follow an evolving paradigm, consisting of tasks that aim to comprehensively capture certain types of predications using questionanswer pairs. the pioneering work in this framework is question answer driven semantic role labeling (qa-srl; @xcite . targeting verbal predicates, qa-srl labels each predicateargument relation with a question-answer pair, both were shot in the confrontation with police and have been recovering in hospital since the attack . qa-srl 1 when was someone shot? in the confrontation ; the attack 2 who was shot? both 3 who shot someone? police 4 where has someone been recovering? in hospital 5 how long was someone recovering from something? since the attack 6 who was recovering from something? both 7 what was someone recovering from? shot qanom 8 who confronted with something? both 9 what did someone confront with? police qadiscourse 10 since when have both been recovering in hospital? since the attack 11 while what were both shot? we find that fine-tuning t5 on the qa-based semantic tasks is favorable over prior approaches, producing state-of-the-art models for all the aforementioned tasks. our experiments suggest that t5 is good at learning the grammar characterizing our semi-structured outputs, and that input and output linearization strategies have a significant effect on performance. we further explore the benefits of joint multi-task training of nominal and verbal qa-srl. our tool, including models and code, is publicly available. foot_0 2 background ## modeling we release a qasem tool for parsing sentences with any subset of the qa-based semantic tasks. our tool first executes sentence-level pre-processing for qa-srl/qanom. it runs a part-of-speech tagger to identify verbs and nouns, foot_1 then applies candidate nominalization extraction heuristics (see §2) followed by a binary classifier for detecting predicative nominalizations @xcite . identified predicates are then passed into the qa-srl or qanom text-to-text parsing models, while the qadiscourse model takes a raw sentence as input with no pre-processing required. the models are described in detail in the following subsections. ## experimental setup datasets we use the qadiscourse and qanom original datasets @xcite . for qa-srl, we make use of the large scale training set collected by @xcite . however, prior work @xcite pointed out that their annotation protocol suffered from limited recall along with multiple, partially overlapping reference answers, hindering parser evaluation. for these reasons, @xcite applied a controlled crowdsourcing procedure and produced a high-quality evaluation set, dedicated for fair comparison of future qa-srl parsers. we adopt their annotations for validation and test. 5 datasets statistics are presented in evaluation metrics for qa-srl and qanom evaluation, we adopt the measures put forward by @xcite . the unlabeled argument detection metric (ua) measures how many of the predicted answers are aligned with ground truth answers, based on token overlap. aligned qas are then inspected for question equivalence to assess semantic label assignment, comprising the labeled argument detection metric (la). consequently, la figures are bounded by ua, as they require to match both the answer and the question to a gold qa to count as a true positive qa. analogously, we embrace the uqa and lqa metrics proposed by @xcite for qadiscourse evaluation. see appendix a.1 for a more detailed description of the evaluation measures. output set linearization experiment as stated, the output of the model is parsed into a set of question-answer pairs at post-processing. thus, the ordering one applies over the linearization of qas into an output sequence can be arbitrary. it is therefore appealing to examine which ordering schemes facilitate model learning more than others. 6 we compare a randomized order (random-order) with two consistent ordering methods. the answer-order method orders the qas according to answer position in the source sentence, teaching the model to "scan" the sentence sequentially in the search for arguments of the predicate. alternatively, qas can be ordered more conceptually, with respect to the semantic role they target. the role-order method sorts qas by their wh-word which is a proxy of semantic role. 7 in contrast to methods that confine the model to a fixed order, one could aim to teach the model to ignore qa ordering altogether. one way to achieve order invariance is to train over various permutations of the qa set rather than a fixed order per instance @xcite . in addition to orderinvariance, training on multiple permutations may enhance performance from a data-augmentation perspective, especially in a realistic medium-size dataset setting. thus, we experiment with three permutationbased augmentation methods. the most straightforward approach is to include all qa permutations of each predicate (all-permutations). 8 nevertheless, in order to cope with the exponential data imbalance toward predicates with more qa pairs, an alternative method samples a fixed number of k permutations for all predicates (fixed-permutations; we set @xmath0. on the other hand, there are reasons to assume that predicates with more qas would be generally harder for the model to learn (see appendix a.5). the third method therefore samples @xmath1, producing linearly imbalanced training data in which instance frequency is proportional to the number of qas in its output (linear-permutations). we train qa-srl and qanom baseline models using each of the above mentioned linearization methods. these models differ both in the semantic task they tackle (i.e. verbs vs. nominalizations) and in the training data scale; thus, in order to distinguish these two effects, we also experiment with training on a random subset of the verbal qa-srl training set with the same size as the qanom training set (qa-srl small). results of comparing the different linearization methods are in section 5.1. ## results in this section, we present the experiments we conducted on the qasem tasks and the corresponding results. we start with results of the experiment testing different linearization methods, and then discuss final performance of best models. we conclude by assessing out-of-domain generalization. ## analyses output validity as mentioned in section 2.2, qa-srl questions adhere to a specialized constrained format. it is therefore not trivial for a model pre-trained on free natural language to acquire these format specifications. nevertheless, we observe that the models have robustly internalized the special grammar of the qa sequences. only a small fraction (1.2%) of output qa-srl/qanom qas were automatically detected as not conforming with qa-srl specifications, of which vast majority (> 95%) are due to answer-sentence misalignment mostly owing to tokenization issues (e.g. answer token is out-of-vocabulary). ## conclusion we propose to bundle three qa-based semantic tasks into a congruent conceptual paradigm. we hence develop and release new state-of-the-art models for these tasks, based on a unified framework for fine-tuning a seq2seq pre-trained language model. specifically, we show the importance of output linearization choices, including permutation-based data augmentation techniques, and propose using joint learning of verbal and nominal qa-srl for further enhancing performance in medium-size dataset settings. we further demonstrate these models' out-of-domain robustness. utilizing these models, the qasem tool we release can be used in various downstream scenarios where an explicit account of textual information units is desired. for example, the recent trend of leveraging qas as an intermediate representation for various summarization-related tasks indicates the perceived attractiveness of this "open" representation style. in particular, questions and answers provide a natural linguistic mechanism for explicitly focusing on concrete information units. however, the common qa datasets (e.g. squad; @xcite , over which prior qa representations have been trained, were developed for modeling qa or reading comprehension as endtasks, but were not designed to provide a systematic semantic representation. hence, qa models trained on such datasets yield a non-systematic set of qas, which might introduce overlapping and non-exhaustive information units, hindering their downstream utility. on the other hand, qasem is designed to produce a systematic -i.e. consistent and comprehensive -set of qas, each targeting an atomic "statement" concerning different predications, thus providing a more precise representation of semantic structure. future work would incorporate upcoming qasem tasks regarding adjectives and noun modifiers into the current seq2seq framework. further, we plan to explore sentence-level modeling for predicting all qasem qas jointly.