Infusing Finetuning with Semantic Dependencies (2021)

Abstract: For natural language processing systems, two kinds of evidence support the use of text representations from neural language models “pretrained” on large unannotated corpora: performance on application-inspired benchmarks (Peters et al., 2018, inter alia), and the emergence of syntactic abstractions in those representations (Tenney et al., 2019, inter alia). On the other hand, the lack of grounded supervision calls into question how well these representations can ever capture meaning (Bender and Koller, 2020). We apply novel probes to recent language models— specifically focusing on predicate-argument structure as operationalized by semantic dependencies (Ivanova et al., 2012)—and find that, unlike syntax, semantics is not brought to the surface by today’s pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning, yielding benefits to natural language understanding (NLU) tasks in the GLUE benchmark. This approach demonstrates the potential for general-purpose (rather than task-specific) linguistic supervision, above and beyond conventional pretraining and finetuning. Several diagnostics help to localize the benefits of our approach.1

## introduction the past decade has seen a paradigm shift in how nlp systems are built, summarized as follows: the newer paradigm encourages an emphasis on corpus curation, scaling up pretraining, and translation of end-user applications into trainable ''tasks,'' purporting to automate most of the labor requiring experts (linguistic theory construction, annotation of data, and computational model design). apart from performance improvements on virtually every task explored in the nlp literature, a body of evidence from probing studies has shown that pretraining brings linguistic abstractions to the surface, without explicit supervision @xcite @xcite . there are, however, reasons to pause. first, some have argued from first principles that learning mappings from form to meaning is hard from forms alone @xcite ). 2 second, probing studies have focused more heavily on syntax than on semantics (i.e., mapping of forms to abstractions of meaning intended by people speaking in the world). @xcite noted that the bert model @xcite offered more to syntactic tasks like constituent and dependency relation labeling than semantic ones figure 1 : an example sentence in the dm (top, blue) and stanford dependencies (bottom, red) format, taken from @xcite and @xcite . based on that finding, we hypothesize that semantic supervision may still be useful to tasks targeting natural language ''understanding.'' in §4, we introduce semantics-infused finetuning (sift), inspired by pre-neural pipelines. input sentences are first passed through a semantic dependency parser. though the method can accommodate any graph over tokens, our implementation uses the delph-in mrs-derived dependencies, known as ''dm'' @xcite , illustrated in figure 1 . the task architecture learned during finetuning combines the pretrained model (here, roberta) with a relational graph convolutional network (rgcn; @xcite ) that reads the graph parse. though the same graph parser can be applied at inference time (achieving our best experimental results), benefits to task performance are in evidence in a ''light'' model variant without inference time parsing and with the same inference cost as a roberta-only baseline. we experiment with the glue benchmarks ( §5), which target many aspects of natural language understanding @xcite . our model consistently improves over both base and large-sized roberta baselines. in §6, we use the hans @xcite and glue @xcite diagnostics to better understand where our method helps on natural language inference tasks. we find that our model's gains strengthen when finetuning data is reduced, and that our approach is more effective than alternatives that do not use the full labeled semantic dependency graph. ## predicate-argument semantics as dependencies though many formalisms and annotated datasets have been proposed to capture various facets of natural language semantics, here our focus is on predicates and arguments evoked by words in sentences. our experiments focus on the delph-in dependencies formalism @xcite , commonly referred to as ''dm'' and derived from minimal recursion semantics @xcite and head-driven phrase structure grammar @xcite . this formalism, illustrated in figure 1 (top, blue) has the appealing property that a sentence's meaning is represented as a labeled, directed graph. vertices are words (though not every word is a vertex), and 59 labels are used to characterize argument and adjunct relationships, as well as conjunction. other semantic formalisms such as psd @xcite , eds @xcite , and ucca @xcite also capture semantics as graphs. preliminary experiments showed similar findings using these. frame-based predicate-argument representations such as those found in propbank @xcite and framenet @xcite are not typically cast as graphs (rather as ''semantic role labeling''), but see @xcite for data transformations and @xcite for methods that help bridge the gap. graph-based formalizations of predicateargument semantics, along with organized shared tasks on semantic dependency parsing @xcite , enabled the development of data-driven parsing methods following extensive algorithm development for dependency syntax @xcite . even before the advent of the pretraining-finetuning paradigm, labeled f 1 scores above 0.9 were achieved @xcite . some similarities between dm and dependency syntax (e.g., the stanford dependencies, illustrated in figure 1 , bottom, red; de @xcite are apparent: both highlight bilexical relationships. however, semantically empty words (like infinitival to) are excluded from the semantic graph, allowing direct connections between semantically related pairs (e.g., technique ← apply, @xmath0, and @xmath1, all of which are mediated by other words in the syntactic graph). dm analyses need not be trees as in most syntactic dependency representations, foot_4 so they may more directly capture the meaning of many constructions, such as control. ## probing roberta for predicate-argument semantics the methodology known as ''linguistic probing'' seeks to determine the level to which a pretrained model has rediscovered a particular linguistic abstraction from raw data @xcite @xcite , inter alia). the procedure is: for both syntactic and semantic parsing, our full ceiling model and our probing model are based on the @xcite parser that underlies many state-of-the-art systems @xcite . our ceiling model contains nonlinear multilayer perceptron (mlp) layers between roberta/bert and the arc/label classifiers, as in the original parser, and finetunes the pretrained representations. the probing model, trained on the same data, freezes the representations and removes the mlp layers, yielding a linear model with limited capacity. we measure the conventionally reported metrics: labeled attachment score for dependency parsing and labeled f 1 for semantic parsing, as well as labeled and unlabeled exact match scores. we follow the standard practice and use ptb sd conll 2015 dm metrics abs δ rel δ ceiling probe abs δ rel δ ceiling probe las/f 1 -13.5 ±0.2 -14.2% ±0.2 95.2 ±0.1 81.7 ±0.1 -23.5 ±0.1 -24.9% ±0.2 94.2 ±0.0 70.7 ±0.2 lem -36.4 ±0.8 -72.4% ±1.1 50.3 ±0.5 13.9 ±0.5 -45.4 ±1.1 -93.5% ±0.5 48.5 ±1.2 3.1 ±0.2 uem -46.3 ±0.7 -73.2% ±0.5 63.3 ±0.8 17.0 ±0.3 -48.8 ±1.0 -92.8% ±0.5 52.6 ±1.0 3.8 ±0.2 (a) base. ptb sd conll 2015 dm metrics abs δ rel δ ceiling probe abs δ rel δ ceiling probe las/f 1 -17.6 ±0.1 -18.5% ±0.1 95.3 ±0.0 77.7 ±0.1 -26.7 ±0.3 -28.3% ±0.3 94.4 ±0.1 67.7 ±0.2 lem -40.0 ±0.6 -77.2% ±0.4 51.9 ±0.6 11.8 ±0.2 -46.6 ±1.1 -94.4% ±0.1 49.3 ±1.1 2.7 ±0.0 uem -50.2 ±0.6 -77.4% ±0.2 64.8 ±0.7 14.6 ±0.2 -50.0 ±1.1 -93.9% ±0.2 53.2 ±1.0 3.3 ±0.1 (b) large. we also report their absolute and relative differences (probe -full). the smaller the magnitude of the difference, the more relevant content the pretrained model already encodes. we report the canonical parsing metric (las for ptb dependency and labeled f 1 for dm) and labeled/unlabeled exact match scores (lem/uem). all numbers are @xmath2. the chu-liu-edmonds algorithm @xcite to decode the syntactic dependency trees and greedily decode the semantic graphs with local edge/label classification decisions. see appendix b for training details. comparisons between absolute scores on the two tasks are less meaningful. instead, we are interested in the difference between the probe (largely determined by pretrained representations) and the ceiling (which benefits also from finetuning). prior work leads us to expect that the semantic probe will exhibit a larger difference than the syntactic one, signalling that pretraining surfaces syntactic abstractions more readily than semantic ones. this is exactly what we see in tables 1 across all metrics, for both roberta-base and roberta-large, where all relative differences (probe -full) are greater in magnitude for parsing semantics than syntax. surprisingly, roberta-large achieves worse semantic and syntactic probing performance than its base-sized counterpart across all metrics. this suggests that larger pretrained representations do not necessarily come with better structural information for downstream models to exploit. in appendix c, we also show that bert-base shows the same qualitative pattern. ## finetuning with semantic graphs given pretrained roberta's relative incapability of surfacing semantic structures ( §3) and the importance of modeling predicate-argument semantics ( §2), we hypothesize that incorporating such information into the roberta finetuning process should benefit downstream nlu tasks. sift, briefly outlined in §4.1, is based on the relational graph convolutional network (rgcn; schlichtkrull et al., 2018). §4.2 introduces a lightweight variant of sift aiming to reduce test time memory and runtime. 4.1 sift sift first uses an external parser to get the semantic analysis for the input sentence. then it contextualizes the input with a pretrained roberta model, the output of which is fed into a graph encoder building on the semantic parse. we use rgcn to encode the dm structures, which are labeled graphs. the model is trained end-to-end. figure 2 diagrams this procedure. rgcn. rgcn can be understood as passing vector ''messages'' among vertices in the graph. the nodes are initially represented with roberta token embeddings. at each rgcn layer, each node representation is updated with a learned composition function, taking as input the vector representations of the node's neighbors as well itself. each dm relation type is associated with a separately parameterized composition function. for tasks such as text classification or regression, we max-pool over the final rgcn layer's output to obtain a sequence-level representation for onward computation. readers are referred to appendix a and @xcite for further details. ## experiments we next present experiments with sift to test our hypothesis that pretrained models for natural language understanding tasks benefit from explicit predicate-argument semantics. ## analysis and discussion in this section, we first analyze in which scenarios incorporating semantic structures helps roberta. we then highlight sift's data efficiency and compare it to alternative architectures. we show ablation results for architectural decisions in appendix d. all analyses are conducted on roberta-base. ## conclusion we presented strong evidence that roberta and bert do not bring predicate-argument semantics to the surface as effectively as they do for syntactic dependencies. this observation motivates sift, which aims to incorporate explicit semantic structures into the pretrainingfinetuning paradigm. it encodes automatically parsed semantic graphs using rgcn. in controlled experiments, we find consistent benefits across eight tasks targeting natural language understanding, relative to roberta and a syntax-infused roberta. these findings motivate continued work on task-independent semantic analysis, including training methods that integrate it into architectures serving downstream applications. che et al. (2019 ) semantic parser's conll 2019 tokenization, we align the two tokenization schemes using character level offsets, as illustrated in figure 3 . for each node i, we find wordpieces [t j , • • • , t k ] that it aligns to. we initialize its node embedding by averaging the vectors of these wordpiece followed by an learned affine transformation and a relu nonlinearity: here w e is a learned matrix, and the e vectors are the wordpiece representations. the superscript on h denotes the layer number, with 0 being the input embedding vector fed into the rgcn layers.