Linguistic representations for fewer-shot relation extraction across domains (2023)

Abstract: Recent work has demonstrated the positive impact of incorporating linguistic representations as additional context and scaffolds on the in-domain performance of several NLP tasks. We extend this work by exploring the impact of linguistic representations on cross-domain performance in a few-shot transfer setting. An important question is whether linguistic representations enhance generalizability by providing features that function as cross-domain pivots. We focus on the task of relation extraction on three datasets of procedural text in two domains, cooking and materials science. Our approach augments a popular transformer-based architecture by alternately incorporating syntactic and semantic graphs constructed by freely available off-the-shelf tools. We examine their utility for enhancing generalization, and investigate whether earlier findings, e.g. that semantic representations can be more helpful than syntactic ones, extend to relation extraction in multiple domains. We find that while the inclusion of these graphs results in significantly higher performance in few-shot transfer, both types of graph exhibit roughly equivalent utility.

## introduction in many specialized domains, such as healthcare or finance, one of the principal limitations for the implementation of machine learning based nlp methods is the availability of high quality data, which tends to be both time-consuming and expensive to acquire. while pre-trained language models allow impressive performance gains across a number of tasks, those gains frequently fail to generalize to specialized domains. robust representations that allow models to both take advantage of pretrained models and generalize across domains are therefore highly desirable. recent works such as @xcite have demonstrated the significant potential of using human-annotated linguistic information as scaffolding for learning language models. other works such as @xcite and @xcite use automatically generated semantic annotations. these works depend on the idea that the structure that the linguistic frameworks provide allows models to better learn salient features of the input. in addition, however, linguistic structures offer abstraction over the variation of natural language, providing representations that might express meaning in domain-general ways. we therefore extend earlier work to investigate whether including linguistic representations encourages learning domain-agnostic representations of relations such that models can generalize better in a few-shot setting, i.e. learning from less high-quality data in new domains. we focus on the case of automatically generated linguistic annotations, to evaluate the impact they can have on downstream tasks without expensive human annotation of parse data. we use two linguistic formalisms, to evaluate and compare their impact: dependency parses, and abstract meaning representations (amr). amr @xcite seeks to represent meaning at the level of a sentence in the form of a rooted, directed graph. amr is based on propbank @xcite , and factors out syntactic transformations due to verb alternations, passivization, and relativization, leading to a less sparse expression of textual variance. dependency parsing, by contrast, remains at a low level of abstraction, with structures that do not nest outside of the words in the original text. we study the problem of relation extraction on procedural text. we use procedural text because of what we term their implicit schemas. across domains, our datasets describe the process of combining ingredients under certain conditions in a sequential fashion to produce a desired product. these range from preparing a cooking recipe to synthesizing a chemical compound to extracting materials from ores. as a result, the relations that we derive from each dataset share a loose semantic correspondence, both to each other and to basic semantic relations such as verb arguments and locations. for example, the actions "boil" and "heat" in "boil the mixture in a medium saucepan" and "heat the solvent in the crucible" are similar. we hypothesize that the underlying semantics of all of these datasets are similar enough that models should be able to better generalize across domains from the explicit inclusion of syntactic and semantic structural features. we use three datasets across two domains: recipes for cooking, and materials science synthesis procedures. each of these datasets defines the task of generating a comprehensive, descriptive graph representation of a procedure. we simplify this task into relation extraction in order to better compare the impact of different linguistic formalisms. we augment a popular transformer-based relation extraction baseline with features derived from amr @xcite and dependency parses and investigate their impact in a few-shot setting both in-domain and across domains. experiments show that both amr parses and dependencies significantly enhance model performance in few-shot settings but that the benefit disappears when models are trained on more data. we additionally find that while cross-domain transfer can degrade the performance of purely text-based models, models that incorporate linguistic graphs provide gains that are robust to those effects. we make our code available with our submission foot_0 . ## methodology we design our methodology to test whether the inclusion of amrs and dependency parses can improve the few-shot re performance across datasets, by incorporating features from linguistic representations. we show an overview of our architecture in figure 1 , and go into further detail in section 3.4. our three datasets have their goal of generating a complete graph representation of a specified procedure. this graph is constructed by first finding salient entities in the procedural text, and then correctly identifying the appropriate relations between them. while this joint task is both challenging and useful, we restrict ourselves to the re task for two reasons. firstly, entity recognition results, as measured by baselines proposed in each of the dataset papers, vary widely, and entity recognition accuracy imposes an upper bound on end-to-end relation classification. secondly, re presents a common way to frame the tasks in each of these datasets. ## datasets we consider three datasets across two different domains for this transfer: cooking and materials science procedures. our cooking datasets are the risec @xcite and english recipe flow graph (efgc) @xcite ) corpora, and we introduce a much wider domain gap with the materials science procedural text corpus (mscorpus) from @xcite . we do not standardize labels across datasets; we retain the original labels from each dataset, though we combine some relations in mscorpus to make it more comparable to the other datasets (see below for details). summary statistics for each dataset (including for the definition of "relation" described in section 3.1) are shown in the efgc dataset takes a more domain-specific approach, and defines a labeling schema specialized for cooking, including coreference relations segmented by whether the coreferent entities are tools, foods, or actions. many of the descriptors of actions that are given explicit labels in risec such as temporal relations and descriptions of manner, are collapsed into a single class in this dataset, with the authors choosing to focus on physical components, their amounts, and operational relationships. the mscorpus dataset splits its relations into three categories: relations between operations and entities, relations between entities, and one relation indicating the flow of operations. mscorpus defines a rich set of relations between entites, which is atypical for the other datasets. we thus combine some of these labels to bring mscorpus into alignment with the other annotation schemas. ## results and discussion we expect that more powerful linguistic representations than plain text will aid in few shot transfer between domains. in order for few shot transfer to be successful, the target data points used for transfer need to increase the relevant shared representation between the source and target datasets. because of this, we expect that any effect of representation on test set performance will depend upon how much shared representation there was between the two domains to begin with and how much the few added examples closes the gap. a more efficient representation may lose its advantage once there are enough target domain examples to obviate the need for efficiency. in this section, we aim to answer a number of questions. do linguistic representation aid in either indomain or cross-domain transfer? we present our in-domain results on the complete datasets in we do, however, see differences in performance between the baseline and graph-aware cases in the few-shot transfer setting. in figure 2 , we visualize the difference between the macro-averaged f1 performance in each of our graph-aware cases and the baseline against the few-shot setting. we see that while in the 1-shot case, our results are highly variable, the 5-, 10-, and 20-shot cases yield noticeable improvement, peaking in the 5-and 10-shot settings. in our best-performing results, we see a 6-point absolute gain in f1 score. we find that both dependency parse and amr representations show a statistically significant positive effect on performance. in particular, we test the significance of the effect with an anova model with multiple independent variables: namely, source and target dataset (efgc, risec, mscorpus), representation case (baseline, +dep, +amr), few-shot setting @xcite @xcite @xcite , and transfer setting (in-domain vs out-of-domain). the dependent variable is test set f1. the data table for the analysis includes 3 runs for each combination of variables each with a separate random seed. we train our models under a full-factorial experimental design, i.e. we ran trials for all combinations of variables. this design allows us to test the reliability of the effect of our variables under a variety of conditions while making the necessary statistical adjustments to avoid spurious significant effects that may occur when multiple statistical comparisons are made. we use this design rather than pairwise significance tests so that we can measure the effect of introducing linguistic formalisms as a whole, rather than arguing the statistical significance of individual, pairwise comparisons. we expect that the similarity between source and target datasets, the variation in the target dataset, and the few-shot setting could all either dampen or magnify any effect of representation on the performance. we therefore include pairwise interaction terms in the anova model for case by source dataset, case by target dataset, case by transfer setting, and case by few-shot setting. the examples added for the few shot setting in the transfer case are sampled from the training split of the target dataset. thus, while we expect for the crossdomain case the few shot setting has an effect, we do not expect an effect in the in-domain case, since the target domain examples added to the training data simply replicate examples that were already part of the dataset. to account for this, we include a final interaction term between few shot setting and transfer setting in the anova model. the anova model explains 98% of the variation in f1 scores. the results align well with our intuitions. first, as expected we find a significant effect of transfer setting such that in-domain performance on the entire dataset is better than transfer performance in a few-shot setting: f(1, 679) = 10356.25, @xmath0.0001. in these cases, the original dataset for in-domain training is between 5 and 15 times the size of the target training dataset. we also find a significant effect of the few-shot setting, such that larger numbers of target domain examples are associated with higher performance, f(5, 679) = 716.79, @xmath1.0001. a post-hoc student-t analysis reveals that all pairwise comparisons are significant. notably, there is a significant interaction between transfer setting and few shot setting: f(5, 679) = 733.83, @xmath2.0001, such that the effect of the fewshot setting is restricted to the transfer setting, as expected. our hypothesis is primarily related to the importance of the representation of the data for efficiently enabling transfer between domains. we find a significant effect of representation case: f(2, 679) = 5.26, @xmath3.01. a student-t post-hoc analysis reveals that both +dep and +amr cases are better than plain text, but there is no significant difference between the two. there is also a significant interaction between representation case and transfer setting: f(2, 679) = 8.19, @xmath4.0005. in particular, the effect of case is only significant in the transfer setting. there is also a significant interaction between few shot setting and case: f(2, 679) = 8.19, @xmath5.0005. a student-t post-hoc analysis reveals that the effect is only significant for the 5and 10-shot settings. thus, 1 target example is too small to yield a significant effect whereas 20 or more is too many such that the representational advantage disappears. we also find a significant interaction between representation case and target dataset, but not with source dataset: f(4, 679) = 2.61, @xmath6.05, such that the effect of representation is significant for risec and mscorpus but not for efgc. we present all of our few-shot results in how important is the choice of the source domain on the transfer performance? we see several interesting patterns in our 5-and 10-shot results when we take our few-shot models trained from scratch into account. we visualizes differences in performance between the from-scratch models and models trained with a different source domain in interestingly, an intuitive notion of "domain distance" fails to explain when transfer will be helpful. efgc and risec both come from the cooking domain, but though risec and mscorpus negatively influence each other in transfer, mscorpus and efgc in the baseline case have very little difference from the transfer case. transfer between the abstract categories of "cooking" dataset and "materials science" dataset is highly variable. notably, we observe that the benefits we derive from transfer seem asymmetrical: even datasets that transfer well in one direction might not in the other direction. we see markedly better results transferring from efgc to risec, for instance, than we see in the reverse direction, and we see a similar result (though less consistent) for transfer from efgc to mscorpus as compared to the reverse. what is the impact of linguistic structure on the performance of few-shot re in-domain? when factoring in the effect of our graph-aware models, we see that they help models generalize, both in the few-shot in-domain setting, as well as the transfer setting. where transfer itself causes the performance of the baseline model to degrade, however, we see that the addition of linguistic representations sometimes makes up for that gap almost entirely. in the case of the 10-shot mscorpus to risec transfer, we see that the baseline transfer model performs an average 2.7 points worse than the baseline from-scratch model (48.5 vs. 45.3), but that the dependency models perform very similarly (51.5 vs. 51.6). in cases where the transfer pairs are well-matched, however, we see that while the baseline results remain similar, the benefit that the models derive from the linguistic representations is much more pronounced in the transfer setting. in the 10-shot transfer in both directions between efgc and mscorpus, as well as the efcg to risec case, transfer models that incorporate dependencies and amrs overperform their in-domain counterparts by between 3 and 7 points. ## conclusion and future work we experiment with using linguistic formalisms as additional context for learning robust representations that facilitate few-shot transfer among domains for the task of relation extraction. our experiments show that the inclusion of linguistic formalisms significantly boosts models' ability to transfer to new datasets. they additionally show that that benefit is robust to whether transfer learning helps in the baseline case. this suggests that using linguistic formalisms as a scaffold for learning in data scarce, specialized domains could be a powerful technique. future work could focus on several directions. with regards to the use of semantic frameworks, more work is needed to understand how best to incorporate highly abstract formalisms such as amr. for example, how can we better use the node features in amr, rather than just the structure? these questions also apply to more abstract syntactic frameworks like constituency parsing. with regards to our transfer learning process, we aim to understand what features of a pair of datasets make them suited to transferring by studying a wider array of datasets in diverse domains, as well as to study the impact of domain adapting our syntactic and semantic parsers to our target domains.