Bootstrapping Relation Extractors using Syntactic Search by Examples (2021)

Abstract: The advent of neural-networks in NLP brought with it substantial improvements in supervised relation extraction. However, obtaining a sufficient quantity of training data remains a key challenge. In this work we propose a process for bootstrapping training datasets which can be performed quickly by non-NLP-experts. We take advantage of search engines over syntactic-graphs (Such as Shlain et al. (2020)) which expose a friendly by-example syntax. We use these to obtain positive examples by searching for sentences that are syntactically similar to user input examples. We apply this technique to relations from TACRED and DocRED and show that the resulting models are competitive with models trained on manually annotated data and on data obtained from distant supervision. The models also outperform models trained using NLG data augmentation techniques. Extending the search-based approach with the NLG method further improves the results.

## introduction the goal of relation extraction (re) is to find and classify instances of certain relations in raw text. we denote a binary relation instance, i.e. a relation instance with two arguments, with a tuple @xmath0, e 1 , e 2 , r), where @xmath1, e 1 , e 2 are entity mentions within s corresponding to the first and second relation argument, respectively, and @xmath2, or an indication of 'no-relation'. in binary classification our goal is to classify whether, according to s, the entity mentions, e 1 and e 2 , satisfy r, the relation label. for such classification we require a training dataset x, comprised of x p , a set of positive examples, representing the relation of interest, and x n , a set of negatives examples. the success of recent papers @xcite in supervised re is fueled by advances in deep learning, but also, crucially, by the availability of a large training set such as tacred @xcite , containing tens of thousands of training examples. for most relations of interest, such training data is not available. in this work we examine methods to inexpensively construct x p and x n , in cases where a training set is not available. we are especially interested in constructing the positive set, x p . in contrast to common nlp tasks like pos tagging, entity extraction and dependency parsing, the task of relation extraction exhibits a much larger degree of label sparsity. for some relations, even when considering only sentences with entities of the relevant types, the ratio between positive and negative examples is highly skewed toward the latter and obtaining a modest amount of positive examples will require a laborious annotation effort (see §3). while manual annotation of large datasets is a viable approach, it typically requires contracting a team of professional annotators @xcite or crowd workers @xcite and is not well suited for smaller projects or for ad-hoc extraction tasks. our main contribution in this paper is a new methodology built on top of @xcite for cheaply obtaining large datasets ( §6). @xcite proposed a syntactic search engine that given a lightly annotated example sentence, retrieves new sentences with a similar syntactic structure from a pre-annotated dataset. our syntactic search bootstrapping method requires a small number of manually curated positive example sentences. then the search engine matches are used as training data for ml models. we evaluate this approach comparing to human annotated data of varying sizes. while this method shows promising results with very few user input examples, we also test the impact on performance when more examples are used. one technique for obtaining an abundance of examples uses recent natural language generation (nlg) models ( §7.1). it has been shown in recent papers @xcite @xcite @xcite that generating abundance of training examples can improve classifier performance. we aim to check whether this can improve our syntactic search method as well. we evaluate the proposed methodologies by training dl classifiers on the obtained data. we show that: (1) syntactic patterns are competitive at bootstrapping training data for ml, even with as little as 3 patterns; the code for all our experiments alongside the generation outputs is publicly available 1 . another system which can optionally utilize rules instead of external kbs is snorkel @xcite . snorkel is implementing the dataprogramming paradigm @xcite where ml models are trained in three stages: (i) users write labeling functions that weakly label data points using arbitrary heuristics (e.g. extraction rules); (ii) the system learns a re-weighted combination of the labeling functions by explicitly modeling the actual distribution of each class. the results are often precise but low-recall; and (iii) the system uses discriminative models to increase recall while preserving precision. the techniques used by @xcite and snorkel can be effective in increasing the accuracy of the initial labeling rules, but coming up with "good enough" initial rules remains a major challenge. in this sense, the search-based methods suggested in this work for bootstrapping re datasets are complimentary and can be plugged in as a first step in these multi-step solutions. only few papers can be directly compared to our paper and use matches as training-data for ml classifiers. one paper similar in that sense is @xcite which claims that training a classifier using search-based examples works better than traditional bootstrapping methods. see §6.2 for further compression with @xcite . augmentation through generation. similarly to our example generation approach, recent papers @xcite suggest using pre-trained language models for data augmentation. in both these papers, the authors suggest prepending class labels to generative models in order to augment the number of instances for classes with a small number of examples. in contrast to these papers we use language models in a zero-shot context, and rather than requiring existing labeled examples of the relevant relation, we propose to manually label the generated samples. ## the data annotation challenge in contrast to linguistic annotation tasks such as parts-of-speech, syntactic-trees or semantic roles, annotating data for relation-extraction does not require special expertise. annotation can be easily performed by a motivated native speaker of the language (in case of "every-day" relations such as those available in tacred and docred) or by a domain expert (in case of "specialized" relations such as in biomedicine or law). annotating a given sentence for a given relation takes roughly the time it takes to read and understand the sentence. so what stops us from obtaining large amounts of annotated data for ml? the annotation challenge lies in relation sparsity in the wild. in an attempt to get a perspective on this issue, let's consider the founded-by relation between a person and an org, as attested in the tacred corpus. assuming we consider only sentences that contain both a person mention and an organization mention, how many sentences do we have to annotate before we reach, for example, 10 positive examples? the tacred training set has 124 founded-by instances, as well as 6947 "negative" instances with matching entity types ("negative" examples are either other relations, or no-relation). this 1-out-of-57 ratio indicates that we will likely sample 56 "negative" sentences before hitting a positive instance. foot_0 this ratio is overly optimistic, as the annotations in the tacred corpus are already very skewed in favor of positive examples. even under this very optimistic scenario, we will need to annotate 570 sentences to recover 10 positive examples. the cost of annotation, then, is not in annotating each individual positive sentence, but in finding the sentences to annotate in the first place. therefore, we should seek for methods that point towards probable positive instances. in this paper, we present two methods, the first returns close to 1-out-of-1 positive ratio, although with low syntactic diversity, and a second method with roughly 1-out-of-3 positive ratio. ## problem statement and setup we are interested in the problem of obtaining a relation classifier for a binary relation, when no a-priori annotated training data for this relation is available. we seek a methodology that will allow to create an effective extractor, using a minimal amount of data annotation effort. we compare four approaches -manual annotation, syntactic-search, manual annotation over generated examples, and a combination of the last two -to be described in later sections. here, we discuss setup which is shared to all experiments. in order to evaluate the methodology on multiple datasets with similar relations, we chose a set of relations that appear in both the tacred @xcite and docred @xcite datasets with at least 50 development examples foot_1 . to quantify the performance of our methodology we assess it comparing to varying amounts of manually annotated data. in our settings, large amounts of supervised examples represent upper bound for our bootstrapping methods and are not expected. while relation extraction is often considered as a multi-class classification problem ("find the occurrences of any of these possible relations"), we instead treat the relations separately, training a binary classifier for each one. we believe this is more representative of a user who wishes to target a low number of relations, who is likely to conduct data collection and evaluation for one relation at a time. obtaining negative examples when training a binary classifier, it is required to include a set of negative examples alongside the list of positive examples. in all our experiments we obtain negative examples by looking for sentences that contain entity types that are compatible with the relation (i.e, for the founded-by relation we sample sentences that include both a person and an org). in our syntactic based methods we sample from the same domain as our positive examples (wikipedia) and then filter this list by removing sentences in which the entities are connected by a syntactic pattern which is attested by the positive examples. for the supervised baselines of various sizes, we obtain negative examples by sampling them from the annotated training set, without replacement. 4 datasets we used two datasets to explore our different methods. tacred @xcite , a large-scale multi-class relation extraction dataset built over newswire and web text. and do-cred @xcite , a dataset for document level re, and similarly designed for multi-class prediction. per our setup above, we changed the setting of both datasets to per relation binary classi-fication. as our main goal in this paper is to evaluate different bootstrapping methods, and not novel methods for document-level relation extraction, we chose to include only instances with single supporting sentence in docred (i.e. sentence level relations). as docred's labelled test set is not publicly available, we used the development set as our test set and used 20% of the train set as development set. ## manual annotation baseline setup. our comparison point throughout the paper is a model trained on traditionally-collected annotated data. we sample increasing-sized annotated sets from tacred and docred, containing 55, 110, 220, 550, and 1100 examples. these correspond to 5, 10, 20, 50, 100 positive examples with 50, 100, 200, 500, 1,000 negatives examples. additionally, we measure the performance on these datasets when using all available positive examples for each relation. results listed in the top rows of importantly, in all these experiments, the number of annotated examples used is significantly higher than the number used in our syntactic search experiments (3 examples in total). ## syntactic search by example we consider this section to be the main contribution of the work. we show that: (i) with modern dl modeling, effective relation extractors can be trained using sentences derived from less than a handful of syntactic patterns; and (ii) through the use of by-example syntactic search engines, one can construct these patterns very quickly, without needing to understand syntax. to explain the suggested workflow, let's consider a user who wants to train a relation extraction binary classifier for the founded by relation, and has a single example sentence, "paul founded microsoft in april 1975". patterns over syntactic structures and entity types are very effective for deriving high-precision extraction templates. for example, searching for sentences containing the word "founded" with an nsubj dependency of type person and dobj dependency of type org, will return many matches for the founded-by relations. there are two issues with this approach (1) while high-precision, the recall of the patterns is low; and the premise of this paper is that the low recall can be offset by machine learning. the sentences resulting from syntactic search over a few patterns are diverse enough that an ml model trained over them manages to generalize from the specific syntactic pattern and identify a broader range of cases, increasing recall substantially. we show this is indeed the case. to overcome the need for linguistic expertise we propose using a by-example syntactic search engine @xcite foot_4 which allows users to execute syntactic queries based on example sentences: the user enters a sentence satisfying the relation of interest and annotates it with light markup indicating the arguments and the trigger words. the system then automatically translates the markup into a syntactic pattern, matches it against a large pre-annotated corpus (e.g. all wikipedia sentences), and returns results. the user does not need to be familiar with syntactic formalisms or with advanced nlp. ## augmenting syntactic patterns with natural language generation we showed how the syntactic search by example method works with only a few human annotated examples. in this section we would like to pursue nlg based methods to expand the number of exemplary patterns. generative language models, compared to other methods for data augmentation (e.g. iterative bootstrapping and distant supervision) are highly accessible and require low technical expertise (sometimes passing a prompt is enough). moreover, recent papers @xcite @xcite @xcite ) report high impact of such models for the closely related data augmentation task. we therefore present numerous methods that take advantage of such models for re bootstrapping. first we show how a user can produce a high number of generated sentences using gpt2 @xcite . then we demonstrate how the generated sentences can be integrated in the syntactic search by example method ( §7.1). finally, in order to validate the necessity of the syntactic search in this flow we compare it to feeding the raw generations as inputs to a classifier ( §7.2). ## applicability to other languages we explored only english in this work. however, we argue that our main method -example-based syntactic search followed by dl-training -is not strongly tied to english, and we encourage other researchers to experiment with it in their languages of interest. we provide details of what is needed to adapt the system to a different language. the syntactic search by example method requires (1) an automatically dependency-parsed corpora in the language. these can be readily produced by the many syntactic parsers that are available for many languages @xcite @xcite . (2) an indexing engine that supports efficient queries over parse trees. @xcite uses the open-source odinson engine @xcite for this purpose. (3) a component that translates a query in spike's "by example" syntax to the indexing engine's query syntax. this requires finding the minimal (in terms of number of nodes) sub-graph that connects all relation arguments (and predicates if available), then search for sentences with similar sub-graphs in the index. with these three components, a syntacticsearch system can be readily implemented. the rest of the components are straightforward application of dl methods. indeed, we suspect the major obstacle in application to a new language will be the availability of evaluation data. ## conclusion we show that with modern dl classifiers and a dataset bootstrapped using syntactic search with as few as 3 seed patterns can be as effective as a dataset with hundreds of manually annotated samples. using lms help to further diversify the dataset and improve results. overall, our results are positively optimistic for bootstrapping methods. however, this work is only an initial step in exploring methods for bootstrapping relation extractors using minimal user effort, supported by strong pretrained neural lms. we hope to encourage further work in this direction. yuhao zhang, victor zhong, danqi chen, gabor angeli, and christopher d. manning. 2017. positionaware attention and supervised data improve slot filling. in proceedings of the 2017 conference on empirical methods in natural language processing (emnlp 2017), pages 35-45.