Syntactically Rich Discriminative Training: An Effective Method for Open Information Extraction (2022)

Abstract: Open information extraction (OIE) is the task of extracting facts "(Subject, Relation, Object)” from natural language text. We propose several new methods for training neural OIE models in this paper. First, we propose a novel method for computing syntactically rich text embeddings using the structure of dependency trees. Second, we propose a new discriminative training approach to OIE in which tokens in the generated fact are classified as “real” or “fake”, i.e., those tokens that are in both the generated and gold tuples, and those that are only in the generated tuple but not in the gold tuple. We also address the issue of repetitive tokens in generated facts and improve the models’ ability to generate implicit facts. Our approach reduces repetitive tokens by a factor of 23%. Finally, we present paraphrased versions of the CaRB, OIE2016, and LSOIE datasets, and show that the models’ performance substantially improves when trained on augmented datasets. Our best model beats the SOTA of IMoJIE on the recent CaRB dataset, with an improvement of 39.63% in F1 score.

## introduction oie @xcite is a branch of information extraction (ie) that focuses on extracting structured information @xcite from unstructured natural language text. this structured information is a set of tuples of the form "(subject, relation, object)", also called facts in oie. for instance, given the sentence "machine learning is a subfield of ai.", the tuple ⟨machine learning, is a subfield of, ai⟩ can be extracted, where the relation phrase "is a subfield of" indicates the semantic relationship between the subject "machine learning" and the object "ai". oie is useful in many downstream natural language processing (nlp) tasks like natural language understanding @xcite , multi-document question answering and summarization @xcite , and knowledge base construction from text @xcite . several neural oie methods in the literature approach oie as either a sequence labeling @xcite @xcite @xcite or a sequence generation problem @xcite @xcite . sequence labeling approaches label each token in the input text as either belonging to the subject, relation, or object, while sequence generation approaches generate facts one word at a time given the input text. for the task of oie, it is a common practice to make use of part-of-speech (pos) and dependency tags in addition to the actual input text as a way of incorporating syntactic information @xcite @xcite @xcite . in such work where these tags are used, the embeddings for the tags are just concatenated to the embeddings of the corresponding text tokens. this formulation does not fully use the rich syntactic information, especially the one that is expressed in the structure of dependency trees. dependency trees' head-dependent relations provide a good approximation of the semantic relationships between predicates and their arguments @xcite . as a result, they are directly applicable to a wide range of applications, including ie. in this work, we compute token representations based on the structure of dependency trees in order to benefit from their rich syntactic and semantic information. furthermore, sequence generation approaches are susceptible to generating facts that often express redundant information and are also prone to generating repetitive text in facts. @xcite and @xcite looked into the issue of generating the same facts more than one time, which is redundant. no work has been done to control the generation of repetitive text in facts. additionally, sequence generation approaches are capable of introducing words that are not in the input sentence into a generated fact. this capability enables such approaches to generate facts that are implicitly stated in the input sentence. however, the approaches employed by @xcite and @xcite are restrictive in how the models "pick" words when generating facts, which restrains the models' flexibility in generating implicit facts. in this paper, our approach uses a sequence generation approach to generate facts from natural language text one word at a time. unlike the previous approaches, we compute syntactically rich vector representations of input text tokens guided by the structure of dependency trees. for each input sentence, we construct a visibility matrix of its tokens based on the structure of its dependency tree by considering tokens that are directly related in the dependency tree as visible to each other. a graph neural network (gnn) @xcite encoder takes as input token embeddings and the corresponding visibility matrix to compute new token embeddings guided by the visibility matrix. furthermore, we also introduce a new method for training neural oie models. we add an extra module, the discriminator, that takes the generated tuple as input and classifies its tokens as either "real" or "fake", where "real" tokens are those that are in both the generated and gold tuples, while "fake" tokens are those that are only in the generated tuple but not in the gold tuple. to ensure that such a model does not generate repetitive text, we use a coverage vector to monitor the degree of coverage that words in the input text have received so far. when generating the next word at timestep t, the coverage vector at that instant is a sum of all attention distributions computed over the input text tokens from timesteps 0 to t-1. this coverage vector is used as part of the input when computing the next attention distribution. intuitively, this informs the current attention mechanism's decision of the previous decisions and makes it easier to avoid repeatedly attending to the same words in the input text, hence avoiding generating repetitive text in a fact. as a means to explicitly guide the model's choice of either generating a word from a vocabulary or to "pick" a word from the input text, we explicitly compute the probability of picking a word from a vocabulary or input text using the model's context vectors. lastly, we also investigate the effect of data-augmentation on the performance of our models. figure 1 illustrates our approach. our main contributions are briefly summarized as follows: ## task formulation in this work, we approach oie as a sequence generation task. as illustrated in fig. 1 , given an input text, we use a generator to generate a tuple one word at a time. then, we pass the generated tuple through the discriminator, which classifies tokens in the tuple as either "real" or "fake", where "real" tokens are those that are both in the generated and gold tuples, while "fake" tokens are in the generated tuple but not in the gold tuple. thus, the discriminator performs binary classification of the tokens in the generated tuple. we only consider binary extractions from sentences. given a sentence-tuple pair ⟨x, y ⟩, where @xmath0, . . . , w n ⟩ and @xmath1, . . . , y m ⟩ are sequences of tokens in the input sentence and expected tuple, respectively, we define the generator's conditional given a generated tuple, @xmath2, . . . , y m ⟩, and the input context vector, d * , the discriminator seeks to label each token in y as either "real" or "fake". the input context vector is weighted average of the vector representations of the input sequence. to that end, we define the discriminator's likelihood ## our approach we now define and describe the modules that we designed to solve oie as defined in section 2 and illustrated in fig. 1 . ## experiments and results in this section, we discuss the various experiments that we conducted. we present descriptions of the oie datasets that we used and the data preparation routines involved. we also discuss the evaluation framework that was used and finally present the results of each experiment in terms of the f 1 and area under the precision-recall curve (auc-pr) values. experiments, training, and evaluation. we set up different combinations of the neural network (nn) modules in blocks discussed in section 3, and each unique combination resulted in a different model. thus, we had 15 unique embedder-decoder combinations, and we present results for each combination. we run experiments for all 15 models ‡ in 6 experimental setups: (a) default setup with embedders and decoders only; (b) + discriminator, where we add a discriminator to the default setup; (c) + transformer encoder, where we add a transformer-based encoder to the default setup; (d) + gnn encoder, where we add a gnn-based encoder to the default setup; (e) + transformer encoder + discriminator, where we add both a transformer-based encoder and discriminator to the default setup; and (f) + gnn encoder + discriminator, where we add both a gnn-based encoder and discriminator to the default setup. for the models with bert and electra embedders, we used pre-trained § bert-base-uncased and electra-small-discriminator versions, respectively. due to resource constraints ¶ , the parameters of the pretrained models were frozen. all configurations for different module combinations are in appendix e. all models were trained by minimizing the loss defined in section 3.5. during training, we define a teacher-forcing ratio and randomly choose whether to use teacher forcing or not depending on the value of the randomly generated number compared to the teacher-forcing ratio. the confidence of the extracted fact was calculated by multiplying the probabilities of all tokens in the fact. the checkpoints that were saved after each epoch were then evaluated on the test partition of each dataset using the carb @xcite evaluation framework. the saved checkpoints did not include the discriminator, as the discriminator was only being used during training. datasets. our models were trained and evaluated on three benchmark oie datasets: oie2016 @xcite , carb @xcite , and lsoie @xcite . in appendix a , we provide detailed descriptions of the datasets and their statistics. ## results analysis and ablation studies in this section, we discuss the results presented in section 4. we also present a detailed analysis of the contributions of each experimental setup. performance across experimental setups. we considered 6 experimental setups, which we have defined in section 4. taking syntactic information into account when computing embeddings not only improves model performance but also improves extraction in some cases. for example, given a sentence "the twins who came are identical but are uniquely mannered.", the best model with the transformer encoder, which does not consider the topology of the dependency tree, yields ⟨who, are, uniquely mannered⟩, whereas the extraction with the gnn encoder, which does consider the topology of the dependency tree, yields ⟨the twins, are, uniquely mannered⟩. this example shows that syntax can be useful in oie. thus, we conclude that taking into account the dependency tree's structure when computing embeddings and the discriminative training approach are the main cause of the high boost in the performance of our models. this is consistently shown on all three benchmark datasets. ## related work in the recent past, neural oie models have been developed, and they tackle oie as either sequence labeling or sequence generation. the former @xcite @xcite et al., 2020) involves tagging each word in the input text with an appropriate tag that indicates whether the word belongs to either the subject, relation, or object. methods that approach oie as sequence generation generate facts one word at a time. these include copyattention @xcite , logician @xcite , and imojie @xcite . copyattention is an encoder-decoder model enhanced with copying and attention mechanisms. logician is another encoder-decoder model that uses coverage attention and gated-dependancy attention to extract facts from chinese text. imojie outputs a variable number of different facts per sentence. the next fact from a sentence is best determined in context of all other facts extracted from it so far. hence, imojie uses a decoding strategy that generates facts in a sequential fashion, one after another, each one being aware of all the ones generated prior to it. our work substantially differs from previous oie works. firstly, we are the first to compute embeddings of input sequences based on the structure of the corresponding dependency trees. this is done by first forming a visibility matrix based on which words are directly related in the dependency tree. then, a gnn encoder computes syntactically rich embeddings of the input sequences controlled by the formed visibility matrices. this formulation is different from previous approaches where the embeddings of pos and dependency tags are just concatenated to the embeddings of the according text tokens at embedding level. secondly, this is the first work that uses an additional module that classifies tokens in the generated tuple as either "real" or "fake" during training. thirdly, unlike previous neural approaches that tackle oie as sequence generation, we consider the entire generated sequence by timestep t, ⟨y 1 , . . . , y t-1 ⟩, to compute a decoder context vector, as discussed in eqs. 3-4 in section 3.3. additionally, @xcite and @xcite consider redundant information as the issue of generating the same facts multiple times, while we consider redundancy at token level in generated facts. furthermore, @xcite and @xcite are restrictive in how the models "pick" words when generating facts, which restrains the models' flexibility in generating facts. @xcite only limit the sample space only to the source sentence, and @xcite only consider the source sentence and keywords, while we consider the source sentence and the entire vocabulary. ## summary and outlook this work has shown that computing syntactically rich embeddings of text based on the structure of dependency trees significantly improves the performance of neural oie models and the quality of extractions. furthermore, the novel discriminative training method for oie models boosts models' performance. additionally, we show that generation probability and coverage mechanisms substantially improve the models' ability to introduce new words into generated facts and reduce repetitive tokens in generated facts. finally, we have presented paraphrased versions of oie2016, carb, and lsoie, and shown that the models' performance substantially improves when trained on datasets augmented by such data. our approach, syntactically rich discriminative training, beats the sota of imojie on the latest carb benchmark dataset: the model improves the f 1 score of imojie by 39.63%. this model also presents competitive results on other benchmark datasets: oie2016 and lsoie. in future work, we will build on this work and investigate further whether the proposed approaches could also be beneficial to sequence labelling approaches for oie. furthermore, models trained on original oie datasets perform poorly on paraphrased versions of the datasets. this is clear from the results shown in