Systematic Comparison of Neural Architectures and Training Approaches for Open Information Extraction (2020)

Abstract: The goal of open information extraction (OIE) is to extract facts from natural language text, and to represent them as structured triples of the form <subject,predicate, object>. For example, given the sentence “Beethoven composed the Ode to Joy.”, we are expected to extract the triple <Beethoven, composed, Ode to Joy>. In this work, we systematically compare different neural network architectures and training approaches, and improve the performance of the currently best models on the OIE16 benchmark (Stanovsky and Dagan, 2016) by 0.421 F1 score and 0.420 AUC-PR, respectively, in our experiments (i.e., by more than 200% in both cases). Furthermore, we show that appropriate problem and loss formulations often affect the performance more than the network architecture.

## introduction the field of information extraction (ie) focuses on the automatic acquisition of information from text in a structured format @xcite , and methods from this field are used to automatically collect desired information from large corpora of text. traditionally, ie methods were developed for specific domains with homogeneous corpora and fixed sets of predicates and entities @xcite . such methods are unable to generalize beyond their domains, as they are limited by their predefined collections of entities and predicates. the area of open information extraction (oie) was introduced as an alternative approach to ie @xcite , where predicates and entities are not restricted to a specific domain. more formally, the aim of an oie algorithm is to extract all facts entailed by the input text in the format of subject, predicate, object triples. alternative formulations allow for longer tuples, however, most of the work (including ours) focuses on binary predicates only. given a sentence »sam succeeded in convincing john.«, an oie system should extract a tuple: sam, succeeded in convincing, john . the relation phrase »succeeded in convincing« indicates the semantic relationship between »sam« and »john«. oie plays a key role in several downstream natural language processing (nlp) applications, such as knowledge base construction from text @xcite , question answering @xcite , information retrieval @xcite ), text comprehension, and natural language understanding @xcite . in this work, we focus on the extraction of binary relations, and introduce several novel neural approaches to oie. we construct our models from three blocks, namely, an embedding block, an encoding block, and a prediction block, and introduce several possible implementations of each of them. furthermore, we exhaustively test all possible combinations of their use, and show that output encoding and loss function strongly influence the results. to that end, we introduce and test three different training schemes, and investigate their influence on the performance of a model. all experiments have been conducted on the oie16 benchmark by @xcite . as part of the systematic architecture search, several existing neural architectures for oie @xcite @xcite have been tested and compared under equal conditions. the main contributions of this work are: ## related work textrunner @xcite stands out as the first oie system. after textrunner, a number of oie systems have been developed, such as clau-seie @xcite , props @xcite , and openie4 @xcite . these systems are mostly rule-based and use the language syntax to extract triples from sentences. for example, clauseie makes use of linguistic knowledge of the grammar of a sentence to detect clauses, and to identify the type of any such. ope-nie4 uses semantic role labeling to extract tuples from a sentence, and props relies on the proposition structure of syntax and dependency trees of the input sentence. however, rule-based systems suffer from severe error propagation when applied to examples outside of expected language patterns @xcite . recently, several approaches based on neural network (nn) architectures have been introduced @xcite @xcite . as one of the first neural methods, @xcite treat oie as a sequence-tagging task, and use a bidirectional long short-term memory (bilstm) architecture with a softmax layer to predict a bio tag @xcite , as defined below, for each word in the input sentence. in contrast to this, @xcite consider oie as a problem of neural machine-translation (nmt) from english into a triple, and approach it with an attention-based sequence-to-sequence model @xcite . jia and xiang (2019) use the same sequence-tagging problem formulation as @xcite and extend their work with systematic tests of various nn architectures, such as (bi)lstms, convolutional neural networks (cnns), and their combinations with conditional random fields (crfs). finally, they develop a hybrid model that combines bilstms, cnns, and crfs to achieve a maximal performance. zhan and zhao (2019) introduce a span model for n-ary oie to replace the previously adopted sequence-tagging formulation. their bilstmbased model finds predicate spans separately and uses a separate bilstm module to find arguments (entities), given a predicate as an input. ## task formulation as defined earlier, we aim to extract binary relations from single sentences. each fact is represented as a triple subject, predicate, object and elements have to be non-overlapping contiguous phrases in the sentence. following @xcite , we treat oie as a sequence-tagging task, where each word is labelled with a bio tag. to that end, each element of a triple is implicitly extracted by labeling the according sequence of words with a beginning-tag followed by an arbitrary number of inside-tags, and words that are not part of any extracted phrase are marked with the outsidetag. tags are prefixed with a0 if they refer to a triple's subject, p for the predicate, and a1 to refer to the object of the extracted triple. we use a total of seven different labels, denoted as @xmath0, a0-i, p-b, p-i, a1-b, a1-i, o}, and figure 1 illustrates how they are used to extract triples from text. more recently, zhan and zhao ( 2019 ) introduced an alternative formulation that is based on spans rather than sequence tags. a span is a subsequence of the sentence and an extracted triple is a set of three disjunct spans, corresponding to subject, predicate, and object. they generate all possible candidate triples and use a model to score them. to restrict the exponentially growing number of possible triples, additional restrictions are put into place, such as maximum length or syntactic requirements (cf. @xcite . unlike its formal counterpart, semantic parsing, the task of oie is defined more vaguely, and hence leaves some space for interpretation. for instance, one could generally consider both ludwig van beethoven, was, a world-famous composer and ludwig van beethoven, was, a world-famous composer of classical music valid extractions for the first example provided in figure 1 , but ultimately, this is a design choice that depends on the concrete application. also, there is no fixed schema of relations to be extracted. this introduces another level of complexity, as memorization of encountered patterns for a given set of relations is insufficient for a good performance and generalization. we highlight that single sentences frequently induce more than one triple. as an example of this, consider the lower part of figure 1 , which illustrates a sentence that consolidates two different pieces of information. ludwig van beethoven was a world -famous composer of classical music . a0-b a0-i a0-i p-b a1-b a1-i a1-i a1-i a1-i o o o o ⇒ ludwig van beethoven, was, a world-famous composer beethoven , who died in 1827 , composed the ode to joy . a0-b o o p-b p-i a1-b o o o o o o a0-b o o o o o o p-b o a1-b a1-i a1-i o ⇒ beethoven, died in, 1827 beethoven, composed, ode to joy any problem instance as a tuple x, y , where @xmath1, w 2 , . . . , w m describes a sentence as a sequence of words, and @xmath2, y 2 , . . . , y n } defines a set of valid extractions. depending on the problem definition being used, the elements in y are either sequences of bio tags, i.e., y @xmath3, or triples specified in terms of spans. independently of the employed input and output formulation, our aim in this work is to model the conditional distribution p {y | x}. ## methodology in this work, we consider several variations and extensions of existing architectures for oie viewed both as sequence-tagging and as span-prediction task. to that end, we subdivide models conceptually into three blocks, which we call embedding, encoding, and prediction (listed from the bottom to the top), and investigate the impact of different nn modules for each of them below. the blocks serve the obvious purposes indicated by the according designations, and are described in detail in the rest of this section. embedding. the embedding block represents the bottom part of our models, and serves the purpose of mapping text, given as sequence of tokens, to a sequence of embedding vectors. traditionally, embedding vectors were (pre-)trained for a fixed vocabulary of tokens, and used directly in place of any actual tokens in a processed input sequence @xcite . more recently, however, different models for computing contextualized vector representations of tokens in an input sequence have been used to represent text input @xcite @xcite , which induced notable progress on a multitude of nlp tasks. we consider both approaches, and use either just word-piece embeddings @xcite , which we refer to as simple embedding block, or al-bert @xcite for computing contextual-ized representations. for the task of oie, it is a common practice to make use of part-of-speech (pos) tags in addition to the actual input text @xcite @xcite . we follow this, and append an embedding representing the according pos tag to every vector produced by the used embedding block. in the case of word pieces, each sub-word token was attributed the same pos tag as the full word it belongs to. encoding. the encoding block constitutes the middle part of the considered models, which processes an embedded input sequence, as provided by the module used in the embedding block, and outputs an encoded sequence of equal length. in this work, we make use of three different nn modules for encoding embedded sequences: a bilstm, the encoder part of a transformer, and a bilstm combined with a cnn, as introduced by @xcite . for the simple bilstm encoder, we concatenate the top-layer hidden states of both directions, and use these as encodings of the respective tokens in the input sequence. the lstm-cnn encoder processes a provided sequence in parallel with a bilstm and a cnn that are independent of each other. the used cnn consists of just one convolutional layer followed by max-pooling, and maps the entire embedded input sequence, viewed as matrix, to a single output vector. this vector is then concatenated with every step in the encoded sequence that is provided by the bilstm to yield the final output. again, the encoding provided by the bilstm consists of the concatenated top-layer hidden states. prediction. the top block of a model takes an encoded sequence as input, and computes a probability distribution over all possible triples to extract. we consider four different model architectures as prediction blocks, three for the sequence-tagging setting, based on lstms, crfs, and multi-layer perceptrons (mlps), and the recently introduced crfs are commonly used in combination with sequential models, such as recurrent neural networks (rnns), since they provide a convenient way of modeling the joint distribution of entire label sequences rather than just step-wise conditional distributions. furthermore, using crfs on top of a recurrent model frequently results in an increased prediction accuracy (e.g., @xcite . for our purposes, we employ a standard linear-chain crf as prediction block. mlps are another family of nn modules that is frequently used for predicting labels in the context of oie. in contrast to the previously mentioned predictors, however, they compute labels independently for each step of an input sequence, disregarding those computed for any surrounding positions. hence, employing mlps is based on the simplifying assumption that finally, we use the spanoie model as the only prediction block that is based on the span-formulation of oie. for an encoded input sequence, this model computes scores independently for each triple in a set of candidates to extract, which are then normalized to yield a probability distribution. we defer the interested reader to @xcite for any details. training loss. since prediction blocks aim to model the probability distribution over all candidate triples to extract, the negative log-likelihood (nll) lends itself as the natural loss function to use. if we consider oie as a sequence-tagging task, however, then optimizing the nll requires us to deal with one problematic aspect. among all bio tags, the o-tag, indicating tokens that are not part of subject, predicate, or object, tends to appear at a much higher frequency than other tags. this, in turn, means that the ability to correctly predict otags has the strongest direct influence on the nll, and encourages models to focus too much on this label. to account for this issue, we explore three novel training schemes, which disregard the probabilities of certain positions in a sequence for computing the nll, as illustrated in figure 2 . a straightforward attempt to decrease the influence of o-tags on the loss term is to disregard them entirely (cf. figure 2a ). since we model probability distributions over tag sequences as opposed to computing unnormalized scores, the law of total probability allows for a network to still learn when to predict o-tags, even though it is not explicitly trained to do so. informally, this means that p {o} = 1 -p {b} -p {i}, which is why the trained network learns to predict o-tags when the probabilities of all remaining tags are small. the second training scheme is based on the observation that the critical aspect of predicting a sequence of tags is identifying transitions between a block of o-tags and an element of the triple, i.e., the subject, the predicate, and the object, or directly between two of the latter. hence, this training scheme disregards all probabilities except for those of positions that appear right before or right after a transition (cf. figure 2b ). intuitively, this makes sense, as all the disregarded tags can be determined immediately, once we know the boundaries of the subject, the predicate, and the object. combining the two previously outlined ideas results in our third trained scheme, which optimizes only the first and the last position of every element of the extracted triple (cf. figure 2c ). again, remaining tags can be determined from the knowledge of these tags only. finally, notice that the presented schemes are used during the training of a model only. for inference and sampling, we make use of all probabilities, including those of o-tags. ## experimental evaluation dataset. for our experiments, we made use of the oie16 benchmark dataset @xcite . with a total of 5,078 training samples, the dataset is rather small, though, making it hard to train models that generalize to unseen problem instances. for this work, we thus augmented the oie16 training data with samples from another dataset created by @xcite . the latter is a huge dataset, consisting of more than 36m training samples that have been generated using openie4 @xcite , and contains examples with lower quality than the oie16 data. furthermore, there is usually just one target triple to extract per sentence in this dataset, while oie models are generally expected to find all of them. to make effective use of the low-quality dataset, we had to perform a number of preprocessing steps (described in detail in appendix a, which left us with a total of 1.7m training samples that were combined with the training partition of the oie16 benchmark dataset. ## conclusion in this work, we systematically compared a range of different nn architectures for oie as well as different schemes for training them. in doing so, we improved the state of the art on the oie16 benchmark by 0.421 f 1 and 0.420 auc-pr, respectively, (i.e., by more than 200% in both cases) with a novel model, consisting of an albert embedding block, a transformer encoding block, and an lstm prediction block, which was trained by means of a training scheme using a newly introduced loss formulation. subsequent analysis revealed that choos-ing the right training scheme is as important as selecting the neural model architecture, as the standard nll loss attributes too much importance to non-essential aspects of the data, and consistently leads to inferior results.