Syntactic Multi-view Learning for Open Information Extraction (2022)

Abstract: Open Information Extraction (OpenIE) aims to extract relational tuples from open-domain sentences. Traditional rule-based or statistical models were developed based on syntactic structure of sentence, identified by syntactic parsers. However, previous neural OpenIE models under-explored the useful syntactic information. In this paper, we model both constituency and dependency trees into word-level graphs, and enable neural OpenIE to learn from the syntactic structures. To better fuse heterogeneous information from the two graphs, we adopt multi-view learning to capture multiple relationships from them. Finally, the finetuned constituency and dependency representations are aggregated with sentential semantic representations for tuple generation. Experiments show that both constituency and dependency information, and the multi-view learning are effective.

## introduction open information extraction (openie) aims to generate structured tuples from unstructured open-domain text @xcite . the extracted tuples are in the form of ⟨subject, relation, object⟩ for binary relation, and ⟨arg 0 , relation, arg 1 , . . . , arg n ⟩ for nary relation. it has been a critical nlp task as it is domain-independent and does not rely on predefined ontology schema. the structured relational tuples are beneficial to many downstream tasks such as question answering @xcite , knowledge base population @xcite and word embedding generation @xcite . in general, traditional openie systems are either statistical or rule-based. they extract relational tuples mainly based on certain sentence patterns heuristically defined on syntactic structures. the likes playing plush toys ; ; < mary's @xmath0: an example of constituency tree, dependency tree, and n-ary openie tuple to be extracted for sentence "mary's cat likes playing plush toys in the room." limitation of patterns causes traditional openie systems to be ineffective in handling complex sentences. recently, neural openie systems have been developed and showed promising results. neural openie systems no longer depend on pre-defined patterns. instead, they learn to extract relational tuples directly from unstructured text in an end-toend manner. however, utilizing syntactic information is under-explored among the neural openie systems, although syntactic information is widely explored in other information extraction tasks such as semantic role labeling (srl) @xcite and relation extraction (re) @xcite . as syntactic information was proved to be useful for the traditional openie systems, we argue that it is important for neural openie systems as well. figure 1a shows an n-ary tuple expected from an example sentence. figure 1b displays the sentence's constituency tree, where the phrases are labeled with constituency tags. the dependency tree (shown in figure 1c ) represents syntax through directed and typed edges between words, instead of between phrases. we observe that the boundary of the openie tuple (shown in figure 1a ) highly coincides with edges from constituency and dependency trees. as such, we believe both constituency and dependency trees provide useful and complementary syntactic information for openie systems. the next question is: how to effectively incorporate syntactic information to neural openie models? to fully explore the syntactic information from both parse trees, we convert them into 'nodesharing' graphs, i.e., constituency graph (denoted as const-graph) and dependency graph (denoted as dep-graph), respectively. a dependency relation specifies a relationship between two words, and we can represent words as nodes and the relationship as corresponding edge. the key challenge is to map constituency tree to a graph in which all of its nodes are words, not phrases. meanwhile, the graph needs to largely capture the constituency syntactic information. in this work, we present a novel method for the conversion of constituency trees. with the converted const-graph modelled at the word level, we can now easily integrate it with dep-graph. moreover, both graphs can be directly integrated with pre-trained language model (plm) which provides word-level representation. in order to leverage heterogeneous syntactic information from both const-graph and dep-graph, we propose a novel neural openie model: smile-oie (syntactic multi-view learning for open information extraction). it first encodes a sentence using bert @xcite , and subsequently uses two syntactic encoders, namely constencoder and dep-encoder. the model represents constituency and dependency relations of the sentence with the corresponding bert representations and applies two graph convolutional networks (gcn) (denoted as const-gcn and dep-gcn) to learn graph representations for const-graph and dep-graph separately. the representations from bert, const-gcn, and dep-gcn are aggregated and finally used for tuple generation. to better fuse the heterogeneous syntactic graph representations, smile-oie introduces a subtask, multi-view learning, to learn multiple types of relationships among const-graph and dep-graph. the multi-view learning loss is used to finetune the graph representations along with openie loss. in summary, our contributions are threefold: • we propose a novel strategy to map phraselevel relations in constituency tree into word-level relations, and to enhance each word's representation with constituency path information. • we propose smile-oie, the first neural ope-nie system that incorporates heterogeneous syntactic information through gcn encoders and multi-view learning. • our experimental results show that the proposed neural openie model achieves better performance than state-of-the-art methods. 2 related work syntax usage in openie. open information extraction (openie) was first proposed by yates et al. (2007), and textrunner is the first openie system that generates relational tuples in open domain. before deep learning era, many statistical and rule-based systems have been proposed, including reverb @xcite , r2a2 @xcite , ollie @xcite , clausie (corro and gemulla, 2013), stanford openie @xcite , ope-nie4 @xcite , nestie @xcite , and minie @xcite . most of these models extract relational tuples based on syntactic structures such as part-of-speech (pos) tags and dependency trees. in this sense, syntactic information has been essential to openie. recently, neural openie systems @xcite @xcite @xcite @xcite have been developed and showed promising results. neural ope-nie systems are able to extract relational tuples end-to-end based on the semantic encoding of input sentence. the analysis of syntactic structure of sentence, which was required by traditional models, seems no longer necessary. as a result, the usage of syntactic information is under-explored in neural openie models. nevertheless, there exist some neural openie systems that utilize some forms of syntactic information. for example, rn-noie @xcite projects pos tag of each word into pos embedding and concatenates it with word embedding as input to sentence encoder. senseoie @xcite further concatenates word embedding with dependency embedding. cigl-oie @xcite ) finds all head verbs in the sentence and pre-defines a few pos patterns to explicitly constrain the model training. mgd-gnn @xcite connects words, if they are in dependency relations, in an undirected graph and applies graph attention network (gat) to the graph @xcite as its graph encoder. although mgd-gnn uses some graphic information of dependency, it loses other information like the directness and types of dependency relations. in short, we observe that existing neural ope-nie systems fail to explore some syntactic features and that the integration of syntax is in a shallow manner. compared to the them, our smile-oie is able to leverage full features of heterogeneous syntactic information from both constituency and dependency trees. ## graph modelling in this section, we elaborate on our graph modelling strategy to convert constituency and dependency trees into graphs @xmath1, e), where u indicates the set of nodes and e the set of edges. they are g @xmath2, e con ) for const-graph, and g @xmath3, e dep ) for dep-graph. the two graphs' nodes correspond to the same set of input sentence's words. however, the labels of the nodes in the two graphs are different (constituency path for const-graph, and dependency relation type for dep-graph), preserving the syntactic information of constituency and dependency trees. also, their edge sets are different, where the edges represent word-to-word syntactic relations. ## smile-oie model the overall architecture of smile-oie is illustrated in figure 5 . smile-oie is based on bert encoder to get contextualized representations of an input sentence. the bert representations are then integrated with constituency and dependency information by const-gcn and dep-gcn, respectively. finally, smile-oie aggregates the bert, const-graph, and dep-graph representations in order to predict output tuples. beyond the openie tagging loss, smile-oie further performs multiview learning on the const-graph and dep-graph representations, and generates additional multi-view losses to enhance openie tagging accuracy. ## experiments we mainly conduct our experiments on lsoie (solawetz and larson, 2021), a large-scale openie data converted from qa-srl 2.0 in two domains, i.e., wikipedia and science. it is 20 times larger than the next largest human-annotated openie data, and thus is reliable for fair evaluation. foot_2 lsoie provides n-ary openie annotations and gold tuples are in the ⟨arg 0 , relation, arg 1 , . . . , arg n ⟩ format. the dataset has two subsets, and we use both, namely lsoie-wiki and lsoie-sci, for comprehensive evaluation. lsoie-wiki has 24,251 sentences and lsoie-sci has 47,919 sentences. carb @xcite dataset is the largest crowdsourced openie dataset. 7 however, carb only provides 1,282 annotated sentences, which are insufficient for training neural openie models. as a result, we use the carb dataset purely for testing. we follow @xcite to convert bootstrapped openie4 tuples as labels for distant supervised model training. carb provides binary openie annotations and gold tuples are in the form of ⟨subject, relation, object⟩. finally, we summarize the statistics of the training and testing datasets of lsoie-wiki, lsoie-sci, and carb in ## conclusion we design a novel strategy to map constituency tree into constituency graph only with word nodes, paving way for integrating constituency syntax with bert and dependency syntax. with the aid of const-gcn and dep-gcn, we propose a new ope-nie system smile-oie which combines heterogeneous syntactic information through multi-view learning. experiment results show that leveraging syntactic information can benefit openie task significantly, and multi-view learning can effectively guide the heterogeneous syntactic information fusion. in future work, we will explore other types of structured information to further improve openie.