Combinatory Grammar Tells Underlying Relevance among Entities (2022)

Abstract: Relation extraction (RE) is an important task in natural language processing which aims to annotate the relation between two given entities, which requires a deep understanding of the running text. To import model performance, existing approaches leverage syntactic information to facilitate the relation extraction process, where they mainly focus on dependencies among words while paying limited attention to other types of syntactic structure. Considering that combinatory categorial grammar (CCG) is a lexicalized grammatical formalism that carries the syntactic and semantic knowledge for text understanding, we propose an alternative solution for RE that takes advantage of CCG to detect the relation between entities. In doing so, we perform a multi-task learning process to learn from RE and auto-annotated CCG supertags, where an attention mechanism is performed over all input words to distinguish the important ones for RE with the attention weights guided by the supertag decoding process. We evaluate our model on two widely used English benchmark datasets (i.e., ACE2005EN and SemEval 2010 Task 8 datasets) for RE, where the effectiveness of our approach is demonstrated by the experimental results with our approach achieving state-of-the-art performance on both datasets.

## introduction given two entities in a sentence, relation extraction (re) extracts the relation between them and thus serves as an important task in natural language processing (nlp). recent neural approaches for re @xcite @xcite @xcite @xcite with powerful encoders (e.g., transformers) have shown outstanding performance on figure 1 : an example sentence with the ccg supertags of all words, where the supertag "(s\np/np)" of "produces" provides important cues to predict the relation between two given named entities "food factory" and "ice cream" (highlighted in red). benchmark datasets because the encoders are superior in capturing contextual information and thus obtain a deep understanding of the running text. to further improve model performance, extra knowledge resources, especially the syntactic information, have been widely used for re and demonstrated to be effective, because they provide structure information that is helpful for text understanding @xcite @xcite . specifically, existing approaches mainly focus on dependencies among words while paying limited attention to other types of syntactic structure, such as combinatory categorial grammar (ccg). as an important part in the a lexicalized grammatical formalism, the ccg supertags provide the lexical category of the associated words, which provides both syntactic and semantic knowledge for text understanding and thus is potentially beneficial for re. figure 1 shows a typical example. herein, the supertag of "produces" (which is "(s\np/np)") indicates the predicate requires to nominal arguments and the supertags of the two given entities (which are highlighted in red) suggests that they could serve as good candidates. therefore, the supertags suggest that "produces" contributes more to extracting the relation between the two entities and thus guide a model to make a correct prediction. in this paper, we propose to leverage ccg supertags to detect the relation between entities. in doing so, we use an existing ccg supertager to annotate the supertags of the input text and then run a multi-task learning process to learn from humanannotated re and auto-annotated supertags, where an attention mechanism is performed over all input words to distinguish the important ones for re with the attention weights guided by the supertag decoding process. therefore, our model is able to learn ccg information through supertag decoding rather than using the supertags as input features, which allows our approach to run efficiently in inference. experimental results on two english benchmark datasets for re, i.e., ace2005en and semeval 2010 task 8, demonstrate the effectiveness of our approach, where our approach outperforms strong baselines and achieve state-of-the-art performance on both datasets. ## preliminaries re is conventionally regarded as a text classification task with the given input sentence (which is denoted as @xmath0, • • • , x n ) and two entities (which are denoted as e 1 and e 2 ) in it, which can be formalized as then, we concatenate the entity representations @xmath1. ## the proposed approach to leverage the information carried by ccg supertags, one strightforward approach is to use an off-the-shelf ccg supertager to annotate the supertags of each input word and then use them as extra word-level features by concatenating them with the input words before sending them to the text encoder. however, such approach requires the ccg supertagging as a pre-processing step in inference, which is not efficient especially when the data to be processed is relative large. considering multi-task learning serves as an effective approach to learn from different tasks and it does not require the label from different tasks as extra input, we propose to learn the ccg information through a multi-task learning process and then use the ccg information to guide re though an attention mechanism over all input words. the overall architecture of our model is illustrated in figure 2 , where the backbone model for re following the standard process illustrated on the left and the ccg supertag decoding process as well as the attention mechanism illustrated on the top right. for ccg supertag decoding, we firstly take the hidden vector h i of the word x i obtained from the encoder and pass it through a mlp: then, for entity e j , we compute the attention weight p j,i assigned to the value v i through afterwards, we apply p j,i to the value vector v i and obtain the weighted sum vector a j via finally, we concatenate a j with the entity representation o j to obtain the enhanced entity representation h e @xmath2. once the enhanced representations of the two entities are computed, we concatenate them and feed the resulting vector to the softmax classifier, following the standard re decoding process. in training, the model is optimized on re and ccg supertagging, which allows our model to learn ccg information and use it to enhance the entity representation through the attention mechanism with the attention weights assigned to different input words guided by the learnt ccg information. ## conclusion in this paper, we propose a neural approach for improving re through a ccg guided attention mechanism, where our model learns the ccg information through a multi-task learning process to predict re and ccg supertags simultaneously and uses the learnt ccg information to compute the attention weights assigned to different words. in doing so, our approach is able to learn the ccg information through ccg supertag decoding rather than using it as additional input features, which allows our model to run efficiently in inference. experimental results on two english benchmark datasets (i.e., ace05 and semeval) for re demonstrate the effectiveness of our approach, where state-of-the-art performance is obtained on both datasets.