Supertagging C ombinatory C ategorial G rammar with Attentive Graph Convolutional Networks (2020)

Abstract: Supertagging is conventionally regarded as an important task for combinatory categorial grammar (CCG) parsing, where effective modeling of contextual information is highly important to this task. However, existing studies have made limited efforts to leverage contextual features except for applying powerful encoders (e.g., bi-LSTM). In this paper, we propose attentive graph convolutional networks to enhance neural CCG supertagging through a novel solution of leveraging contextual information. Specifically, we build the graph from chunks (n-grams) extracted from a lexicon and apply attention over the graph, so that different word pairs from the contexts within and across chunks are weighted in the model and facilitate the supertagging accordingly. The experiments performed on the CCGbank demonstrate that our approach outperforms all previous studies in terms of both supertagging and parsing. Further analyses illustrate the effectiveness of each component in our approach to discriminatively learn from word pairs to enhance CCG supertagging.

## introduction combinatory categorial grammar (ccg) is a lexicalized grammatical formalism, where the lexical categories (also known as supertags) of the words in a sentence provide informative syntactic and semantic knowledge for text understanding. therefore, ccg parse often provides useful information for many downstream natural language processing (nlp) tasks such as logical reasoning @xcite and semantic parsing @xcite . to perform ccg parsing in different languages, most studies conducted a supertagging-parsing pipline (clark and curran, 2007; kummerfeld et al., † corresponding author. 1 our code and models for ccg supertagging are released at https://github.com/cuhksz-nlp/nest-ccg . building an accurate supertagger in a sequence labeling process requires a good modeling of contextual information. recent neural approaches to supertagging mainly focused on leveraging powerful encoders with recurrent models @xcite @xcite , with limited attention paid to modeling extra contextual features such as word pairs with strong relations. graph convolutional networks (gcn) is demonstrated to be an effective approach to model such contextual information between words in many nlp tasks @xcite @xcite ; thus we want to determine whether this approach can also help ccg supertagging. however, we cannot directly apply conventional gcn models to ccg supertagging because in most of the previous studies the gcn models are built over the edges in the dependency tree of an input sentence. as high-quality dependency parsers are not always available, we do not want our ccg supertaggers to rely on the existence of dependency parsers. thus, we need another way to extract useful word pairs to build gcn models. for that, we propose to obtain word pairs from frequent chunks (n-grams) in the corpus, because those chunks are easy to identify with co-occurrence counts. to appropriately learn from n-grams, one requires the gcn to be able to distinguish different word pairs because such information in n-grams are not explicitly structured as that in dependency parses. because existing gcn models are limited in treating all word pairs equally, we propose an adaptation of conventional gcn for ccg supertagging. figure 1 : the architecture of our ccg supertagger with a-gcn and an example input sentence with its supertagging and parsing output. the supertagging process for "buy" is highlighted in green. the adjacency matrix illustrates the edges of the graph that is built upon the chunks (n-grams) extracted from the lexicon n , with the chunks illustrated in the red boxes. in this paper, we propose attentive gcn (a-gcn) for ccg supertagging, where its input graph is built based on chunks (n-grams) extracted with unsupervised methods. in detail, two types of edges in the graph are introduced to model word relations within and across chunks and an attention mechanism is applied to gcn to weight those edges. in doing so, different contextual information are discriminatively learned to facilitate ccg supertagging without requiring any external resources. the validity of our approach is demonstrated by experimental results on the ccgbank @xcite , where state-of-the-art performance is obtained for both tagging and parsing. ## the approach we treat ccg supertagging as a sequence labeling task, where the input is a sentence with n words the graph is built based on n-grams in the input sentence that appear in a lexicon n . this lexicon consists of n-grams automatically extracted from raw corpora by unsupervised methods. the overall architecture of our tagger is illustrated in figure 1 , with an input sentence and corresponding supertagging and parsing output. the details of the main components in the architecture are provided below. ## conclusion in this paper, we propose a-gcn for ccg supertagging, with its graph built from chunks extracted from a lexicon. we use two types of edges for the graph, namely, in-chunk and cross-chunk edges for word pairs within and across chunks, respectively, and propose an attention mechanism to distinguish the important word pairs according to their contribution to ccg supertagging. experimental results and the ablation study on the english ccgbank demonstrate the effectiveness of our approach to ccg supertagging, where state-of-the-art performance is obtained on both ccg supertagging and parsing. further analysis is performed to investigate using different types of edges, which reveals their quality and confirms the necessity of introducing attention to gcn for ccg supertagging. easyccg the dow jones industrials closed at 2569.26 np[nb]/n (n/n)/(n/n) n/n n s[dcl]\np ((s\np)\(s\np))/np @xmath0.26 np[nb]/n n/n n/n n (s[dcl]\np)/pp pp/np @xmath1.6 % on black monday np[nb]/n n ((s[dcl]\np)/pp)/np n/n n pp/np n/n @xmath2.6 % on black monday np[nb]/n n s[dcl]\np ((s\np)\(s\np))/((s\np)\(s\np)) (s\np)\(s\np) (s\np)\(s\np)/np n/n @xmath3