Graph-based Syntactic Word Embeddings (2020)

Abstract: We propose a simple and efficient framework to learn syntactic embeddings based on information derived from constituency parse trees. Using biased random walk methods, our embeddings not only encode syntactic information about words, but they also capture contextual information. We also propose a method to train the embeddings on multiple constituency parse trees to ensure the encoding of global syntactic representation. Quantitative evaluation of the embeddings show a competitive performance on POS tagging task when compared to other types of embeddings, and qualitative evaluation reveals interesting facts about the syntactic typology learned by these embeddings.

## introduction distributional similarity methods have been the standard learning representation in nlp. word representations methods such as word2vec, glove, and fasttext @xcite @xcite aim to create vector representation to words from other words or characters that mutually appear in the same context. the underlying premise is that "a word can be defined by its company" @xcite . for example, in the sentences, "i eat an apple every day" and "i eat an orange every day", the words 'orange' and 'apple' are similar as they share similar contexts. recent approaches have proposed a syntax-based extension to distributional word embeddings to include functional similarity in the word vectors by leveraging the power of dependency parsing @xcite . syntactic word embeddings have been shown to be advantageous in specific nlp tasks such as question type classification @xcite , semantic role labeling @xcite , part-of-speech tagging @xcite , biomedical event trigger identification @xcite , and predicting brain activation patterns @xcite . one limitation of these methods is that they do not encode the hierarchical syntactic structure of which a word is a part due to its reliance on nonconstituency parsing such as dependency parsing. while the latter analyzes the grammatical structure of a sentence by establishing a directed binary head-dependent relation among its words, constituency parsing analyzes the syntactic structure of a sentence according to a phrase structure grammar. syntactic hierarchy has advantages in tasks such as grammar checking, question answering, and information extraction @xcite . it has also been encoded in neural models such as recursive neural tensor network and has proved it can predict the compositional semantic effects of sentiment in language @xcite . moreover, it can uniquely disambiguate the functional role of some words and therefore the overall semantic meaning. figure 1 shows the modal verb should in the following sentences: (1) let me know should you have any question. and (2) i should study harder for the next exam. even though the word should is a modal verb (md) in both sentences, it exhibits two different grammatical functions: conditionality and necessity respectively. similarly, the word is in (3) the king is at home. and (4) is the king at home? has a similar semantic meaning in both sentences, yet it exhibits two different syntactic roles (statement-forming and question-forming). traditional word embeddings methods give a contextual, semantic representation to words like is and should, but they make no distinction of their grammatical function due to the absence of information on syntactic hierarchy. on the other hand, constituency parse trees provide a syntactic representation that can easily capture such distinction. figures (a) and (b) show constituency parse tree of sentences (1) and ( 2 ) respectively. the difference in the position of the modal verb should in both sentences indicates a difference in the grammatical function, especially if it is compared to words with similar grammatical function in other sentences such as the one in (figure (c )). comparing figures (a) and (c), we can note that should hold the same sense of conditionality the word if has. to this end, we propose a simple, graph-based framework to build syntactic word embeddings that can be flexibly customized to capture syntactic as well as contextual information by leveraging information derived from either manually or automatically constituency-parsed trees. while recent transformer-based models such as bert @xcite have proved to be more sophisticated than word embeddings, the latter remains a popular choice due to its simplicity and efficiency. thus, the contribution of this work is two-fold: (1) bridge the research gap in the literature of word embedding by introducing hierarchical syntactic embeddings based on constituency parsing (2) propose a graphtheoretic training method that cluster words according to their syntactic and constituent role without sacrificing the original context in which a word appears. ## related work the nlp literature is rich with studies suggesting an improvement to original word embeddings models by incorporating external semantic resources like lexicons and ontologies @xcite @xcite @xcite . however, very few studies were dedicated to syntactic embeddings. one of the earliest methods was dependency-based word embeddings @xcite , which generalizes the skip-gram algorithm to include arbitrary word context. instead of using bag-of-word context, they use context derived automatically from dependency parse trees. specifically, for a word w with modifiers m 1 , ..., m k and head h, the contexts (m1, lbl 1 ), ..., (m k , lbl k ), (h, lbl 1 h ), where lbl is a type of dependency relation between the head. and the modifier (e.g. nsub, dobj, etc). for example, the context for the word scientist in "australian scientist discovers star with telescope" is australian/amod and discovers/nsubj 1 . another modification to word2vec model was proposed by @xcite to improve the word embeddings to syntax-based tasks by making it sensitive to the positioning of the words, and thereby accounting for its lack of order-dependence. the modification does not involve incorporating external parsing information, but it includes using 2 output predictors for every word in the window context each of which is dedicated to predicting position-specific value. results on syntax-based tasks such as pos tagging and parsing show an improvement over classic word2vec embeddings. more recently, a new approach, named syngcn, for learning dependency-based syntactic embeddings is introduced by @xcite . syngcn builds syntactic word representation by using graph convolution network (gcn). using gcn allows syngcn to capture global information from the graph on which it was trained while remaining efficient at training due to parallelization. experiments show that syngcn obtains improvement over state-of-the-art approaches when used with methods such as elmo @xcite . most syntactic word embeddings methods rely on dependency parsing, and to the best of our knowledge that our work is the first utilizing constituency parsing to build syntactic representation. ## method our goal is to learn word embeddings that not only capture the sentence-level syntactic hierarchy encoded by the constituency parse tree, but also capture a global (suprasentential) syntactic representation, and because the constituency parse tree only provides sentence-level syntactic representations, we need a method to combine multiple constituency parse trees. we also need a flexible algorithm to learn the embeddings from those combined trees. in this section, we present a method of parse tree combination (namely graph unionization) as well as the node2vec algorithm. graph unionization given a training dataset of constituency parse trees, we compose one graph (henceforth supergraph, figure 2 ) by unionizing all the sentence trees in the training dataset. formally, let g(v, e) be a graph in the training corpus, where v represents a lexical or a non-lexical vertex in a constituency parse tree and e is the edge between them, and let ## data and experiment for the purposes of training the syntactic embeddings, we use the penn treebank corpus @xcite , which contains over 43,000 constituency parse trees to sentences collected from the wall street journal (wsj). next, we unionize all the parses trees into one supergraph. the supergraph has 51071 vertices and 65895 edges, and it has an average degree of 2.5805 and a density of 5.05 × 10 -5 . we chose to unionize all the trees in the training corpus for simplicity, but we certainly could have grouped the sentences into clusters of thematic or semantic identity prior to applying unionization. after that, we train the embeddings with node2vec algorithm using sgd of 10,000 epochs and a learning rate of 0.025 with a weight decay of 0.005. in terms of node2vec hyperparameters, we chose a random walk of length 200 and a batch size of 100, and the return parameter p and the in-out parameter q are both set to 10 -6. lower p values keep the walk close to the starting node, and lower q values encourage the walk to behave in a dfs manner @xcite . the training took 51 seconds on 1 tesla k80 gpu using the graphvite python package @xcite . we initialize the word vectors randomly for simplicity, but initialization with other types of distributional embeddings such as word2vec or glove is possible. we decide to explore the latter in future work. ## results we conduct two types of evaluations: qualitative and quantitative. in the qualitative evaluation, we examine the extent to which the learned embeddings can encode grammatical information about the words using words analogies and word arithmetics. in addition, we compare its performance against glove vectors @xcite and syngcn @xcite on one stream task, pos tagging. ## conclusion and future work we presented a simple and efficient framework to learn syntactic embeddings from constituency parse trees using a combination of multiple graph unionization and biased random walk. our framework can be flexibly customized to learn purely contextual and non-contextual syntactic embeddings, and it can be also used as a post-hoc method for other kinds of (distributional) word embeddings. thus, for future studies, we would like to investigate training constituency-based vectors on a larger corpus and examine the effect of different initialization on more mainstream tasks such as machine translation and automatic speech recognition.