Semi-supervised Parsing with a Variational Autoencoding Parser (2020)

Abstract: We propose an end-to-end variational autoencoding parsing (VAP) model for semi-supervised graph-based projective dependency parsing. It encodes the input using continuous latent variables in a sequential manner by deep neural networks (DNN) that can utilize the contextual information, and reconstruct the input using a generative model. The VAP model admits a unified structure with different loss functions for labeled and unlabeled data with shared parameters. We conducted experiments on the WSJ data sets, showing the proposed model can use the unlabeled data to increase the performance on a limited amount of labeled data, on a par with a recently proposed semi-supervised parser with faster inference.

## introduction dependency parsing captures bi-lexical relationships by constructing directional arcs between words, defining a head-modifier syntactic structure for sentences, as shown in figure 1 . dependency trees are fundamental for many downstream tasks such as semantic parsing @xcite , machine translation @xcite , information extraction @xcite and question answering @xcite . recently, efficient parsers @xcite @xcite have been developed using various neural architectures. while supervised approaches have been very successful, they require large amounts of labeled data, particularly when neural architectures are used, which usually are over-parameterized. syntactic annotation is notoriously difficult and requires specialized linguistic expertise, posing a serious challenge for low-resource languages. semi-supervised parsing aims to alleviate this problem by combining a small amount of labeled data and a large amount of unlabeled data, to improve parsing performance on using labeled data alone. traditional semi-supervised parsers use unlabeled data to generate additional features in order to assist the learning process @xcite , together with different variants of self-training @xcite . however, these approaches are usually pipe-lined and error-propagation may occur. in this paper, we propose variational autoencoding parser, or vap, extends the idea of vae, illustrated in figure 3 . the vap model uses unlabeled examples to learn continuous latent variables of the sentence, which can be used to support tree inference by providing an enriched representation. we summarize our contributions as follows: 1. we proposed a variational autoencoding parser (vap) for semi-supervised dependency parsing; 2. we designed a unified loss function for the proposed parser to deal with both labeled and unlabeled data. 3. we show improved performance of the proposed model with unlabeled data on the wsj data sets, and the performance is on a par with a recently proposed semi-supervised parser @xcite , with faster inference. ## related work most dependency parsing studies fall into two major groups: graph-based and transition-based @xcite . graph-based parsers @xcite regard parsing as a structured prediction problem to find the most probable tree, while transition-based parsers @xcite treat parsing as a sequence of actions at different stages leading to a dependency tree. while earlier works relied on manual feature engineering, in recent years the hand-crafted features were replaced by embeddings and deep neural network architectures were used to learn representation for scoring structural decisions, leading to improved performance in both graph-based and transition-based parsing @xcite @xcite @xcite @xcite . the annotation difficulty for this task, has also motivated work on unsupervised (grammar induction) and semi-supervised approaches to parsing @xcite @xcite @xcite @xcite . it also leads to advances in using unlabeled data for constituent grammar @xcite similar to other structured prediction tasks, directly optimizing the objective is difficult when the underlying probabilistic model requires marginalizing over the dependency trees. variational approaches are a natural way to alleviate this difficulty, as they try to improve the lower bound of the original objective, and have been applied in several recent nlp works @xcite @xcite . variational autoencoder (vae) (kingma and welling, 2014) is particularly useful for latent representation learning, and has been studied in semi-supervised context as the conditional vae (cvae) @xcite . note our work differs from vae as vae is designed for tabular data but not for structured prediction, as the input towards vap is the sequence of sentential tokens and the output is the dependency tree. ## graph-based dependency parsing a dependency graph of a sentence can be regarded as a directed tree spanning all the words of the sentence, including a special "word"-the rootto originate out. assuming a sentence of length l, a dependency tree can be denoted as @xmath0, m 1 >, . . . , < h l-1 , m l-1 >), where h t is the index in the sequence of the head word of the dependency connecting the tth word m t as a modifier. our graph-based vap parser is constructed based on the following standard structured prediction paradigm @xcite . in inference, based on the scoring function s λ with parameter λ, the parsing problem is formulated as finding the most probable directed spanning tree for a given sentence x: it is common to factorize the score of the entire graph into the summation of its substructures-the individual arc scores @xcite : ## variational autoencoding parser vap (illustrated in figure 3b ) is a semi-supervised parser able to make use of unlabeled data in addition to labeled data, extending the idea of variational autoencoder (vae, illustrated in figure 3a ) to dependency parsing. vap learns, using both labeled and unlabeled data, a continuous latent variables representation, designed to support the parsing task by creating contextualized token-representations that capture properties of the full sentence. typically, each token in the sentence is represented by its latent variable z t , which is a high-dimensional gaussian variable, to be aggregated as a group of latent variables z. this configuration ensures the continuous latent variable retains the contextual information from lower-level neural models to assist finding its head or its modifier; as well as forcing the representation of similar tokens to be closer. the latent variable group z is modeled via p (z|x). in addition, we model the process of reconstructing the input sentence from the latent variable through a generative story p (x|z). we adjust the original vae setup in our semisupervised task by considering examples with labels, similar to recent conditional variational formulations @xcite @xcite . we propose a full probabilistic model for a given sentence x, with the unified objective to maximize for both supervised and unsupervised parsing as follows: this objective can be interpreted as follows: if the training example has a golden tree t with it, then the objective is the log joint probability p θ,ω (t , x); if the golden tree is missing, then the objective is the log marginal probability p θ (x). the probability of a certain tree is modeled by a tree-crf with parameters ω as p ω (t |x). given the assumed generative process p θ (x|z), directly optimizing this objective is intractable, thus instead we optimize its evidence lower bound (elbo): we show j lap is the elbo of j in the appendix a.1. ## conclusion in this study, we presented variational autoencoding parser (vap), an end-to-end parser, capable of utilizing the unlabeled data together with labeled data to improve the parsing performance, without any external resources. the proposed vap model performs on a par with a recently published @xcite semi-supervised parsing system on the wsj data set, with faster inference, showing its potential for low-resource languages.