Semi-supervised Autoencoding Projective Dependency Parsing (2020)

Abstract: We describe two end-to-end autoencoding models for semi-supervised graph-based projective dependency parsing. The first model is a Locally Autoencoding Parser (LAP) encoding the input using continuous latent variables in a sequential manner; The second model is a Globally Autoencoding Parser (GAP) encoding the input into dependency trees as latent variables, with exact inference. Both models consist of two parts: an encoder enhanced by deep neural networks (DNN) that can utilize the contextual information to encode the input into latent variables, and a decoder which is a generative model able to reconstruct the input. Both LAP and GAP admit a unified structure with different loss functions for labeled and unlabeled data with shared parameters. We conducted experiments on WSJ and UD dependency parsing data sets, showing that our models can exploit the unlabeled data to improve the performance given a limited amount of labeled data, and outperform a previously proposed semi-supervised model.

## introduction dependency parsing captures bi-lexical relationships by constructing directional arcs between words, defining a head-modifier syntactic structure for sentences, as shown in figure 1 . dependency trees are fundamental for many downstream tasks such as semantic parsing @xcite , machine translation @xcite , information extraction @xcite and question answering @xcite . as a result, efficient parsers @xcite @xcite have been developed using various neural architectures. despite the success of supervised approaches, they require large amounts of labeled data, particularly when neural architectures are used. syntactic annotation is notoriously difficult and requires specialized linguistic expertise, posing a serious challenge for low-resource languages. semi-supervised parsing aims to alleviate this problem by combining a small amount of labeled data and a large amount of unlabeled data, to improve parsing performance over labeled data alone. traditional semi-supervised parsers use unlabeled data to generate additional features, assisting the learning process @xcite ## related work most dependency parsing studies fall into two major groups: graph-based and transition-based @xcite . graph-based parsers @xcite regard parsing as a structured prediction problem to find the most probable tree, while transition-based parsers @xcite treat parsing as a sequence of actions at different stages leading to a complete dependency tree. while earlier works relied on manual feature engineering, in recent years the hand-crafted features were replaced by embeddings and deep neural network architectures that were used to learn representation for scoring structural decisions, leading to improved performance in both graph-based and transitionbased parsing @xcite @xcite @xcite @xcite . the annotation difficulty for this task, has also motivated work on unsupervised (grammar induction) and semi-supervised approaches to parsing @xcite @xcite @xcite @xcite . it also leads to advances in using unlabeled data for constituent grammar @xcite similar to other structured prediction tasks, directly optimizing the objective is difficult when the underlying probabilistic model requires marginalizing over the dependency trees. variational approaches have been widely applied to alleviating this problem, as they try to improve the lower bound of the original objective, and were employed in several recent nlp works @xcite @xcite . variational autoencoder (vae) @xcite is particularly useful for latent representation learning, and is studied in semi-supervised context as the conditional vae (cvae) @xcite . note our work differs from vae as vae is designed for tabular data but not for structured prediction, as in our circumstance, the input are the sentential tokens and the output is the dependency tree. the work mostly related to ours is @xcite 's as they consider the dependency tree as the latent variable, but their work takes a second approximation to the variational lower bound by an extra step to sample from the latent dependency tree, without identifying a tractable inference. we show that with the given structure, exact inference on the lower bound is achievable without approximation by sampling, which tightens the lower bound. ## graph-based dependency parsing a dependency graph of a sentence can be regarded as a directed tree spanning all the words of the sentence, including a special "word"-the root-to originate out. assuming a sentence of length l, a dependency tree can be denoted as @xmath0, m 0 >, < h 1 , m 1 >, . . . , < h l-1 , m l-1 >), where h t is the index in the sequence of the head word of the dependency connecting the tth word m t as a modifier. our graph-based parsers are constructed by following the standard structured prediction paradigm @xcite . in inference, based on the parameterized scoring function s λ with parameter λ, the parsing problem is formulated as finding the most probable directed spanning tree for a given sentence x: it is common to factorize the score of the entire graph into the summation of its substructures: the individual arc scores @xcite : ## preliminaries to explain the lap and gap models, we briefly review variational autoencoders (vae) and tree crfs. ## locally autoencoding parser (lap) lap is a fast, semi-supervised parser able to make use of unlabeled data in addition to labeled data. the illustration of this model is displayed in figure 2a . lap learns, using both labeled and unlabeled data, a continuous latent variables representation, designed to support the parsing task by creating contextualized token-representations that capture properties of the full sentence. typically, each token in the sentence is represented by its latent variable z t , which is a high-dimensional gaussian variable. this configuration ensures the continuous latent variable retains the contextual information from lower-level neural models to assist finding its head or its modifier; as well as forcing the representation of similar tokens closer. we adjust the original vae setup in our semi-supervised task by considering examples with labels, similar to recent conditional variational formulations @xcite @xcite . we propose a full probabilistic model for a given sentence x, with the unified objective to maximize for both supervised and unsupervised parsing as follows: this objective can be interpreted as follows: if the training example has a golden tree t with it, then the objective is the log joint probability p θ,ω (t , x); if the golden tree is missing, then the objective is the log marginal probability p θ (x). the probability of a certain tree is modeled by a tree-crf in eq. 1 with parameters ω as p ω (t |x). given the assumed generative process p θ (x|z), directly optimizing this objective is intractable, therefore we instead optimize its evidence lower bound (elbo) : we show j lap is the elbo of j in the appendix a.1. in practice, similar as vae-style models, ## globally autoencoding parser (gap) lap autoencodes the input locally at the sequence level, focusing on the textual representation in isolation by connecting them to the parsing algorithm. a better approach is to treat the entire dependency tree as a structured latent variable to reconstruct the input. we design a different model, gap, by building a model containing both a discriminative component and a generative component to jointly learn the downstream representations and the dependency structure construction. the discriminative component builds a neural crf model for dependency tree construction, and the generative model reconstructs the sentence from the factor graph as a bayesian network, by assuming a generative process in which each head generates its modifier. concretely, the latent variable in this model is the dependency tree structure. ## comparison of complexity we briefly compare the complexity of lap, gap and their competitors in ## conclusion in this paper, we present two semi-supervised parsers, namely locally autoencoding parser (lap) and globally autoencoding parser (gap). both are end-to-end learning systems enhanced with neural architectures, capable of utilizing the latent information within the unlabeled data together with labeled data to improve the parsing performance, without using external resources. more importantly, gap outperforms the previous published @xcite semi-supervised parsing system on the wsj data set. we attribute this success to two reasons: first, gap consists both a discriminative component and a generative component, which are constraining and supplementing each other such that final parsing choices are made in a checked-and-balanced manner to avoid over-fitting. second, instead of sampling from posterior of the latent variable (the dependency tree), gap analytically computes the expectation and marginalization, hence the decoder's global optima can be found, leading to improved performance.