Semi-Supervised Semantic Dependency Parsing Using CRF Autoencoders (2020)

Abstract: Semantic dependency parsing, which aims to find rich bi-lexical relationships, allows words to have multiple dependency heads, resulting in graph-structured representations. We propose an approach to semi-supervised learning of semantic dependency parsers based on the CRF autoencoder framework. Our encoder is a discriminative neural semantic dependency parser that predicts the latent parse graph of the input sentence. Our decoder is a generative neural model that reconstructs the input sentence conditioned on the latent parse graph. Our model is arc-factored and therefore parsing and learning are both tractable. Experiments show our model achieves significant and consistent improvement over the supervised baseline.

## introduction semantic dependency parsing (sdp) is a task aiming at discovering sentence-internal linguistic information. the focus of sdp is the identification of predicate-argument relationships for all content words inside a sentence @xcite . compared with syntactic dependencies, semantic dependencies are more general, allowing a word to be either unattached or the argument of multiple predicates. the set of semantic dependencies within a sentence form a directed acyclic graph (dag), distinguishing sdp from syntactic dependency parsing tasks, where dependencies are usually tree-structured. extraction of such high-level structured semantic information potentially benefits downstream nlp tasks @xcite . several supervised sdp models are proposed in the recent years by modifying syntactic dependency parsers. their parsing mechanisms are either transition-based @xcite ., * corresponding author. one limitation of supervised sdp is that labeled sdp data resources are limited in scale and diversity. due to the rich relationships in sdp, the annotation of semantic dependency graphs is expensive and difficult, calling for professional linguists to design rules and highly skilled annotators to annotate sentences. this limitation becomes more severe with the rise of deep learning, because neural approaches are more data-hungry and susceptible to over-fitting when lacking training data. to alleviate this limitation, we investigate semi-supervised sdp capable of learning from both labeled and unlabeled data. while a lot of work has been done on supervised sdp, the research of unsupervised and semisupervised sdp is still lacking. since parsing results of semantic dependencies are dags without the tree-shape restriction, most existing successful unsupervised @xcite @xcite and semi-supervised @xcite @xcite learning models for syntactic dependency parsing cannot be applied to sdp directly and it would be non-trivial to extend these models for sdp. there also exist several unsupervised @xcite and semi-supervised @xcite @xcite methods for semantic parsing, but these models are designed for semantic representations different from dependency graphs, making their adaptation to sdp difficult. in this work, we propose an end-to-end neural semi-supervised model leveraging both labeled and unlabeled data to learn a dependency graph parser. our model employs the framework of conditional random field autoencoder @xcite , modeling the conditional reconstruction probability given the input sentence with its dependency graph as the latent variable. our encoder is the supervised model of @xcite , formulating an sdp task as labeling each arc in a directed graph with a simple neural network. analogous to a crf model @xcite , our encoder is capable of computing the probability of a dependency graph conditioned on the input sentence. the decoder is a generative model based on recurrent neural network language model @xcite , which formulates the probability of generating the input sentence, but we take into account the information given by the dependency parse graphs when generating the input. our model is arc-factored, i.e., the encoding, decoding and reconstructing probabilities can all be factorized into the product of arc-specific quantities, making both learning and parsing tractable. a unified learning objective is defined that takes advantage of both labeled and unlabeled data. besides, compared with previous semi-supervised approaches based on variational autoencoder @xcite , our learning process does not involve sampling, promising better stability. we evaluate our model on semeval 2015 task 18 dataset (english) @xcite and find that our model consistently outperforms the supervised baseline. we also conduct detailed analysis showing the benefits of different amounts of unlabeled data. ## model our model is based on the crf autoencoder framework @xcite which provides a unified fashion for structured predictors to leverage both labeled and unlabeled data. a crf autoencoder aims to produce a reconstruction of the input x from the original input x with an intermediate latent structure y. it is trained to maximize the conditional reconstruction probability p ( @xmath0. ideally, successful reconstruction implies that the latent structure captures important information of the input. we adopt the following notations when describing our model. we represent a vector in lowercase bold, e.g., s, and use a superscript for indexing, e.g., s i for the i-th vector. we represent a scalar in lowercase italics, e.g., s, and use a subscript for indexing, e.g., s i for the i-th element of vector s. an uppercase italic letter such as y denotes a matrix. a lower case letter with a subscript pair such as y i,j refers to the element of matrix y at row i and column j. an uppercase bold letter, e.g., u, stands for a tensor. we maintain this convention when indexing, e.g., y i is the i-th row of matrix y . in our model, the input is a natural language sentence consisting of a sequence of words. a sentence with m words is represented by @xmath1, s 1 , s 2 , . . . , s m ), where s 0 is a special token top. the latent variable produced by our encoder is a dependency parse graph of the input sentence, represented as a matrix of booleans @xmath2, 1} (m+1)×(m+1) , where y i,@xmath3. the reconstructed output generated by our decoder is a word sequence ŝ = (ŝ 1 , ŝ2 , . . . , ŝm ). our encoder with parameters θ computes p θ (y |s), the probability of generating a dependency parse graph y given a sentence s. our decoder with parameters λ computes p λ (ŝ|y ), the probability of reconstructing sentence ŝ conditioned on the parse graph y . the encoder and decoder in combination specify the following conditional distribution. to compute the conditional probability p (ŝ|s), we sum out the latent variable y . note that throughout our model, we only consider dependency arc predictions (i.e., whether an arc exists between each word pair). arc-labels will be learned separately as described in section 3. we leave the incorporation of arc-label prediction in our model for future work. ## learning since we want to train our model in a semisupervised manner, we design loss functions for labeled and unlabeled data respectively. for each training sentence s, the overall loss function is defined as a combination of supervised loss l l and unsupervised loss l u . supervised loss for any labeled sentence (s, y * ), where s stands for a sentence and y * stands for a gold parse graph, we can compute the discriminative loss. following the derivation of eq.6, we have: usage source sentences tokens train wsj sec.00-20 35,656 802,717 test (id) wsj sec.21 1,410 31,948 test (ood) brown 1,849 31,583 gold parses also provide a label for each dependency. we follow @xcite and model dependency labels with a purely supervised module on top of the bilstm layer of the encoder. its parameters are learned by optimizing a crossentropy loss function. unsupervised loss for any unlabeled sentence s, we maximize the conditional reconstruction probability p (ŝ = s|s). the unsupervised loss is: derivations of eq.9 are provided in the appendix a. given a dataset containing both labeled and unlabeled sentences, our model can be trained end-toend by optimizing the loss function eq.7 over the combined dataset using any gradient based method. ## related work work on unsupervised or semi-supervised dependency parsing, to the best of our knowledge, is dominated by tree-structured parsing @xcite @xcite . recently, @xcite introduced an approximate inference method with a variational autoencoder @xcite for semi-supervised syntactic dependency parsing. our decoder is inspired by their work, but differs from theirs in that our decoder handles parse graphs and is arc-factored. @xcite used the framework of crf autoencoder @xcite to perform unsupervised syntactic dependency parsing. the same framework has been used by @xcite for semi-supervised sequence labeling. our work also adopts the crf autoencoder framework, but with both the encoder and the decoder redesigned for semantic dependency parsing. existing unsupervised and semi-supervised approaches to semantic parsing focused on semantic representations different from dependency graphs, e.g., general-purpose logic forms @xcite and formal meaning representations @xcite . @xcite presented the first unsupervised semantic parser to transform dependency trees into quasi-logical forms with markov logic. following this work, @xcite proposed a non-parametric bayesian model for unsupervised semantic parsing using hierarchical pit-manyor process @xcite . @xcite described a semi-supervised approach to framesemantic parsing. @xcite proposed a semi-supervised semantic parsing approach making use of unpaired logical forms with sentence being unobserved. recently, @xcite proposed a variational autoencoding model for semisupervised semantic parsing of tree-structured semantic representations. take yin et al. (2018) for example. to extend their approach for sdp, one needs to design a different transition system for their encoder for graph parsing and design a graph linearization method for their sequence-tosequence decoder. in addition, sdp-specific constraints (e.g., the graph structure contains exactly the same set of words as the sentence) shall be incorporated into their model. therefore, previous semi-supervised semantic parsing models cannot be applied to sdp directly and modifying them for sdp is non-trivial. we leave for future work such modification and extension of previous semisupervised semantic parsing approaches to sdp. ## conclusion in this work, we proposed a semi-supervised learning model for semantic dependency parsing using crf autoencoders. our model is composed of a discriminative neural encoder producing a dependency graph conditioned on an input sentence, and a generative neural decoder for input reconstruction based on the dependency graph. the model works in an arc-factored fashion, promising end-to-end learning and efficient parsing. we evaluated our model under both fullsupervision settings and semi-supervision settings. our model outperforms the baseline on multiple target representations. by adding unlabeled data, our model exhibits further performance improvements. in particular, our semi-supervised model performs well in the low resource setting and on the out-ofdomain test set. this points to future directions of applying our model to low-resource languages and cross-domain settings. our code is publicly available at https://github.com/jzxxx/semi-sdp .