A Deep Decomposable Model for Disentangling Syntax and Semantics in Sentence Representation (2021)

Abstract: Recently, disentanglement based on a generative adversarial network or a variational autoencoder has significantly advanced the performance of diverse applications in CV and NLP domains. Nevertheless, those models still work on coarse levels in the disentanglement of closely related properties, such as syntax and semantics in human languages. This paper introduces a deep decomposable model based on VAE to disentangle syntax and semantics by using total correlation penalties on KL divergences. Notably, we decompose the KL divergence term of the original VAE so that the generated latent variables can be separated in a more clear-cut and interpretable way. Experiments on benchmark datasets show that our proposed model can significantly improve the disentanglement quality between syntactic and semantic representations for semantic similarity tasks and syntactic similarity tasks.

## introduction recently, disentangled representations have significantly advanced the performance of several applications in nlp. for example, disentanglement has been used to separating representation of attributes such as sentiment from contents @xcite , understanding subtleties in component modeling @xcite , detecting anomalies @xcite , and learning sentence representations that split the syntax and the semantics @xcite . they are also used to boost text generation @xcite or calculating the semantic or syntactic similarity between sentences @xcite . in this paper, we focus on the task of separating syntax and semantics in sentence representation learning. unlike previous supervised approaches that usually resort to syntactic parsers to handle syntax processing, our approach separates syntactic and semantic variables by disentangling hidden states of deep neural nets in a self-learning and unsupervised fashion. the first work focusing on the separation of syntax and semantics from hidden variables is @xcite . they proposed a deep generative model based on vae with two latent variables to represent syntax and semantics. the generative model comprises von mises fisher (vmf) and gaussian priors on the semantic and syntactic latent variables, and a deep bow decoder conditioning on these latent variables. following previous work, they train this model by optimizing the evidence lower bound (elbo) with a vae-like (kingma and welling, 2014) objective. however, their approach still generates a rough decomposition and thus may fail to disentangle syntax and semantics at a finer granularity. to address this weakness, we propose a decomposable variational autoencoder (decvae) to allow hidden variables factorizable. from a modeling perspective, factorizable representations with statistically independent variables usually obtained in an unsupervised or semi-supervised manner can distill information into a compact form, which is semantically useful for downstream tasks. from an application perspective, different words or phrases in sentences represent various entities with variant roles. it is necessary to utilize decomposable latent variables to capture a variety of entities with different semantic meanings. towards building a finer-grained disentanglement, motivated by factorvae @xcite , we extend the work in @xcite and use total correlation @xcite (tc) as a penalty term to obtain a deeper and meaningful factorization of syntactic and semantic latent variables. to make tc more discriminative, we also integrate multi-head attention into this framework. decvae can identify and cluster hierarchically independent semantic components in natural language text, which exhibits hierarchical linguistic structure @xcite , and the corresponding syntax and semantics interact with each other. for experiments, we evaluate learned semantic representations on the semeval semantic textual similarity (sts) tasks. following the protocol in @xcite , we predict the syntactic structure of an unseen sentence to be the one similar to its nearest neighbor, determined by the latent syntactic representation in a large dataset of annotated sentences. experiments show that decvae achieves the best performance on all tasks when learned representations are mostly disentangled. ## proposed approach in this work, we are developing a generative model named decomposable vae (decvae). although our proposed approach is applicable to any disentangled tasks in nlp, we focus on disentangling semantic and syntactic information from sentence representations. we extend vgvae model @xcite to incorporate the total correlation as a penalty term to enable latent variable factorization. ## experiments following @xcite , we sampled 50m paraphrase pairs from paranmt-50m @xcite as our training set. we use the semeval semantic textual similarity (sts) task 2017 @xcite as the development set. the sts task and its benchmark as the test set for similarity evaluation. the implementation was based on the paddlepaddle deep learning platform. ## conclusion we propose decvae, a framework to disentangle syntax and semantics in a sentence. it extends the original vae so that the latent variables can be separated in more interpretable way. experiments show that decvae achieves better results in semantic and syntax similarity than that of sota. one future direction is fine-grained representation learning for words and sentences, which is essential for many downstream applications such as controllable text generation. besides, continual and interactive feature distillation may help improve more discriminate disentanglement @xcite .