Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAE s (2022)

Abstract: We propose a generative model for text generation, which exhibits disentangled latent representations of syntax and semantics. Contrary to previous work, this model does not need syntactic information such as constituency parses, or semantic information such as paraphrase pairs. Our model relies solely on the inductive bias found in attention-based architectures such as Transformers. In the attention of Transformers, keys handle information selection while values specify what information is conveyed. Our model, dubbed QKVAE, uses Attention in its decoder to read latent variables where one latent variable infers keys while another infers values. We run experiments on latent representations and experiments on syntax/semantics transfer which show that QKVAE displays clear signs of disentangled syntax and semantics. We also show that our model displays competitive syntax transfer capabilities when compared to supervised models and that comparable supervised models need a fairly large amount of data (more than 50K samples) to outperform it on both syntactic and semantic transfer. The code for our experiments is publicly available.

## introduction disentanglement, a process aimed at obtaining neural representations with identified meaning, is a crucial component of research on interpretability @xcite . a form of disentanglement that received a lot of interest from the nlp community is the separation between syntax and semantics in neural representations @xcite @xcite @xcite . previous works perform disentanglement using paraphrase pairs as information for semantics, and/or constituency parses as information for syntax. the dependence of models on labeled data is known to entail high cost (see @xcite on syntactic annotation), and to often require new labels to handle problems such as concept drift @xcite and domain adaptation @xcite . in light of the above, we propose an unsupervised model which directs syntax and semantics into different neural representations without semantic or syntactic information. in the transformer architecture @xcite , the attention mechanism is built upon a query from a set q, which pools values v through keys k. for each query, values are selected according to their matching score computed by the similarity between their corresponding keys and the query. building on an analogy between the (k, v ) couple and syntactic roles with their lexical realizations (explicited in §4.2) we present qkvae foot_1 , a transformer-based variational autoencoder (vae; @xcite . to build our model, we modify a previous transformer-based vae, called the attention-driven vae (advae; @xcite . using cross-attention, our model encodes sentences into two latent variables: z sem to infer values for v , and z syn to assign keys in k for values in v . these keys and values are then used in the attention mechanism of a transformer decoder to generate sentences. we show that z syn tends to contain syntactic information, while z sem tends to represent semantic information. additionally, comparisons with a supervised model show that it needs a considerable amount of data to outperform our model on syntactic and semantic transfer metrics. our contributions can be summarized as follows: ## related work we broadly divide works on explainability in nlp into two research directions. the first seeks post hoc explanations for black-box models, and led to a rich literature of observations on the behavior of neural models in nlp @xcite @xcite @xcite @xcite . along with these observations, this line of works also led to numerous advances in methodology concerning, for instance, the use of attention as an explanation @xcite , the validity of probing @xcite , or contrastive evaluation with minimal pairs @xcite . the second research direction on explainability in nlp seeks to build models that are explainable by design. this led to models with explicit linguistically informed mechanisms such as the induction of grammars (rnng; @xcite or constituency trees (on-lstm; @xcite . disentangled representation learning is a subfield of this second research direction which aims at separating neural representations into neurons with known associated meanings. this separation was performed on various characteristics in text such as style @xcite , sentiment and topic @xcite , or word morphology @xcite . in works on disentanglement, consequent efforts have been put in the separation between syntax and semantics, whether merely to obtain an interpretable specialization in the embedding space @xcite @xcite , or for controllable generation @xcite @xcite . however, all these works rely on syntactic information (constituency parses and pos tags) or semantic information (paraphrase pairs). to the best of our knowledge, our work is the first to present a method that directs syntactic and semantic information into assigned embeddings in the challenging unsupervised setup. from a broader machine learning perspective, using knowledge of the underlying phenomena in our data, we design our model qkvae with an inductive bias that induces understandable behavior in an unsupervised fashion. among the existing line of applications of this principle @xcite @xcite , advae @xcite , the model on which qkvae is based, is designed to separate information from the realizations of different syntactic roles without supervision on a dataset of regularly structured sentences. ## background in this section, we go over the components of our model, namely vaes, attention in transformers, and advae, the model on which qkvae is based. ## qkvae: using separate latent variables for keys and values in this section, we describe the architecture of our model, the behavior it entails, and how we deal with the optimization challenges it poses. ## experiments 5.1 setup data to compare our model to its supervised counterparts, we train it with data from the english machine-generated paraphrase pairs dataset paranmt @xcite . more specifically, we use the 493k samples used by @xcite foot_6 to train their model vgvae. since our model is unsupervised, we only use the reference sentences (half the training set) to train our model. using the development and test sets of paranmt, chen et al. (2020) also provide a curated set of triplets formed by a target sentence (target), a semantic source (sem_src),and a syntactic source (syn_src). the semantic source is a paraphrase of the target sentence, while the syntactic source is selected by finding a sentence that is syntactically close to the target (i.e. edit distance between the sequence of pos tags of both sentences is low foot_7 ) and semantically different from the paraphrase (has low bleu score with it). contrary to paraphrases in the training set of paranmt, paraphrases from this set were manually curated. these triplets are divided into a development set of 500 samples and a test set of 800 samples. we display results on the test set in the main body of the paper. the results on the development set, which lead to the same conclusions, are reported in appendix a. training details & hyper-parameters encoders and decoders in qkvae are initialized with pa-rameters from bart @xcite . after manual trial and error on the development set, we set the sizes of z syn and z sem to 768, and l to 4. further hyper-parameters are in appendix b. we train 5 instances of our model and report the average scores throughout all experiments. baselines we compare our system to 4 previously published models, where 2 are supervised and 2 are unsupervised: i) vgvae @xcite : a vae-based paraphrase generation model with an lstm architecture. this model is trained using paraphrase pairs and pos tags to separate syntax and semantics into two latent variables. this separation is used to separately specify semantics and syntax to the decoder in order to produce paraphrases; ii) synpg @xcite : a paraphrase generation seq2seq model based on a transformer architecture which also separately encodes syntax and semantics for the same purpose as vgvae. this model is, however, trained using only source sentences with their syntactic parses, without paraphrases; iii) optimus @xcite : a large-scale vae based on a fusion between bert @xcite and gpt-2 (radford et al., 2019) with competitive performance on various nlp benchmarks; iv) advae: this model is qkvae without its syntactic variable. the size of its latent variable is set to 1536 to equal the total size of latent variables in qkvae. official open-source instances 10 of the 4 models above are available, which ensures accurate comparisons. the off-the-shelf instances of vg-vae and synpg are trained on paranmt with glove 11 @xcite embeddings. we fine-tune a pre-trained optimus on our training set following instructions from the authors. similar to our model, we initialize advae with parameters from bart @xcite and train 5 instances of it on paranmt with @xmath0. ## discussion and conclusion in we presented qkvae, an unsupervised model which disentangles syntax from semantics without syntactic or semantic information. our experiments show that its latent variables effectively position sentences in the latent space according to these attributes. additionally, we show that qkvae displays clear signs of disentanglement in transfer experiments. although the semantic transfer is moderate, syntactic transfer with qkvae is competitive with synpg, one of its supervised counterparts. finally, we show that vgvae, a supervised model, needs more than 50k samples to outperform qkvae on both syntactic and semantic transfer. we plan to extend this work in three directions: i) finding ways to bias representations of each z sem l towards understandable concepts; ii) applying qk-vae to non-textual data since it is data agnostic (e.g. to rearrange elements of a visual landscape.); iii) investigating the behavior of qkvae on other languages.