Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational A uto E ncoders (2024)

Abstract: The injection of syntactic information in Variational AutoEncoders (VAEs) can result in an overall improvement of performances and generalisation. An effective strategy to achieve such a goal is to separate the encoding of distributional semantic features and syntactic structures into heterogeneous latent spaces via multi-task learning or dual encoder architectures. However, existing works employing such techniques are limited to LSTM-based VAEs. This work investigates latent space separation methods for structural syntactic injection in Transformer-based VAE architectures (i.e., Optimus) through the integration of graph-based models. Our empirical evaluation reveals that the proposed end-to-end VAE architecture can improve theoverall organisation of the latent space, alleviating the information loss occurring in standard VAE setups, and resulting in enhanced performances on language modelling and downstream generation tasks.

## introduction injecting explicit syntactic information in variational autoencoders (vaes) @xcite has led to improved performance on several language generation tasks, such as paraphrasing and translation @xcite @xcite . among existing techniques, a line of research explores syntactic injection via sentence-level semantics-syntax disentanglement, which consists in the explicit separation of distributional semantic and structural syntactic features through the optimisation of heterogeneous latent spaces @xcite @xcite . such methods, implemented under multi-task learning or dual encoder architectures, have been demonstrated to improve: (i) generation controllability and interpretability @xcite , (ii) robustness and generalisation, (iii) fine-grained representation and latent space organisation @xcite , and more importantly (iv) injecting syntactic features into vaes can allow for optimization in low-dimensional and regularized latent gaussian space, rather than complex discrete sequence spaces as investigated in previous work @xcite @xcite , which represents an efficient to improve text generation @xcite . however, most of these methods focus on lstmbased vaes, and their effectiveness for larger architectures based on transformers, such as optimus @xcite , is still under-explored. to combine the benefits of larger pre-trained vaes and latent separation methods, this paper focuses on the injection of structural syntactic information in transformer-based vaes (i.e., optimus @xcite ). specifically, we investigate a first overarching research question: "rq1: how can we best capture explicit syntactic information in the latent space of transformer-based vaes?" we address this question by directly intervening on the optimus architecture to induce a latent space separation via graph-based @xcite and sequential neural encoders @xcite . specifically, our hypothesis is that graph neural networks (gnns) @xcite @xcite can induce specialised and complementary latent representations that can better capture structural syntactic relations and alleviate the information bottleneck in vaes' semantic encoder @xcite (i.e. trade-off between semantics and syntax). subsequently, we focus on the problem of leveraging multiple, specialised latent spaces derived from the dual encoder architecture for decoding. this leads to several challenges (figure 1 ) since (i) the syntactic representations may not possess a one-to-one mapping with the semantic representations (i.e., one syntactic structure can correspond to multiple sentence representations), (ii) the optimisation of heterogeneous latent spaces can result in different latent distributions, a feature that can affect decoding and language generation performance, and (iii) compared with an lstm decoder, transformer-based decoders (e.g., gpt2) are typically larger and contain information acquired during pre-training, being more difficult to control. those challenges lead to our second research question: "rq2. how can multiple, specialised latent spaces be effectively injected into the vae decoder?" to answer it, we investigate injection mechanisms for transformer-based vaes via the following methods: (i) we separately inject syntax and semantic representations into the attention weights of the decoder (i.e., query and key-value), and (ii) consider low-rank injections, including addition, memory @xcite , and tensor fusion @xcite , which directly operate over the attention matrices and potentially reduce information redundancy @xcite . we perform extensive experiments to evaluate the resulting vae architectures on both mathematical expressions @xcite and natural language explanatory sentences @xcite . overall, our contributions can be summarised as follows: 1. we propose a dual encoder architecture for transformer-based vaes integrating graph-based and sequential models to better capture and disentangle semantic and structural syntactic features in multiple, specialised latent spaces. 2.we explore the injection of such representations into the decoder of transformerbased vaes via low-rank vector operations to better guide the generation process. 3. we perform extensive experiments showing that the adoption of a graph-based encoder coupled with a transformer encoder can reduce the loss of information in the sentence bottleneck, resulting in improved reconstruction and language modelling. overall, we found that the proposed vae architecture can significantly improve performance and generalisation when compared to sentence-level vae baselines. our complete experimental code is available online to encourage future work in the field 1 . ## preliminaries latent space injection. in optimus, the transformation between latent (i.e., gaussian) and observed (i.e., generated sentences) spaces can be done by intervening on the key-value attention weights of the decoder (i.e., gpt2) via memory injection @xcite . specifically, the latent representation z produced by the encoder (i.e., bert) is concatenated into the original key-value weights of gpt2 as follows: where q has dimension r 64×seq , and [z; k], [z; v ] have dimension r 64×(seq+1) (where 64 is the dimension of gpt2 attention, seq is sequence length). in order words, the decoder model is explicitly guided in the generation process by conditioning kv on z. in this work, however, we focus on heterogeneous representations encoding distributional semantic and structural syntactic features in separate latent spaces (see figure 1 ). such a separation requires going beyond the memory injection setup and developing different methods to effectively condition the decoding process in gpt2. ## methodology our methodology consists of two main phases. first, we investigate different encoding strategies to explicitly capture syntactic and structural information in a separate latent space. subsequently, we explore techniques to fuse syntactic and semantic features and inject them into the decoder model. regarding the encoding phase, we explore four architectures based on two different configurations (i.e., multi-task learning and dual encoder) integrating both sequential and graph-based models under optimus (bert-gpt2) memory setup (see figure 6 ). regarding the decoding phase, we consider the best encoding configuration in terms of syntactic representation and propose different injection mechanisms via low-rank operations over attention-weight matrices of gpt2. the following sections describe each phase in detail (sections 3.1 and 3.2), including how the encoding and decoding stages are integrated into an end-to-end vae architecture (section 3.3). ## empirical evaluation following the stages in our methodology, we first evaluate different encoding setups for injecting syntactic information into vaes (section 3.1). subsequently, we consider the best encoding configuration to examine which decoding strategy (section 3.3) can lead to better language modelling performances. finally, we evaluate the best architectural setup for downstream tasks. to experiment, we focus on both explanatory sentences and mathematical expressions. the rationale behind this choice is that (1) explanatory sentences @xcite @xcite provide a semantically challenging yet sufficiently well-scoped scenario to evaluate the syntactic and semantic organisation of the space; (2) mathematical expressions @xcite follow a welldefined syntactic structure and set of symbolic rules that are notoriously difficult for neural models. all experimental details are provided in appendix a. ## conclusion and future work this work focused on the semantic-syntax separation through language vaes, especially optimus (bert-gpt2), architecture. we first implement several encoding baselines and reveal that languagegraph encoding setups can better capture syntax information and maintain semantic-syntax separation. however, the language-graph encoding setup leads to low reconstruction performance. to solve this problem, we explored the integration of heterogeneous latent spaces via injection mechanisms. experimental results showed that our setup can greatly improve language modelling performance, and revealed that the semantic-syntax separation can assist the language modelling task since the language encoder pays more attention to fine-grained lexical semantics, avoiding the distraction of syntax information captured by the separated syntax encoder, which can alleviate the information bottleneck of the language encoder. in the future, we will investigate graph-to-text generation through vaes for bridging structural and distributional semantics via latent gaussian space. by learning the structural semantics distribution as approximated posterior, this type of representation can shorten the gap between deep latent semantics and formal linguistic representations @xcite , integrating the flexibility of distributionalneural models with the properties of linguistically grounded representations, facilitating both interpretability and generative control.