Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networks (2024)

Abstract: Disentangled latent spaces usually have better semantic separability and geometrical properties, which leads to better interpretability and more controllable data generation. While this has been well investigated in Computer Vision, in tasks such as image disentanglement, in the NLP domain, sentence disentanglement is still comparatively under-investigated. Most previous work have concentrated on disentangling task-specific generative factors, such as sentiment, within the context of style transfer. In this work, we focus on a more general form of sentence disentanglement, targeting the localised modification and control of more general sentence semantic features. To achieve this, we contribute to a novel notion of sentence semantic disentanglement and introduce a flow-based invertible neural network (INN) mechanism integrated with a transformer-based language Autoencoder (AE) in order to deliver latent spaces with better separability properties. Experimental results demonstrate that the model can conform the distributed latent space into a better semantically disentangled sentence space, leading to improved language interpretability and controlled generation when compared to the recent state-of-the-art language VAE models.

## introduction most previous work on controlled text generation have concentrated on style transfer tasks: modifying sentences with regard to markers of sentiment, formality, affirmation/negation @xcite @xcite @xcite @xcite (figure 1 top). disentanglement of language generative factors over variational autoencoder (vae) spaces has been a key mechanism to deliver this type of generative control @xcite @xcite . recently, @xcite demonstrated that a more general form of semantic control can be achieved in the latent space of optimus @xcite , the first interpolation path: animals require oxygen for survival 1. animals require oxygen to survival 2. animals require water 3. animals require water and oxygen ... 9. animals require food for survival animals require food for survival localised/formal semantic control (optimus) interpolation path: animals require oxygen for survival 1. animals require oxygen to survival 2. producer lives in an environment 3. human needs water and oxygen ... 9. animals eat food for survival animals require food for survival sentiment: positive sentiment: negative topic: sport topic: science style-transfer attribute space coarse-grained attribute control sport + negative: i hate basketball. science + positive: i love physics and chemistry. this work aims to improve the localisation and semantic control of latent sentence spaces, by delivering a model which can better separate and control syntactic-semantic features (e.g. predicateargument) and their associated lexical semantics content. this type of representation can provide the foundation to shorten the gap between deep latent semantics and formal linguistic representations @xcite @xcite , integrating the flexibility of distributional-neural models with the properties of linguistically grounded representations, facilitating both interpretability and generative control. to deliver this type of semantic control within the distributional sentence space, following the methodological framework introduced by @xcite , we target on improving the semantic separability of sentences by focusing on explanatory sentences 1 , rather than synthetic or style transfer datasets @xcite , in which the semantic structure of sentences can be isolated and controlled. inspired by the work of @xcite , we integrate a flowbased invertible neural network (inn) @xcite as a plug-in control component to learn the bijective transformation between the distributional hidden space of the transformer-based language autoencoder (bert-gpt2) and the smooth gaussian space of the inn (figure 2 ). specifically, we first pre-train an autoencoder (ae) to learn sentence representations from the transformers' latent spaces. then, we freeze the ae weights and train the inn to map the ae representations to a gaussian space. since inn models define a bijective transformation, we can control the autoencoder generation by manipulating the inn latent spaces, which is more efficient and significantly less resource intensive than re-training a language ae end-to-end. more importantly, we propose a supervised train1 the rationale for choosing explanatory sentences and their semantic details are provided in appendix a. interpreting and controlling sentence generation from the perspective of the geometric manipulation of the latent space is still largely unexplored within nlp. to the best of our knowledge, this is the first work which focuses on the introduction of invertible nn-based mechanisms to support latent spaces with better separated argument structure/semantic features, allowing for a more universal form of sentence generation control. ## preliminaries in this section, we first introduce the formal semantics and define the sentence representation model based on argument structure theory (ast), linking with the associated disentanglement/generative factors and then proceed with the description of the proposed flow-based inn mechanism. controllability and interpretation in formal semantics. formal semantics, which provides a canonical, granular, and rigid representation, have been investigated for thousands of years, with well established theoretical frameworks such as montague semantics @xcite , davidsonian semantics @xcite , neo-davidsonian semantics @xcite , etc. one typical characteristic of these formal semantics is the localisation or composition property. for example, in the sentence, animals require oxygen for survival, the words are functionally combined into sentence semantics: λx(animals(x) → require(x, oxygen)) (1) where x is the variable representing any entity within a logical structure. in this case, we can localise the sentence semantics by replacing x with birds, fishes, etc. this localised process indicates the interpretation in cognitive science @xcite . disentanglement @xcite can potentially provide such localisation in the context of distributional latent representations, which has been widely investigated to localise image generation @xcite @xcite . therefore, we link several key notions-disentanglement, formal semantics, and localization-to investigate formal control and interpretability in language models. sentence semantic disentanglement. ast @xcite @xcite provides a model for representing sentence structure and meaning of sentences in terms of the interface between the their syntactic structure and the associated semantic roles of the arguments within those sentences. it delineates how verbs define the organisation of their associated arguments and the reflection of this organisation in a sentence's syntactic realisation. ast abstracts sentences as predicate-argument structures, where the predicate p (associated with the verb) has a set of associated arguments arg i , where each argument has an associated positional component i and a thematic/semantic roles r i , the latter categorising the semantic functions of arguments in relation to the verb (e.g. agent, patient, theme, instrument). in the context of this work, the ast predicate-argument representation is associated with a lexical-semantic representation of the content c i of the term t i . in this work, we simplify and particularise the relationship between the argument structure and the distributional lexical semantic representation as a role-content relation, where the structural syntactic/semantic relationship is defined by its shallow semantics, i.e. as the composition of the content of the terms, their position in the predicate-argument (parg) structure (arg i ) and their semantic roles (srs) (r i : pred, arg). therefore, this work uses the notion of sentence semantic disentanglement as the cluster separation of the content under the parg/srs structure (the corresponding role in rolecontent), aiming to induce a latent space which geometrically encodes the ast structure, better disentangling and separating role-content clusters. where t i (c i , r i ) = c i ⊗ r i represents the semantics of term t i with content c i (i.e., animals) and srl r i (i.e., arg0) in context s, ⊗: connects the meanings of words with their roles, using the compositionaldistributional semantics notation of @xcite . ⊕: connects the lexical semantics (word content + structural role) to form the sentence semantics. this work applies distinct symbols aiming to emphasise the disentanglement aspects associated with the ast structure. if the sentence representation can be semantically disentangled under ⊕, the sem(s) can be decomposed into: ## proposed approach we encode each sentence x with a frozen autoencoder (e.g., bert-gpt2) and consider its sentence representation e(x) as the input of inns (figure 2 ). we propose two training strategies to map the hidden representations into the gaussian space. ## experiments for the experiments, we start by focusing on the effect of the supervised inn mechanism to examine its impact on the sentence semantic separability of the distributional latent space defined in section 2 (detailed in section 4.1). next, we examine the localised semantic generation control enabled by such semantic separability via latent interpolation (section 4.2). further details of the autoencoder model and dataset are provided in appendix a. ## conclusions and future work this work focused on an inn-based mechanism to support better disentangled and separated latent sentence spaces over language autoencoders. by aligning the predicate-argument structure of sentences to the latent representations, we aimed to build a bridge between the formal and distributional semantics perspectives for sentence representation. we define the sentence semantic disentanglement from the perspective of formal semantics, aligning the predicate-argument structure to disentanglement and cluster separation properties, and exploiting the invertibility and bijection properties of inns to facilitate such alignment. experimental results indicate that the invertibility mechanisms can transform the distributed hidden space of an autoencoder into a latent space where ast-level syntactic and semantic transformations can be localised, interpolated and controlled. secondly, we propose a supervision approach, which leads to an improved disentangled and separated space. this property can facilitate localised interpolation control. lastly, we utilise these geometric properties to support a semantically controlled data augmentation to assist the disentanglement process. since our work connects distributional and formal semantics via disentanglement, one future direction is to explore the safety and control of the formal semantic properties of large language models. besides, recent work @xcite revealed that distinct factors can be composed by modelling the moving of latent vectors via ordinary differential equations, which can be adapted to sentence representations to deliver more complex sentence transformations within the latent space.