Learning Disentangled Representations for Natural Language Definitions (2023)

Abstract: Disentangling the encodings of neural models is a fundamental aspect for improving interpretability, semantic control and downstream task performance in Natural Language Processing. Currently, most disentanglement methods are unsupervised or rely on synthetic datasets with known generative factors. We argue that recurrent syntactic and semantic regularities in textual data can be used to provide the models with both structural biases and generative factors. We leverage the semantic structures present in a representative and semantically dense category of sentence types, definitional sentences, for training a Variational Autoencoder to learn disentangled representations. Our experimental results show that the proposed model outperforms unsupervised baselines on several qualitative and quantitative benchmarks for disentanglement, and it also improves the results in the downstream task of definition modeling.

## introduction learning disentangled representations is a fundamental step towards enhancing the interpretability of the encodings in deep generative models, as well as improving their downstream performance and generalization ability. disentangled representations aim to encode the fundamental structure of the data in a more explicit manner, where independent latent variables are embedded for each generative factor @xcite . previous work in machine learning proposed to learn disentangled representations by modifying the elbo objective of the variational autoencoders (vae) @xcite , within an unsupervised framework @xcite @xcite . on the other hand, a more recent line of work claims the benefits of supervision in disentanglement @xcite and it advocates the importance of designing frameworks able to exploit structures in the data for introducing inductive biases. in parallel, disentanglement approaches for nlp have been tackling text style transfer, and evaluating the results with extrinsic metrics, such as style transfer accuracy @xcite @xcite . while style transfer approaches investigate the ability to disentangle and control syntactic factors such as tense and gender, the aspect of understanding and disentangling the semantic structure in language is under-explored, but with recent attempts of separating syntactic and semantic latent spaces showing promising results @xcite . furthermore, evaluating disentanglement is challenging, because it requires knowledge of generative factors, leading most approaches to train on synthetic datasets @xcite . in this work, we argue that recurrent semantic structures at sentence level can be leveraged both as inductive biases for enhancing disentanglement (rq1) but also for providing meaningful generative factors that can be employed for evaluating the degree of disentanglement (rq2). we also inves-tigate whether organizing the generative factors in groups may facilitate learning and disentanglement (rq3). as a result, this work focuses on natural language definitions, which are a textual resource characterised by a principled structure in terms of semantic roles, as demonstrated by previous work which proposed the extraction of structural and semantic patterns in this kind of data @xcite . seeking to address the highlighted issues and answer the research questions, we make the following contributions, also depicted in figure 1 . ## disentangling framework in this section we first describe the framework designed for improving disentanglement in natural language definitions with semantic role labels. secondly, we present three models, shown in figure 2 based on the variational autoencoder (vae) @xcite architecture for achieving disentanglement. ## evaluation framework we first present the evaluation framework that for measuring disentanglement, then describe and justify the generative factor setup used in the experiments. ## empirical analysis in this section, we firstly describe the empirical setup for experiments, secondly, we provide qualitative evaluation and thirdly, we measure various quantitative metrics. finally, we demonstrate the capacity of the proposed models in the downstream task of definition modeling. ## conclusion we propose a novel vae-based framework for learning and evaluating disentangled representations in natural language definitions. we leverage the semantic structure present in dictionaries as inductive biases for improving disentanglement in vaes, and as generative factors during evaluation. our evaluation shows, both with qualitative investigations and with quantitative metrics, that the proposed framework is able to produce encodings with a higher degree of disentanglement. finally, our models outperform existing baselines on a definition modeling application, demonstrating the generalization capabilities of disentangled representations.