Learning Disentangled Representations of Negation and Uncertainty (2022)

Abstract: Negation and uncertainty modeling are long-standing tasks in natural language processing. Linguistic theory postulates that expressions of negation and uncertainty are semantically independent from each other and the content they modify. However, previous works on representation learning do not explicitly model this independence. We therefore attempt to disentangle the representations of negation, uncertainty, and content using a Variational Autoencoder. We find that simply supervising the latent representations results in good disentanglement, but auxiliary objectives based on adversarial learning and mutual information minimization can provide additional disentanglement gains.

## introduction in formal semantics, negation and uncertainty are operators whose semantic functions are independent of the propositional content they modify @xcite 2 . that is, it is possible to form fluent statements by varying only one of these aspects while leaving the others the same. negation, uncertainty, and content can thus be viewed as disentangled generative factors of knowledge and belief statements (see figure 1 ). disentangled representation learning (drl) of factors of variation can improve the robustness of representations and their applicability across tasks @xcite . specifically, negation and uncertainty are important for downstream nlp tasks such as sentiment analysis @xcite , question answering @xcite , and information extraction @xcite . disentangling nega-trees might not have leaves. tion and uncertainty can therefore provide robust representations for these tasks, and disentangling them from content can assist tasks that rely on core content preservation such as controlled generation @xcite and abstractive summarization @xcite . still, no previous work has tested whether negation, uncertainty, and content can be disentangled, as linguistic theory suggests, although previous works have disentangled attributes such as syntax, semantics, and style @xcite @xcite @xcite . to fill this gap, we aim to answer the following research questions: rq1: is it possible to estimate a model of statements that upholds the proposed statistical independence between negation, uncertainty, and content? rq2: a number of existing disentanglement objectives have been explored for text, all giving promising results. how do these objectives compare for enforcing disentanglement on this task? ## background we here provide relevant background on negation and uncertainty processing, disentangled representation learning in nlp, as well as discussion of how this study fits in with previous work. ## proposed approach we describe our overall model in section 3.1. section 3.2 enumerates three specific desiderata for disentangled representations, and sections 3.3 and 3.4 describe how we aim to satisfy these desiderata. ## experiments we describe our datasets, preprocessing, and data augmentation methods in section 4.1. section 4.2 describes our evaluation metrics and how these target the desiderata for disentanglement given in section 3.2. ## conclusion motivated by linguistic theory, we proposed a generative model of statements in which negation, uncertainty, and content are disentangled latent variables. we estimated this model using a vae, comparing the performance of existing disentanglement objectives. via a suite of evaluations, we showed that it is indeed possible to disentangle these factors. while objectives based on adversarial learning and mi minimization resulted in disentanglement and consistency gains, we found that a decent balance between variable disentanglement and reconstruction ability was obtained by a simple supervision of the latent representations (i.e., the inf objective). also, our 1-dimensional negation and uncertainty representations achieved high predictive performance, despite their simplicity. future work will explore alternative latent distributions, such as discrete distributions @xcite , which may better represent these operators. this work has some limitations. first, our model does not handle negation and uncertainty scope, but rather assumes that operators scope over the entire statement. our model was estimated on relatively short, single-statement sentences to satisfy this assumption, but future work will investigate how operator disentanglement can be unified with models of operator scope in order to apply it to longer examples with multiple clauses. second, while our models achieved high disentanglement, they fell short on the controlled generation task. we found that this was likely due to the models memorizing sentence length, constraining the reconstructions in way that is incompatible with the addition of negation and uncertainty cue tokens. @xcite also noticed this tendency for sentence length memorization in vaes and future will will explore their suggested remedies, such as encoder pretraining. yiyun zhao and steven bethard. 2020. how does bert's attention change when you fine-tune? an analysis methodology and a case study in negation scope. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 4729-4747, online. association for computational linguistics.