R ev U p: Revise and Update Information Bottleneck for Event Representation (2023)

Abstract: The existence of external (“side”) semantic knowledge has been shown to result in more expressive computational event models. To enable the use of side information that may be noisy or missing, we propose a semi-supervised information bottleneck-based discrete latent variable model. We reparameterize the model’s discrete variables with auxiliary continuous latent variables and a light-weight hierarchical structure. Our model is learned to minimize the mutual information between the observed data and optional side knowledge that is not already captured by the new, auxiliary variables. We theoretically show that our approach generalizes past approaches, and perform an empirical case study of our approach on event modeling. We corroborate our theoretical results with strong empirical experiments, showing that the proposed method outperforms previous proposed approaches on multiple datasets.

## introduction in this work, we are interested in addressing limitations in how computational event modeling can make use of relevant, supplementary semantic knowledge. this is because when modeling text descriptions of complex situations, such as newspaper descriptions of real world events, learning how to encode richer information about those descriptions can be a fruitful way of improving modeling performance @xcite . e.g., if we are dealing with sequences of events, like a newspaper report of a stock or commerce transaction, then being able to encode that "buying" or "selling," even though they may have nuanced connotation differences, are instances of the same general event (a transaction event) can improve downstream predictive performance on what events might happen next @xcite . however, there is not an obvious single way to learn to encode this richer information, as three dif-< l a t e x i t s h a 1 _ b a s e 6 4 = " w e b n e x e p x w / e o x z u x k q k o v 8 u 6 t @xmath0, z) = ˆ r (t|z) + (1 ˆ )p d (t|x) < l a t e x i t s h a 1 _ b a s e 6 4 = " u 6 i 0 + k z t d / x r a s e c c 0 5 t u k + k e z @xmath1, z)||r (t|z) < l a t e x i t s h a 1 _ b a s e 6 4 = " k w u 3 d e r 6 m v 2 i u k t w a b a i q c 1 c m 7 @xmath2 figure 1 : revise and update steps for an observed node. we update the proposed distribution r γ (t|z) with the empirical distribution p d (t|x) to produce the revised distribution p γ (t|x, z) (dashed). we then minimize the kl divergence between the proposed and revised distributions to update the proposed distribution. for unsupervised nodes, we rely on r γ (t|z) without updating. ferent questions naturally come to mind: (1) if the model is representationally limited, can we address these representational limitations of the model itself, such as through developing richer latent representations z of the input x? (2) if there is available background or side information t that may be especially relevant for the modeling task at hand, can we develop systems that make use of it? (3) even when side information is available, it may be noisy, e.g., it may not always be available (missing data) or it may contain errors: how can we make our models robust to this noisy side information? in the context of a text-based sequence modeling problem, we propose an approach that addresses all three of these questions. we provide a conceptual overview of our method-revup-in fig. 1 . the building blocks of revup are revision of modeling side information t: forming a new distribution by combining the (red) learned proposal distribution with (blue) empirical information about when particular aspects of side knowledge appeared in training, and updat-<l a t e x i t s h a 1 _ b a s e 6 4 = " + 1 c n v m o h v u c 1 m 3 e b e x q 0 d x f e u d @xmath3 c j a r 9 d p d u k l h z m h 6 l k q s m b n m 5 + f o 3 d o r r g 6 s t c 2 j 7 l z 9 p t e l i t g t j l s d c c g r w f z y 8 t + v l 2 f 8 e 0 y 5 t d n k k i 4 w x z l w u b n 5 7 2 7 e n a m o j p y q q r m 9 1 a u j o g l f m 1 a e g r / 8 8 i p p x 9 t 9 q 7 r / c f l r 3 b z x l o e e t u e c f l i g b t x d e 1 p a y q z p 8 a p v t u q 8 o o / o x 6 k 1 5 b q z x / a h z u c p 9 k + p u @xmath4. . . < l a t e x i t s h a 1 _ b a s e 6 4 = " p f m k x q 3 t v n w e j v w q c x o m o x b 0 k n @xmath5. . . < l a t e x i t s h a 1 _ b a s e 6 4 = " p f m k x q 3 t v n w e j v w q c x o m o x b 0 k n @xmath6 r w 7 f q 3 o d 0 l / e h 5 i k g e i 0 c j + c v s p y x r n k e o x p + l 6 g y q e a b z o 8 v w 5 y w z n g 1 9 d h t u s t u n y e x e c m h t 2 w s o u 2 u 2 0 r q t p q f 0 4 u o i z p q t h 2 k s a r m + r 1 x f + 8 z o 7 t / b a q s z y j t 9 j 3 r + 1 c u k x p p x p a e p o z l f 1 l g g l h d 6 x s c j q w t m m v b q j + 6 m l / y c v 2 1 d + t + m c 7 l d r h m i 4 s w s x r z j p 4 z i / u y d e 5 j x x c y d 1 5 j m / k x x l w n p x x 5 + 2 7 d c w z z q y q x 3 d e v w d k q z g @xmath7 figure 2 : components of revup for event modeling. we encode the sequence of events x into gaussian latent z and discrete knowledge t. red nodes such as t 1 are latent: [verdict] for the convicted man of murdering event. revup predicts these latent nodes by sampling from the gumbelsoftmax distribution. blue nodes such as t m are observed: @xcite for the sentenced man to death event. we use this observed node to modify the proposed distribution and then we draw a sample from the revised distribution. our approach is inspired by recent work @xcite , which argued that a key to the success of neural advances in nlp @xcite @xcite is that they fall within the information theoretic-based infomax framework @xcite . in this viewpoint, neural advances can be attributed to implicitly maximizing the mutual information between different representations of the same document. however, we additionally want to use "side" or "external" knowledge to aid our modeling. the use of side information has long been desired for effective representation @xcite . while this side information can occur in a variety of forms @xcite , we broadly view it as a concise, abstract view of information conveyed in the main input. our work centers on the idea that the latent neural representations z and side knowledge t act as complementary representations of the same input x: z provides a compact representation of the input itself, while t provides more generic information about the data or task. as such, we use an infomaxinspired formulation to incorporate external knowledge into the modeling and latent representation aspects. since recent work @xcite has suggested that providing some sort of "guidance" to these latent variables is beneficial and can lead to these guided models outperforming both fully supervised and fully unsupervised methods, learning with less-than-full observation of structured side knowledge is a requirement. we illustrate this in fig. 2 , where the correct value for the first piece of side information (t 1 = verdict) is missing but the last piece of side information is known (t @xmath8. our model must be able to operate over this partially observed sequence of side knowledge as a way of modeling the original event description. we propose a unified approach, revup (revise and update information bottleneck), that maximizes the mutual information between the external knowledge and latent representation i(z; t). we consider the external knowledge t to be a discrete random variable in a lightly structured, semisupervised setting. we demonstrate the effectiveness of this approach on multiple modeling tasks. our contributions are: our code is available at https://github. com/mmrezaee/revup . ## background in this section we present the related work and discuss the connections to our approach. ## revup: revise and update previous work @xcite has empirically shown that incorporating external knowledge in discrete gumbelsoftmax parameters improves model performance with different metrics such as classification, event modeling and training speed. we seek to re-frame this view with information theory terms and generalize it. for fairness and consistency we stick with the same types of computational event modeling tasks that ssdvae was developed for. we first describe the probabilistic model we develop (section 3.2) and the loss/training methodology (section 3.3). these enable smoothly incorporating side knowledge into a probabilistic neural model. recall we provide a conceptual overview of revup in fig. 1 . ## experimental results following previous work @xcite , we experiment on event modeling tasks using concretely annotated versions of new york times gigaword (nyt) and wikipedia datasets @xcite . both are english and have framenet semantic frames provided via se-mafor @xcite . we perform a direct comparison with the previous state-of-theart work @xcite by using the same frame types, which were derived from the framenet annotations on wikipedia articles. training, validation and test splits for nyt and wikipedia have 320k/17k/7k and 240k/8k/11k documents, respectively. for both the validation and test phases we set l @xmath9. we average results over three runs (standard deviations in the appendix). we tuned the hyperparameters on the validation dataset. see appendix i for additional implementation and data details. ## related work information bottleneck the concept of using side information for discrete source coding was explored by @xcite ; @xcite . the information bottleneck (ib) principle was then introduced by @xcite to compress input variables while predicting a target. @xcite proposed incorporating negative side information through an auxiliary loss in a supervised manner. our method stands apart by handling both supervised and semi-supervised settings with ease. the variational information bottleneck (vib) was introduced by @xcite , which improved the ib estimation through amortized variational methods. @xcite extended the vib method for semi-supervised classification. in relation to our work, there have been numerous studies focused on maximizing the mutual information between different views and discarding the non-shared information @xcite @xcite @xcite . variational autoencoders (vaes) kingma and welling (2013) and rezende et al. (2014) introduced the reparameterization technique for variational inference. recently, huang and xiao (2021) applied adversarial training to enhance the ability of vae in processing sequential data, while qiu et al. (2020) explored the possibility of vae handling multi-view temporal data. our focus is limited to objectives that can be expressed with discrete latent variables. jang et al. (2016) and maddison et al. (2016) independently presented the gumbel-softmax distribution, which allows backpropagation through latent discrete variables in variational autoencoders. for more information, refer to the review by huijben et al. (2022). van den oord et al. (2017) proposed vq-vae, which uses vector quantization to represent discrete values with embeddings. kingma et al. (2014) proposed a vae for semisupervised classification, where labels in unsupervised settings are treated as latent variables and predicted when available. joy et al. (2022) later improved this approach by separating latent variables to differentiate label characteristics from values. lo and lim (2020) modified the vae objective to incorporate external knowledge with embedding vectors, and feng et al. (2020) proposed a new elbo to incorporate label prediction loss. event modeling in recent years, much research has been dedicated to the challenge of modeling the sequence of events @xcite @xcite . one such contribution was made by @xcite , who introduced a tensor-based composition to effectively capture semantic event relations. @xcite proposed a self-supervised contrastive learning approach based on co-occurring events. another approach suggested by @xcite involves the use of recurrent neural networks (rnns) to model the hierarchical latent structures that exist in sequences of events. this methodology was further developed in a subsequent study by @xcite , where external knowledge was incorporated into the latent layer for enhanced modeling, which is highly relevant to our proposed approach. ## conclusion we show how to incorporate noisy partially observed side knowledge source along with latent variables. to do so, we generalized the main idea of parameter injection and maximizing the mutual information between external knowledge and latent variables. our experiments show that our approach is more robust to noisy knowledge and outperform other baselines for the event modeling task this material is based in part upon work supported by the national science foundation under grant no. iis-2024878. some experiments were conducted on the umbc hpcf, supported by the national science foundation under grant no. cns-1920079. this material is also based on research that is in part supported by the army research laboratory, grant no. w911nf2120076, and by the air force research laboratory (afrl), darpa, for the kairos program under agreement number fa8750-19-2-1003. the u.s. government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright notation thereon. the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either express or implied, of the air force research laboratory (afrl), darpa, or the u.s. government.