Learning to Abstract with Nonparametric Variational Information Bottleneck (2023)

Abstract: Learned representations at the level of characters, sub-words, words, and sentences, have each contributed to advances in understanding different NLP tasks and linguistic phenomena. However, learning textual embeddings is costly as they are tokenization specific and require different models to be trained for each level of abstraction. We introduce a novel language representation model which can learn to compress to different levels of abstraction at different layers of the same model. We apply Nonparametric Variational Information Bottleneck (NVIB) to stacked Transformer self-attention layers in the encoder, which encourages an information-theoretic compression of the representations through the model. We find that the layers within the model correspond to increasing levels of abstraction and that their representations are more linguistically informed. Finally, we show that NVIB compression results in a model which is more robust to adversarial perturbations.

## introduction learning representations of language using selfsupervision has become a cornerstone of nlp @xcite @xcite . however, these representations are specific to their tokenisation (e.g. byte-pair @xcite , wordpiece @xcite , sentencepiece @xcite , characters @xcite , and even bytes @xcite ), which restricts the level of abstraction from the input text which their representations are able to convey. work like canine @xcite and charformer @xcite avoid problems with tokenisation by modeling individual characters or bytes, and thereafter use a stride-based downsampling to reduce the representation length. the stride pattern is fixed and thus can't be considered as learning to abstract. @xcite recently introduced the task of learning a higher level of abstraction in a set-of-vector space by proposing dynamic capacity slot attention. in this work, we propose a novel character-level model of representation learning which learns different levels of abstraction in different layers of the same model. contributions we adapt the nonparametric variational information bottleneck regulariser (nvib) @xcite for application to selfattention in the stacked layers of a transformer encoder. 1 the resulting model has greater abstraction than a standard transformer due to selectively dropping some vectors in higher attention layers. interestingly, we observe that the learned abstract units are intuitive, often corresponding to words. by employing different analysis methods, we demonstrate that our model is better at encoding semantically and linguistically meaningful information than a standard transformer baseline. moreover, it exhibits an enhanced level of robustness, further consolidating its advantage. ## the model our model consists of standard transformer encoder-decoder layers @xcite , where the encoder block has been augmented with an nvib regulariser on the self-attention layers, as seen in figure 1 . ## related work modeling language at the level of characters has the advantage of providing an end-to-end framework for the models to operate, without the need for tokenization as a preprocessing step @xcite @xcite @xcite . this is at the cost of longer sequence lengths and the need for greater model depth to reach the understanding level of subword-based models. while canine @xcite and charformer @xcite are some attempts to bypass these shortcomings, they do so by fixed architectural design choices. our work differs in that it allows the model to learn how to abstract and compress the input without a hard-coded abstraction structure. our inspiration comes from @xcite who introduced the task of learning a higher level of abstraction and proposed a method based on slot attention @xcite for this purpose. our work is also related to hm-rnns @xcite as it tends to learn a hierarchy of units within its layers, though it does not make discrete decisions on unit boundaries. our approach to learning meaningful disentangled abstractions by encouraging the models to learn compressed representations through a bottleneck is shared with vaes (kingma and welling, 2014) and other work in that line @xcite . ## experiments our proposed model's abstractness is analyzed qualitatively through attention visualisations (section 4.2) and quantitatively through a challenging sub-topic classification task (section 4.3.1). each layer is probed to analyse the linguistic information captured (section 4.3) and finally we examine the models' robustness to adversarial, synthetic noise (section 4.4). we provide additional details of these experiments in the appendices f to i. ## conclusions we propose a novel method for inducing abstract representations of text. we adapt the nonparametric variational information bottleneck @xcite regulariser for application to selfattention in the stacked layers of a transformer encoder. our model learns how many vectors are needed at each layer, thereby inducing different levels of abstraction in different layers of the same model. we find that these abstract units are intuitive, more robust, and better at encoding semantically and linguistically meaningful information.