Balaur: Language Model Pretraining with Lexical Semantic Relations (2023)

Abstract: Lexical semantic relations (LSRs) characterize meaning relationships between words and play an important role in systematic generalization on lexical inference tasks. Notably, several tasks that require knowledge of hypernymy still pose a challenge for pretrained language models (LMs) such as BERT, underscoring the need to better align their linguistic behavior with our knowledge of LSRs. In this paper, we propose Balaur, a model that addresses this challenge by modeling LSRs directly in the LM’s hidden states throughout pretraining. Motivating our approach is the hypothesis that the internal representations of LMs can provide an interface to their observable linguistic behavior, and that by controlling one we can influence the other. We validate our hypothesis and demonstrate that Balaur generally improves the performance of large transformer-based LMs on a comprehensive set of hypernymy-informed tasks, as well as on the original LM objective. Code and data are made available at https://github.com/mirandrom/balaur

## introduction pretrained language models (lms) trained on everincreasing compute and data have achieved stateof-the-art performance on a wide variety of nlp benchmarks. however, they are still known to struggle on certain tasks, notably those involving reasoning and world knowledge @xcite . in particular, previous work has found that these models make errors on tasks involving lexical semantic relations (lsrs) such as hypernymy: notably cloze completions such as "a fox is a type of ", where the hypernym "canine" is a valid completion @xcite ; and monotonicity-based inferences such as "bob saw a fox" entailing "bob saw a canine" @xcite . this represents an important limitation of current lms. besides the specific classes of inferences in which they are involved, lsrs are crucial because token inputs screen lm vocab embeddings dog ##ja residents balaur concept embeddings (age_group) (animal) (canine) (corgi) token prediction objective dog [mask] screen ##ja residents lm encoder contextual embeddings balaur heads transformed embeddings balaur constraints dog (canine) (animal) fox young old (age_group) hypernymy vector space young (aged) old (young) antonymy vector space [mask] (corgi) (puppy) hyponymy vector space language model balaur they provide a dimension along which lms can perform class-based generalization to new events in a sample-efficient manner. for example, knowing that a xoloitzcuintli is a dog breed allows a model to infer many plausible properties about them, even without many occurrences of this word in training. this improved modelling of lsrs could in turn improve generalization and data efficiency in lms. we propose balaur (figure 1 ), an approach to transformer lm pretraining which directly models lsrs in the latent representations of the lm. balaur consists of a modular neural architecture with multiple heads, each head modeling a distinct lsr (e.g. hypernymy, synonymy, or antonymy). each lsr is modelled as a learned vector transformation, with constraints injected in the resulting vector space to reflect structural properties of the lsr. concretely, a balaur head transforms contextual token embeddings (e.g. fox) and static concept embeddings (e.g. (canine)) such that related pairs are similar in the corresponding lsr vector space (e.g. hypernymy). previous work has proposed to integrate lsrs with lms, for example augmenting lm pretraining with hypernym class prediction @xcite . however, these approaches do not encode known structural properties of lsrs, modeling these indirectly without accounting for e.g. hypernymy being transitive, or antisymmetric with hyponymy. in contrast, our method draws inspiration from work on semantic specialization, which models lsrs as constraints in the vector space of static word embeddings (e.g. @xcite . we show how similar insights can be applied to modern transformer lm architectures such that the resulting lm's hidden states have inductive biases useful for lsr-informed tasks. more broadly, we demonstrate that our method creates an interface between the model's latent representations and its linguistic behaviour, and that we can control the latter by injecting lsr information into the former. we evaluate balaur on several tasks which require knowledge of hypernymy. these include cloze completion, monotonicity-informed natural language inference (nli) , and finetuning efficiency on the previous two tasks in a transfer learning setting. we find that balaur generally improves performance on these hypernymy-informed tasks, as well as the on the original lm objective. ## the balaur head architecture in this section, we present the neural architecture of balaur heads, describing how they model lsrs in the hidden states of lms, how this is translated into an optimizable loss function, and how they interface with lms during pretraining. assumptions we take the term (neural) language model (lm) to refer to a neural network that predicts a token given its context, including commonly used masked lms like bert @xcite and auto-regressive lms like gpt @xcite . these lms encode a sequence of tokens into latent representations known as hidden states or contextualized token embeddings t , using these as inputs to a classification head that predicts the target token. meanwhile, we represent a lexical semantic relation r as a set of related lexical item pairs (x @xmath0, noting that lsrs can be directed, e.g. (@xmath1. ## method in this section, we detail our methods for lm pretraining with balaur, using lsrs and concepts extracted from wordnet. we also present the architecture and hyperparameters for the lm in our experiments, a variant of bert large suitable for academic budgets. while our experiments are limited to masked language modeling and lsrs, our method can be extended to autoregressive language modeling and other forms of relational knowledge. ## conclusion in this work, we set out to align the linguistic behavior of lms with our knowledge of lsrs and improve their performance on hypernymy-informed tasks. we presented balaur, an approach that aims to guide the linguistic behavior of lms in such a way by modeling lsrs directly in their hidden states throughout pretraining. underlying this proposed approach was the hypothesis that lm latent representations can provide an interface to their linguistic behavior, and that controlling one can help guide the other. to verify our hypothesis, we characterized the effect of bal-aur on a series of evaluations, targeting several distinct ways in which lms might capture hypernymy in their linguistic behavior. our findings show that balaur can robustly improve performance on diverse hypernymyinformed tasks, validating the effectiveness of our method while supporting our original hypothesis. notably, we demonstrated that balaur also improves performance on the original language modeling objective, indicating our method's improvements are not limited to hypernymy-informed tasks and can extend to more general linguistic behavior. however, we found that aligning the linguistic behavior of lms with balaur still poses several challenges. we further discuss these limitations in §7 and outline implications for future work. more broadly, balaur is a general-purpose architecture for modeling relations in the latent representations of neural network models. while our work has focused on modeling lsrs in lm hidden states throughout pretraining, balaur can in principle be applied to different modalities, architectures, relations and optimization settings. how well our results and hypothesis generalize to such different settings remains an open question.