Better Language Model with Hypernym Class Prediction (2022)

Abstract: Class-based language models (LMs) have been long devised to address context sparsity in n -gram LMs. In this study, we revisit this approach in the context of neural LMs. We hypothesize that class-based prediction leads to an implicit context aggregation for similar words and thus can improve generalization for rare words. We map words that have a common WordNet hypernym to the same class and train large neural LMs by gradually annealing from predicting the class to token prediction during training. Empirically, this curriculum learning strategy consistently improves perplexity over various large, highly-performant state-of-the-art Transformer-based models on two datasets, WikiText-103 and ARXIV. Our analysis shows that the performance improvement is achieved without sacrificing performance on rare words. Finally, we document other attempts that failed to yield empirical gains, and discuss future directions for the adoption of class-based LMs on a larger scale.

## introduction over the course of the past decades, language modeling (lm) has transitioned from n-gram to neural models @xcite @xcite . performance improvement of today's neural lms is often achieved at the cost of increased computational resources. for example, to capture long-term dependencies, various extensions of transformerbased lms have been proposed @xcite . these modifications bring about significant improvements on held-out perplexity, but training cost also increases significantly due to large gpu memory consumption and more computations at each training step. in parallel, alternative training strategies have also been proposed @xcite a final torch used to enter empire stadium that was made of stainless steel and powered by a magnesium candle original text: replaced with hypernym class: a final instrumentality.n.03 used to enter empire structure.n.01 that was made of alloy.n.01 alloy.n.01 and powered by a metallic_element.n.01 instrumentality.n.03 figure 1 : an example of word prediction training text and hypernym class prediction training text. and rush, 2019; @xcite . in this paper, we explore the effectiveness of class-based language models @xcite in the context of neural lms. clms group individual words into coarser-grained classes and has proven effective in alleviating context sparsity in n-gram lms @xcite . it has been also used to improve computational efficiency in neural lms @xcite . more recently, @xcite pretrain masked lms @xcite by predicting wordnet supersense labels. however, the work focuses on word-sense disambiguation tasks and doesn't provide clear evidence of gains in terms of perplexity. in this paper, we revisit clm and assign words to classes by leveraging hypernym relations from the wordnet @xcite . our proposal, dubbed hypernym class prediction (hcp) is simple and effective: for each batch, we substitute a subset of the tokens with their wordnet hypernyms (see figure 1 ). then, we train an autoregressive lm on the resulting sentences using a mixed vocabulary composed of hypernyms and tokens. crucially, we anneal the substitution rate during training, i.e., we gently switch from hypernym prediction to token prediction, following a curriculum learning approach. note that this approach does not require wordnet information at inference time nor increases training time. our approach is motivated by two hypotheses. firstly, mapping words to their hypernyms gives rise to a natural gradation of difficulty in the prediction task. prior work has shown that lm benefits from training on instances of increasing difficulty @xcite . we thus postulate that, when coupled with the right curriculum, hcp can improve lm training and perplexity. secondly, we hypothesize that hcp can improve rare word generalization through implicit context sharing. neural models still struggle to learn reliable representations for rare words @xcite . with clm-based models, data sparsity for rare words can be abated, e.g., when the representation of their contexts are potentially drawn closer to those of their more frequent siblings by way of label (hypernym) sharing. empirically, the proposed method consistently yields about 0.6-1.9% relative reduction in perplexity over baselines on the wikitext-103 dataset @xcite , and 1.3-3.1% on the arxiv dataset @xcite . these improvements are observed with respect to memory-augmented @xcite and segmentaware @xcite lms. importantly, the proposed method improves performance for both rare and frequent words. we also observe that this is in contrast with performance improvements in regular lms, which seem to be achieved at the cost of worsened performance on rare words. to the best of our knowledge, this is the first work that shows how perplexity of transformer lms can be improved by leveraging hypernymy relationships. we provide an extensive ablation study highlighting crucial elements of hcp. amongst those, we found particularly important to adopt a curriculum learning approach, rather than multiobjective learning or adaptive-softmax, and excluding frequent words from the hypernym prediction task. we highlight the simplicity and effectiveness of the proposed method as our main contribution, and hope this study would facilitate further exploration in this line of research. ## related work transformer-based models are now popular language models. @xcite propose transformer-xl by extending the vanilla transformer @xcite with a memory segment, which can encode more context tokens to predict the next token. rae et al. (2020) extend transformer-xl with a compressed memory segment to further encode long-time context memory. other works explore different sparse transformers to encode much longer sequences for lm @xcite . @xcite propose a segment-aware transformer (segatron) to encode more positional information for language modeling. despite their effectiveness, neural models still struggle to learn reliable representations for rare words. some approaches have been proposed to tackle this challenge by way of morphology @xcite , lexical similarity @xcite , context similarity @xcite and tokenization @xcite . in addition to the model modifications, other work investigated curriculum learning to train lms. @xcite first find that curriculum learning could benefit lm training by training with highfrequency tokens first and low-frequency tokens later. @xcite find that curricula works well when the training data is noisy or the training data is too large to iterate multiple epochs. @xcite find that training transformer-based lms with short sequences first could improve convergence speed and perplexity. related work aimed at integrating wordnet information into pretrained language models. @xcite propose sensebert by adding the word sense (wordnet supersense) prediction as an additional task during bert @xcite pre-training. sensebert outperforms bert on both word supersense disambiguation @xcite task and word in context (pilehvar and camacho-collados, 2019) task. recently, @xcite use wordnet hypernymy chains as input to a roberta @xcite model to predict the plausibility of input events. in this work, our focus is to improve performance of auto-regressive lms. we show that a multi-task strategy harms performance in this setting, and give a successful recipe to consistently boost lm performance with class-based predictions. ## method coupling class-based lm (clm) and curriculum learning, hcp is to gradually anneal class prediction to token prediction during lm training. in this section, we first describe how we instantiate word classes by leveraging hypernym relation from the entity.n.01 physical_entity.n.01 matter.n.03 substance.n.01 chemical_element.n.01 abstraction.n.06 relation.n.01 part.n.01 entity.n.01 physical_entity.n.01 matter.n.03 substance.n.01 chemical_element.n.01 iron.n.01 abstraction.n.06 relation.n.01 part.n.01 metallic_element.n.01 entity.n.01 physical_entity.n.01 object.n.01 whole.n.02 artifact.n.01 instrumentality.n.03 furnishing.n.02 furniture.n.01 table.n.02 desk.n.01 magnesium.n.01 metallic_element.n.01 figure 2 : hypernym-paths of synsets "magnesium.n.01", "iron.n.01", and "desk.n.01", corresponding to the word magnesium, iron, and desk respectively. if @xmath0: continue for synset in wordnet.synsets( token ): for path in synset.hypernym_paths(): if len(path)>=d and noun in path[d-1]: rtn[token] = path[d-1] break if token in rtn: break return rtn code 1: pseudocode for token to class mapping. wordnet. we then present how to incorporate the proposed hypernym class prediction task into lm training via curriculum learning. ## experiments we conduct experiments on two datasets. wikitext-103 @xcite is a large wordlevel dataset with long-distance dependencies for language modeling. there are 103m tokens and 28k articles (3.6k tokens per article on average). the original vocabulary size is 271121, among which we find 3383 hypernym classes for 71567 tokens with @xmath1.1). arxiv @xcite is collected from publicly available arxiv abstracts foot_0 with an average of 172 words per abstract and partitioned into training @xcite , evaluation (aug-dec 2017), and test (2018-2019). following @xcite , we use the bpe @xcite tokenization for this dataset. the final vocabulary size is 48935, and we find 1148 hypernym classes for 5969 tokens among the vocabulary with @xmath2. several variants of the transformer model have been used for our experiments: the input lengths are 150 for the base model and 384 for the large model. the memory length is equal to the input length for both training and testing. the hyper-parameters used for the arxiv dataset are as same as the wikitext-103, except the arxiv base model's input length is 384. the number of training steps varies greatly for the large model in previous work, so we experiment on both the lower (80k) higher (350k) ends. ## conclusion in this work, we propose a new lm training strategy with wordnet's super-subordinate relation and curriculum learning. although wordnet is an external resources, it's not clear how to get lower perplexity using wordnet before this work. consistent perplexity reduction can be observed over various models. both rare and frequent tokens can be modeling better with our proposed method while other optimization method may sacrifice the performance on rare tokens. we'd like to address the limitations of this work: other methods to map words to classes; lm experiments with other languages; pre-training lm with our proposed method and testing on downstream tasks. we hope to investigate these directions in the future.