B aby HGRN : Exploring RNN s for Sample-Efficient Language Modeling (2024)

Abstract: This paper explores the potential of recurrent neural networks (RNNs) and other subquadratic architectures as competitive alternatives to transformer-based models in low-resource language modeling scenarios. We utilize HGRN2 (Qin et al., 2024), a recently proposed RNN-based architecture, and comparatively evaluate its effectiveness against transformer-based baselines and other subquadratic architectures (LSTM, xLSTM, Mamba). Our experimental results show that, our HGRN2 language model, outperforms transformer-based models in both the 10M and 100M word tracks of the challenge, as measured by their performance on the BLiMP, EWoK, GLUE and BEAR benchmarks. Further, we show the positive impact of knowledge distillation. Our findings challenge the prevailing focus on transformer architectures and indicate the viability of RNN-based models, particularly in resource-constrained environments.

## introduction in recent years, natural language processing (nlp) has been revolutionized by transformer-based language models (lms), like bert @xcite or @xcite and their derivatives, achieving state-of-the-art results @xcite across a wide range of tasks such as machine translation, question answering, and text generation. however, despite their dominance, transformers come with notable limitations: they require extensive training data @xcite and enormous computational resources, which pose challenges for their use in resource-constrained environments. these limitations led to an increasing interest in more sample-efficient alternatives and approaches with lower computational requirements @xcite . the shared task of the babylm challenge @xcite systematically explores this trend by training lms on datasets of limited size (10m words in the "strict-small" and 100m words in the "strict" setup). the resulting models are then evaluated on linguistic and general language understanding tasks. while most participants in babylm challenge focus on adapting transformers to low-resource settings, we propose revisiting recurrent neural networks (rnns). once foundational to sequence modeling tasks @xcite , rnns have been largely overshadowed by transformers due to their sequential nature which does not easily allow for parallelization. potential of rnn-architectures. in this paper, we investigate whether the inductive biases of rnn architectures, such as their sequential processing and memory states, provide advantages in dataconstrained settings. this question is especially relevant given that state-of-the-art transformer models depend on quadratic self-attention, which requires calculating the inner product between all tokens. in particular, we investigate the potential of the hgrn2 @xcite , a novel subquadratic rnn-based architecture based on hierarchical gating. we train our model using knowledge distillation @xcite and evaluate our approach, babyhgrn, against state-of-the-art transformer models and other efficient rnn architectures (e.g. xlstm @xcite or mamba @xcite ). our experiments demonstrate that our resulting model yields better performance compared to both transformer-based and other rnn-based architectures. we summarize our contributions as follows: our results show that babyhgrn outperforms transformer-based baselines on both tracks of the babylm challenge. ## babyhgrn we utilize hgrn2 as our backbone architecture with a hidden size of 2048 and 18 layers, resulting in a total parameter count of 330m. we train our model either with (1) the default dataset of the babylm challenge or (2) a sub-sampled split of thepile @xcite . further, we employ knowledge distillation training using a teacherstudent setup. in the following, we will discuss the details of our design choices. ## empirical evaluation in section 3.1, we shortly present the evaluation benchmarks of the babylm challenge and the bear knowledge probe. in sections 3.2 to 3.4, we evaluate babyhgrn compared with other efficient rnn architectures, its training dynamics, and the influence of different datasets. finally, in section 3.5, we evaluate babyhgrn using knowledge distillation. ## related work in recent years, there has been a resurgence of interest in recurrent neural network (rnn) architectures for sequence modeling, particularly in the context of large language models (llms). this renewed focus has led to the development of several rnnbased architectures that aim to combine the efficiency of recurrent models with the expressiveness of more complex architectures like transformers. hgrn and hgrn2 the hierarchically gated recurrent neural network (hgrn) @xcite introduces a novel gating mechanism that allows for more effective modeling of long-term dependencies. the key innovation of hgrn is its hierarchical structure, in which forget gates have monotonically increasing lower bound values from bottom layers to upper layers. this design enables lower layers to model short-term dependencies while upper layers capture long-term relationships in the data. hgrn achieves efficient training by reformulating its recurrent computation as a parallel scan operation to enable parallelization across sequence length while maintaining linear time complexity. building upon hgrn, @xcite introduced hgrn2 which further enhances the capabilities of gated linear rnns. hgrn2 addresses some limitations of its predecessor by incorporating a state expansion mechanism. this innovation significantly increases the recurrent state size without introducing additional parameters, leading to improved expressiveness. the mamba architecture @xcite improves on state space models (ssms) by introducing selective state spaces. building on structured ssms @xcite , mamba achieves linear-time sequence processing through inputdependent ssm parameters, enabling selective information propagation across sequences. this mechanism is conceptually similar to gating in classical rnns @xcite while maintaining modern computational benefits. the architecture consists of repeated blocks that combine selective ssms with feed-forward components, in contrast to more complex predecessors like h3 @xcite and @xcite . though attention-free, mamba matches or exceeds transformer performance @xcite across various domains. its recurrent computation pattern eliminates the need for attention caches during inference, leading to 5× faster inference compared to similar-sized transformers. this combination of linear scaling and efficiency, without sacrificing model quality, makes mamba a significant development in sequence modeling. the development of hgrn2, xlstm, and mamba is part of a broader trend in revisiting and improving rnn architectures @xcite . ## conclusion we presented babyhgrn, an rnn-based language model that utilizes the hgrn2 architecture. our experimental results on the evaluation datasets of the babylm challenge and the bear probe indicate that babyhrgn is competitive. indeed, despite relatively little hyperparameter optimization, our approach significantly outperforms strong transformer-based baselines on the evaluation datasets. revisiting our research question posed in section 1, we conclude that rnn-based language models are indeed competitive in low-resource language modeling scenarios. based on these results, we believe that advanced rnn-based architectures such as hgrn and mamba may hold promise for research in sample-efficient language modeling. accordingly, future work could explore further optimizations of the underlying rnn architectures, investigate their performance on a broader range of tasks, and examine their scalability to larger datasets and model sizes.