The Importance of Context in Very Low Resource Language Modeling (2021)

Abstract: This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low-resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model’s performance in the low-resource setting, finding that limiting the model’s self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.

## introduction with the advent of the transformer (vaswani et al., 2017) and masked language model (mlm) pretraining (devlin et al., 2018), attention-based neural networks have proven quite effective at a variety of language tasks, provided that large amounts of data are available for pretraining. however, the performance can drop significantly as the number of sentences used for mlm pretraining decreases. this poses an issue for low-resource settings such as for underrepresented languages, where there is a limited amount of monolingual data. under low-resource conditions, attention-based models have difficulty learning from mlm, and as such statistical language models (slms) can outperform neural language models (nlms). we demonstrate this by using a popular slm toolkit, kenlm (heafield, 2011), and test its accuracy on the mlm task compared to that of a transformer model. @xcite the results ( ## methods when comparing the general function of an slm to an nlm, we consider the largest difference to be the context size considered. a tri-gram slm will consider only the context of the adjacent two words on either side. for example, the score we use for word c in the sequence a b c d e f g is log(p(c|a, b) × p(d|b, c) × p(e|c, d)). meanwhile, self-attention allows a transformer to consider the entire context, which in xlm is 256 tokens by default (lample and conneau, 2019). since xlm is trained with continuous streams of text, the input size is therefore always 256, and can consist of multiple sentences. in the low-resource setting, it may be difficult to learn important features from such a large context size. to tackle this, we consider three alternative approaches. first, we put the strictest limitation on context by limiting the length of the input ( § 2.1). second, we use a limited attention scope, thereby limiting the context within the first layer of the transformer, but allowing information to flow from larger contexts in subsequent layers ( § 2.2). finally, we put no explicit restriction on context size, but rather we expect the model to learn to limit itself via distillation from the limited statistical model ( § 2.3). if context size is indeed the issue, we would expect the strictest form of limitation to perform best, as it would not need to learn to limit itself during training. this may be however too limiting for tasks which require a larger context, where we would expect that limiting attention would perform best. if context size is not the issue, we would expect that distilling knowledge from the statistical model would perform best, as its context is not limited, and the statistical model would still help the neural model learn a better strategy for language modelling than it is capable of on its own. ## experimental setup we test each of our three methods on english (en), hindi (hi), and turkish (tr). we train our models with xlm (lample and conneau, 2019), starting from a random initialization. we use the first 10 or 40 thousand sentences per language foot_4 from the wmt2007 newscrawl for english (following xlm), wmt2013 newscrawl for hindi, and the wmt2016 newscrawl for turkish. @xcite for all of the tests, the data is tokenized with udpipe (straka and straková, 2017), foot_6 truecased with moses (koehn et al., 2007), and 10 thousand bpe (sennrich et al., 2015) joins are used. the architecture behind our models is a 6-layer transformer with 8 attention heads, an embedding dimension size of 1024, dropout set at 0.1, and gelu (hendrycks and gimpel, 2016) activation. for pretraining, we use a batch size of 32, and the adam optimizer (kingma and ba, 2014), with a learning rate of 1e-4. we lower the learning rate to 2.5e-5 for the fine-tuning tasks. we use an early stopping criterion of no improvement in accuracy (mlm accuracy for pretraining, nli or pos tag accuracy for fine-tuning) on the validation set for 20000 iterations, with a patience of 10. 9 ## results we now compare the results of the slm, normal nlm, and our 3 improvements to the nlm: limited context (nlm-c), limited attention (nlm-a), and the hybrid training objective (nlm-h). for nlm-c and nlm-a, we experiment with different context sizes and attention window sizes, ranging from 5 to 13. the slms are trigram models, and nlm-h uses these models for its soft labels. ## conclusion despite the ubiquity of pre-trained neural language models (nlms) in state-of-the-art nlp, in the lowresource setting they are outperformed by statistical language models (slms). their general formulation assumes a large amount of data for pretraining, so in this work we adapt them to better perform in low-resource conditions. we found that the complexity of self-attention on large contexts is a major inhibitor. as a solution to this, we propose shortening the attention span (nlm-a), which we show can increase the model's performance on downstream tasks. we believe that an ideal limitation of attention span would be initially very limited, but the span would increase dynamically during training. we plan to look into this further in future work. for the best performance on mlm accuracy during pretraining itself, we propose limiting the size of the input (nlm-c), improving upon the standard method for training neural models. this achieves slm-level performance on the lowest resource setting (10 thousand sentences), and outperforms an slm on slightly higher-resource settings (40 thousand sentences). in addition, the neural model with a limited context can, unlike the slm, be transferred to downstream tasks. while limiting the input size (nlm-c) performs better than limiting the attention span (nlm-a) for pretraining, the opposite is the case for downstream tasks. as a potential solution for this, we propose for future work a second pretraining step in which the non-limited input is used. finally, our work primarily serves to investigate how attention-based models function with very little data. however in many real-world scenarios, transfer learning from large multilingual models is often used. looking at the impact of these methods with multilingual transfer learning employed alongside is something we plan to do in the future.