BRENT : Bidirectional Retrieval Enhanced N orwegian Transformer (2023)

Abstract: Retrieval-based language models are increasingly employed in question-answering tasks. These models search in a corpus of documents for relevant information instead of having all factual knowledge stored in its parameters, thereby enhancing efficiency, transparency, and adaptability. We develop the first Norwegian retrieval-based model by adapting the REALM framework and evaluate it on various tasks. After training, we also separate the language model, which we call the reader , from the retriever components, and show that this can be fine-tuned on a range of downstream tasks. Results show that retrieval augmented language modeling improves the reader’s performance on extractive question-answering, suggesting that this type of training improves language models’ general ability to use context and that this does not happen at the expense of other abilities such as part-of-speech tagging, dependency parsing, named entity recognition, and lemmatization. Code, trained models, and data are made publicly available.

## introduction retrieval-based language models meet some important shortcomings associated with pre-trained language models (plms): they are more dynamic, allowing for updating of knowledge without having to re-train the model from scratch; they are more transparent, allowing backtracking the source of returned statements; and they are more efficient, as retrieval provides a non-parametric memory. the accentuated benefit of these models has been the ## related work the basic setup for most retrieval-based approaches to nlp is that for a query q, be it a question for qa or a premise in natural language inference, the model must retrieve a set of passages relevant to q. relevant candidates are then typically appended to q before being passed to a classification layer. while earlier work approached this using heuristics and sparse retrieval methods like bm25 @xcite , recent work has focused on learning this retrieval step. most of these use an architecture with an encoder and a reader: the encoder learns to represent q and the retrieved passages in a representation space that makes it possible to match documents using the inner product operation, while the reader learns how to utilize the retrieved passage for downstream prediction. recent work by @xcite shows how it is also possible to model this interaction using a single transformer model @xcite with a retrieval as attention technique, as compared to having separate encoders and readers. lee et al. (2019) note that it is computationally impractical to learn to make predictions conditioned on a retrieval corpus from scratch and thus proposed to pre-train the encoders with an inverse cloze task (ict) in order to "prime" the model for retrieval. this is also done in @xcite . we outline more details on this and how we use ict in the following section. the most direct application of retrieval is to use a supervision signal such as openqa to train the context and passage encoders, such as in @xcite . however, @xcite show how this setup can also be used for language modeling. using english wikipedia as the retrieval corpus, they perform mlm conditioned on retrieved passages. passages are retrieved using mips over an index that is asynchronously updated during training. they also use a masking technique that prioritizes named entities in order to incentivize the usage of world knowledge from the retrieved passages. a similar approach to language modeling is also done in @xcite , but over a corpus consisting of trillions of tokens. for both works, the lms are trained for a number of steps with retrieval before being fine-tuned on a downstream task, as is the typical workflow with plms. @xcite demonstrate how the encoder-reader architecture can be used for language generation as well. they propose both a sequence model where the generation is conditioned on the same set of retrieved documents for the entire sequence and a token model where a different document is used per target token. the retriever is based on dense passage retrieval (dpr) @xcite , which uses the same general approach to retrieval as @xcite , where a plm like bert @xcite is used as the encoder. the reader model is swapped with a generator based on bart @xcite . ## method as in @xcite , the architecture of brent can be separated into two parts: a retriever and a reader. our architecture is modified to improve the speed of training, to ensure that the retrieved documents affect the predictions, and to incentivize the retrieval of world knowledge from the retrieval corpus instead of the reader memorizing it. this section puts forth the architecture and these modifications. ## experiments we evaluate brent on a wide range of norwegian nlp tasks. we do this both without retrieval using the extracted reader, and with retrieval turned on using the full model. by doing this, we highlight both the improved capacity of the reader to use context and show how retrieval in general affects performance on nlp tasks other than qa. this section describes the specific datasets and models we use during experimentation. ## ablations as with other retrieval-augmented lms, brent is a pipeline model -consisting of multiple parts that interact according to a series of design choices that impact the outcome. due to the computational cost of pre-training, it is not feasible to quantitatively determine the effect of all these choices, resulting in a poor understanding of some aspects of these models. to mitigate this, we study the effect of some of these choices with respect to the overall loss during pre-training with a series of ablations. we do this for a reduced number of steps, but with 208 the values of all three runs are smoothed with an exponential moving average, using α = 0.99. the same retrieval corpus and with the same gpu setup as described in section 3.3 and section 4.3. ## conclusion we develop the first norwegian retrieval augmented language model, brent, based on the realm method proposed by @xcite . the model uses an encoder-reader architecture, and we train it on a relatively small corpus consisting of 735k wikipedia documents. in addition to the model itself, our contribution has been to demonstrate how the use of continued pre-training with retrieval benefits the context utilization of the reader, which we extract from the pipeline. the reader performs better than comparable baselines on the extractive qa task without losing performance on morpho-syntactic tasks. we also evaluate our full retriever model on sentiment analysis with two different corpora as the retrieval corpus, but here we observe a decrease in performance overall. contrary to some previous work, our ablation studies find that the effect of having a null document and using ict as a warmup task is minimal.