GPT or BERT : why not both? (2024)

Abstract: We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transformer stack – GPT-BERT can be transparently used like any standard causal or masked language model. We test the pretraining process that enables this flexible behavior on the BabyLM Challenge 2024. The results show that the hybrid pretraining outperforms masked-only or causal-only models. We openly release the models, training corpora and code.

## introduction language models have become fundamental tools in natural language processing, with two dominant paradigms: causal language models (clm) and masked language models (mlm). six years ago, gpt by @xcite demonstrated the generative abilities of transformer-based causal language models. just a few months after this publication, bert by @xcite heavily outperformed the causal gpt models when finetuned on downstream nlp tasks, showcasing the major advantage of masked language modeling. these two 'historical' models define the main use-cases of the two paradigms up to this date. the difference between these paradigms lies in how they process text. clms can only look at previous tokens when making predictions, mimicking the left-to-right reading process. this makes them particularly well-suited for efficient text generation. mlms, on the other hand, can access both previous and following tokens, allowing them to causal and masked language modeling can be easily unified by shifting both outputs by one token to the right. then we can train one language model on both paradigms at the same time just by modifying the input tokens, output tokens and attention masks. gpt-bert this motivated us to introduce gpt-bert, a hybrid language model that combines the strengths of both clm and mlm approaches. our key insight is that the two objectives can be unified by reformulating how output tokens are handled in the mlm framework. instead of predicting masked tokens at their original positions, we shift the predictions one position to the right, aligning them with the clm's next-token prediction pattern. this simple modification allows us to train a single model that can seamlessly switch between masked and causal modes without any architectural changes or additional parameters. this paper demonstrates the benefits of the hybrid approach across multiple benchmarks. we evaluate gpt-bert on the babylm challenge 2024 @xcite , which provides a controlled environment for comparing language models trained on limited data. additionally, we explore the impact of varying the ratio between mlm and clm, and we test the model's ability to perform in-context learning and text generation. the results suggest that integrating mlm and clm objectives during pretraining leads to more robust and capable language models, even in lowresource scenarios, without any extra training cost. our approach opens up new possibilities for developing more efficient and versatile models for a wide range of natural language processing tasks. ## pretraining and evaluation the main purpose of this section is to evaluate if the mlm and clm training objectives can be merged, and to evaluate the effect of this. we base the experiments on the babylm challenge @xcite . ## related work baby language models this paper describes a submission to the second iteration of the babylm challenge @xcite . our submission is heavily inspired by the last-year's winner, elc-bert (georges gabriel charpentier and samuel, 2023), and by its inspiration, ltg-bert @xcite . our modifications to these approaches are described in section 2.1 and section 2.2. hybrid masked-causal models our work is not the first to attempt to merge bidirectional masked language modeling with generative causal modeling: t5 @xcite , bart @xcite and glm @xcite proposed autoregressive fill-in-the-blank training objectives, cm3 is based on a causal-mask objective @xcite , prefix language models use a partiallybidirectional causal modeling @xcite , and ul2 further improves the t5 encoder-decoder with more training objectives @xcite . our approach differs by its simplicity -not requiring any architectural changes nor novel training objectives -it just combines a standard causal language model with a (shifted) masked language model; the resulting hybrid can then be used as any gpt-like or bert-like model out-of-the-box. ## conclusion we introduced gpt-bert, a novel approach that unifies masked and causal language modeling objectives within a single transformer architecture. through extensive experiments on the babylm challenge 2024, we demonstrated that this hybrid approach offers several key advantages over singleobjective models: our results suggest that the traditional dichotomy between mlm and clm architectures may be unnecessary, and that future work might benefit from exploring more unified approaches to language model pretraining.