A nt LM : Bridging Causal and Masked Language Models (2024)

Abstract: Causal Language Modeling (CLM) and Masked Language Modeling (MLM) are two mainstream learning paradigms based on Transformer networks, specifically the Decoder-only and Encoder-only architectures. The strengths of each paradigm in downstream tasks have shown a mix of advantages and disadvantages. In the past BabyLM Challenge 2023, although the MLM paradigm achieved the best average performance, the CLM paradigm demonstrated significantly faster convergence rates. For the BabyLM Challenge 2024, we propose a novel language modeling paradigm named \textbf{AntLM} , which integrates both CLM and MLM to leverage the advantages of these two classic paradigms. We chose the strict-small track and conducted experiments on two foundation models: BabyLlama, representing CLM, and LTG-BERT, representing MLM. During the training process for specific foundation models, we alternate between applying CLM or MLM training objectives and causal or bidirectional attention masks. Experimental results show that combining the two pretraining objectives leverages their strengths, enhancing overall training performance. Under the same epochs, AntLM_{BabyLlama} improves Macro-average by 1%, and AntLM_{LTG-BERT} achieves a 2.2% increase over the baselines.

## introduction language modeling (lm) is a core task in nlp and a key technology for natural language understanding and generation, supporting a wide range of applications including machine translation @xcite , speech recognition @xcite , sentiment analysis @xcite , and information extraction @xcite . over the past decades, lm has seen significant development, evolving from simple models like n-grams @xcite to more sophisticated models, such as recurrent neural networks @xcite , long short-term memory networks @xcite , and more recently, transformer-based large language models (llms) like gpt @xcite and bert @xcite . llms have demonstrated human-like or even superhuman performance in language modeling. however, the tremendous success of llms relies on learning from massive corpora, which is not as data-efficient and low-energy as human language learning. the babylm challenge 2023 @xcite and 2024 @xcite is a shared task over two consecutive years. it aims to encourage the discovery of more effective methods for training models using limited data. considering that a 13-year-old child has encountered fewer than 100 million words in their lifetime, the shared task has introduced the strict-small track foot_0 . these tracks confine pre-training data to 10 million and 100 million words. these datasets consist of child-accessible materials, such as books, conversations, and wikipedia entries, to enhance the relevance of language model pre-training to human language learning processes. compared to 2023, the 2024 competition removed the children's book test @xcite and qcri educational domain corpus datasets @xcite . the 2024 competition also reduced the proportion of opensubtitles @xcite dataset while increasing the proportions of childes @xcite and project gutenberg @xcite ) datasets. the current investigation of lms primarily adopts two predominant modeling paradigms: causal language models (clms), represented by gpt @xcite , and masked language models (mlms), represented by bert @xcite . clms employ next-token prediction as their training objective, which is predicting the next token given the preceding context, and they perform exceptionally well on generative tasks. on the other hand, the training objective of mlm is the random selection and masking of some tokens in the input text, following which the model is trained to predict the original unmasked tokens. due to its global information modeling capabilities, this approach excels in tasks necessitating the capture of bidirectional contextual information, such as text classification. considering these modeling paradigms' strengths, this paper raises an important question: could the two modeling methodologies be seamlessly integrated? intuitively, performing the mlm task allows the model to learn bidirectional contextual encoding of text, while the clm task enables the model to predict and generate text based on prior content. these two learning objectives are not in conflict and could potentially be integrated. analogous to a child learning a new language via practicing both cloze exercises and writing assignments, the training mechanism for a model can similarly employ a multi-task strategy. therefore, we consider enabling our model to learn both tasks concurrently. to achieve this, we adopt a unified model architecture and alternate the training objective between mlm and clm tasks. this approach attempts to mimic the human learning process, hence helping the model acquire deeper knowledge from a limited amount of text data. to examine the effect of integrating mlm and clm pretraining tasks on model performance, we conducted experiments using ltg-bert and babyllama foot_1 as base models, testing on the babylm2024 10m datasets. ltg-bert, an encoder-only model, and babyllama, a decoderonly model, are notable architectures from the 2023 babylm challenge .the results indicate that both ltg-bert and babyllama showed improvements in macroaverage scores. these experiments confirm that the integration of these two pretraining objectives can positively impact model training. masked language model is a training approach used to develop deep bidirectional representations of context, often referred to as a cloze task @xcite . specifically, a special token [mask] is employed to randomly mask a proportion of input tokens, and the model is trained to predict these masked tokens. this training task was first innovatively introduced in bert @xcite and has been adopted in subsequent models like roberta @xcite and albert @xcite . research has also led to improvements in mlm tasks, such as in spanbert @xcite , where the model is trained to predict spans of words instead of individual tokens, enhancing its ability to capture long-range dependencies. unified modeling refers to using a single model architecture to handle multiple training and evaluation tasks. in the t5 @xcite ) model, various downstream tasks were reformulated as textto-text tasks, significantly enhancing the model's ability for multitask learning. moreover, many related works @xcite have also applied unified modeling for multitask training and evaluation, making it a common approach to improve the generalization ability of models. unilm @xcite , based on the bert architecture, is one of the significant endeavors in unified modeling. by employing specific self-attention masks, unilm controls the contextual information used during prediction. when predicting tokens, it not only trains like an autoencoding language model by leveraging the context of masked tokens but also performs left-to-right training like an autoregressive language model. additionally, unilm can function similarly to encoder-decoder architectures by encoding the first input text and then generating sequences from left to right. by switching the attention matrix, it seamlessly transitions between different training tasks and downstream application scenarios. existing methods have unified clm and mlm networks regarding model architecture and parameter sharing. however, research on unifying their training objectives remains unexplored. this paper is the first to bridge the two classic training objectives. ## conclusion in this study, we propose antlm, a model that applies to multiple natural language-related tasks in the babylm challenge by alternating between causal language modeling (clm) and masked language modeling (mlm) during training. experimental results demonstrate that antlm achieves either superior or comparable performance to the baseline across all evaluation tasks. additionally, we found that clm and mlm have different impacts on various evaluation tasks, suggesting that these training tasks guide the model 6 329 to learn distinct aspects of human language. we believe this difference is the key reason why integrated training yields effective results, as the model benefits from the knowledge learned from both training approaches. this finding also raises an intriguing question: do different training tasks allow models to capture only specific portions of natural language knowledge? due to resource limitations, we were unable to explore additional ideas and approaches in this study. in future work, we plan to address these limitations by expanding our resources and support, allowing us to further investigate these potential directions. moreover, we conducted experiments with varying numbers and sequences of alternating training, and the results suggest that specific integrated training methods are more effective in achieving optimal evaluation outcomes.