On the Utility of Pretraining Language Models on Synthetic Data (2024)

Abstract: Development of pre-trained language models has predominantly relied on large amounts of datasets. However, this dependence on abundant data has limited the applicability of these models in low-resource settings. In this work, we investigate the utility of exploiting synthetic datasets acquired from different sources to pre-train language models for Arabic. Namely, we leverage data derived based on four different methods: optical character recognition (OCR), automatic speech recognition (ASR), machine translation (MT), and generative language models. We use these datasets to pre-train models in three different architectures: encoder-only (BERTBase), encoder-decoder (T5), and decoder-only (GPT-2). We test the capabilities of resulting models on Arabic natural language understanding (NLU) tasks using the ORCA benchmark. Our results show that utilizing synthetic data can achieve performance comparable to, or even surpassing, those trained on gold data. For example, our model based on a GPT-2 architecture trained on a combined synthetic dataset surpasses the baseline model ARBERTv2. Overall, our models pre-trained on synthetic data demonstrate robust performance across various tasks. This highlights the potential of synthetic datasets in augmenting language model training in low-resource settings.

## introduction the quality and quantity of training data significantly affect model performance in many natural language processing (nlp) tasks. often, large amounts of training data are essential for neural models to achieve strong performances across a wide range of tasks @xcite . in areas such as natural language understanding (nlu) @xcite and neural machine translation (nmt) @xcite consistent improvements with increased training datasets have been shown, underscoring the significance of abundant training data. however, this becomes particularly challenging in low-resource settings where preparing extensive annotated data is costly, time-consuming, and infeasible to relabel every example whenever new data comes in. this highlights the need for research towards learning with limited labeled data for various nlp tasks @xcite . there has been growing interest in data augmentation (da) strategies to generate new data by modifying existing data through transformations @xcite . moreover, with the recent surge of pre-trained large language models (llms) and multimodal large language models (mllms), capabilities in generating data across domains such as images, audio, and text have expanded @xcite . renowned generative models like chatgpt (for text) @xcite and stable diffusion (for images) @xcite are now readily accessible via apis, making the generation of high-quality synthetic data across various domains more accessible @xcite . in this study, we investigate how training data from various sources and methods affect the performance of pre-trained language models (plms) on arabic nlu tasks on the orca benchmark @xcite . we primarily focus on the arabic language due to its complex grammar, rich morphology, and limited resources, which present significant challenges. we compare the performance of three different model architectures: encoder-only (bert base architecture), encoderdecoder (t5 architecture), and decoder-only (gpt-2 architecture), trained on data extracted through automatic speech recognition (asr), optical character recognition (ocr), machine translation (mt), and text generation (tg). this evaluation aims to determine whether synthetically generated text data can match or potentially replace gold standard data. our contributions in this paper are as follows: multimodal augmentation. multimodal augmentation (ma) leverages generative advancements in ai to synthesize data across multiple domains, including image, audio, text, and video (bewersdorff et al., 2024). these techniques either generate entirely new content or modify existing content to maintain semantic relationships, similar to how traditional data augmentation (da) techniques adapt training examples (liu et al., 2022). the notable growth and results from publicly released generative models have established ma as a critical component in numerous research sectors, particularly those involving multimodal data (alemohammad et al., 2023). ma has demonstrated significant effectiveness in various domains, such as developing sota image models like stable diffusion using the synthetic laion-5b dataset (schuhmann et al., 2022). in the vision domain, innovations like mixgen excel in joint data augmentation by combining images and text to create new image-text ensembles @xcite . in the audio domain, advancements such as audiolm push the boundaries of speech synthesis, capturing the unique voice and nuances of previously unheard speakers @xcite . these achievements highlight the convergence of self-supervised representation learning and language modeling, pointing to a promising future for the field @xcite . arabic nlp. arabic natural language processing (nlp) faces significant challenges due to its characteristics as a low-resource language, which has historically hindered achieving sota results due to data scarcity @xcite . despite this, there have been multiple efforts to address these challenges. for instance, the kind dataset collects nuanced dialectal data through social collaboration @xcite . similarly, the arquad dataset, a comprehensive, expert-annotated corpus designed for arabic machine reading comprehension, emphasizes the urgent need for substantial, high-quality datasets to advance arabic nlp @xcite . furthermore, recent efforts in specific tasks such as grammatical error correction (gec) @xcite and task classification @xcite 3 datasets & experimental setup ## conclusion in this work, we highlight the importance of leveraging different data extraction techniques for the augmentation and development of lms. our results show noticeable benefits from leveraging different sources of text to augment the performance of lms. our findings indicate that models trained on diverse synthetic data sources can achieve performance comparable to those trained on gold data. specifically, the diverse domain sources in asr data and the informal content included in such datasets significantly enhance model capabilities in various tasks. while mt data enable lower performance due to its translation-based generation, tg data demonstrate improved results due to its novelty and the robust language model used in its creation. our work, emphasizes the value of incorporating a variety of data sources to improve the effectiveness of language models in handling different linguistic tasks.