Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis (2024)

Abstract: Training LLMs in low resources languages usually utilizes machine translation (MT) data augmentation from English language. However, translation brings a number of challenges: there are large costs attached to translating and curating huge amounts of content with high-end machine translation solutions; the translated content carries over cultural biases; and if the translation is not faithful and accurate, the quality of the data degrades causing issues in the trained model. In this work, we investigate the role of translation and synthetic data in training language models. We translate TinyStories, a dataset of 2.2M short stories for 3-4 year old children, from English to Arabic using the open NLLB-3B MT model. We train a number of story generation models of size 1M-33M parameters using this data. We identify a number of quality and task-specific issues in the resulting models. To rectify these issues, we further pre-train the models with a small dataset of synthesized high-quality stories generated by a capable LLM in Arabic, representing 1% of the original training data. We show, using GPT-4 as a judge and dictionary learning analysis from mechanistic interpretability, that the suggested approach is a practical means to resolve some of the translation pitfalls. We illustrate the improvement through case studies of linguistic and cultural bias issues.

## introduction large language models (llms) have transformed the landscape of natural language processing (nlp) significantly. with the realization of unprecedented capabilities, the availability of under-figure 1 : the proposed tinystories arabic dataset is formed by translating 2m tiny stories from english to arabic using nllb-3b and synthesizing 20k arabic tiny stories using command r+ llm. the former data is used to pre-train small language models (slms) with different architectures. the latter is used for continual pre-training. the models are qualitatively and quantitatively evaluated using a gpt judge. further we train sparse auto-encoder (sae) on a selected slm to analyze the model behavior. however, newer small models have been recently shown to compete with 10x bigger models in challenging tasks. examples include min-ima @xcite and the phi family @xcite . these small language models (slms) offer faster training and lower deployment cost at the expense of being more task-specific and less general-use, which is not an issue for application-oriented language models. the key ingredient for the success of these smaller models seems to be the use of sophisticated and aggressive data curation, and highquality synthetic data generated by bigger llms. this new trend, however, does not map equally to languages that are not as privileged with huge amounts of high quality content (or content in quantities that allow aggressive filtering), nor the availability of strong models that can be employed to generate diverse synthetic data in substantial quantities in a cost-effective manner, as is the situation with arabic and many other low-resourced languages @xcite . a commonly-adopted workaround to the issue of data shortage is to turn to machine translation (mt) to benefit from the available content in english, which is evident in the data mixtures of the more capable arabic models, e.g. jais @xcite . however, the use of machine translation does not come without pitfalls. specifically, 1) cultural biases that are stored within a language corpus get imported when translation is used, leading to misaligned models, e.g. @xcite and 2) based on the quality of the translation, certain linguistic intricacies of languages might not be respected @xcite , leading to degradation in the quality of the data and thus degradation in the capability of the final model with regard to the quality of the target language. in this work, we study this phenomenon using the recently-released tinystories @xcite , a synthetic dataset introduced to explore emergent properties in small language models. tinystories comprises 2.2m short stories in english of about 200 words generated by gpt-3.5 and gpt-4. the relatively small models trained on tinystories have shown interesting capabilities in generating coherent and creative short stories with correct grammar. we translate tinystories to arabic using the open-source translation model nllb-3b to simulate medium-quality translation. we train models with different sizes using the arabic-translated tinystories, and benchmark the trained models against several other arabic llms using gpt-4 as a judge following previous works @xcite in the task of story generation using three metrics: grammar correctness, consistency with the provided context, and creativity. we also identify some cultural and linguistic issues that arise from the translation. then, we synthesize a high quality small dataset of 20k stories to explore the efficacy of continual pre-training in recovering from the issues brought along by low-quality translated data. the contributions of this work are as follow: the rest of the paper is organized as follows: first, we describe how we prepare training data. then we discuss the limitations of translated data. in section 3.1, we show how continual pre-training with small amount of high-quality improve the models. ## navigating pitfalls of translated data there are issues faced when we train models with medium-quality translation: 1) cultural biases: translated data carries over the culture of the source data into the target language, which might not be desirable. we choose in this work one example from this category to focus on, which is that the translated stories come with english person names, leading to an arabic model that can only generate stories with such names; 2) grammatical and style issues: languages differ in how they express the same sentence, and weaker translations might fail to correct for nuanced style issues. we choose a single example related to how the speaker could follow the saying in english but not in arabic. for example, the direct arabic translation of the english sentence "'i'm happy!', said tim." to is not appropriate. ## conclusion in this work, we investigate the role of machine translated data in training language models for story generation in arabic. we identify linguistic and cultural bias issues in the translated data. we propose to address these issues by further pre-training the models with a small amount of high-quality synthetic data. we investigate the effects of this intervention via dictionary learning tools. the trained sparse auto-encoders shows a shift in the learned features towards corrected linguistic properties and reduced cultural bias.