From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes (2024)

Abstract: Language models are typically trained on large corpora of text in their default orthographic form. However, this is not the only option; representing data as streams of phonemes can offer unique advantages, from deeper insights into phonological language acquisition to improved performance on sound-based tasks. The challenge lies in evaluating the impact of phoneme-based training, as most benchmarks are also orthographic. To address this, we develop a pipeline to convert text datasets into a continuous stream of phonemes. We apply this pipeline to the 100-million-word pre-training dataset from the BabyLM challenge, as well as to standard language and grammatical benchmarks, enabling us to pre-train and evaluate a model using phonemic input representations. Our results show that while phoneme-based training slightly reduces performance on traditional language understanding tasks, it offers valuable analytical and practical benefits.

## introduction the use of orthographic text to train neural networks is so commonplace that it is considered the default. this has not always been the case. when neural networks were first applied to language, models were primarily trained on continuous streams of phonemes or graphemes, rather than orthographic text with its written artefacts. these early neural models demonstrated a striking ability to acquire phonology, syntax and semantics @xcite @xcite . as technology scaled, subword representations became the dominant representation, offering key advantages such as reducing computation costs and better character tokenization no word boundaries phonemic representation h i t h e r e hi there h a ɪ ð ɛ r hithere haɪ ðɛr figure 1 : an illustration of all three adjustments that we make to convert text input to continuous streams of phonemes. the success of these models on downstream tasks has motivated researchers to examine the internal representations of llms and analyze their ability to learn grammatical generalizations (hewitt and manning, 2019; @xcite . however, their phonological capabilities remain understudied due to the orthographic nature of training data. an alternative input representation for text-based language models is to use phonemes rather than graphemes, corresponding to how words are pronounced, rather than how they are written. the use of phonemes, such as those described by the international phonetic alphabet (ipa), as an underlying input representation, presents the following analytical and practical benefits over an orthographic representation that is the modern-day default. analytical: a phoneme-based representation is useful when using language models to study the distributional properties of phonemes @xcite and phonological systems of languages more broadly @xcite . many language acquisition studies prefer using phonemes as a representation that more closely represents the human learning environment, which facilitates statistical learning experiments ranging from word segmentation @xcite , to past-tense formation @xcite , and broader lexico-syntactic knowledge @xcite . practical: ipa-encoded text has been found to be beneficial for a variety of nlp tasks including lyric generation @xcite ), text-to-speech @xcite and lowresource language modeling @xcite . phonemes also benefit multi-lingual language modeling by establishing a universal representation shared between languages @xcite . despite the analytical and practical advantages of training language models with phonemes, a key question remains: can modern language model architectures encode grammatical knowledge and succeed at language understanding tasks when trained with phoneme-based representations? answering this question is challenging for two reasons. first, training and evaluation data need to be provided to a model in both a phonemic and graphemic representation. second, it is nontrivial to select the transformations to convert orthographic text into phonemic representations and to evaluate how these individually affect a model's performance across a wide variety of benchmarks. in this work, we address these challenges as follows. we first present a method for converting training data and evaluation benchmarks into a unified ipa representation. this enables language models to be trained and evaluated on graphemic and phonemic representations of the same data. we then identify three key transformations which enable us to map from the written representation typically used to train language models to the phonemic representation often used in analytical studies (see fig. 1 ). finally, we conduct a careful ablation of the three transformations: we train a language model on the same corpus of 100 million words with all combinations of the three transformations (2 3 configurations), evaluating the model's gram-matical capabilities and its resulting performance on downstream language understanding tasks. we find that large language models are powerful statistical learners capable of learning grammar from a phonemic input representation. although we observe a decrease in performance on some tasks, the degradation is not as substantial as has been anecdotally suggested by previous studies. our ablation studies indicate that the impact of each transformation that we use to convert orthographic text to continuous phoneme streams depends on the downstream task; tasks in the blimp supplement set are particularly sensitive to the use of phonemes, while those in glue are sensitive to character tokenization. a deeper analysis into these ablations reveals that many evaluation instances rely on information only present in written text (such as punctuation). finally, we take advantage of the fact that we train models using phonemic streams and evaluate our models for phonological knowledge using the babyslm benchmark. our models achieve the best scores on this benchmark to date. ## related work the standard input representation for training large language models consists of written text split into subword units. by contrast, studies that train models using a phonemic input representation tend to split words into individual phonemes, without word boundaries (as spoken utterances are produced continuously, without clear pauses between words). we identify three key transformations that bring us from the standard input representation used by language models to this alternative phoneme stream representation: ## phoneme stream pipeline to convert the data to a phonemic representation, we developed the corpus phonemizer tool: ## experimental setup we evaluate the effect of our proposed input adjustments by training a gpt-2 model @xcite using the babylm challenge framework @xcite . the model is trained eight times with each combination of the three input adjustments. following the strict track of the babylm challenge, we train on a provided corpus of 100 million words and evaluate on a series of benchmarks assessing the grammatical knowledge and the downstream capabilities of each model. we additionally evaluate on babyslm @xcite which provides syntactic and lexical scores specifically for speech-based models. our phonemized dataset, trained models and tokenizers are hosted on huggingface. the babylm 2024 pretraining data contains 100 million words sourced from nine different corpora @xcite . over 50% of the data consists of transcribed or scripted speech and over 40% comes from child-directed sources (written or spoken). we apply minor cleaning operations to the dataset, removing extraneous spaces and formatting anomalies using regular expressions. ## results in the base gpt-2 model with no input adjustments outperforms the two baselines for blimp, blimp supplement and glue, validating our selection of hyper-parameters and choice of architecture as described in appendix a. comparing the gpt-2 model with no input transformations (top row) to the same model with all three transformations applied (bottom row), we notice a decrease in performance across all benchmarks. although this indicates that the gpt-2 architecture is best suited for the standard orthographic input representation (word boundaries, graphemes and subword tokenization), the decrease in performance when the three transformations are applied is not substantial and scores remaining competitive with the baseline models (all combinations still outperform ltg-bert on blimp). it is clear that the model is still capable of learning grammatical rules and excelling at downstream tasks when the input consists of individual phonemes with no word boundaries. in section 5.1 we investigate this result further through an ablation of the three transformations, noting the effect of punctuation and context size. in section 5.2 we focus on the babyslm metrics, which demonstrate a different pattern to the other benchmarks. finally, in section 5.3 we investigate the consequences of removing punctuation in our phonemic transcriptions. ## discussion in this work, we set out to establish whether modern language model architectures can encode grammatical knowledge and succeed at language understanding tasks when trained with phonemic input representations. by identifying three key transformations, carefully ablating them and evaluating our models on a wide variety of benchmarks, we found that these transformations do lead to decreased performance on standard benchmarks, but that this decrease is not substantial, and the effect of each transformation varies according to the evaluation. generally, we conclude that language models are capable learners and training with these input representations is completely viable. in this section, we consider explanations for the difference in performance across the benchmarks and discuss the limitations of phonemic transcriptions and our monolingual approach. our work also has implications for human acquisition investigations and studies that train models directly from raw audio, which we discuss in appendix c. ## conclusion our study explores the effect of training language models using phonemic input representations, which offer both analytical and practical advantages. we develop a pipeline to convert orthographic datasets into a continuous stream of phonemes and leverage this pipeline to train a language model on phoneme streams and evaluate its grammatical and language understanding abilities. our findings suggest that while phoneme-based input representations result in a slight decrease in model performance on traditional language understanding tasks, it is nonetheless a feasible training paradigm, facilitating future language modeling work, improving phonological interpretability and enhancing speech-based applications.