Graphemes vs. phonemes: battling it out in character-based language models (2024)

Abstract: We present grapheme-llama and phoneme-llama, character-based language models trained for the 2024 BabyLM challenge. Through these models, we explore an under-researched approach to downsizing: replacing subword-based tokenization with character-level tokenization, drastically reducing the vocabulary size. The grapheme model is trained on a standard BabyLM dataset, while the phoneme model uses a phoneme-converted version of this dataset. Results show that grapheme-based models perform better overall, achieving scores comparable to subword-based models on grammatical benchmarks. Despite lower performance, phoneme models also demonstrate promising grammatical learning. We argue that our results challenge conventional wisdom on language modeling techniques and open up novel research questions with character- and phoneme-based models as objects of inquiry.

## introduction while large language models continue to beat benchmarks, their parameter numbers, amounts of training corpora and training flops are ever increasing. more recently, however, a new research focus on ecologically friendly, data-efficient and possibly cognitively plausible language modelsso called babylms -has emerged. but what makes a language model a babylm? for the babylm challenges @xcite , babylms are defined by extremely constrained data settings. in this constrained data setting, the best scoring models in the 2023 challenge employed highly sophisticated and large-ish architectures: elc-bert (charpentier and samuel, 2023) used numerous architectural improvements over standard encoders, while babyllama @xcite was distilled from various larger teacher models. models with architectures downsized similarly to their training data (e.g. by @xcite @xcite did not fare as well on standard benchmarks. as our submission to the 2024 babylm challenge @xcite , we present graphemellama foot_0 and phoneme-llama foot_1 . we replace the standard subword-based tokenization algorithms with naive character-based tokenization, leading to a drastic decrease in vocabulary size. we show that when such simplifications are combined with stateof-the-art architectures like llama @xcite , the resulting models still achieve considerable grammatical proficiency and provide useful inductive biases for further fine-tuning. while the grapheme model is trained on the standard 100m babylm data, our phoneme model is trained on a version of this data set converted to phonemes foot_2 . although it performs generally worse than its grapheme counterpart, the phoneme model still manages to learn the grammatical phenomena in a matched blimp data set quite well. in the light of these results, we offer some discussion points for phoneme-based language modeling, the pitfalls it is currently facing and its general potential. in sum, we argue that these results open fruitful avenues for further research on small language models and question "common wisdom" in current language modeling practices. ## related work small lms/downsizing: recently, there has been a surge in interest in small-ish language models. the arguably first babylm, babyberta @xcite , followed a combined (i.e. data and architecture) downsizing approach and showed that dramatically less training data can result in remarkable linguistic proficiency with a small model architecture. on the other hand, current "small" models often employ more complex strategies to achieve compactness, e.g. distillation with teacher and student models @xcite , or reduction of number precision @xcite . these models' "smallness" is only achieved after complex training procedures. in contrast to these developments, the babylm 2023 submissions by veysel çağatan (2023), @xcite and @xcite used a priori small models (in terms of parameter size) to show the lower bounds of knowledge learnability from small data. they all showed that very small models (even models with a parameter size below 1m) can achieve scores equal to much larger baselines on standard evaluation tasks like blimp or glue. as such, these successful experiments give impetus for our current models: against common wisdom, the reduction of certain models hyperparameters does not have to have a detrimental effect on performance (a fact also corroborated by @xcite . comparable studies have neither focused on character-level tokenization nor on phoneme-based representations (see paragraphs below for the most comparable studies available), so we pioneer into this uncharted territory with our models. ## discussion general remarks: there are two commonly presented arguments against character-level tokenization (e.g. presented in @xcite : (i) such models achieve subpar results on evaluations; and graphemes vs. phonemes: the comparison between our grapheme and phoneme models undoubtedly concludes with a win for the grapheme models. across all benchmarks, they outperform the phoneme models on average. no clear tendencies spring to mind when analyzing the detailed results -however, all four models achieve best scores on some sub-tasks in benchmarks. separating noise from signal in these results remains an open task for future studies. as of now, we can only speculate why the phoneme models perform this worse. an easy explanation could be the absence of punctuation in phoneme models. as dots, commas and other punctuation marks perform important semantic functions in texts (see @xcite , their absence quite possibly has a negative effect on the acquired grammatical system of a language model. another problem could lie in the quality of our g2p system. alphabetic writing systems generally associate letters to sounds, and vice versa. however, especially for english, the correspondences between graphemes and phonemes are not trivial and (can seem) arbitrary @xcite @xcite . graphemes are arranged according to orthographic conventions which usually do not directly reflect a language's underlying phonological system. grapheme-to- whitespace: finally, the detrimental effect of whitespace removal also deserves explanation and discussion. whitespace encodes important linguistic information about word boundaries (or approximations thereof) --information which is not available in spoken language (there, pauses between stretches of connected speech serve different purposes). instead, prosody (e.g. word stress or intonation), provides cues to segmentation at different levels of linguistic abstraction (like words and phrases). this is, apart from whitespace, not reflected in orthographic texts and also often missing from phonetic transcriptions foot_6 . as such, data without whitespaces is a developmentally/cognitively/linguistically more plausible form of input. as this added plausibility comes with the loss of information, it is not surprising that scores for non-whitespace models are generally lower. a notable exception is the high score of the non-whitespace grapheme model for the matched blimp supplement. this might be a side effect of our very small context size. the blimp supplement contains inter alia dialogue phenomena with long dependencies. the models without whitespace can take in more (non-whitespace) characters, and in the light of our rather small context size, it might be the case that the whitespace models cannot process enough information to actually grasp these phenomena. ## conclusion this paper has shown two things: (i) characterbased tokenization is a viable alternative for small language models and (ii) phoneme-based lms can also perform reasonably well on common benchmarks, although grapheme models are superior. with the drawbacks (e.g. the computational complexity increase in large models) of character-based tokenization, we of course do not want to replace sub-word tokenization. however, we believe that our models deserve a place in the toolbox of developmentally more plausible language models. they can be used to test what kind of linguistic knowledge can be learned from raw input and answer questions about the learnability of linguistic knowledge from an even poorer stimulus @xcite than the "stimulus" of subword models. in combination with phoneme representations, they open up new avenues of inquiry, e.g. for phenomena on the phonological/phonetic or lexical levels of linguistic analysis -phenomena which are not captured by the coarse-grained structure of subword tokens. moreover, character-based language models open new pathways into experiments with multilingual models. the latin script, for example, offers a shared vocabulary for many languages, whereas the ipa even offers a shared vocabulary for practically all languages.