MYTE : Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling (2024)

Abstract: A major consideration in multilingual language modeling is how to best represent languages with diverse vocabularies and scripts.Although contemporary text encoding methods cover most of the world’s writing systems, they exhibit bias towards the high-resource languages of the Global West. As a result, texts of underrepresented languages tend to be segmented into long sequences of linguistically meaningless units. To address the disparities, we introduce a new paradigm that encodes the same information with segments of consistent size across diverse languages. Our encoding convention (MYTE) is based on morphemes, as their inventories are more balanced across languages than characters, which are used in previous methods. We show that MYTE produces shorter encodings for all 99 analyzed languages, with the most notable improvements for non-European languages and non-Latin scripts. This, in turn, improves multilingual LM performance and diminishes the perplexity gap throughout diverse languages.

## introduction multilingual language models have become the state-of-the-art solution for performing tasks on a wide range of languages @xcite @xcite . however, it is challenging to ensure high performance for all languages due to differences in data availability, especially for the long tail of low-resource languages @xcite . this challenge is compounded by choices of how words are represented during tokenization; past studies have shown that multilingual models either cannot accurately represent texts in rare languages @xcite or do so via over-segmentation, which is detrimental both to model performance and inference cost @xcite . 8 72 6f 75 67 68 6c 79 61 74 31 32 myte 52 82 a3 93 6c 79 61 74 31 32 cs: byte-level models aim to solve these challenges. rather than words or subword tokens, they use byte-level text representations that achieve high coverage @xcite , as common encodings such as utf-8 support most of the world's scripts. nevertheless, the over-segmentation problem still exists even at the byte level, as byte sequences for single characters are overly long for many non-latin script languages @xcite . this problem has an immense effect on modeling these scripts in nlp systems, as operating on longer sequences significantly increases the computation costs of training and inference in models, while also making learning less sample efficient. furthermore, the billing for apis such as chatgpt (openai.com/chatgpt) is often associated with the segmented sequence length, disadvantaging speakers of specific languages @xcite . in this work, we propose a novel method to derive byte representations of text, enabling equitable segmentations across languages and scripts. in our approach, we replace the current convention of assigning byte codes to characters with a morphology-driven approach, as morphemes foot_0 are more informatively comparable constituents of text across languages than characters @xcite . specifically, we introduce a novel algorithm for representing text as byte sequences that is based on unsupervised morphological segmentation @xcite . we demonstrate that our new paradigm for byte representation improves the segmentation of diverse languages of various scripts and morphological inventories. furthermore, the segmentation of parallel sentences across languages converges to comparable lengths. we test our method's effectiveness in creating equitable text representation -representations that given parallel texts have similar encoded sequence lengths. we then evaluate the applicability of the method to multilingual language modeling across 99 typologically diverse languages. our contributions can be summarized as follows @xcite we propose a novel byte-encoding method that is morphologically driven; (b) we show empirically that the resulting representations are more equitable across languages than vanilla byte, character, or subword segmentation; (c) we analyze the typical lengths of these representations and show decreased sequence length across all analyzed languages, significantly reducing computation cost and benefiting non-latin script languages the most; (d) we train a language model with our new representation scheme and demonstrate that it maintains balanced and better lm performance across diverse languages and exhibits faster inference speed. this improvements holds across different model scales. our models match sota byt5 performance across multiple tasks for diverse low-resource languages while being more efficient in training and inference. we will release our code and models to facilitate further research in this direction. ## background: utf-8 bytes the vast majority of texts online foot_1 are represented as bytes via utf-8 convention, which is defined by the unicode standard (the unicode consortium, 2011). in utf-8, each character (or codepoint) is represented as a sequence of one to four bytes. due to the gradual development of communication standards, utf-8 first allocated one-byte representation ascii symbols, which cover primary latinscript characters (see 00 to 7f in figure 2 ). other characters are represented as multi-byte codes starting with a byte from range c2 to f4 denoting the number of bytes in the codepoint and followed by continuation bytes from range 80 to bf. in utf-8 convention, characters in non-latin alphabetic scripts (cyrillic, armenian, georgian), diacritics, and abjads foot_2 usually have two-byte codes, while the byte length increases to three or four for brahmic abugidas foot_3 and cjk (chinese, japanese, korean) logographs. as a result, the granularity of byte codes varies significantly across languages; this means that texts conveying the same information across languages tend to be represented by byte sequences of significantly different lengths @xcite . ## method: morphology-driven bytes as discussed in the prior section and shown in figure 1 , utf-8 convention produces longer byte sequences for some languages due to the development choices. to make byte representation more equitable, we introduce an encoding paradigm that aims to assign byte codes of similar lengths to morphemes across languages. we base our encoding scheme on morphological analysis because morphemes are the shortest meaningful constituents and are independent of the writing convention @xcite . we assume that the number of morphemes in sentences with the same information load is more balanced across languages than the number of characters, bytes, or tokens. thus, we enforce balanced segmentation granularity across languages. an alternative approach to encoding morphological representations would be treating the union of multilingual morpheme inventories across languages as one large subword vocabulary. to cover the morphemes of many languages in this manner, the vocabulary would be much larger than the ones usually applied to models. 5 this would incur additional computational costs and, similar to other subword representations, would likely not generalize well to new, unseen languages. ## equitable multilingual segmentation with myte we first analyze the properties of our proposed morphology-driven encoding. following the setting of @xcite , we measure whether myte produces the segmented sequences of comparable length across languages. we compute parity across languages using the multi-parallel corpus flores 200 @xcite . parity is defined as |t(s l )|/|t(s en )|, where s l and s en stand for parallel sentences in language l and in english, respectively. |t(s)| is the length of sequence s with segmentation method t. we compare the myte encoding from section 3.2 to several baselines of common input representation: (a) vanilla byte-level encoding via utf-8; (b) character-level encoding; (c) subwords produced by sentencepiece algorithm @xcite . in comparison, we focus on the equitability of sequence lengths produced by the methods for diverse languages. furthermore, we compare our morphological byte encoding sequence compression rate against the utf-8 convention. compression is essential for an effective text representation as it affects nlp systems' efficiency and usage cost @xcite . finally, we check whether our method more effectively compresses languages and scripts unseen in myte algorithm described in section 3.2. ## myt5: language modeling with myte this section investigates the benefits of myte as an encoding scheme for byte-level language modeling. for that purpose, we have trained t5 language models on myte representation. we refer to these models as myte t5 models, or myt5 for short. ## conclusion in this paper, we introduce myte encoding, a fairer byte-level representation for multilingual language modeling that is based on morphological segmentation. we show that adapting a morphological analyzer to unsupervised segmentation allows us to represent multi-parallel corpora with comparable encoding lengths across a wide range of languages. additionally, our new representation significantly improves language modeling, especially of lowresource and non-latin script languages, and provides efficiency benefits over traditional byte-level models. these trends hold across model sizes, with improvement increasing at scale. overall, myte bridges the gap in encoding efficiency between high and low-resource languages, benefiting (to varying extent) all 99 analyzed languages.