Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer (2023)

Abstract: We introduce and demonstrate how to effectively train multilingual machine translation models with pixel representations. We experiment with two different data settings with a variety of language and script coverage, demonstrating improved performance compared to subword embeddings. We explore various properties of pixel representations such as parameter sharing within and across scripts to better understand where they lead to positive transfer. We observe that these properties not only enable seamless cross-lingual transfer to unseen scripts, but make pixel representations more data-efficient than alternatives such as vocabulary expansion. We hope this work contributes to more extensible multilingual models for all languages and scripts.

## introduction multilingual model vocabularies are finite and typically smaller than the possible set of unicode characters, inherently leaving some languages and scripts under-represented. as coverage increases, parameter allocation to each language decreases, resulting in a trade-off between capability, capacity, and coverage. recent work on pixel representations @xcite provides an appealing alternative to past approaches, because they do not have a discrete model vocabulary or finite embedding matrix, and can represent all scripts with complete parameter sharing. recent work @xcite has also shown that pixel-based models can be directly finetuned across scripts without vocabulary extensions, adapters, or transliteration. however, pixel representations have previously only been trained or finetuned on individual languages at a time, rather than multilingually. this leaves unanswered questions about the effects of multilingual co-training, such as whether similar scripts will interfere with or boost performance, or if architectural changes will be needed given the larger input space. in this work we demonstrate how to effectively parameterize and train multilingual translation models with pixel representations, leading to improvements of up to 9 bleu on two multilingual datasets with diverse language and script coverage. we explore various properties of pixel representations in order to understand their potential benefits and limitations, including positive transfer and representational similarity between languages, parameter sharing, and frequency-based relationships. finally, we show that not only can pixel representations be finetuned cross-lingually or to unseen scripts, but can do so more data-efficiently than alternatives such as vocabulary expansion, with significant improvements for unseen scripts. ## our approach covering the larger character sets foot_0 in multilingual models commonly results in significant parameter increases in the embedding matrix and softmax, creating a vocabulary bottleneck. while sampling data by language to balance vocabularies is common for large-scale multilingual systems @xcite , sampling may cause common vocabulary to be outof-vocabulary (oov) for languages with longer-tail character distributions like chinese @xcite . 2 one alternative is to move to bytebased representations, which combats exploding model parameters by reducing the set of embeddings to 256. however, this approach increases sequence lengths up to 12× compared to characters, determined by the script's unicode encoding, making optimal batch sizes prohibitively large and slow for our computational resources. rendering text to images bypasses many of the vocabulary challenges posed by multilingual modeling. pixel-based representations have the advantage of no predetermined static vocabularies, no exploding embedding matrix parameters or sequence lengths, and complete parameter sharing across similar word forms at a sub-character level regardless of the underlying unicode or byte structure. below we present the technical details of our approach and comparisons before proceeding to experimental settings and results. ## multilingual translation with pixels we experiment with two datasets to investigate the performance and properties of multilingual pixel representations for machine translation. we perform initial experiments with the balanced 7 language pair multi-target ted data @xcite used by @xcite , which we will refer to as ted-7, to compare performance to prior work with pixel representations and explore any necessary architectural changes in the multilingual setting. we then scale up using the larger 59 language pair ted talk corpus from @xcite , or ted-59. in all cases, our models are many-to-one multilingual translation models with english as the target. we list the languages in each corpus with the number of training examples in app. a. results for all datasets are shown in ## data-efficient cross-lingual transfer it has been shown that using pretrained multilingual models for cross-lingual transfer can provide significant performance improvements, particularly when the target language is under-resourced. however, adapting models to unseen scripts with no lexical coverage in the original model typically requires techniques such as expanding the embedding matrix to include new vocabulary @xcite or language-specific adapters @xcite . in contrast, models with pixel representations can be finetuned directly on new languages and scripts without requiring any architectural changes @xcite . we hypothesize that the model properties discussed in § 4 will not only allow transfer without model exscript unigram bigram trigram gain gain seen? coverage coverage coverage bpe ext bpe ro ✓ 96% 91% 84% +11% +11% pl ✓ 95% 88% 73% +13% +14% fa ✓ 99% 79% 66% +18% +20% vi ✓ 86% 66% 41% +22% +21% he ✗ 23% 5% 1% +30% +4700% to evaluate the data-efficiency of cross-lingual transfer, we adapt our multilingual models to language pairs with five new source languages, each with different degrees of script coverage to those observed in pretraining as quantified in whether model vocabularies cover a particular script is typically described as binary, but even with observed scripts new languages introduce unseen character sequences and diacritics which will not be appropriately represented. we observe that for unicode-based models, transfer capability is strongly reflected in lexical coverage; vocabulary expansion improves performance slightly for languages with higher n-gram coverage, and significantly for hebrew with minimal coverage, particularly with more data to train new language-specific embeddings, as seen in figure 10 . however, pixel representations enable models to perform better still than vocabulary expansion, particularly with less data. we believe this is because with complete parameter sharing across all scripts, all parameters for new languages are more strongly initialized. this direction may lead to more data-efficient crosslingual transfer, particularly for under-resourced languages and tasks. ## related work previous work has shown allocating additional encoder capacity to be beneficial for smaller granularity inputs, both for characters and bytes @xcite and other modalities @xcite . deep encoders and shallow decoders have been used to improve model efficiency and latency with subword inputs @xcite @xcite , and deeper and narrower encoders have been shown to scale more effectively @xcite . significant prior work has been devoted to broader and more effective language coverage, through full unicode character coverage and downsampling @xcite , clustered vocabularies for efficient modeling of large vocabularies @xcite , bytelevel modeling @xcite , bytes in conjunction with bpe to combat data sparsity and memory issues (bbpe: @xcite or bytefallback @xcite . mapping characters to a smaller set of common representations across scripts through transliteration @xcite or graphemeto-phoneme systems @xcite have also been shown beneficial for multilingual and cross-lingual transfer for re-lated languages across scripts, though they may also introduce collisions which can negatively affect performance. post-hoc vocabulary expansion @xcite or language adapters @xcite to increase vocabulary coverage have also been shown to be very effective. recently, pixel representations have been proposed as a vocabulary-free alternative @xcite , though not trained yet multilingually. we refer readers to the bigscience survey for greater discussion @xcite . ## conclusions we introduce and demonstrate how to effectively train multilingual pixel representations for machine translation. we experiment with two different data scales with a variety of language and script coverage, demonstrating improved performance compared to the traditional subword approach. we analyze various properties of pixel representations to better understand where they may provide potential benefits and the impact of different scripts and data representation. we observe that these properties not only enable cross-lingual transfer to unseen scripts, but make pixel representations more data-efficient than alternatives such as vocabulary expansion. we hope this work contributes to more extensible multilingual models for all languages and scripts.