Phone-ing it in: Towards Flexible Multi-Modal Language Model Training by Phonetic Representations of Data (2022)

Abstract: Multi-modal techniques offer significant untapped potential to unlock improved NLP technology for local languages. However, many advances in language model pre-training are focused on text, a fact that only increases systematic inequalities in the performance of NLP tasks across the world’s languages. In this work, we propose a multi-modal approach to train language models using whatever text and/or audio data might be available in a language. Initial experiments using Swahili and Kinyarwanda data suggest the viability of the approach for downstream Named Entity Recognition (NER) tasks, with models pre-trained on phone data showing an improvement of up to 6% F1-score above models that are trained from scratch. Preprocessing and training code will be uploaded to https://github.com/sil-ai/phone-it-in .

## introduction pre-trained language models are increasingly applied in ways that are agnostic to targeted downstream tasks @xcite . this usage has led to a proliferation of large language models trained on enormous amounts of data. for example, the recent megatron-turing nlg 530b model was trained on the pile, which includes 800gb+ of text @xcite , and other large language models utilize large portions of the 200tb+ common crawl data. 1 these large data sets include impressive amounts of text, but all languages are not represented equally (or at all) in that text. the reality is that only a negligible fraction of the 7000+ currently spoken languages @xcite have sufficient text corpora to train state-of-theart language models. this data scarcity results in systematic inequalities in the performance of nlp tasks across the world's languages @xcite . in this paper, we propose a multi-modal approach to train both language models and models for downstream nlp tasks using whatever text and/or audio data might be available in a language (or even in a related language). our method uti-lizes recent advances in phone recognition and text/grapheme-to-phone transliteration to convert input audio and text into a common phonetic representation (the ipa phone inventory). we then pre-train character-based language models in this phone-space. finally, we fine-tune models for downstream tasks by mapping text-based training data into the phonetic representation. thus, in addition to flexibility in pre-training, our method provides a way to reuse labeled text data for common nlp tasks, like named entity recognition or sentiment analysis, in the context of audio inputs. we demonstrate our phonetic approach by training named entity recognition (ner) models for swahili [swh] 3 using various combinations of swahili text data, swahili audio data, kinyarwanda [kin] text data, and kinyarwanda audio data. these two languages both originate from from the same language family, bantu, and are spoken by millions of people in eastern africa, often within the same country, resulting in some overlap in loan words, etc. 4 however, they are both considered low-resource languages. kinyarwanda in particular, though spoken by approximately 13-22 million people 5 , has very little text data available in that language, with fewer than 3,000 articles on the kinyarwanda-language wikipedia, and swahili comparatively ahead but still poorly resourced at approximately 68,000 articles, far less than many european languages. 6 , though some datasets have been created such as kinnews @xcite . on the other hand, kinyarwanda is uniquely placed as a language to leverage speech-based technologies, due to well-organized efforts 7 to collect voice data for that language. it is in fact one of the largest subsets available on the common voice dataset @xcite , with 1,183 hours of voice clips collected and validated. choosing these two languages allowed us to test the use of the technique on legitimately low-resourced languages that could benefit from improved nlp technology, and which as part of the same family of languages 3 language codes formatted according to iso 639-3 standard: https://iso639-3.sil.org/ 4 see for example @xcite , which describes english loan words entering kinyarwanda "very often via kiswahili" we find that simple ner models, which just look for the presence or absence of entities, can be trained on small amounts of data (around 2000 samples) in the phonetic representation. models trained for complicated ner tasks in the phonetic representation, which look for entities and their locations within a sequence, are improved (by up to 6+% in f1 score) through pre-training a phonetic language model using a combination of text and audio data. we see this improvement when finetuning either a swahili or kinyarwanda language model for downstream swahili tasks, which implies that one could make use of text and audio data in related languages to boost phonetic language model performance. the utility of the method in data scarce scenarios and importance of pre-training depends on the complexity of the downstream task. ## related work there have been a series of attempts to utilize phonetic representations of language to improve or extend automatic speech recognition (asr) models. some of these jointly model text and audio data using sequences of phonemes combined with sequences of text characters. @xcite , for example, uses a joint transformer architecture that encodes sequences of phonemes and sequences of text simultaneously. however, this joint model is utilized to learn representations that are more robust to transcription errors. the architecture still requires text inputs (from asr transcriptions) and generates outputs in both text and phoneme representations. in contrast, our approach allows for text input, audio input, or text plus audio input to language models. similarly, in @xcite and @xcite investigate the potential of phoneme-based or phoneme aware representations and models, showing gains in performance, language transfer, and flexibility across written scripts. these works conduct training on text-based data only, using epitran to convert to phonemes. @xcite transforms unlabeled text (i.e., not aligned with corresponding audio files) into phonemes in a scheme to train speech recognition models without any labeled data. this scheme involves a generator model trained jointly with a discriminator model. the generator model converts audio, segmented into phonetic units into predicted phonemes, and the discriminator model attempts to discriminate between these predicted phonemes and the phonemes transliterated from unlabeled text. although both text and audio are utilized in this work, they are not input to the same model and the primary output of the training scheme is a model that creates good phonetic speech representations from input audio. outside of speech recognition focused work, @xcite (and other researchers cited therein) attempt to "fuse" audio and text at the word level for emotion recognition. they introduce another architecture that internally represents both audio and text. however, the so-called wise framework relies on speech recognition to generate the text corresponding to audio frames in real-time. the current work explicitly avoids reliance on speech recognition. the 2021 multimodal sentiment analysis (muse) challenge continues this vein of research integrating audio, video, text, and physiology data in an emotion recognition task @xcite . contributions to this challenge, such as @xcite , introduce a variety of ways to "fuse" audio and text inputs. however, these contributions are squarely focused on emotion/sentiment analysis and do not propose methods for flexible, phonetic language models. lakhotia et al. (2021) introduced functionality for "textless" nlp. they explored the possibility of creating a dialogue system from only audio inputs (i.e., without text). as part of this system, language models are directly trained on audio units without any text. this advances the state-of-the-art with regard to self-supervised speech methods, but it does not provide the flexibility in audio and/or text language modeling introduced here. ## methodology our approach is inspired by the fact that many languages are primarily oral, with writing systems that represent spoken sounds. we convert both text and audio into single common representation of sounds, or "phones," represented using the international phonetic alphabet, or ipa. then, we perform both language model pre-training and the training of models for downstream tasks in this phonetic representation. well-tested architectures, such as bert-style transformer models @xcite , are thus flexibly extended to either speech or audio data. regarding the conversion process of text and audio data, we leverage recent advances to transliterate this data into corresponding sounds represented by ipa phonetic symbols. this transliteration is possible for speech/audio data using tools such as the allosaurus universal phone recognizer, which can be applied without additional training to any language @xcite , though it can benefit from fine-tuning @xcite . to convert text data to phonemes we can use tools such as the epitran grapheme-to-phoneme converter @xcite , which is specifically designed to provide precise phonetic transliterations in low-resource scenarios. fig. 1 shows how downstream models for certain nlp tasks, like named entity recognition (ner), are performed in the phonetic representation. labeled data sets for nlp tasks need to be mapped or encoded into the phonetic representation to train downstream models. however, once this mapping is accomplished, models trained in the phonetic representation can perform tasks with audio input that are typically restricted to processing text input. ## experiments in order to evaluate the quality of learned phonetic representations, we transliterate several text and audio data sets in the swahili [swh] language. we pre-train phonetic language models on various combinations of these data sets and evaluate downstream performance on ner tasks. see fig. 2 for a detailed overview of these various combinations. we refer to these combinations as denoted by downstream tasks (sner for swahili ner), and pre-training language ((k for kinyarwanda, s for swahili) as well as data modality (t for text, a for audio). by way of example, the sner+st2 model results from pre-training using 2 swh text datasets (st2) and fine-tuning on the swh ner (sner) task, whereas the sner+sat model results from pre-training using swh audio and text data (sat). kinyarwanda @xcite data is used in our experiments as a language related to the target language (swh) with existing text and audio resources that, in some ways, surpasses those available in the target language. thus, we pre-train some models on kin data while fine-tuning for the downstream ner task using swh data. three different formulations of the ner task, from more simple (ner1) to more complicated/granular (ner3), are used (see fig. 2 ) to help determine the applicability of our methods to less challenging (ner1) to more challenging (ner3) tasks. the ner1 task tries to determine the presence or absence of certain kinds of entities within an input. for our task we use per, org, date, and loc entities. the ner2 task additionally requires models to predict the correct numbers of these entities within an input. finally, the ner3 task requires models to determine entities at the correct locations with an input sequence of phones. for all of these tasks, we first convert text data to phones using epitran and audio data to phones using allosaurus. then, we pre-train on various combinations of data, before fine-tuning on ner. ## results and discussion the importance (or relative impact) of pretraining phonetic language models increases with the complexity of the ner task. fig. 4 shows the maximum percentage improvement due to pretraining for each of our ner tasks. this suggests that simple nlp tasks with a small number of output classes are much easier to port to phonetic representations, even without pre-training, while more complicated nlp tasks may require a more significant amount of text and/or audio data for pretraining. we expect this trend to carry through to tasks like sentiment analysis, which could be formulated as a simple classification task with neg, neu, and pos sentiment labels or a more complicated aspect based sentiment analysis task. ## conclusions and further work the proposed method for multi-modal training using phonetic representations of data has minimum viability for simple ner tasks. for more complicated ner tasks, pre-training phonetic language models boosts downstream model performance by up to 6% in f1 scores. this pre-training can be performed in the target language or in a related language using text and/or audio data. thus, the method provides flexibility in the data needed to train language models, while also allowing for audio and/or text inputs to models trained on downstream nlp tasks. we anticipate exploring various extensions to and validations of this method in the future. specifically, we would like to explore methods that might mitigate performance degradation due to a lack of word boundaries in our method. subword tokenization techniques, such as byte-pair encodings (bpe) @xcite , or character-based word segmentation techniques might help in detecting and exploiting repeating patterns within the phonetic representation. furthermore, the word embedding techniques used by @xcite or @xcite have been shown to work well, and would be worth investigating how the removal of spacedelimited word boundaries would affect this. we would also like to validate our methods on a variety of other data sets and tasks. we selected the masakhaner dataset for evaluation because we specifically wished to evaluate results on ac-tual low-resource languages supported by both allosaurus and epitran. while there are still, we argue, detectable improvements in downstream results with our method, further work would benefit from additional evaluations on other data sets or tasks. in particular, the swahili news classification corpus @xcite corpus may provide a useful evaluation. we did not investigate going from audio to phones, then phones to words/characters, judging that information losses and errors would likely compound in multiple stages of processing. instead, we focused on what could be achieved with the allosaurus "universal phone transcriber" without any language-specific finetuning. a truly universal transcriber would increase flexibility when training for truly low-resource scenarios. nevertheless, it has been shown by @xcite that it is possible to improve phone recognition with even small amounts (approximately 100 sentences) of annotation. it may be possible to improve phonetic language modeling results by performing this fine-tuning in the target language. experiments involving other languages with, e.g. languages that are not related would help to isolate the role of relatedness, lexical overlap, or related sound systems/phonology. while we do not claim that conversion to phones provides better performance generally, we believe that our experiments show that the fundamental idea of converting either text or audio data to the common phone representation provides a viable path to more flexible approach to certain downstream nlp tasks, worthy of further development.