Bi-Phone: Modeling Inter Language Phonetic Influences in Text (2023)

Abstract: A large number of people are forced to use the Web in a language they have low literacy in due to technology asymmetries. Written text in the second language (L2) from such users often contains a large number of errors that are influenced by their native language (L1).We propose a method to mine phoneme confusions (sounds in L2 that an L1 speaker is likely to conflate) for pairs of L1 and L2.These confusions are then plugged into a generative model (Bi-Phone) for synthetically producing corrupted L2 text. Through human evaluations, we show that Bi-Phone generates plausible corruptions that differ across L1s and also have widespread coverage on the Web.We also corrupt the popular language understanding benchmark SuperGLUE with our technique (FunGLUE for Phonetically Noised GLUE) and show that SoTA language understating models perform poorly. We also introduce a new phoneme prediction pre-training task which helps byte models to recover performance close to SuperGLUE. Finally, we also release the SuperGLUE benchmark to promote further research in phonetically robust language models. To the best of our knowledge, FunGLUE is the first benchmark to introduce L1-L2 interactions in text.

## introduction we live in a multilingual world with over 7,000 languages spoken across the globe @xcite . however, technology asymmetrically supports only a few specific languages. for instance, the internet is mostly in english with over 60% of websites using the language despite just around 16% share of its speaking population around the world foot_0 @xcite . increasingly, people are forced to navigate and produce content on the web in languages they have not been formally trained on. the english text produced by esl (english as second / l2 language) writers is heavily influenced by their native language (l1). research in the field of second-language acquisition has found evidence of phoneme-shift based misspellings stemming from l1 influence in l2 text for specific language pairs @xcite @xcite @xcite . studies in natural language understanding (nlu) have been limited to spelling correction @xcite ; @xcite and native language identification @xcite ; @xcite in english learners. these studies predominantly use the toefl11 dataset @xcite which deals with very specific demographics such as test-takers who have formal training in the l2 language. we make the following four key observations about prior work in the study of l1-l2 influences in text and speech. first, current models for l1-l2 influence on textual spelling are limited to certain language pairs and tasks. we argue that l1-l2 influence phenomenon is much more broad and is language and task agnostic. second, there is no large scale study to examine the prevalence of this phenomenon on the open web. third, given that this is an important problem especially for multilingual, new-to-the-internet communities there is no standardized benchmark to study the robustness of natural language understanding(nlu) and natural language generation (nlg) models to inter-language phonetic noise. finally, there is very sparse literature on architecture / pre-training strategies to introduce phonetic robustness into large language models. in this paper, we present modeling techniques,data analyses and a new benchmark to address the gaps mentioned above. we summarise our contributions as follows: and second language (l2). our method exploits the "hidden knowledge" contained in @xmath0. we also propose a generative model bi-phone that is able to synthetically produce spelling corruption in accordance with l1-l2 confusions (sections 3.1, 3.2). ## related work we divide the presentation of related work in two sections. (i) first, we discuss prior work spanning multiple research areas regarding phonetic influences in text and how it relates to our work. (ii) second, we discuss work in the speech domain which studies phonetic variations occurring due to inter-language interference in multi-lingual scenarios. ## method in this section we introduce our method for creating inter-language influenced phonetic misspellings (or corruptions). we present the technique in two parts. section 3.1 presents a method for mining native-language influenced phonetic confusions. section 3.2 contains details of bi-phone, our model that uses mined phonetic confusions to create misspellings. ## evaluations we evaluate the misspellings generated by our model along two distinct dimensions. ## the funglue benchmark significant progress has been made in recent research to substantially improve performance of language understanding tasks. superglue @xcite is a very popular benchmark with ten diverse and hard language understanding tasks. these tasks are boolq, commitmentbank (cb), multi-sentence reading comprehension (mul-tirc), choice of plausible alternatives (copa), reading comprehension with commonsense reasoning (record), recognizing textual entail- split description contains phonetic noise train train split from superglue as is no dev dev split from superglue as is no test dev split from superglue noised with biphone yes ## phoneme prediction as a pre-training task given the inadequacy of existing state-of-the-art models in handling phonetic noise in inputs, we propose a novel pre-training task of phoneme prediction. we posit that the task of predicting phoneme sequences will have the effect of teaching the model "phonetic information". since different lexicalizations of the same sound will have the same phoneme sequence, the model will learn to embed these close. additionally since close sounds often appear in similar intra-word contexts, their graphemic representations will also be pushed closed together. however, to perform nlp tasks, semantic similarity is still crucial. in current models this is often achieved through some variation of the span corruption task (corrupting a span in the input and predicting it on the output). we propose a mixture of these two tasks where a small amount of the phoneme prediction task (20%) is mixed into the standard span corruption task. figure 5 demonstrates our proposal through two example instances. in the first instance the span "sofa design" is masked in the input (replaced with a sentinel) and is expected to be produced on the output. this teaches the model that adjectives like "exquisite" are semantically close. the second instance has the word "building" in the input and the phoneme sequence corresponding to this word (b, ih, l, d, ih, ng) on the output. this task teaches the model that all tokens that produce the same sound (like "ui" or "e" for ih) should be embedded close. we train both mt5 and byt5 checkpoints for an additional 100,000 steps (10% additional steps) on this mixture task. we call this step of additional pre-training, "phonetic pre-training". finally, we fine-tune these models on the standard clean su-perglue training set. the phoneme prediction data is created by taking roughly 2,000,000 highest frequency words from the common crawl english data and getting their pronunciations from an offthe-shelf grapheme to phoneme model. as we will see later, this kind of noisy supervision (not human labelled) is still useful in making models phonetically robust. the last two rows in the byt5 model does however have certain drawbacks. since input sequences are much longer with byte level representations, both training and inference times are much slower than a sub-word tokenized alternative (like mt5). additionally, the byte-level representation also restricts input sequence lengths. using these phonetically robust byte-level models as teachers for sub-word tokenized student models remains an interesting direction for future work. ## conclusion language is a significant barrier to technology especially for new internet users. for such users, english often is not their first language. the speech community has made significant progress in making technology (asr for instance) accessible for such users by making models robust to account for inter-language interactions. we argue that a similar line of effort is needed in the natural language understanding for text community as well. to this end, we first propose a generative model bi-phone that can account for l1-l2 interactions in text. next we show the inter-language perturbations generated by bi-phone are indeed present in non-trival amount in the common crawl corpus. we also release a new benchmark funglue to help further research in this area. we also present our early yet very promising explorations on making natural language understanding models robust to l1-l2 phonetic shifts through a novel phoneme prediction based pre-training.