Phonetic and Visual Priors for Decipherment of Informal R omanization (2020)

Abstract: Informal romanization is an idiosyncratic process used by humans in informal digital communication to encode non-Latin script languages into Latin character sets found on common keyboards. Character substitution choices differ between users but have been shown to be governed by the same main principles observed across a variety of languages—namely, character pairs are often associated through phonetic or visual similarity. We propose a noisy-channel WFST cascade model for deciphering the original non-Latin script from observed romanized text in an unsupervised fashion. We train our model directly on romanized data from two languages: Egyptian Arabic and Russian. We demonstrate that adding inductive bias through phonetic and visual priors on character mappings substantially improves the model’s performance on both languages, yielding results much closer to the supervised skyline. Finally, we introduce a new dataset of romanized Russian, collected from a Russian social network website and partially annotated for our experiments.

## introduction written online communication poses a number of challenges for natural language processing systems, including the presence of neologisms, codeswitching, and the use of non-standard orthography. one notable example of orthographic variation in social media is informal romanization 2speakers of languages written in non-latin alphabets encoding their messages in latin characters, for convenience or due to technical constraints (improper rendering of native script or keyboard com/ryskina/romanization-decipherment 2 our focus on informal transliteration excludes formal settings such as pinyin for mandarin where transliteration conventions are well established. in this paper, we focus on decoding informally romanized texts back into their original scripts. we view the task as a decipherment problem and propose an unsupervised approach, which allows us to save annotation effort since parallel data for informal transliteration does not occur naturally. we propose a weighted finite-state transducer (wfst) cascade model that learns to decode informal romanization without parallel text, relying only on transliterated data and a language model over the original orthography. we test it on two languages, egyptian arabic and russian, collecting our own dataset of romanized russian from a russian social network website vk.com. qto mo et byt luqxe? [latent cyrillic] čto možet byt' lučše? since informal transliteration is not standardized, converting romanized text back to its original orthography requires reasoning about the specific user's transliteration preferences and handling many-to-one (figure 2 ) and one-to-many (figure 1 ) character mappings, which is beyond traditional rule-based converters. although user behaviors vary, there are two dominant patterns in informal romanization that have been observed independently across different languages, such as russian @xcite , dialectal arabic (darwish, 2014) or greek @xcite : phonetic similarity: users represent source characters with latin characters or digraphs associated with similar phonemes (e.g. m /m/ → m, l /l/ → l in figure 2 ). this substitution method requires implicitly tying the latin characters to a phonetic system of an intermediate language (typically, english). visual similarity: users replace source characters with similar-looking symbols (e.g. q / > ts j / → 4, u /u/ → y in figure 2 ). visual similarity choices often involve numerals, especially when the corresponding source language phoneme has no english equivalent (e.g. arabic /q/ → 3). taking that consistency across languages into account, we show that incorporating these style patterns into our model as priors on the emission parameters-also constructed from naturally occurring resources-improves the decoding accuracy on both languages. we compare the proposed unsupervised wfst model with a supervised wfst, an unsupervised neural architecture, and commercial systems for decoding romanized russian (translit) and arabic (arabizi). our unsupervised wfst outperforms the unsupervised neural baseline on both languages. ## related work prior work on informal transliteration uses supervised approaches with character substitution rules either manually defined or learned from automatically extracted character alignments @xcite . typically, such approaches are pipelined: they produce candidate transliterations and rerank them using modules encoding knowledge of the source language, such as morphological analyzers or wordlevel language models @xcite . supervised finite-state approaches have also been explored @xcite ; these wfst cascade models are similar to the one we propose, but they encode a different set of assumptions about the transliteration process due to being designed for abugida scripts (using consonant-vowel syllables as units) rather than alphabets. to our knowledge, there is no prior unsupervised work on this problem. named entity transliteration, a task closely related to ours, is better explored, but there is little unsupervised work on this task as well. in particular, @xcite propose a fully unsupervised version of the wfst approach introduced by @xcite , reframing the task as a decipherment problem and learning cross-lingual phoneme mappings from monolingual data. we take a similar path, although it should be noted that named entity transliteration methods cannot be straightforwardly adapted to our task due to the different nature of the transliteration choices. the goal of the standard transliteration task is to communicate the pronunciation of a sequence in the source language (sl) to a speaker of the target language (tl) by rendering it appropriately in the tl alphabet; in contrast, informal romanization emerges in communication between sl speakers only, and tl is not specified. if we picked any specific latinscript language to represent tl (e.g. english, which is often used to ground phonetic substitutions), many of the informally romanized sequences would still not conform to its pronunciation rules: the transliteration process is characterlevel rather than phoneme-level and does not take possible tl digraphs into account (e.g. russian sh /sx/ → sh), and it often involves eclectic visual substitution choices such as numerals or punctua-tion (e.g. arabic @xcite @xmath0, russian dl [dlja, 'for'] → dl9| ). finally, another relevant task is translating between closely related languages, possibly written in different scripts. an approach similar to ours is proposed by @xcite . they also take an unsupervised decipherment approach: the cipher model, parameterized as a wfst, is trained to encode the source language character sequences into the target language alphabet as part of a character-level noisy-channel model, and at decoding time it is composed with a word-level language model of the source language. recently, the unsupervised neural architectures @xcite have also been used for related language translation and similar decipherment tasks @xcite , and we extend one of these neural models to our characterlevel setup to serve as a baseline ( §5). ## methods we train a character-based noisy-channel model that transforms a character sequence o in the native alphabet of the language into a sequence of latin characters l, and use it to decode the romanized sequence l back into the original orthography. our proposed model is composed of separate transition and emission components as discussed in §3.1, similarly to an hmm. however, an hmm assumes a one-to-one alignment between the characters of the observed and the latent sequences, which is not true for our task. one original script character can be aligned to two consecutive latin characters or vice versa: for example, when a phoneme is represented with a single symbol on one side but with a digraph on the other (figure 1 ), or when a character is omitted on one side but explicitly written on the other (e.g. short vowels not written in unvocalized arabic but written in transliteration, or the russian soft sign representing palatalization being often omitted in the romanized version). to handle those alignments, we introduce insertions and deletions into the emission model and modify the emission transducer to limit the number of consecutive insertions and deletions. in our experiments, we compare the performance of the model with and without informative phonetic and visual similarity priors described in §3.2. ## datasets here we discuss the data used to train the unsupervised model. unlike arabizi, which has been explored in prior work due to its popularity in the modern online community, a dataset of informally romanized russian was not available, so we collect and partially annotate our own dataset from the russian social network vk.com. ## experiments here we discuss the experimental setup used to determine how much information relevant for our task is contained in the character similarity mappings, and how it compares to the amount of information encoded in the human annotations. we compare them by evaluating the effect of the informative priors (described in §3.2) on the performance of the unsupervised model and comparing it to the performance of the supervised model. ## results and analysis the cer values for the models we compare are presented in original latin r /r/ r (.93), p (.05) b /b/ b (.95), 6 (.02) v /v/ v (.87), 8 (.05), w (.05) /w, u:, o:/ w (.48), o (.33), u (.06) /x/ 5 (.76), k (.24) ## conclusion this paper tackles the problem of decoding nonstandardized informal romanization used in social media into the original orthography without parallel text. we train a wfst noisy-channel model to decode romanized egyptian arabic and russian to their original scripts with the stepwise em algorithm combined with curriculum learning and demonstrate that while the unsupervised model by itself performs poorly, introducing an informative prior that encodes the notion of phonetic or visual character similarity brings its performance substantially closer to that of the supervised model. the informative priors used in our experiments are constructed using sets of character mappings compiled for other purposes but using the same underlying principle (phonetic keyboard layouts and the unicode confusable symbol list). while these mappings provide a convenient way to avoid formalizing the complex notions of the phonetic and visual similarity, they are restrictive and do not capture all the diverse aspects of similarity that idiosyncratic romanization uses, so designing more suitable priors via operationalizing the concept of character similarity could be a promising direction for future work. another research avenue that could be explored is modeling specific user preferences: since each user likely favors a certain set of character substitutions, allowing user-specific parameters could improve decoding and be useful for authorship attribution.