Enhancing Cross-lingual Transfer via Phonemic Transcription Integration (2023)

Abstract: Previous cross-lingual transfer methods are restricted to orthographic representation learning via textual scripts. This limitation hampers cross-lingual transfer and is biased towards languages sharing similar well-known scripts. To alleviate the gap between languages from different writing scripts, we propose PhoneXL, a framework incorporating phonemic transcriptions as an additional linguistic modality beyond the traditional orthographic transcriptions for cross-lingual transfer. Particularly, we propose unsupervised alignment objectives to capture (1) local one-to-one alignment between the two different modalities, (2) alignment via multi-modality contexts to leverage information from additional modalities, and (3) alignment via multilingual contexts where additional bilingual dictionaries are incorporated. We also release the first phonemic-orthographic alignment dataset on two token-level tasks (Named Entity Recognition and Part-of-Speech Tagging) among the understudied but interconnected Chinese-Japanese-Korean-Vietnamese (CJKV) languages. Our pilot study reveals phonemic transcription provides essential information beyond the orthography to enhance cross-lingual transfer and bridge the gap among CJKV languages, leading to consistent improvements on cross-lingual token-level tasks over orthographic-based multilingual PLMs.

## introduction despite recent advances in cross-lingual pretrained language models (plm) such as mbert @xcite , xlm-r @xcite , plms remain heavily biased towards highresourced languages due to the skewed amount of available pre-training data under parameter capacity constraints. this heavily affects the downstream task performance of less-represented languages during pre-training. in addition, as most high-resourced languages share the similar scripts (i.e. mostly latin scripts), plms tend to perform well on languages sharing similar scripts with those languages @xcite @xcite . as most challenging lowresourced languages do not share similar scripts with common high-resourced languages, this limitation leaves them significantly behind. to alleviate the challenges on low-resource or zero-resource target languages, recent works attempt to transfer knowledge from high-resourced languages (typically english (en)) to lowresourced target languages via augmentations from bilingual dictionaries and parallel corpora. however, these approaches are restricted to english source language and results in less significant performance gain on languages that are more distant from english @xcite . languages can be considered distant due to the differences in orthography, phonology, morphology and grammatical structures. the fact that performance drop occurs on distant languages when transferring from english indicates that additional works are needed to exploit connectivity between closely related languages, especially under extreme parameter constraints. besides purely relying on the orthographic representation in the format of written scripts, additional modality of languages such as articulatory signals can provide essential information beyond the written scripts to enhance textual representations @xcite . phonemic transcriptions which capture linguistic articulatory signals are beneficial to understanding non-latin-based languages when integrated with plms @xcite . they can also facilitate for knowledge transfer between languages sharing lexical similarities in phonology but possessing different writing scripts. as demonstrated in motivated by the inherent lexical similarities in terms of phonology among cjkv languages, we propose a novel cross-lingual transfer framework to integrate and synthesize two specific linguistic modalities (1) textual orthographic input scripts, (2) phonemic transcription, represented in international phonetic alphabet (ipa) format. our unified cross-lingual transfer framework aims to effectively (1) align both orthographic and phonemic transcriptions via multi-modality learning, (2) capture additional alignment between the two modalities via contextual information, (3) enhance crosslingual transfer of the two modalities with additional bilingual dictionary. our work specifically targets chinese-vietnamese-japanese-korean languages which are not well-studied in cross-lingual transfer and possess lexical similarities with one another in terms of phonology. our contributions can be summarized as follows: cross-lingual transfer recent works in crosslingual transfer focus on generating multilingual contextualized representation for different languages based on the pre-trained language models (plm) via bilingual dictionaries @xcite and/or machine translation approaches @xcite . @xcite proposes a comprehensive code-switching technique via random selection of languages, sentences, and tokens to enhance multilingual representations, leading to improved performance on target languages on different downstream tasks. on the other hand, other approaches leverage parallel corpora generated by machine translation to (1) distill knowledge from source languages to target languages @xcite or augment source language data with target language knowledge during training @xcite . however, current cross-lingual efforts concentrate on single source language (en) to multiple target languages. under parameter capacity constraints, cross-lingual transfer has been shown to be biased towards high-resourced languages which share similar scripts and possess larger corpora of unlabeled data during pre-training @xcite . unlike previous works, we specifically target enhancing performance of low-resourced languages by exploiting inherent linguistic similarities between closely-connected languages @xcite . multi-modality learning multi-modality learning @xcite @xcite was initially proposed for the task of visual-question answering @xcite . the objective is to find alignment between the given images and textual input (i.e. caption). the two aligned modalities are trained to maximize the agreement with ground truth textual-image alignment. despite its simple objectives, the clip @xcite pre-training mechanism is considered the state-ofthe-art in multi-modality representation learning. motivated by multi-modality learning, we integrate multi-modality learning approaches in unifying two modalities of transcriptions (orthographic and phonemic) for better representation enrichment. ## problem formulation in this work, we study the problem of cross-lingual transfer in a bilingual setting where there exists annotated data collection of high-resource language s, namely formally, given an i-th input source language utterance with the length of m orthographic tokens ## proposed framework in this section, we introduce our proposed crosslingual transfer framework, namely phonexl, with 3 major learning objectives: (1) orthographic-phonemic alignment (l align ), (2) contextual cross-modality alignment (l m lm ) , (3) contextual cross-lingual alignment (l xm lm ). the overall learning objective is summarized as follows: ## result & discussion our experimental results for ner and pos tasks are summarized in in ner task, the larger and state-of-the-art multilingual plm xlm-r yields worse performance than mbert in both of our language pair performance on source and target languages. on the other hand, for pos task, xlm-r based architecture only results in marginal performance gains when compared with mbert. interestingly, our mbert-based framework achieves competitive performance with xlm-r backbone architecture on pos task despite leveraging smaller vocabulary size and less pre-training data. we hypothesize this might be due to the fact that xlm-r has been trained with more languages, leading to biases towards certain types of languages during pre-training that might not share common properties with cjkv languages. despite the performance gain over xlm-r based architecture observed in ## conclusion & future work in our work, we propose phonexl, a novel mechanism to integrate phonemic transcription with orthographic transcription to further enhance representation capability of language models in cross-lingual transfer settings. by encouraging the alignment between the two linguistic modalities via direct one-to-one alignment, indirect contextual alignment and additional code-switching via bilingual dictionaries, our proposed phonexl yields consistent performance improvements over the backbone orthographic-based plm architecture in downstream cross-lingual token-level task among the cjkv languages. we also release the first aligned phonemic-orthographic datasets for cjkv languages for two popular token-level tasks (ner and pos). in future work, we plan to train our proposed unsupervised objectives with larger cjkv corpora as pre-training mechanisms to evaluate effectiveness of the representations in multigranularity downstream tasks (i.e. sentence-level classification tasks to question-answering tasks). further extensions towards few-shot learning settings @xcite where a small number of target language examples can be leveraged to exploit orthographic-phonemic similarity between source and target languages is a promising direction for our future work.