S cript M ix: Mixing Scripts for Low-resource Language Parsing (2024)

Abstract: Despite the success of multilingual pretrained language models (mPLMs) for tasks such as dependency parsing (DEP) or part-of-speech (POS) tagging, their coverage of 100s of languages is still limited, as most of the 6500+ languages remains “unseen”. To adapt mPLMs for including such unseen langs, existing work has considered transliteration and vocabulary augmentation. Meanwhile, the consideration of combining the two has been surprisingly lacking. To understand why, we identify both complementary strengths of the two, and the hurdles to realizing it. Based on this observation, we propose ScriptMix, combining two strengths, and overcoming the hurdle.Specifically, ScriptMix a) is trained with dual-script corpus to combine strengths, but b) with separate modules to avoid gradient conflict. In combining modules properly, we also point out the limitation of the conventional method AdapterFusion, and propose AdapterFusion+ to overcome it. We empirically show ScriptMix is effective– ScriptMix improves the POS accuracy by up to 14%, and improves the DEP LAS score by up to 5.6%. Our code is publicly available.

## introduction recently, pretrained language models (plms) have become the de facto standard for solving various natural language processing tasks, such as dependency parsing and part-of-speech tagging. to support a wide range of languages, multilingual plms (mplms) have been designed with a shared architecture that incorporates multiple languages, yielding promising results. for example, mbert @xcite trains with wikipedia articles in 104 languages to share a common feature space. however, considering the availability of 6500+ languages, the majority of languages are still unseen by mbert. building another mplm to include new languages incurs massive retraining costs. even if such cost is afforded, performance is reportedly low, due to resource imbalance and limited model capacity @xcite . a better alternative is to adapt mplm by "specializing" foot_0 it for an unseen language using the following two approaches, as illustrated in figure 1 . when representing the word a from an unseen language, sharing the same semantics with a ′ from a high-resource language (hrl), vocabulary augmentation (va) @xcite augments lowresource language (lrl) tokens, by treating unk as original lrl script such as ' ' in the figure. transliteration (tl) @xcite replaces unk with the script of an hrl, e.g., romanize a, which can generate a common token with a ′ , shown as a blue word, 'lad'. we first observe complementary aspects of tl and va (figure 1 ). tl, by sharing a common token, transfers the semantics from hrls better, than va using the original script that cannot overlap with hrl. however, this comes with the risk of a "false positive" transfer. to illustrate, consider two red tokens ' ' and ' ', sharing the same sound with different semantics. tl maps them to the same token 'bang', even if the semantics originally differ. meanwhile, va does not suffer from false positives by preserving original scripts, but cannot increase token overlaps. inspired by the complementary strengths of va and tl, we propose a novel method, scriptmix, that combines the advantages of both techniques. script-mix, as the name suggests, mixes both scripts to obtain token overlap with hrls by tl, while resolving potential false positives by source script from va. we implement scriptmix upon an adapterbased architecture @xcite , a stateof-the-art architecture for specialization at this moment, which devises language module (la) and task module (ta) as in figure 2a ). the first component of scriptmix is dual-script corpus, which mixes the original corpus from va and the transliterated corpus from tl ({s s }, {s t } in figure 2b ). however, we argue this component by itself does not fully realize complementarity, as we later attribute to gradient conflict (figure 3 ). we devise our second component to overcome this hurdle: language module separation. to avoid gradient conflict from different scripts, we train separate adapters for each script (la s and la t in figure 2c ) . then we linearly combine the outputs from these conflict-free adapters with a dynamically calculated weight vector. to calculate such a vector, the most natural way has been adapterfusion @xcite . despite the benefits of adapterfusion, we have found it to have a harmful inductive bias that compromises its performance. to overcome this issue, we propose our third component, adapterfusion+, which effectively eliminates the harmful inductive bias and improves overall performance. we evaluate scriptmix on dependency parsing and part-of-speech tagging-the human-annotated tasks with more low-resource languages than other nlp tasks. combining all three components, scriptmix could extract complementary knowledge from both scripts. to demonstrate the generalizability of scriptmix, we also apply it for fusing unipelt @xcite , where existing fusion techniques fail to lead to performance gains. script-mix improves the pos accuracy by up to 14%, and improves the dep las score by up to 5.6%. our contribution can be summarized as follows: expanding mplm new languages can be added during pretraining multilingual models, though lrl performance is degraded due to resource imbalances @xcite . specializing mplm specializing mplm is a more effective alternative to overcome imbalances. from the data perspective, tl @xcite and va @xcite @xcite , compared and contrasted in figure 1 , are proposed as specialization methods. from the architecture perspective, one may add additional modules with few parameters for specialization. out of the multilingual domain, modules such as adapters @xcite , or unipelt @xcite have been proposed. among them, adapters have been more broadly applied for cross-lingual transfer. mad-x @xcite trains language adapters (las) and task adapters (tas). then mad-x stacks them to utilize knowledge from both unlabeled and labeled datasets, achieving a state-of-the-art. we use this as our baseline architecture, and also generalize to unipelt. our distinction our distinction is observing the complementarity of va and tl, for better specialization to an unseen language. va preserves the original script, and tl increases token overlaps, with the risk of false positives (figure 1 ). our contribution is achieving the complementarity between va and tl, through realizing and resolving the hurdles for the goal. the easiest way to integrate knowledge from two different corpora would be a polyglot corpus @xcite . we devise a dual-script corpus inspired by this approach, but we argue that training one module with a polyglot corpus is suboptimal. to overcome, we propose to train two different adapters and add a component to fuse the knowledge. to fuse two adapter outputs, adapterfusion @xcite devises an attentionbased fusion mechanism. however, we observe that adapterfusion is suboptimal in our scenario, and propose to reduce inductive bias from it. existing work on reducing inductive bias includes mlp-mixer @xcite -it reduces inductive bias from convolutional neural networks by replacing them with mlps, which was effective for our purpose of bias reduction from adapterfusion. let h be the output of a layer from an mplm. mad-x prepares la @xmath0, where ## experiments we mainly evaluate our method on few-shot training, which assumes a more low-resource scenario. we also validate that scriptmix is effective when more train data is allowed. ## conclusion and discussion we studied how to specialize mplm to unseen languages and introduced scriptmix, the complementary approach to merge the benefits of tl and va. we verified our effectiveness across a comprehensive array of languages. while our focus was to show the feasibility of script-mixing, future work could explore combining tl with improved va methods, such as @xcite .