Towards Low-resource Language Generation with Limited Supervision (2023)

Abstract: We present a research narrative aimed at enabling language technology for multiple natural language generation (NLG) tasks in low-resource languages (LRLs). With approximately 7,000 languages spoken globally, many lack the resources required for model training. NLG applications for LRLs present two additional key challenges: (i) The training is more pronounced, and (ii) Zero-shot modeling is a viable research direction for scalability; however, generating zero-shot well-formed text in target LRLs is challenging. Addressing these concerns, this narrative introduces three promising research explorations that serve as a step toward enabling language technology for many LRLs. These approaches make effective use of transfer learning and limited supervision techniques for modeling. Evaluations were conducted mostly in the zero-shot setting, enabling scalability. This research narrative is an ongoing doctoral thesis.

## introduction recently, there has been remarkable progress in natural language processing (nlp) research, primarily due to advancements in large pre-trained language models (plms). the global linguistic landscape comprises approximately 7,000 spoken languages worldwide foot_1 . a notable disparity is evident in nlp research, with the majority of studies conducted on english data @xcite . this is concerning as the vast majority of the global population -roughly 95%does not speak english as their primary language, and a staggering 75% do not speak english at all foot_2 . according to @xcite , out of the 7,000 languages, approximately 400 languages have more than 1 million speakers, and about 1,200 languages have more than 100,000 speakers. despite this, only around 100 languages are incorporated into large pre-trained models, and limited resources are available for building nlp models for lrls. furthermore, a study presented at acl 2008 @xcite revealed that 63% of all papers focused only on english. a more recent study during acl 2021 @xcite concluded that nearly 70% of the papers were evaluated on english. even a decade later, there has been little change. the nlp application involving text generation (nlg tasks) in lrls presents additional challenges in model development: (1) the scarcity of nlg resources for model development in lrls is more pronounced than other nlp tasks. (2) lrls often exhibit a long tail, with many lacking annotated data. the preferred solution is zero-shot modeling, though this approach introduces additional challenges for cross-lingual generation tasks. it has been observed that zero-shot generation models frequently encounter issues like catastrophic forgetting @xcite or accidental translation @xcite . due to these problems, the zero-shot generated text is either codemixed or not in the intended target language. (3) lrl modeling typically employs a transfer learning setup, where supervision is transferred from hrls to lrls. however, performance tends to degrade for lrls that are different from their hrl and (4) many lrls lack monolingual or parallel data, and their representations are absent from plms. these lrls are referred to as extremely lrls (elrls) or dialects. despite having millions of speakers, there is a noticeable absence of nlp technology for these elrls. this thesis is a step towards addressing these challenges and aims to enable language technology for lrls, thereby democratizing nlp research for the general population/audience. prior to the emergence of transformers-based plms, most works in cross-lingual generation were primarily reliant on machine translation (mt) systems. existing models either directly employed the mt system within the modeling @xcite or generate training data using mt @xcite to develop models. this dependence on mt not only limits scalability but also propagate error with translation. to address these limitations, multilingual plms (mplms) have emerged @xcite , where a large set of languages share a common latent representation space. the cross-lingual models built on top of these mplms lead to the remarkable advancement @xcite in the cross-lingual transfer in zero-shot or few-shot settings. however, most of these advancements are limited to nlu tasks. furthermore, existing cross-lingual nlg models incorporate one or more challenges mentioned above. with this thesis, our contributions are as follows: in this section, we provide high-level details of the proposed models. this also includes insights into how we build more recent proposed models based on earlier models and advance the field. then, we look back and position our research efforts by contextualizing a broader spectrum of multilingual research, specifically for low-resource language generation. finally, we list our learnings from failed and successful modeling. ## mitigating catastrophic forgetting to enable zero-shot language generation our research mission to enable language technology for nlg tasks in lrls started with zmbart @xcite work. zmbart is an unsupervised cross-lingual transfer and generation framework that focuses on generative tasks for lrls in zero-shot and few-shot settings. a typical zero-shot cross-lingual generation modeling involves two main steps: (1) training with hrls: train (fine-tune) a model (plm) using a large annotated dataset from hrls, typically english. for instance, training with english abstractive text summarization (ats) dataset. (2) zero-shot generation in lrls: utilize the trained model for zeroshot inference. for instance, when given input in an lrl (e.g., hindi), the model generates a summary in the same lrl (hindi). unlike natural language understanding (nlu) tasks, the cross-lingual generation task in zero-shot scenarios is particularly challenging. this is because the zero-shot generated text needs to be in the target lrl, which generally suffers from catastrophic forgetting (cf; van de ven et al. (2022)) or accidental translation (at; xue et al. (2021)) problems. due to this, the model fails to generate text in the target lrl or produce code-mixed output with both high-resource and lrls. with this work, our objective is to alleviate cf and at problems with an unsupervised framework, meaning we do not rely on any parallel or pseudo-parallel/back-translated data. instead, we harness multilingual pre-trained checkpoints, specifically the mbart model @xcite , to seamlessly enable the generation of well-formed text in lrls across multiple generative tasks. prior to zmbart, existing cross-lingual generation models were grounded with either machine translation (mt) or parallel/back-translated datasets. @xcite employed the mt pipeline to facilitate cross-language document summarization. this involves the translation of non-english input into english. subsequently, the english ats model was employed to procure the summaries, which were finally translated back into non-english languages. similar approaches are adapted by @xcite and @xcite . this direction is not feasible as mt systems are not available for many lrls and the imperfect translations propagate errors. considering this, @xcite and @xcite use back-translated (need mt system) and parallel datasets to develop the few-shot cross-lingual question and answering (q&a) and zero-shot cross-lingual ats, respectively. these approaches require an mt system or annotated dataset which limits the model development to a few hrls. unlike these, we propose zmbart, the first unsupervised scalable model based on mbart specialized for zero-shot crosslingual transfer and generation. additionally, we have also created hidg foot_3 , a high-quality distractor generation dataset in the hindi language. ## meta-learning approach to improve zero-shot language generation the effort with the zmbart helps in effectively mitigating ct/at problems and generating zeroshot outputs in target lrls seamlessly. in this work, we leverage these findings and extend the study to improve the cross-lingual supervised signals to boost the performance for zero-shot genera-tion. there are more than 7000 languages across the globe. 95% of the world's population does not speak english as their first language and 75% does not speak english at all foot_4 . however, the majority of nlp research is focused on the english language @xcite . to democratize the nlp research for the benefit of the large global community, it is essential to focus on non-english languages. recently, cross-lingual transfer learning @xcite has emerged as a promising research direction where a model is trained on hrl(s) and transfer supervision to lrl(s). however, the supervision transfer is uneven across languages, which leads to large performance gaps. such performance gaps are observed because models do not account for cultural and linguistic differences in the modeling @xcite . this work was a step towards bridging this performance gap. meta-learning or learning to learn @xcite has emerged as an active research direction to learn shareable structures across multiple tasks with limited annotated data. the only constraint is all tasks should share some common structure (or come from a task distribution). different languages in the world follow this constraint as they come into existence with a common goal of communication and share some structure. so, we consider languages as tasks. the meta-learning approach has been actively applied to multiple nlp tasks @xcite including text classification @xcite , ner @xcite , dialogue systems and q&a @xcite . there were few efforts made in the multilingual setup @xcite ; however, these are limited to machine translation or nlu tasks only. this work -to the best of our knowledgewas the first attempt to study meta-learning techniques for cross-lingual natural language generation (x nlg ). particularly, we focus on zero-shot x nlg for low-resource languages. unlike nlu tasks, the zero-shot nlg is a more challenging setup due to the typological diversities of languages and cf/at problems. we refer to this framework as metax nlg foot_5 @xcite , a framework for effective cross-lingual transfer and gen-eration based on language clustering and model-agnostic meta-learning (maml) algorithm @xcite . following are the main contributions: (1) we propose a novel metax nlg framework based on language clustering and meta-learning to improve zero-shot generation performance for typologically diverse lrls. (2) we have conducted an extensive empirical evaluation with 30 languages (29 lrls), covering two tasks (qg and ats) and using 5 popular datasets (xl-sum, wikilingua, mlqa, tydiqa, and xquad). ## utilizing lexical similarity to enable zero-shot mt for extremely lrls the efforts with zmbart, metax nlg , and the nlp research community on multilingual modeling have extended the coverage of nlp technologies for many lrls. however, there is a long-tail of languages for which there is no parallel/pseudoparallel data, no/limited monolingual data, and their representations from the multilingual language model are absent. these fall into categories of extremely low resource languages (elrls) or dialects. with this work @xcite , we made a step towards enabling technology for el-rls where resources are limited (zero-shot setting). in particular, our focus was on the machine translation (mt) task, driven by the availability of a true evaluation test set from recently released sources such as @xcite . fortunately, many of these elrls are lexically similar to closely related hrls. lexical similarity refers to languages sharing words with similar form (spelling and pronunciation) and meaning. 7 this includes cognates, lateral borrowings, and loan words. for example, the word lgta (lagta) in hindi (hrl) is spelled as lagata (laagata) in bhojpuri (lrl). existing cross-lingual transfer methods based on common embedding spaces work best between related languages @xcite . so, if we make the hrl model robust to spelling variations, it will improve cross-lingual transfer to related elrls. to achieve this, we introduce unigram character and character-span noise augmentation approaches, charspan, to improve generalization in zerosshot. the noise injection acts as a regularizer. a sample example is presented in fig. 3 . formally, we look at a machine translation task from an elrl to another language (english) with transfer enabled by a related hrl on the source side. the character-level noise augmentation has been employed to improve the robustness and adversarial testing @xcite @xcite ) for mt systems. there are general noise augmentation techniques @xcite ) that help in cross-lingual transfer. @xcite introduced unigram character noise augmentation for nlu tasks such as ner, pos tagging, and topic classification. in contrast, we propose charspan noise augmentation for the more challenging mt task. there is another line of works that leverages lexical similarity based on vocabulary overlap @xcite , non-deterministic segmentations @xcite , and soft decoupled encoding @xcite . while these approaches typically require certain amounts of monolingual data, our proposed model operates without such constraints, eliminating the need for monolingual data. with this work, our key contributions are: (a) we show that unigram character and character-span level noise augmentation can improve zero-shot translation from elrls to english. charspan model outperforms the unigram model. (b) the proposed approach is generalized across three typologically diverse language groups which include 6 hrls and 12 elrls. ## conclusion with this thesis, we have presented a coherent narrative of our efforts in the field of text generation for multiple lrls with limited supervision. we began by enabling zero-shot well-formed text generation, then progressed to improving cross-lingual generation, and ultimately enabled zero-shot machine translation for elrls and dialects. our modeling approaches are aligned with adaptive training, meta-learning, language clustering, lexical similarity, and noise augmentation. the evaluations were conducted across a wide range of lrls across language families, multiple nlg tasks, and datasets. through these endeavors, we have taken a step towards facilitating language technology for the long tail of languages that possess limited or no linguistic resources. this advancement aims to benefit the general audiences where text needs to be generated in local languages. in the future, we will explore the following directions: (1) extend the existing modeling framework to cover 7000+ spoken languages of the world. (2) design a single unified and scalable framework for many nlg tasks and lrls. (3) develop a better modeling approach to adapt the existing multilingual plm representations to new/unseen lrls. (4) since for many elrls there are no evaluation datasets, we will explore a modeling technique where the performance of lrls is evaluated without reference. (5) creating a large-scale multilingual nlg benchmark similar to @xcite . (6) investigating active learning, prompting, and other trending methodologies to advance cross-lingual transfer and generation research with limited supervision.