Does Meta-learning Help m BERT for Few-shot Question Generation in a Cross-lingual Transfer Setting for Indic Languages? (2022)

Abstract: Few-shot Question Generation (QG) is an important and challenging problem in the Natural Language Generation (NLG) domain. Multilingual BERT (mBERT) has been successfully used in various Natural Language Understanding (NLU) applications. However, the question of how to utilize mBERT for few-shot QG, possibly with cross-lingual transfer, remains. In this paper, we try to explore how mBERT performs in few-shot QG (cross-lingual transfer) and also whether applying meta-learning on mBERT further improves the results. In our setting, we consider mBERT as the base model and fine-tune it using a seq-to-seq language modeling framework in a cross-lingual setting. Further, we apply the model agnostic meta-learning approach to our base model. We evaluate our model for two low-resource Indian languages, Bengali and Telugu, using the TyDi QA dataset. The proposed approach consistently improves the performance of the base model in few-shot settings and even works better than some heavily parameterized models. Human evaluation also confirms the effectiveness of our approach.

## introduction qg can be defined as the task of generating an appropriate question based on the answer tokens and the context. the previous state-of-the-art qg models are built using neural networks @xcite @xcite , and are trained on high-resource languages with availability of vast amount of manually annotated data for training. collecting and annotating such vast data for training on low-resource languages can be challenging and costly. crosslingual transfer learning has shown its effectiveness in many nlp applications @xcite @xcite ## methodology qg task is defined as to generate a (syntactically and semantically correct) question based on a paragraph and the relevant sequence of answer tokens present in it. in our cross-lingual transfer setting, we denote the source language labelled training data as d s train and the target language test data as d t test . the aim of our qg meta-learning algorithm is to train a model with d s train using minimum or zero resource of target language labelled data, such that it performs well on d t test . our base model in detail, as well as our proposed approach, is described below. ## experiments we evaluate our meta-learning based qg model in zero-shot and few-shot settings. this section covers details about the dataset used in our experiments followed by the implementation details with evaluation metrics. dataset: we conduct experiments on low resource indian languages having minimum amount of annotated data for qg. we use tydi qa 1 @xcite gold passage 1 https://github.com/google-research-datasets/tydiqa dataset for our experiments. the dataset contains triplets of passage, question and answer for 9 languages. we evaluate our method on bengali and telugu dataset. the sizes of the bengali and telugu dataset (train, dev), in terms of number of examples, are (2390, 113), and (5563, 669), respectively. for cross-lingual knowledge transfer, we additionally use english triplets from the same dataset (@xmath0; @xmath1. one should note that since the aforementioned dataset contains no test data, we consider development set as test data for all our experiments. for evaluating bengali, we consider english and telugu as the source languages, while we use english and bengali as the source languages for telugu. the purpose of mixing one indian languages is to learn different language distributions rather than single-source distribution. please note that we follow the same zeroshot and few-shot approach to our base models for fair comparison. experiment setup: we implement our algorithm using pytorch 1.1.0. our base model uses bert base multilingual cased with 12 transformer blocks, 12 self-attention heads and 768 hidden dimension, gelu activation, and dropout is 0.1. the maximum sequence length is set to 512 for the input. for the creation of pseudo-meta-qg task, we take only two @xmath2. each meta-training step performs two inner-update and a meta-update on a batch of 16 tasks. we train our model up to 6,000 metatraining steps. as described in @xcite , we freeze the embedding and the first three layers of the base transformer model, while the other layers are further fine-tuned for each task. other hyper-parameter settings are same as in @xcite . we use adam @xcite optimizer with learning rates of lr inner , lr @xmath3. we set learning rate of lr @xmath4. for few-shot experiment, we fine-tune the meta-trained model up to 60 steps. we evaluate the systems using bleu @xcite , meteor @xcite , and rouge-l @xcite ) scores foot_0 . during the training phase, we train our model using the source language's training data and save the model based on the accuracy of the source language's dev dataset. we carry out the training procedure for four random seeds. for few-shot setting, we randomly sample 5 datasets, and average over 4 training random seeds. ## conclusion in this work, we make use of mbert for qg task in few-shot cross-lingual transfer setting, and interestingly, we find that it actually performs better than mt5-base for all the settings, and better than mbart-50 for 16-shot setting in bengali. we then explore the use of meta-learning with mbert as the base model (meta-qg) and find that it achieves significant performance improvements compared to the mbert as well as mt5base, and surprisingly also outperforms mbart-50 for bengali. in the future, we plan to extend this framework to other natural language generation tasks, and also plan to study the effectiveness of data augmentation approaches.