Cross-lingual Transfer for Automatic Question Generation by Learning Interrogative Structures in Target Languages (2024)

Abstract: Automatic question generation (QG) serves a wide range of purposes, such as augmenting question-answering (QA) corpora, enhancing chatbot systems, and developing educational materials. Despite its importance, most existing datasets predominantly focus on English, resulting in a considerable gap in data availability for other languages. Cross-lingual transfer for QG (XLT-QG) addresses this limitation by allowing models trained on high-resource language datasets to generate questions in low-resource languages. In this paper, we propose a simple and efficient XLT-QG method that operates without the need for monolingual, parallel, or labeled data in the target language, utilizing a small language model. Our model, trained solely on English QA datasets, learns interrogative structures from a limited set of question exemplars, which are then applied to generate questions in the target language. Experimental results show that our method outperforms several XLT-QG baselines and achieves performance comparable to GPT-3.5-turbo across different languages. Additionally, the synthetic data generated by our model proves beneficial for training multilingual QA models. With significantly fewer parameters than large language models and without requiring additional training for target languages, our approach offers an effective solution for QG and QA tasks across various languages.

## introduction automatic question generation (qg) aims to generate questions based on a given context. qg models have been employed not only to augment questionanswering (qa) datasets but also to generate educational materials and develop chatbots. several qa datasets have been proposed, including @xcite , hotpotqa @xcite , and quac @xcite . however, the majority of these datasets are in english, resulting in a significant lack of data for other languages. moreover, translating english datasets into other languages or creating new qa datasets, despite the availability of similar english datasets, is often inefficient in terms of both time and financial resources. recently, researchers have concentrated on cross-lingual transfer (xlt) to address data deficiencies in non-english languages @xcite @xcite @xcite . xlt involves deploying models trained on english datasets to other languages when annotated data in the target language is limited or unavailable. additionally, in recent years, multilingual large language models (mllms), such as gpt-4 @xcite , bloom @xcite , and palm @xcite , have exhibited remarkable performance across various natural language generation (nlg) tasks, often achieving high efficacy through zero or few-shot inference. however, significant cost burdens are associated with utilizing commercial apis, and employing open-source llms requires substantial computational resources. previous studies on xlt for qg (xlt-qg) have typically utilized target language data, such as monolingual corpora, sourcetarget parallel corpora, or a limited number of qa examples @xcite @xcite @xcite . nevertheless, incorporating languagespecific data during model training can lead to inflexibility in language scalability, necessitating additional training efforts for applications in new languages. in this paper, we present a simple and efficient xlt-qg method that generates questions by learning interrogative structures in target languages (quist). quist comprises two stages: 1) ques-tion type classification (qtc) and 2) qg utilizing question exemplars. we categorize questions into eight types based on english interrogative words, and the qtc model determines the type of question to be generated based on the input context and answer. once the question type is identified, it is used to select the corresponding question exemplars for the qg stage. the qg model generates questions based on a given input context, answer, and question exemplars. during training with english data, the model learns to identify the interrogative structures specific to each question type from the provided exemplars. this approach enables the model to generate questions that are not only semantically aligned with the input context and answer but also syntactically similar to the exemplars. by training exclusively on english data, we ensure that the model can generate questions in other languages without requiring additional training. in our experiments, we evaluate the performance of quist across nine linguistically diverse languages. through both automatic and human evaluation, we show that quist outperforms various xlt-qg baselines and achieves performance comparable to gpt-3.5-turbo in several languages. furthermore, we confirm that synthetic questions generated by quist are more effective for training high-performance multilingual qa models than those generated by gpt-3.5-turbo. our contributions can be summarized as follows: ## cross-lingual transfer for automatic question generation the zero-shot xlt approach-leveraging multilingual pretrained language models (mplms) fine-tuned exclusively on english data for target languages-has shown promising performance across various classification tasks @xcite @xcite @xcite . however, when applied to natural language generation (nlg) tasks, this approach often results in catastrophic forgetting of the target language. to mitigate this issue,maurya et al. (2021) proposed fine-tuning only the encoder layers of mplms while keeping the word embeddings and all decoder layer parameters frozen. finnish synthetic question (mt5) : how long is pyhäjärven pituus? synthetic question (mbart) : how pitkä on pyhäjärven muoto? ground truth : kuinka pitkä pyhäjärvi on? translation: how long is pyhäjärvi? korean synthetic question (mt5) : when did 카를 마르크스 죽었다? synthetic question (mbart) : when was 카를 하인리히 마르크스's birthday? ground truth : 마르크스는 언제 사망하였는가? translation: when did marx die? telugu synthetic question (mt5) : how many వే రు వే రు మహాసముద్ర ా లు ఉన్రాయి? synthetic question (mbart) : how many మహాసాగరాలుగా గురత ి సా ి రు? ground truth : మహా సముద్ర ా లు ఎన్నా ఉన్రాయి? translation: how many great oceans are there? figure 1 : questions generated by mt5 @xcite and mbart @xcite fine-tuned on english qa datasets. the questions often contain english interrogative expressions such as "how long" and "when did." in our preliminary investigation, we found that this technique did not completely prevent codeswitching in xlt-qg, as shown in figure 1 . specifically, the models struggled to fully grasp interrogative structures in the target language, a phenomenon we refer to as "interrogative codeswitching." in this study, we propose a method that enables small mplms to learn interrogative structures without relying on target language data during training. as illustrated in figure 2 , we divide the task into two stages. in the qtc stage, a classification model identifies the type of question to be generated. we focus on wh-questions, categorizing them into eight types based on english interrogative words. while the type of question is primarily influenced by the type of answer, the model considers both the answer and the context. this is crucial, when what how … [cls] caprivi [sep] namibia is … the highest rainfall occurs in the caprivi … [sep] [cls] 1930 [sep] ... 월드컵은 1930년에 첫 대회가 열렸다. ... [sep] (the first world cup was held in 1930.) where does the highest rainfall occur in namibia? 월드컵은 언제 처음 시작되었나요? (when did the world cup first start?) [training stage] where what when … [inference stage] question exemplars: where do you find a lead out? where does morphine come from? … where are the most visited monuments located in paris? answer: caprivi context: namibia is … the highest rainfall occurs in the caprivi … question exemplars: 최초의 신호등은 언제 세워졌나요? 항균 내성 확산이 자주 발생하는 시기는 언제입니까? ...하드캔디는 언제 출시됐나요? (when was the first traffic light erected? when does the spread of antimicrobial resistance often occur? ...when was hard candy released?) answer: 1930 context: ...월드컵은 1930년에 첫 대회가 열렸다. ... (the first world cup was held in 1930.) question generator question type classifier as the same answer can result in different types of questions depending on the context. for example, the number "911" could refer to a quantity, year, or proper noun. the set of question exemplars corresponding to the question type identified by the qtc model is used in the qg stage. these exemplars are precreated for each question type and language, as detailed in section 3.1. by leveraging the shared interrogative structures among the exemplars, the qg model generates questions using the provided answer and context. both the qtc and qg models are trained exclusively on english qa data and can be deployed to new languages without the need for additional training with target language data. ## experimental setup in this section, we describe the datasets and baseline models we used in our experiments. details regarding the implementation and evaluation metrics are provided in appendices b and c.1, respectively. ## main results comparison with baselines quist notably outperforms baseline enc across all languages. note that both models have the same number of trainable parameters during the finetuning stage. these results indicate that exposing the model to interrogative structures during the inference stage significantly enhances its ability to generate questions in the target language. despite baseline m ulti learning questions in the target language via the denoising task, it exhibited poor performance, even scoring lower than baseline enc . upon reviewing the generated results of baseline m ulti , we frequently observed instances where the questions were unrelated to the input context or answer. these findings suggest that utilizing a small number of question exemplars during the training stage may lead to overfitting, resulting in a decline in model performance. ## conclusion in this paper, we proposed a simple yet effective xlt-qg approach, where the question generation model is trained solely on an english qa dataset and leverages a small set of target language questions during inference. by incorporating question exemplars from target languages, our method enables the model to learn the interrogative structures of those languages, effectively addressing the issue of code-switching. experimental results demonstrate that this approach significantly outperforms several xlt-qg baselines and achieves performance comparable to gpt-3.5-turbo across a variety of languages. additionally, we validated the utility of our method's synthetic data for training multilingual qa models. a key strength of our method lies in its scalability and parameter efficiency, as it relies exclusively on english qa data during training. this enables the seamless extension to new languages without the need for additional parameter updates. moreover, in contrast to llms, our approach employs smaller backbone models, offering the advantages of lower deployment costs and reduced computational requirements, making it more accessible for practical use in diverse multilingual settings.