Polyglot Prompt: Multilingual Multitask Prompt Training (2022)

Abstract: This paper aims for a potential architectural improvement for multilingual learning and asks: Can different tasks from different languages be modeled in a monolithic framework, i.e. without any task/language-specific module? The benefit of achieving this could open new doors for future multilingual research, including allowing systems trained on low resources to be further assisted by other languages as well as other tasks. We approach this goal by developing a learning framework named Polyglot Prompting to exploit prompting methods for learning a unified semantic space for different languages and tasks with multilingual prompt engineering. We performed a comprehensive evaluation of 6 tasks, namely topic classification, sentiment classification, named entity recognition, question answering, natural language inference, and summarization, covering 24 datasets and 49 languages. The experimental results demonstrated the efficacy of multilingual multitask prompt-based learning and led to inspiring observations. We also present an interpretable multilingual evaluation methodology and show how the proposed framework, multilingual multitask prompt training, works. We release all datasets prompted in the best setting and code.

## introduction the emergence of multilingual pre-trained language models @xcite @xcite enables different languages to be represented in a unified semantic space. as a result, a fine-tuned model of a data-rich language such as english can achieve decent transfer (e.g., zero-shot) performance in geographically, syntactically, or phonetically similar languages @xcite . the insufficient features learned by languages under lower-resource settings can thus be compenqa, sum, and nli represent different tasks, namely question answering, summarization, and natural language inference here. plm represents pre-trained language model, and "zh", "en", "fr", "ja", "de","es" denote different languages. despite the preliminary success in the lowresource scenarios using shared knowledge across languages in multilingual language models @xcite @xcite , the cross-lingual transfers have mostly occurred only within the boundary of the same task or similar tasks. can more conceptually-diverse tasks from different languages communicate together? while researchers have made some preliminary progress towards this direction @xcite @xcite @xcite , the scope of the "different" tasks had remained relatively narrow, such as limiting to tasks with the same sequence labeling form @xcite (e.g., named entity recognition, chunking, and part-of-speech), or different domains for the same task @xcite . unifying different tasks into one framework can be challenging if we are to avoid introducing additional task-specific parameterized modules. recently, the success of the prompting methods @xcite ) has provided us with new clues on unifying different tasks in the same framework without task-specific parameters by formulating all tasks as a pre-training problem with various frameworks such as the mask language model @xcite or the encoder-decoder model @xcite @xcite . in this paper, we leverage prompt techniques to cross the boundaries of different tasks and languages so that multiple tasks in different languages can be placed in a monolithic framework (as shown in fig. 1-(c ) as opposed to single task single language (fig. 1-(a) ) as well as multiple task single language learning (fig. 1-(b) )) to benefit from one another without requiring any task/languagespecific modules. we name this multilingual multitask training model as polyglot prompting (polyprompt). different tasks from different languages can then be seamlessly connected together by being reformulated as pre-training tasks. architecturally, we choose the encoder-decoder pre-training framework so that more nlp tasks could be unified, as compared to other architectures such as the mask language model that favors classification-based tasks. our explorations in this paper are driven by following research questions: q1: can different tasks from different languages benefit from each other by a monolithic framework? if the answer is "yes", can the performance be further improved by introducing more high-resource datasets that are more readily available? ## related work multitask & multilingual learning the developments of neural networks have made it easier to share information across tasks or languages. as such, in the past few years, there has been much work on multitask learning within the same language @xcite @xcite , or multilingual learning in the same or similar types of tasks such as sequence labeling @xcite and machine translation @xcite @xcite . however, the task of learning different languages and tasks simultaneously in a unified learning framework without task-or language-dependent parameters has remained unexplored. prompting methods prompting is a technique that aims to make better use of pre-trained knowledge by reformulating tasks at hand accordingly @xcite and thus allowing us to do more with one model by unifying signals cross tasks @xcite , languages @xcite , even modality @xcite . in this paper, we expand what a system can do by proposing multilingual multitask learning with prompting methods for connecting geographically diverse languages and linguistically different tasks, thereby allowing them to leverage one another effectively. answer the question based on the paragraph. | question: [question] | paragraph: [context] news: 連邦政府は１ ５０億マルクの税収 不足… answer the question based on the news. | question : what topic does this news belong to ? | (a) business and industry. (b) economy. (c) government and society. (d) market. | news : [news] text1: ฉั นก็ เลยไม่ แน่ ใ จจริ งๆ ว่ าทํ าไม... text2: sฉั นมั 7 นใจถึ งเหตุ ผล... answer the question based on paragraph 1 and paragraph 2 . | question : do paragraph 1 and paragraph 2 mean the same thing ? | ( a ) yes . ( b ) no . ( c ) maybe . | paragraph 1 : [text1] paragraph 2 : [text2] text: দু গ াপু জায় মিহষাসু র বেধ মধ িদেয় অ ভর ওপর ভর িবজয় দখােনা হয়... [text] in summary, plain text question answering prompt template 卡万•肖特 market topic classification nli polyglot prompt summarization no িহ ু বাঙালীরা য সমেয় তােদর সবেথেক বড় উৎসব দু গ াপূ জা উদযাপন কেরন ... ## multitask multilingual prompt training we unify different tasks from different languages by reformulating each nlp task as a sequenceto-sequence problem @xcite so that they can be connected by a multilingual pre-trained language model (e.g., mt5 @xcite ) that also adopts a sequenceto-sequence training objective. fig. 2 shows the overview of our proposed framework. each sample from different tasks will be re-formatted as a (prompt, answer) pair using pre-defined templates and then be fed into a multilingual pre-trained language model. formally, given a task set @xmath0, t 2 , ⋯, t n } with n tasks and corresponding prompt templates @xmath1, k 2 , ⋯, k n } (without loss of generality, we shall assume that each task has one prompt for easy understanding). first, we transform the samples in each task into a form that is understandable by the language model based on the predefined templates k. assume that (x i,j , y i,j ) ∈ z is the input and output pair for the j-th sample of the i-th task, where z contains all input and output sample pairs. the input-output pair (x i,j , y i,j ) for the j-th sample of the i-th task can be converted to (x i,j , ŷi,j ) through the predefined template k i , which can be formulated as: we choose a sequence-to-sequence language model to achieve multilingual multitask prompt training, where samples from n tasks will be the input of the chosen language model. the loss function is to maximize the log-likelihood of the output text and can be defined as: lang. and acc. denote "language" and "accuracy". summ., ner, qa, tc, sc, and spc are abbreviations for summarization, named entity recognition, question answering, topic classification, sentiment classification, and sentence pair classification. "-" indicates that the task is not used for evaluation. "misc." indicates that the dataset was artificially constructed or of unclear origin. ## conclusions we can provide the following preliminary empirical answers to our research questions. (1) can different tasks from different languages benefit from each other by a monolithic framework? yes. what's more, introducing more highresource datasets can further improve the tasks' performance involved in multitask prompt training.