Resources and Few-shot Learners for In-context Learning in S lavic Languages (2023)

Abstract: Despite the rapid recent progress in creating accurate and compact in-context learners, most recent work focuses on in-context learning (ICL) for tasks in English. However, the ability to interact with users of languages outside English presents a great potential for broadening the applicability of language technologies to non-English speakers. In this work, we collect the infrastructure necessary for training and evaluation of ICL in a selection of Slavic languages: Czech, Polish, and Russian. We link a diverse set of datasets and cast these into a unified instructional format through a set of transformations and newly-crafted templates written purely in target languages. Using the newly-curated dataset, we evaluate a set of the most recent in-context learners and compare their results to the supervised baselines. Finally, we train, evaluate and publish a set of in-context learning models that we train on the collected resources and compare their performance to previous work. We find that ICL models tuned in English are also able to learn some tasks from non-English contexts, but multilingual instruction fine-tuning consistently improves the ICL ability. We also find that the massive multitask training can be outperformed by single-task training in the target language, uncovering the potential for specializing in-context learners to the language(s) of their application.

## introduction the emergent ability of very large language models to understand unseen tasks from natural input text @xcite , referred to as in-context learning (icl), recently motivated a large body of work focused specifically on creating more efficient models able to understand a new task from human 1 all our templates and models are available on https: //github.com/fewshot-goes-multilingual/ slavic-incontext-learning sąd najwyższy sąd najwyższy нет нет учитывая {{premise}}, следует ли из этого, что " {hypothesis}}"? да, нет или возможно? учитывая {{premise}}, следует ли из этого, что " {hypothesis}}"? да, нет или возможно? "{{text}}" jaka encja typu nazwa organizacji znajduje się w poprzednim akapicie? "{{text}}" jaka encja typu nazwa organizacji znajduje się w poprzednim akapicie? {{review}} je táto recenze pozitivní nebo negativní? {{review}} je táto recenze pozitivní nebo negativní? учитывая {{premise}}, следует ли из этого, что " {hypothesis}}"? да, нет или возможно? "{{text}}" jaka encja typu nazwa organizacji znajduje się w poprzednim akapicie? instructions @xcite @xcite . the icl models presented in these works reduce the number of parameters compared to the first in-context learners by orders of magnitude. in exchange, they assume that the generalization to new tasks emerges from a vast mixture of diverse training tasks seen in the training process. the data volume and diversity requirements might also be the factor that substantially limits the application of current icl models mainly to english. acquiring a large and diverse set of tasks is relatively easy for english, which is in the spotlight of the nlp community. unfortunately, there are fewer datasets in other languages, and the collection of new ones is costly. previous work addresses this problem by automatic translation of some english datasets @xcite , or by a cross-lingual training @xcite and evaluation @xcite . however, such approaches do not resemble the use of instruction models by non-english speakers, expecting the models to interact solely in their native language. this work evaluates the quality of in-context learning achievable in non-english languages to this date, specifically focusing on applicability in few-shot in-context learning for interaction in selected slavic languages (figure 1 ). further, we assess the possibilities of further improvement under the assumption of limited data availability in the target language. we formulate these goals in two research questions: rq1: how well can recent in-context few-shot learners perform in the interaction purely within our chosen, non-english languages? rq2: can the improvements of in-context learning in a large-resource language transfer to lower resource, target languages? given very limited previous work in in-context learning in our target languages, within our work, we first (i) survey and transfer a diverse set of datasets to instructional format through a set of transformations and newly-collected database of prompting templates with both the instructions and labels written in our target language(s). our collected tasks include datasets for named entity recognition, sentiment classification, natural language inference, and question answering in our target languages. after collecting the datasets of diverse tasks in the icl-compatible format, we (ii) survey and evaluate in-context few-shot learners that can be applied to our target languages. finally, we (iii) explore the possibility of further improving the in-context learners specific for our target languages along two axes: (a) by increasing models' exposure to target-language data and (b) by improving icl ability in high-resource language, evaluating the cross-lingual transfer of such improvements. this paper is structured as follows. section 2 overviews the standard settings of in-context fewshot learning and surveys the previous work in this direction. section 3 describes the evaluation datasets that we use and covers datasets' selection and unification process and templates database collection. section 4 presents the settings used for training our in-context learners for czech, polish, and russian. finally, section 5 presents the evaluation results, including existing and newly-trained in-context learners in the supervised baseline. ## background in-context learners in-context learning from both human prompt and a set of input-output examples is initially observed as an emergent ability of gpt-3 @xcite trained on a vast collection of unlabelled texts for causal language modeling (clm) objective @xcite . subsequent work reproduces icl ability and open-sources the resulting models, such as bloom @xcite or opt @xcite . however, in-context learners trained in a solely unsupervised fashion are impractically large and hence, expensive for conventional use; in unsupervised settings, the icl ability seems to emerge only when using far over 10 billion parameters @xcite , thus requiring an extensive infrastructure to perform a single inference. computational overhead is addressed by a series of smaller models trained specifically for incontext learning. the smaller in-context learners are trained with a large mixture of tasks converted to a consistent sequence-to-sequence format via human-written templates (bach et al., 2022) that define the input prompts for each task in the collection. a popular use of this framework includes prefixing the input sequence with natural-language instructions, such as the ones given to human annotators @xcite . large-scale instructionbased prompting in training over 1,600 tasks is also adopted in training tk-instruct (wang et al., 2022) that we assess in our evaluations. recently, more attention has been dedicated to a selection of in-context training tasks under the assumption that some training tasks might be more beneficial for the emergence of in-context learning than others. in this direction, @xcite further extends a database of tasks with the ones requiring multi-step reasoning in a chain-of-thought manner, where additionally to the correct prediction, the model is trained to predict a sequence of steps mapping the input to an output. in-context few-shot learning in-context learners are easily applicable in few-shot evaluation settings, where a small set of demonstrations for a given task exists. given a dataset d : {(x 1 → y 1 ), . . . , (x @xmath0, an in-context few-shot learner θ(x) → y aims to predict a correct y k+1 ≡ y k+1 given input text containing a sequence of k input-output demonstrations, and the predicted input x k+1 @xcite : contrary to the standard supervised learning, in in-context learning, model θ is not updated. thus, it can rely solely on its ability to understand the task from input text. similarly to humans, the specific wording of input, i.e., prompt x j , might play a large difference in the evaluation performance of the model. a prompt formulation optimal for one model type is likely not optimal for another @xcite . therefore, in order to fairly compare different in-context learners, one should evaluate in-context learners on a larger set of diverse prompts @xcite . with this motivation, we also collect multiple prompts for each task, with a focus on their mutual diversity. ## datasets the evaluation and training of new in-context learners for our target languages require (i) a collection of datasets for a representative range of tasks, and (ii) the transformation of these datasets into a unified, self-containing sequence-to-sequence form of inputs and outputs. thus, one of our main contributions is the adaptation of the datasets for czech, polish, and russian in a range of tasks: named entity recognition, sentiment classification, natural language inference, and question answering. the overview of the datasets for our target languages is shown in this section overviews the datasets in the target languages that we transformed, followed by a description of the process of constructing the templates for these datasets. ## experiments making in-context learning in our target languages finally possible through the transformations described in the previous section, our first objective is to assess the current state-of-the-art of the recent in-context few-shot learners when used in the interaction exclusively in the target language (rq1). we follow by outlining the perspectives in further enhancing the quality of target-language incontext few-shot learners by assessing the potential of cross-lingual transfer (rq2). ## results consistently with the previous work @xcite , we jointly report the rouge-l score @xcite over all the evaluation datasets (which we transform and create templates for ( §3)) and all the evaluated in-context learners ( §4.1), including the newly-trained ones introduced in this work ( §4.2). to ease the readability, we split the reports by language, to the results on czech datasets in as a reference of the resulting icl performance, for each dataset, we also train a baseline model that is also based on mt5 model @xcite , fine-tuned on the training split of the dataset transformed to a sequence-to-sequence format through a mixture of all the templates that we curated. details on the training and evaluation configuration that we use can be found in appendix b. multilingual training helps in most cases a comparison of mtk-instruct to tk-instruct of the same size through all languages (tables 3 , 5 , 4) evaluates the significance of including the training data from the target language(s). note that mt5, a base model for mtk-instruct, was pretrained on mc4 balanced over languages, but mtkinstruct was finetuned on only 15 polish, 5 russian, and 2 czech datasets making it about 1% of all data. additionally, the training prompts for these datasets were english. still, we see that mtk-instruct is better than its english-finetuned counterpart in all evaluation datasets, except two czech sentiment classification tasks. however, in some cases, the differences are relatively small; for instance, in the case of polish cdsc, where english tk-instruct ends only 2.8 points behind the multilingual counterpart. the inconsistent benefits of cot training comparing the performance of t5-flan models with tkinstruct models of the corresponding size, we find that t5-flan is superior in 17 out of 28 cases. however, the differences are often relatively small, and the performance of both in-context learners in these cases remains below the usable level nevertheless. therefore, while it seems that fine-tuning to a chain-of-thought reasoning allows the modeling of features that are applicable also in some multilingual settings, these do not generalize over all in-context learning scenarios. notably, t5-flan perhaps surprisingly fails on classification in czech, where it shows an inability to understand the task even from the given demonstrations. on the other hand, we note that in two of four evaluation cases in polish, the larger t5-flan performs superiorly to even multilingual mtk-instruct of the same size. ## conclusion this paper documents our work in creating the evaluation benchmark for in-context learning for czech, polish, and russian. we transform selected datasets into a compatible format, and with the aid of volunteer native speakers, we create templates for these datasets exclusively in the evaluated language. however, our templates can be applied to any other dataset of the supported types (ner, classification, qa, and nli). in the interaction that is purely in the language(s) of our interest, we evaluate a set of recent incontext learners that we consider state-of-the-art in this area. we find that even in-context learners trained dominantly on english data might perform considerably well and even outperform a fully supervised baseline in some cases. however, on average, massive multilingual pre-training and instruction-based fine-tuning still largely improve the icl ability. finally, we train a set of in-context learners specifically for our target languages by mixing the large qa datasets in english with smaller qa datasets in our target languages; in both czech and russian, such-created learners perform better or comparably to mtk-instruct trained on a vastly larger collection of over 2,000 tasks from 55 languages. we believe that this finding will motivate future work in creating specialized but more accurate in-context learners also for other languages outside english. we publicly release all data transformations, templates, and the newly-created in-context learners for any use.