Making Instruction Finetuning Accessible to Non- E nglish Languages: A Case Study on S wedish Models (2023)

Abstract: In recent years, instruction finetuning models have received increased attention due to their remarkable zero-shot and generalization capabilities. However, the widespread implementation of these models has been limited to the English language, largely due to the costs and challenges associated with creating instruction datasets. To overcome this, automatic instruction generation has been proposed as a resourceful alternative. We see this as an opportunity for the adoption of instruction finetuning for other languages. In this paper we explore the viability of instruction finetuning for Swedish. We translate a dataset of generated instructions from English to Swedish, using it to finetune both Swedish and non-Swedish models. Results indicate that the use of translated instructions significantly improves the models’ zero-shot performance, even on unseen data, while staying competitive with strong baselines ten times in size. We see this paper is a first step and a proof of concept that instruction finetuning for Swedish is within reach, through resourceful means, and that there exist several directions for further improvements.

## introduction the use of pretrained language models in natural language processing (nlp) is widespread, with finetuning or zero-shot approaches employed for various tasks. however, not all pretrained models exhibit strong zero-shot performance or are cost-effective to finetune for every new task. to overcome these limitations, instruction finetuningfinetuning on natural language processing tasks *equal contribution. in this paper, we explore how automatic methods for instruction finetuning can be extended to swedish. the work is partly based on unnatural instructions @xcite , a method of bootstrapping the instruction creation process. we use the generated instruction as a teacher to a swedish student, where a translator module acts as an intermediary. the dataset is translated from english to swedish and then used to finetune various swedish and non-swedish models to investigate the effectiveness of the proposed technique. the translations and models are evaluated using both human and automatic methods. we find that the translated instructions generates a significant increase in zero-shot performance, even to unseen data. this paper is a first step, and a proof of concept, that instruction finetuning for swedish is possible and that there exist several directions for further improvements. ## related work language models have demonstrated the capability to solve tasks through following instructions in a zero-shot setting. however, their performance can be enhanced by finetuning on a diverse set of taskspecific instruction data. this allows the model to adapt and generalize to new, unseen tasks, reducing the need for task-specific finetuning and enabling an off-the-shelf solution @xcite @xcite @xcite @xcite . manually procuring data for task specific finetuning can be costly. to mitigate this issue, researchers have explored automatically generating data @xcite . studies have shown that this alternative is highly effective, with performance that is only slightly behind that of large language models, which has been finetuned on manual data @xcite . ## automatic instruction finetuning for swedish we use two different instruction specific datasets and translate them to swedish: the first one to finetune two models, and the second one as a held-out evaluation set. we evaluate the performance of the two models before and after instruction finetuning together with a strong gpt3 baseline to explore the usability of the automatically procured and translated instruction data. the code, model checkpoints, and datasets used in the paper are made available foot_0 . ## results perplexity we first start with a perplexity analysis to measure the language modelling quality of the models on unseen swedish data using the dataset described in section 3.1. when evaluating perplexity using token length normalization, tokenizers that generate sentences with more tokens are favored. however, this approach can be problematic in cross-lingual settings, where tokenizing unknown words may increase the number of tokens generated. to overcome this issue, we use character length normalization as it provides a fairer measurement of perplexity across languages @xcite . ## discussion the results from the perplexity analysis show an increase for all models after instruction finetuning, even though the models become more capable at a broad set of tasks. it has been shown that perplexity does not correlate strongly with downstream task or prompting performance @xcite . however, when using automatically translated data, we need to be aware that noise in the translation process can affect the models' capabilities. the increase in perplexity could partly be explained by unwanted noise. a hypothesis for why we see a larger increase in perplexity for the opt model than the gpt-sw3 model is that stronger foundations in the target language makes the model more robust to translation errors. the significant increase of rouge-l scores for all models, especially on the difficult natural instructions tasks, show that the models can become strong zero-shot generalizers with relatively little finetuning. unsurprisingly, the stronger performance of gpt-sw3 than opt shows that a strong foundation in the target language is helpful. there are some issues with making direct comparisons with the baseline gpt3 models: we do not know what data it has seen during training, the models have not been trained specifically for swedish, and they have followed a more structured instruction-finetuning process. what can be said is that translated automatic instruction seems to be highly useful. gpt-sw3 outperforms the larger curie model on both datasets and even the davinci model, two orders of magnitude larger, on the unnatural instructions test set. however, our human evaluation shows that there are still significant improvements that need to be made to reach parity with the largest model. ## conclusion and future work using automatically created instructions that have been translated to swedish provides a significant increase in zero-shot performance when instruction finetuning a gpt-sw3 and opt model. the gpt-sw3 model shows competitive performance against the hundred times larger instruction-tuned gpt3-davinci model. while the results are promising, this is still a work in progress. human evaluations show that there is significant progress to be made, especially in giving correct answers to instructions. a possible path for performance gain is to study the effects of translation quality on models' performance. we also leave for future work how the automatically translated instruction finetuning interacts with increased model scale. in conclusion, we find that instruction finetuning for swedish is not only within our reach, but it can be achieved with a completely automatic process that yields significant improvements on a broad set of tasks in a zero-shot setting.