Bridging the Resource Gap: Exploring the Efficacy of E nglish and Multilingual LLM s for S wedish (2023)

Abstract: Large language models (LLMs) have substantially improved natural language processing (NLP) performance, but training these models from scratch is resource-intensive and challenging for smaller languages. With this paper, we want to initiate a discussion on the necessity of language-specific pre-training of LLMs.We propose how the “one model-many models” conceptual framework for task transfer can be applied to language transfer and explore this approach by evaluating the performance of non-Swedish monolingual and multilingual models’ performance on tasks in Swedish.Our findings demonstrate that LLMs exposed to limited Swedish during training can be highly capable and transfer competencies from English off-the-shelf, including emergent abilities such as mathematical reasoning, while at the same time showing distinct culturally adapted behaviour. Our results suggest that there are resourceful alternatives to language-specific pre-training when creating useful LLMs for small languages.

## introduction large language models (llms) are a milestone in natural language processing (nlp), introducing massive improvements particularly for generation tasks @xcite @xcite @xcite @xcite . the basic recipe for these improvements is conceptually simple: scaling up model parameter count, training data, and compute. with increased size, models learn more expressive representations and capture more nuanced relationships between words, phrases, and sentences. further improvements can be made, e. g., with instruction fine-tuning @xcite and reinforcement learning based on human feedback @xcite . interestingly, llms show emergent abilities: they perform tasks they were not explicitly trained on in zero-shot or few-shot setups @xcite . some llms can produce accurate answers even on complex reasoning tasks with simple prompting schemes such as chain-ofthought (cot) prompting, where the correct answer is inferred by making intermediate steps explicit @xcite . the resource-intensive nature of pre-training llms poses a challenge, especially for smaller languages such as swedish. while the north germanic languages are privileged regarding data access due to widespread internet usage and centralized resources, procuring large amounts of highquality, diverse, and representative training data is far from trivial, even for large actors. for example, the swedish research center ai sweden, which is training the largest language model for north germanic languages, currently has access to "only" 320b tokens of training data in total foot_0 , which corresponds to approximately 114b tokens of swedish data and even less for other languages, as per the proportions reported by @xcite . assuming continued up-scaling of architectures, it appears unlikely that the nlp community can collect enough data to produce models with abilities comparable to the english models for all languages or even language families. continuously pre-training llms comes with significant environmental and economic costs (direct and opportunity costs) and demands substantial human resources and competency. therefore, pretraining is an unsustainable approach to adapt all relevant model innovations to swedish. with this paper, we want to initiate a discussion on pre-training llms for specific languages or smaller language families -whether such pre-training is a good use of resources given the the current nlp landscape and whether it is necessary to develop specific llms for capabilities at swedish tasks. to support the discussion, we do the following: while conclusions from studies on smaller models reported better performance with monolingual models @xcite @xcite , our findings show that previous views may no longer hold for llms that are vastly larger and instruction fine-tuned. given these results, it is timely to consider more resourceful alternatives to pre-training llms for swedish. ## related work we start by reviewing relevant work on advantages and limitations of multilingual models and on the lingual versatility and language adaption of llms. ## background: multilingual skill sharing to better understand the adaptability of llms to various tasks and languages, we explore the concept of knowledge sharing in multilingual llms. it involves examining how different languages, like swedish and english, interact within these models and whether their multilingual capabilities are supported by separate or shared mechanisms. ## experimental setup we aim to test to what extent current implicit and explicit multilingual llms can be used in swedish. ## discussion in this section we discuss our results from § 5 with respect to the one @xmath0, and what implications we see for the question if we need to pre-train language-specific models. for limitations of our work, see appendix a. ## conclusion continuously pre-training llms comes with significant environment and economic costs (direct and opportunity cost), and demands substantial human resources and competency. as such, pretraining is an unsustainable approach to adapt all new model innovations that arise to swedish. in this paper, we aim to spark a discussion on the necessity of pre-training language-specific models for languages like swedish. we explore the capabilities of existing llms, that are not implicitly trained on swedish, on a set of swedish tasks. we found that gpt-3 transfers capabilities from english to swedish, providing emergent abilities such as mathematical reasoning for swedish. the gpt-3 model also adapts to the cultural setting of its answers based on the language of the prompt. while we may not want to rely on gpt-3 specifically due to its closed nature, our results indicate that there are alternatives that make llms perform well for swedish without language-specific pre-training. we suggest that the shared capabilities in these models need to be explored further. instead of pretraining llms for swedish, we should focus our research resources on understanding and adapting models that already have the functional capabilities that we desire for swedish.