Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT (2025)

Abstract: Smaller LLMs still face significant challenges even in medium-resourced languages, particularly when it comes to language-specific knowledge – a problem not easily resolved with machine-translated data. In this case study on Icelandic, we aim to enhance the generation performance of an LLM by specialising it using unstructured text corpora. A key focus is on preventing interference with the models’ capabilities of handling longer context during this adaptation. Through ablation studies using various parameter-efficient fine-tuning (PEFT) methods and setups, we find that increasing the number of trainable parameters leads to better and more robust language adaptation. LoRAs placed in the feed-forward layers and bottleneck adapters show promising results with sufficient parameters, while prefix tuning and (IA) ^3 are not suitable. Although improvements are consistent in 0-shot summarisation, some adapted models struggle with longer context lengths, an issue that can be mitigated by adapting only the final layers.

## introduction llms have strong multilingual capabilities and top the leaderboards even for less-represented languages @xcite . however, smaller llms still struggle with these languages, hampering fast and resource-efficient inference. instruction tuning on machine-translated data can improve performance compared to english-only tuning @xcite but models still fall short when evaluated on native benchmarks, likely due to missing languagespecific knowledge @xcite . while collecting large amounts of native instruction-tuning data could address this issue, this can be costly or infeasible. this makes techniques for adapting models using unstructured text data valuable. in this paper, we perform ablations with parameter-efficient fine-tuning (peft) methods for language adaptation with unstructured text data after instruction alignment. this diverges from the standard setup for fine-tuning a model: unlike typical fine-tuning, where the adaptation data closely matches the expected output format, the data we use is closer to the expected output in language but likely further from the target task format. therefore, the setup risks interference with the original instruction-tuning objectives, possibly leading to catastrophic forgetting @xcite . in addition, hardware constraints made us choose a maximum context length smaller than the one used in pre-training, risking further performance degradation. therefore, we aim to identify setups that do not interfere with previously learned abilities. we attempt to avoid catastrophic forgetting with peft methods that leave the majority of or all model parameters unchanged: lora @xcite , ia 3 @xcite , bottleneck adapters @xcite and prefix tuning @xcite . we experiment with the number of learnable parameters, the placement of lora matrices in different transformer modules and layers, as well as the training corpus used for adaptation. we use the smallest instruction-tuned llama 3.2 model @xcite with 1b parameters and adapt it to icelandic, evaluating performance on text summarisation. our findings are that: • lora and bottleneck adapters show improvements especially in 0-shot settings, though simply adding target-language task demonstrations also improves the performance substantially. ## conclusion we tested a range of peft methods for language adaptation using unstructured text corpora, finding that lora in the feed-forward modules yielded the most promising results, followed by bottleneck adapters. lora in the attention modules performed less well, was less robust to larger context lengths and needed more parameters for a comparable performance. combining loras in both the attention and feed-forward modules did not improve over feed-forward loras only, and may even lead to slightly decreased performance. prefix tuning and (ia) 3 were not suitable at all. our results show that across architectures, more trainable parameters lead to better scores, showing, perhaps unsurprisingly, that sufficient learning capacity is crucial for language adaptation. some adaptation setups led to a decline in performance as contexts get longer; possibly a result of restricted context lengths during adaptation. however, this issue can be mitigated by training only the last layers. notably, we did not observe any positive effects from using higher-quality pre-training data sourced from narrower domains. moving forward, with a higher resource investment, we see the potential that more training data, possibly with instruction data in the mix, and longer context lengths improve the performance further. however, to truly assess the potential of these methods, we need more diverse, language-native evaluation data, as well as fine-grained human evaluations that assess various aspects of generated language quality and content.