The BRAGE Benchmark: Evaluating Zero-shot Learning Capabilities of Large Language Models for Norwegian Customer Service Dialogues (2025)

Abstract: This study explores the capabilities of open-weight Large Language Models in a zero-shot learning setting, testing their ability to classify the content of customer service dialogues in Norwegian from a single instruction, named the BRAGE benchmark. By comparing results against widely used downstream tasks such as question-answering and named entity recognition, we find that (1) specific instruction models greatly exceed base models on the benchmark, (2) both English and multilingual instruction models outperform the tested Norwegian models of similar sizes, and (3) the difference between base and instruction models is less pronounced than in other generative tasks, suggesting that BRAGE is a challenging benchmark, requiring precise and generalizable instruction-tuning.

## introduction satisfied customers are critical to any telecommunications provider's long-term success and sustainability. an essential piece of this puzzle is to provide the best possible customer service once a problem has occurred and try to avoid any further negative experiences @xcite . advances in automatic speech recognition and text analysis methods have transformed customer service processes, enabling providers to gain aggregated insights from the large volume of daily calls. these insights allow the telecommunications provider to act quickly on issues that influence multiple customers in close to real-time. however, creating models capable of analyzing transcribed conversations remains challenging due to the technical expertise required and the time-intensive development process. additionally, the distribution of the incoming calls may change over time due to concept drift @xcite , requiring frequent updating of models to maintain operational quality -thus increasing costs. in-context learning @xcite and the ongoing efforts on adapting large language models (llms) to lower-resource languages such as norwegian (nora ai, 2024) offer a promising solution to this problem. 1 this study explores the potential of open-weight llms to enable non-expert users to perform zero-shot content classification. to this end, we introduce brage, a private benchmark designed to evaluate zero-shot classification of transcribed conversations in norwegian. using the same instructions provided to human annotators for creating ground truth labels, various llms are tasked with classifying the content of conversations between customers and customer service agents. to assess the feasibility of this approach, we evaluate base-and instructiontuned open-weight llms, including pre-trained and fine-tuned norwegian models. given the sensitive nature of the data, brage is a private benchmark. aggregated results, however, are publicly shared to ensure transparency. additionally, we foot_1 aim to facilitate academic collaboration by performing benchmark evaluations on brage and sharing the results with the public when requested by researchers in the nordics. code for the benchmark is available on github. 3 the customer service process the case process from which we create brage is an analytics unit within a telecommunications provider, supporting customer service with insights on current calls. when customers contact the service center, calls are, if permitted, recorded t ranscr iption ser vice a nalytics unit call recordings r etr ieve new call recordings exter nal vendor: propr iet ary transcr iption pipeline n ew recordings call transcr ipts call transcr ipts classification m odels classify call attr ibutes update dashboards a nnot ated transcr ipts and subsequently transcribed. transcription is performed by a proprietary service from an external vendor, similar to whisper @xcite . the transcripts are then processed and annotated using classification models, predicting business-relevant attributes. figure 1 shows an overview of this process. in our benchmark, we modify the process by replacing the classification model(s) in figure 1 with a single open-weight llm, which is prompted using the codebook (annotation guidelines, see @xcite previously used by the human annotators. ## related work in recent research and development of llms, it has become clear that these models can adapt to the context they are presented @xcite , to such a degree that they do not need further training to adapt to a particular task. this is also known as in-context learning (icl) @xcite , and can be done using a single instruction with no examples (zero-shot) or multiple examples at inference time (few/n -shot). icl dramatically reduces the associated costs in evolving systems, as one no longer relies on expensive training pipelines to support shifts within data and business needs. open-weight models such as llama @xcite , and gemma @xcite have proven to be competitive when considering the efficiency vs performance trade-off. "smaller" models (≤ 70b) have achieved impressive results across numerous tasks @xcite , and we have reached a point where the performance gap between open-weight models and larger proprietary models is quickly diminishing. however, evaluating llms and quantifying this gap is incredibly difficult @xcite , e.g., because of benchmark data leaks @xcite . to circumvent this, lmsys @xcite developed an elo-score system where users rank anonymous models -giving an idea of real-world performance. as of september 2024, several open-weight models (≤ 70b) rank on par with much larger models. narrowing in on scandinavian languages, scan-deval @xcite is a tool for evaluating models on language-specific data for downstream tasks. focusing on constrained generation with llms in this study, the norwegian generative scandeval benchmark is highly relevant, with 134 different models currently benchmarked across 9 different datasets. 4 because our data is sensitive, the models must run locally to avoid transferring the data during inference. open-weight models, particularly those explicitly trained on norwegian data, are included for evaluation. while open models come with the advantage of enabling continued training, supervised fine-tuning (sft), merging @xcite , and jailbreaking @xcite along with an ever-growing set of preference tuning techniques @xcite , the search space of optimization methods is so vast that finding an optimal approach is near impossible. furthermore, @xcite found sft to degrade knowledge and reduce output quality of the pretrained models, with the same observations for using fine-tuned llms for telecommunications @xcite . moreover, research within out-of-domain generalizability in traditional machine learning and llms suggests that domainspecific training will reduce a model's performance @xcite @xcite . @xcite challenges this idea and performs thorough evaluations on the generalization of models for icl and sft in the parameter range of 125m to 30b. while mosbach et al. find compelling results for the case of sft, such that a smaller sft can outperform higherparameter models with icl, these findings diminish as the model sizes grow, and icl performs significantly better when evaluating in-domain than sft for model sizes ⪆ 7b. ## conclusion we have presented brage, a private zero-shot benchmark for classifying transcribed calls between customers and customer service. based on these preliminary results, we observe that the task can be accomplished to a somewhat acceptable level using open-weight llms. based on our results, we can conclude that this is a challenging benchmark and that instruction fine-tuned models generally perform better on this type of zeroshot task. specifically, instruction fine-tuning (fnb) on a multilingual base model, in the case of norskgpt mistral 7b, was superior to any of the other norwegian models on brage. we, therefore, stress the importance of creating more open instruction datasets in norwegian, as this might foster progress in zero-shot settings such as the brage case. surprisingly, we found that the english-only and significantly smaller gemma-2 2b it did better than any of the norwegian models. these results may also apply to other european languages, especially those with a higher presence in multilingual training corpora, e.g., german and spanish. we plan to expand this benchmark by adding new tasks as well as to include all of the scandinavian languages.