An Icelandic Linguistic Benchmark for Large Language Models (2025)

Abstract: This paper introduces a linguistic benchmark for Icelandic-language LLMs, the first of its kind manually constructed by native speakers. We report on the scores obtained by current state-of-the-art models, which indicate room for improvement, and discuss the theoretical problems involved in creating such a benchmark and scoring a model’s performance.

## introduction large language models (llms) have in the last few years become near ubiquitous in the field of language technology (lt) and in their wake follows a growing need to test their capabilities on all kinds of tasks, such as language understanding and generation, mathematics, programming etc. as english is the dominant language in the field and the biggest source of training data for these models, it is only natural that the principal benchmarks for the models (translations aside) also focus on english. however, it is vital to also evaluate the capabilities of the models for lower-resource languages. we introduce a standard benchmarking dataset @xcite to evaluate llms' grammatical 'knowledge' and linguistic accuracy for icelandic, a lower-resource language. such benchmarks can help llm developers to improve their models' icelandic proficiency in a measurable way and provide researchers with further insight into these models' output patterns, limitations and unexpected 'behaviour'. as far as the authors are aware, this is the first benchmark of its kind specifically constructed for icelandic by native speakers and experts in linguistics and lt (see section 2). although the models' capabilities in icelandic are under scrutiny, we use english for all of our prompts in order to facilitate future crosslinguistic research. as one reviewer points out, it might be interesting to contrast these results with the same prompts in icelandic, but we leave that for future work. we do not test for proficiency in standard vs. non-standard icelandic, for instance the widespread use of dative instead of the standard accusative as the subject case of various psych verbs, like langa 'want' or vanta 'lack, need', i.e. mér [dat.] langar instead of mig [acc.] langar 'i want'. we rather aim to focus on features which should be unanimously agreed to be grammatical or ungrammatical by native speakers of icelandic. 1 the published benchmark set contains 1160 hand-written items over 19 subcategories of syntax, morphology and semantics, tested with five different methods (see ## related work in constructing our dataset, we partly look to similar linguistic benchmarks for llms that have been constructed for english. @xcite 's benchmark of linguistic minimal pairs for english (blimp) is perhaps the most commonly cited example. it is based around 67,000 minimal pairs, where one example is considered grammatical and the other ungrammatical, and models are tasked with 'judging' the grammatical acceptability of each sentence. (as this was before instruction-tuned models like chatgpt-3 and the tendency towards closed black-box models, method category no. of items sentence grammaticality check (yes/no)* simple bad/good sentences 40 attributive agreement 88 predicate agreement 28 word order 28 verb agreement 28 subject case 28 island effect sentences 80 wh-movement 20 topicalization 32 gapping 120 reflexivization 40 well-formedness check of compound nouns (yes/no)* word formation 280 fill-in-the-blank anaphoric reference 20 coreference resolution 44 wug test (past tense of verbs) 20 fragment answering fragment answers 40 question answering coreference resolution 44 attributive agreement 30 word sense disambiguation 150 total 1160 the authors simply compared the log probabilities a model assigned to sentences, i.e. making it easy to contrast how likely input sentence a was compared to input sentence b for a given model.) this general blueprint for constructing linguistic benchmarks for llms has been widely followed, for instance by the makers of the zorro test suite @xcite and the scala linguistic acceptability dataset for scandinavian languages (including icelandic) @xcite . these test sets all use automatically constructed examples, which makes it possible for the blimp dataset, for example, to have 1,000 sentence pairs for each of the 67 grammatical tasks tested. in terms of size, our benchmark certainly pales in comparison. on the other hand, it is possible for a human to have an overview of it, whereas blimp is simply too large and lower-quality pairs get lost in the masses (see vázquez @xcite for more detailed criticism). in this case, we find our approach preferable, but we are also aware of its drawbacks (see @xcite for arguments that "expert authorship" can be counterproductive, when researchers have direct, finegrained control over the data, as it may intentionally or unintentionally lead to data "that is oriented toward linguistic phenomena that are widely studied and widely known to be important to the task at hand"). as far as interesting theoretical work on the linguistic capabilities and limitations of llms is concerned, there has been an ongoing and interesting debate between researchers that have used two different approaches to evaluate models in this regard. one group is represented by @xcite , who use acceptability judgments, widely used in traditional linguistic research, that are elicited with prompts. the other group is represented by @xcite , who argue that prompting is not a substitute for probability measurements in llms and that such metalinguistic judgments of acceptability presuppose a model's understanding of grammatical acceptability. their approach is to compare the log probabilities of a model's output on the grounds that this gives a better idea of that model's "linguistic generalization". as much as we would have liked to imitate this approach, it was simply not possible in our one-sizefits-all setup, as closed models such as the ones provided by openai and anthropic offer limited or no access to their log probabilities. 2 we take some inspiration from @xcite , who test the morphological capabilities of chatgpt via a 'wug test', where a model is tasked with forming words from nonsense root forms. we also build on the work of @xcite , who test the capabilities of gpt-4 in icelandic, partly using methods we include in our benchmark set. lastly, the icelandic lt company miðeind maintains an llm leaderboard on huggingface, where a selection of llms are evaluated across six tasks for icelandic: a reduced icelandic version of winogrande, grammatical error detection, inflection, belebele (multiple-choice reading comprehension), machine-translated arc-challenge (multiple-choice question answering) and an icelandic wikiqa dataset. foot_2 ## benchmark composition the benchmark was created by the authors of this paper, who have an academic background in the study of icelandic, theoretical linguistics and lt. as already mentioned in section 2, the point of departure were similar linguistic benchmarks for english, but we were also inspired by previous work and studies on icelandic grammar; we point out some references below, where applicable. some of the tasks can be applicable in a multitude of languages (such as the sentence grammaticality check), whereas others are more specific to icelandic and languages that have more complex morphology and a richer inflectional system than, for instance, english (word formation, fill-in-theblank and fragment-answering). see appendix a for examples of each task. ## limitations the limitations are a few. firstly, we tried to find a suitable base prompt for each task that would be understood in the same way by different models. even though we feel that the uniformness of the resulting answers reflects that we succeeded in this respect, we cannot be sure that some "fine-tuning" of the prompts would not have yielded better results. secondly, although we tried to include clear instructions in english in the prompts on what the output should be, such as "answer only with yes or no", there were some deviations in the answers. these include correct answers in icelandic, correct answers with an additional tail (e.g. "yes. the correct sentence"), answers that include a full stop or other additional punctuation etc. to reduce these deviations, we cleaned the model answers for scoring. a correct answer in icelandic, for instance, was therefore considered correct, as well as answers with a "tail" etc. even though, as one reviewer points out, post-processing methods are fairly common practice and often used by llm evaluation frameworks such as @xcite , for human-alignment comparisons in llms, such lenience has been criticized @xcite , on the grounds that a human would hardly respond in such a way. we acknowledge this, but would again like to stress that this could perhaps have been avoided with more precise prompts. it remains an open question how best to score output. in our setup, a model's answer that matches our reference answer gets one point. an answer that does not, gets none. it could be argued that this method does not highlight the differences in performance between different models sufficiently, as two models both get the same score for a wrong answer if one outputs a single pronoun in icelandic as prompted and the other outputs gibberish. in this regard, our benchmark perhaps is better suited to measure the differences of better-performing models than capturing the differences between lesser models. we would like to encourage the further development of open-source models, which may require an evaluation that can provide information on when one of two wrong answers is more promising than another. focusing on open-source models would also allow one to compare model output with input log probabilities of the test examples, following the work of @xcite . on the other hand, a more for-giving scoring metric, based on e.g. levenshtein distance, would simply not be applicable for our benchmark as the difference between a correct answer and an ungrammatical one is often only one or two letters. ## conclusions we have presented a standard benchmarking dataset for evaluating the linguistic capabilities of llms for icelandic, the first of its kind. we publish the dataset openly and describe its construction in order to hopefully aid further work in this respect for both icelandic and nordic nlp in a wider sense. in order to show the current state of the art for icelandic, we show the results on our set for a variety of currently available models, which indicate considerable room for improvement for some of the tested phenomena. we also discuss some of the still-open questions regarding the best methods for testing the language capabilities of llms.