N o C o LA : The N orwegian Corpus of Linguistic Acceptability (2023)

Abstract: While there has been a surge of large language models for Norwegian in recent years, we lack any tool to evaluate their understanding of grammaticality. We present two new Norwegian datasets for this task. NoCoLA-class is a supervised binary classification task where the goal is to discriminate between acceptable and non-acceptable sentences. On the other hand, NoCoLA-zero is a purely diagnostic task for evaluating the grammatical judgement of a language model in a completely zero-shot manner, i.e. without any further training. In this paper, we describe both datasets in detail, show how to use them for different flavors of language models, and conduct a comparative study of the existing Norwegian language models.

## introduction large pre-trained language models have recently led to a revolution in natural language processing (nlp) as they substantially increased the performance of most nlp tools @xcite . large language models were originally developed for english, but a surge of norwegian-based models has recently followed @xcite @xcite . the remaining issue is that the norwegian linguistic resources do not contain a large range of tasks to evaluate and compare these models on, as opposed to the english benchmark suites like glue @xcite , superglue @xcite or glge @xcite , to name a few. we present two new datasets for evaluating the understanding large language models have of norwegian grammar, jointly called the norwegian corpus of linguistic acceptability (nocola). we hope listing 1: two illustrative examples of incorrect / correct sentence pairs from nocola zero . the english translations: "society would be happier" and "i'm not norwegian, but i enjoy living in norway." while not as practical as the first task, the zeroshot evaluation provides a better estimate of the innate grammatical understanding. we provide a comprehensive evaluation of the existing norwegian language models and release the data and code for an easy evaluation of new norwegian models. foot_0 ## related work the closest equivalents of our nocola class dataset are the english corpus of linguistic acceptability (cola; @xcite and the swedish dataset for linguistic acceptability judgments (dalaj; @xcite . on the other hand, nocola zero roughly follows the the benchmark of linguistic minimal pairs for english and the english (blimp; @xcite ). ## conclusion in this paper we have proposed nocola, the first dataset for linguistic acceptance in norwegian bokmål. we showed how to use it for measuring the linguistic knowledge of language models on both a classification task and a zero-shot probability comparison task. we have described how the datasets were created and what their motivation is, compared them to related work in english nlp and showed how to use them for fine-grained error analysis of language models. lastly, we evaluated existing norwegian masked language models on both proposed tasks. these results suggest that models trained specifically for norwegian or scandinavian languages perform better at discriminating between acceptable an nonacceptable sentences. the classification results also show that linguistic acceptability is a relatively hard task, as only one of the models achieved more than 60% on the main mcc metric. the results on our diagnostic dataset highlight some shortcoming of the existing models. we will release all evaluation sources in the camera-ready version.