Playing by the Rules: A Benchmark Set for Standardized Icelandic Orthography (2025)

Abstract: We present the Icelandic Standardization Benchmark Set: Spelling and Punctuation (IceStaBS:SP), a dataset designed to provide standardized text examples for Icelandic orthography. The dataset includes non-standard orthography examples and their standardized counterparts, along with detailed explanations based on official Icelandic spelling rules. IceStaBS:SP aims to support the development and evaluation of automatic spell and grammar checkers, particularly in educational settings. We evaluate various spell and grammar checkers using IceStaBS:SP, demonstrating its utility as a benchmarking tool and highlighting areas for future improvement.

## introduction digital language infrastructure, not least for spell and grammar checking, is a productive and growing field within icelandic language technology. although various datasets have been produced, which in turn have been used to develop and improve spell and grammar checking software, there is a lack of datasets which provide a 1:1 mapping between spelling errors and formalized rules regarding standard orthography (spelling rules). in this paper, we present the icelandic standardization benchmark set: spelling and punctuation (icestabs:sp, ármannsson et al. 2024), a dataset of examples of text standardization along with thorough explanations of how and why text has been altered. the dataset is based on the official spelling rules for icelandic. 1 our goal is to provide a standardized benchmark for evaluating the performance of spell and grammar checkers, thereby contributing to the improvement of digital language tools for icelandic. the paper is structured as follows: section 2 provides an overview of related work in the field of icelandic spell and grammar checking, most importantly existing datasets. section 3 describes the structure of the icestabs:sp dataset and the methodology behind it. section 4 outlines the evaluation experiment we performed to gauge the efficacy of the dataset as a benchmarking tool for orthography. section 5 presents the results of the evaluation, section 6 discusses the limitations of our approach and section 7 concludes the paper with a discussion of the implications of our findings and suggestions for future work. ## related work the most comprehensive single dataset in the field of spell and grammar checking for icelandic is the icelandic error corpus (iec, @xcite and its subsidiary corpus for errors made by l2 speakers @xcite . it uses a fine-grained error categorization system and has been used for training and evaluating spell and grammar checkers, specifically the rule-based greynircorrect @xcite . the grammatical error correction test set (gects, @xcite ), a handannotated dataset of icelandic text with various spelling and grammatical errors, is annotated on the document level as opposed to the iec, where each individual error is annotated. this, along with a more general error categorization system, makes it more suitable for evaluating recent sequence-to-sequence error correction models by testing the models' context awareness on larger texts. ## suggesting standardized orthography we present the icelandic standardization benchmark set: spelling and punctuation (icestabs:-sp), a dataset of text examples containing non-standard orthography and their standardized counterparts. each item in our set corresponds to an entry in the official spelling rules for icelandic, published by the icelandic language council and applied in icelandic schools. each item contains three examples of standardized text, along with thorough explanations of how and why each text has been altered. the dataset is meant to serve as a key component in the development of automatic spell checking in an educational setting, providing handcrafted explanations which can be expanded or used for instruction tuning. both the text examples showing non-standard orthography and the additional explanations are constructed and reviewed by the authors of this paper, all of whom have a background in icelandic linguistics and one of whom is one of the authors of the most recent version of the official spelling rules. the text examples each show exactly one non-standard text feature in order to clearly demonstrate the applicable standardization and in order to check whether that feature has been correctly captured by a spell-checking system. depending on the orthographic issue being demonstrated, the text examples range from very short and simple sentences to short paragraphs, e.g. to display the prescribed use of punctuation between whole sentences. they are mostly synthetic (and partly based on the examples included in the publication of the spelling rules themselves) but where possible, we have extracted real-world examples from the iec using the error codes in that corpus. this authentic approach was, however, limited by the need to include only one example of non-standard orthography in each example, so some of those examples have been slightly altered. the official spelling rules consist of 33 main chapters and numerous subchapters. some subchapters, as well as all of chapters 30 and 33, are ignored in our set as they are not applicable in the context of automatic correction of spelling and grammar (e.g. some contain only general discussion of phenomena, rather than concrete examples). in other cases, subchapters had to be split into further subsections for our purposes as they dealt with multiple distinct features. these are marked with "(a)", "(b)", etc. in icestabs:sp. in this way, we define a total of 247 rules over 31 chapters. for each of these 247 rules, our dataset contains an entry labeled with a distinct number and consisting of the following parts: to illustrate how this information is structured in the icestabs:sp dataset, the entry for rule 1.2.1 (a) is shown in figure 1 . the aim of the suggestions and explanations in our set is to provide further assistance to potential future users of an automatic spellchecker, not least young people and second language learners of icelandic. therefore, we try to keep our explanations accessible to the average speaker, with as little linguistic terminology as possible (especially in the short suggestions). to as great an extent as possible, we also try to include helpful generalizations in the short suggestion format as opposed to only word-specific corrections. an example would @xmath0, <leiðrétt>, þar sem það er örnefni '<@xmath1, <correction>, as it is a place name', rather than simply '<@xmath2. this is sometimes made difficult, however, by rules that can apply to many different scenarios or are simply too complex to sum up in one short sentence. ## applying icestabs:sp in evaluation to gauge the efficacy of the icestabs:sp dataset as a benchmarking tool for orthography, we performed an evaluation experiment, where various spell and grammar checkers for icelandic were applied on our data and then evaluated statistically. this serves two purposes. firstly, it allows us to evaluate the performance of these tools on a standardized dataset, which can be used to compare the tools to each other and, preferably, to other benchmark sets. secondly, we standardize our methods for evaluating correction tools on our benchmark set. the source code of our evaluation methods is then made available on github 3 for others to use on new tools, as well as the output of the tools we use in our evaluation. ## results we evaluate the performance of the tools on the icestabs:sp dataset using the three metrics described above. the results are shown respectively in figures 2 , 3 , and 4. ## limitations there are various aspects of the icestabs:sp dataset that could be improved in future iterations. these range from superficial to inherent issues, the solutions to which will need further work and discussion. as shown in figure 5 , even though the icestabs:sp data is organized into 31 distinct chapters (reflecting the 33 chapters of the source material), the distribution of examples across these chapters is not uniform. this is due to the fact that some chapters cover more common and straightforward spelling rules, while others deal with more com- chapters 1, 2, and 5 have significantly more entries than the other chapters in the dataset, with chapter 1 (use of upper and lower case letters at the beginnings of words) being particularly prominent. this discrepancy is due to the fact that these chapters cover fundamental and frequently encountered spelling rules in icelandic orthography. on the opposite end of this spectrum are chapters 9, 11, 18, 23 and 25, which all have a single rule entry each (giving 3 examples per chapter in the overview in figure 5 ). even though the dataset structure and evaluation procedure treats these chapters as equal to the others, they are not equal in terms of the number of examples. currently, the icestabs:sp dataset only allows for a single standardized suggestion for each example. this means that the dataset does not account for the possibility of multiple correct solutions to a given spelling or grammar issue. as there are sometimes more than one correct way to write something according to the spelling rules, some entries in our dataset should allow for multiple possible correct alterations. an example would be some non-standard way of writing a specific time, which could be corrected to e.g. '2.30' or '2:30' as both a full stop and a colon are possible ways of separating hours from minutes, according to spelling rules 22.5 and 29.5, respectively. even though the number of these occurrences is low (variation is found in about 20 rules out of 247), this is a limitation that will be addressed in future iterations of the dataset. ## conclusions and future work we have presented the icestabs:sp dataset, a comprehensive benchmark set for icelandic spelling and punctuation. the dataset is based on the official spelling rules for icelandic and provides standardized suggestions for a wide range of spelling and punctuation issues. as such, it is the first of its kind for icelandic. we have evaluated the performance of ten spell and grammar checkers on the dataset, using three main metrics: sentence-level accuracy, token-level f 0.5 score, and gleu score. the results are broadly in line with expected performance, which is encouraging for the utility of the dataset as a benchmarking tool. further work is needed to address limitations in the dataset and explore additional evaluation metrics to provide a more comprehensive assessment of spell and grammar checkers for icelandic. on grammatical error correction. in proceedings of the eighteenth conference on computational natural language learning: shared task, pages 1-14, baltimore, maryland. association for computational linguistics. hulda óladóttir, þórunn arnardóttir, anton karl ingason, and vilhjálmur þorsteinsson. 2022. developing a spell and grammar checker for icelandic using an error corpus. in proceedings of the thirteenth language resources and evaluation conference, pages 4644-4653, marseille, france. european language resources association. vilhjálmur þorsteinsson, hulda óladóttir, and hrafn loftsson. 2019. a wide-coverage context-free grammar for icelandic and an accompanying parsing system. in proceedings of the international conference on recent advances in natural language processing (ranlp 2019), pages 1397-1404, varna, bulgaria. incoma ltd.