Rules Ruling Neural Networks - Neural vs. Rule-Based Grammar Checking for a Low Resource Language (2021)

Abstract: We investigate both rule-based and machine learning methods for the task of compound error correction and evaluate their efficiency for North Sámi, a low resource language. The lack of error-free data needed for a neural approach is a challenge to the development of these tools, which is not shared by bigger languages. In order to compensate for that, we used a rule-based grammar checker to remove erroneous sentences and insert compound errors by splitting correct compounds. We describe how we set up the error detection rules, and how we train a bi-RNN based neural network. The precision of the rule-based model tested on a corpus with real errors (81.0%) is slightly better than the neural model (79.4%). The rule-based model is also more flexible with regard to fixing specific errors requested by the user community. However, the neural model has a better recall (98%). The results suggest that an approach that combines the advantages of both models would be desirable in the future. Our tools and data sets are open-source and freely available on GitHub and Zenodo.

## introduction this paper presents our work on automatically correcting compound errors in real world text of north sámi and exploring both rule-based and neural network methods. we chose this error type as it is the most frequent grammatical error type (after spelling and punctuation errors) and twice as frequent as the second most frequent grammatical error (agreement error). it also regards both spelling and grammar as the error is a space between two words, but its correction requires grammatical context. a grammar checker is a writer's tool and particularly relevant to improve writing skills of a minority language in a bilingual context, as is the case for north sámi. according to unesco (moseley, 2010), north sámi, spoken in the north of norway, sweden and finland, has around 30,000 speakers. it is a low resource language in a bilingual setting, and language users frequently face bigger challenges to writing proficiency as there is always a competing language. @xcite developing a reliable grammar checker with a high precision that at the same time covers a lot of errors has therefore been our main focus. good precision (i.e. avoiding false alarms) is a priority because users get easily frustrated if a grammar checker gives false alarms and underlines correct sentences. in this paper we focus on the correction of compound errors. this type of errors is easy to generate artificially in the absence of large amounts of error marked-up text, and we have a good amount of manually marked-up corpus for evaluation for this error type. compound errors (i.e. one-word compounds that are erroneously written as two words) can be automatically inserted by using a rule-based morphological analyser on the corpus and splitting the word wherever we get a compound analysis. unlike other error types (like e.g. real word errors) they are easily inserted, and existing compounds are seldom errors. in addition, they are interesting from a linguistic point of view as they are proper (complex) syntactic errors and not just spelling errors and serve as an example for higher level tools. two adjacent words can either be syntactically related or erroneous compounds, depending on the syntax. in north sámi orthography, as in the majority languages spoken in the region (norwegian, swedish and finnish), nouns that form a new concept are usually written together. for example, the north sámi word boazodoalloguovlu 'reindeer herding area' consists of three words boazu 'reindeer', doallu 'industry' and guovlu 'area', and thus it is written together as a single compound. the task of our methods is to correct spellings such as boazodoallu guovlu into boazodoalloguovlu in case the words have been written separately in error. we develop both a rule-based and a neural model for the correction of compound errors. the rulebased model (gramdivvun) is based on finite-state technology and constraint grammar. the neural model is bi-directional recurrent (birnn). while the rule-based model has earlier produced good precision, it did not handle unknown compounds well, which is why we were interested in a neural approach. however, neural models depend on large amounts of 'clean' data and synthetic error generation (or alternatively marked-up data). typical for low-resource languages and also north sámi, the corpora are not clean and contain a fair amount of a variety of different spelling and grammatical errors (see @xcite . therefore, efficiently preparing data as to making it available for neural model training is an important part of this paper. in our case, we make use of the existing rule-based tools to both, generate synthetic error data and clean the original data for training. for evaluation, on the other hand, we use real world error data. our free and open-source rule-based tools can be found on giellalt github. 1 the training data and the neural models are freely available on zenodo. 2 we hereby want to promote a wider academic interest in conducting nlp research for the north sámi. ## background sámi open source rule-based language tools have a long and successful tradition (nearly 20 years) @xcite @xcite . north sámi is a low-resource language in terms of available corpus data (32.24m tokens raw data). although there is a fair amount of data, it contains many real errors and only a small amount is marked up for errors. applying neural approaches for highlevel language tasks to low resource languages is an interesting research question due the various limitations of minority language corpora, versus the existing research in the topic in well-resourced, majority languages and artificially constrained setups @xcite . rules have been used and are in a wide-spread use in the context of endangered uralic languages. there is recent work on grammar checking for north sámi @xcite and spell checking for skolt sámi @xcite . other rule-based approaches to grammar checking are extensively described in @xcite . before the era of neural models, it was common to use statistical machine translation (smt) as a method for grammar error correction @xcite @xcite . many recent papers on grammar checking use bi-directional lstm models that are trained to tag errors in an input sentence. such methods have been proposed for latvian @xcite , english @xcite and chinese @xcite . similar lstm based approaches have also been applied for error correction @xcite @xcite . other recent approaches @xcite use methods that take advantage of bert @xcite and other data-hungry models. while such rich sentence embeddings can be used for english and a few other languages with a large amount of data, their use is not viable for north sámi. ## data for evaluation and training the neural model we use the sikor (2018) (the sámi international korpus), which is a collection of texts in different sámi languages compiled by uit the arctic university of norway and the norwegian sámi parliament. it consists of two subcorpora: gt-bound foot_2 (texts limited by a copyright which are available only by request) and gt-free foot_3 (the publicly available texts). as a preprocessing step, we run a rule-based grammar checker @xcite and remove sentences with potential compound errors, as we cannot automatically ensure whether these errors are real or not. this is needed as we want this data to be fully free of any compound errors as it serves as the target side of our neural model. thereafter, we take in each sentence in this error free data and analyse it by a rule-based morphological analyser 5 . when the analyser sees a potential compound word, it indicates the word boundary with a compound (+cmp#) tag. we use this information to automatically split all compounds identified by the rule-based analyser. this results in a parallel corpus of the original sentences as the prediction target and their corresponding versions with synthetically introduced compound errors. many of the compound boundaries are ambiguous, and the algorithm decides the one used in training data based on heuristics: maximum number of compound boundaries where the splitting will not cause any other modifications of the word stems or other content. as an additional data source, we use the north sámi universal dependencies treebank @xcite . we parse the corpus with uralic-nlp @xcite and split the compounds the rule-based morphological analyser identifies as consisting of two or more words in order to synthetically introduce errors. we also run the rule-based morphological analyser and morpho-syntactic disambiguator to add part-of-speech (pos) information to produce an additional data set with pos tags. for the universal dependencies data, we use the pos tags provided in the data set. we then make sure that all sentences have at least one generated compound error and that the only type of error the sentences have is the compound error (no other changes introduced by the rule-based models). we shuffle this data randomly and split it on a sentence level into 70% training, 15% validation and 15% testing. the size of the data set can be seen in for the rule-based model gramdivvun we do not generate synthetic errors. we have hand-selected a large corpus for rule development and as regression tests, consisting of representative sentences from gt-free. the current selection for syntactic compound errors includes 3,291 sentences with real world compound errors (and possibly other errors in addition). ## methods we use a neural models and a rule-based model for compound error correction. ## results we evaluate the models both quantitatively and qualitatively. we evaluate on accuracy, precision and recall, and do a linguistic evaluation. the measurements are defined in this article as follows: accuracy @xmath0, where c is a correct sentence (1:1 string match) and s is corpus size in sentences, precision @xmath1, where tp is true positive, f p is false positive and f n is false negative. the f 1 score is the harmonic mean of precision and recall f 1 = 2 × p ×r p +r . the accuracy is thus sentence level correctness rate-as used in the method section to probe model qualitieswhereas precision measures how often corrections were right and recall measures how many errors we found. the word-level errors are counted once per error in the marked-up corpus. thus, if a threepart compound contains two compounding errors it is counted towards the total as one error, but if a sentence has three separate compounds with wrong splits each, we count three errors. the error marked-up corpus we used includes 140 syntactic compound errors (there are other compound errors that can be discovered by the spellchecker as they are word internal) and is from gt-bound. we chose gt-bound to make sure that the sentences had not been used to develop rules. it is part of our error-marked up corpus, which makes it possible to run an automatic analysis. this error corpus does only contain real world (as opposed to synthetic) errors. ## discussion in the future, we would like to look into hybrid grammar checking of other error types and other (sámi) languages. the neural approach gives us relatively high recall in the real world situation with lower precision, whereas the rule-based model is designed to give us high precision even at the cost of lower recall (user experience), which is why hybrid approaches that combine the best of two worlds are interesting. noisy data is to be expected in any endangered language context, as the language norms are to a lesser degree internalized. we will therefore need a way of preparing the data to train neural networks, which can either consist in creating synthetic data or automatically fixing errors and creating a parallel corpus. when creating synthetic data for neural networks, the amount of data is hardly the main issue. many generative systems are capable of overgenerating data. the main question that arises is the quality and representatives @xcite of the generated data. if the rules used to generate the data are not in line with the real world phenomenon the neural model is meant to solve, we cannot expect very high quality results in real world data. generated sentences can easily be less complex 'text book examples' that are not representative of real world examples. in the case of agreement errors between subjects and verbs, for example, there are long distance relationships and complex coordinated subjects including personal pronouns that can change the structure of a seemingly straightforward relation. therefore, we advocate the use of high quality rule-based tools to prepare the data, i.e. fix the errors and create a parallel corpus. while synthetic error data generation for compound errors is somewhat more straightforward as it only affects adjacent words, the generation of synthetic error corpora for other error types is not as straightforward, in part also because generating synthetic errors of other kind can potentially create valid and grammatically correct sentences with different meanings. we therefore predict that (hybrid) neural network approaches for other error types that either involve specific morphological forms (of which there are many in north sámi) or changes in word order will be more difficult to resolve. ## conclusion in this paper, we have developed both a neural network and a rule-based grammar checker module for compound errors in north sámi. we have shown that a neural compound-corrector for a lowresource language can be built based on synthetic error data by introducing the compound errors using a high level rule-based grammar models. it is based on the rule-based tools to both generate errors and clean the data using both part-of-speech analysis, disambiguation and even the error detector. the rule-based module is embedded in the full-fledged gramdivvun grammar checker and achieves a good precision of 81% and a lower recall of 61%. a higher precision, even at the cost of a lower recall, is in line with our objective of keeping false alarms low, so users will be comfortable using our language tools. the neural network achieves a slightly lower precision of 79% and a much higher recall of 98%. however, the rule-based model has more userfriendly suggestions and some false positives are simply other correct alternatives to the ones in the text, while the neural network's false positives sometimes introduce new and unrelated errors. onthe-fly fixes that avoid false positives are an advantage of rule-based models. rule-based models, on the other hand, are not so good at recognizing unknown combinations. hybrid models that combine the benefits of both approaches are therefore desirable for efficient compound error correction in the future.