Reusing a Multi-lingual Setup to Bootstrap a Grammar Checker for a Very Low Resource Language without Data (2022)

Abstract: Grammar checkers (GEC) are needed for digital language survival. Very low resource languages like Lule Sámi with less than 3,000 speakers need to hurry to build these tools, but do not have the big corpus data that are required for the construction of machine learning tools. We present a rule-based tool and a workflow where the work done for a related language can speed up the process. We use an existing grammar to infer rules for the new language, and we do not need a large gold corpus of annotated grammar errors, but a smaller corpus of regression tests is built while developing the tool. We present a test case for Lule Sámi reusing resources from North Sámi, show how we achieve a categorisation of the most frequent errors, and present a preliminary evaluation of the system. We hope this serves as an inspiration for small languages that need advanced tools in a limited amount of time, but do not have big data.

## introduction language tools for very low resource languages are urgently needed to support language maintenance, but also it takes a long time to develop them. an existing multilingual infrastructure and existing tools that can be reused can speed up the process. in this article, we describe the process of making a lule sámi gec together with a preliminary categorization of frequent lule sámi errors. lule sámi is on the lower end of lower resource language. it can benefit from north sámi which is closely related and has a well-functioning grammar checker. the reuse of existing knowledge is an important concept in effective development of new grammar checkers in multilingual infrastructures. with this work we would like to set an example of how high-end complex nlp tools can be made, in less time, by taking existing tools as a frame. the following tools were already ready-made: an fstbased morphological analyser, a morpho-syntactic disambiguator developed for correct text, and a multi-lingual infrastructure that contains scripts to build the grammar checker (among other applications). our work took altogether 120 hours, (40 hours of meetings of two linguists (one of them native speaker) and 40 hours of work of one native speaker linguist). for related languages we can even reuse rules and sets (prenominal modifiers, sentence barriers). but for example, lexemes have to be translated. this article will show in detail what can be reused, and which factors need special focus as they are language specific -many times it is systematic homonymies, and definitely idiosyncratic homonymies. in addition, we will evaluate the lule sámi grammar checker and point out future steps for improvement. ## setup in this section, we answer the question of how to set up a grammar checker for a new language in giellalt. the resources we need are: unlike machine learning, this approach is not dependent on a large amount of text data or a gold corpus. to develop a grammar checker, we only need several test sentences containing the errors in question. @xcite however, in the absence of a fully error marked-up text corpus, finding frequent errors is a challenge. we therefore provide a scheme based on our experience with finding common errors (for the north sámi grammar checker) as a guideline for work on new languages. this scheme serves any language, but our experience is based on morphologically richer languages. error types can be divided into three main categories: writing/formatting conventions apply to compounding (one vs. several words, hyphen), quotation marks, comma and punctuation in general. morpho-syntactic and syntactic errors can be subdivided into verb-, np-internal and vp-internal issues. np-internal issues can be about prepositions and postpositions and their case restrictions, adjective agreement /forms in attributive/predicative positions, and relative pronoun agreement with its anaphora in number, gender and animacy. verb internal issues concern the auxiliary construction, negation phrases (where negation is expressed by a verb) and other periphrastic verb constructions. vp internal issues, on the other hand, are more global and concern subject-verb agreement, subclauses formation, subcategorisation in general and case marking of object/adverbial and word order. in addition to that, the choice of error types will depend on efficiency as well, that means which error types can rules generalize over, and which error types are very word specific. very word specific work that cannot be generalized may not be so efficient. ## evaluation the first version of the lule sámi grammar checker has 64 rules and 17 rule types, three of which have a regression test of 50 or more test sentences. we also ran an initial evaluation of each regression test, and plan to run the grammar checker on the error-marked up corpus of 32,202 words foot_3 . figure 6 shows an evaluation of three error types with a sufficiently large regression tests. the other error types will be evaluated in the final version of the paper. the rules for relative pronoun and numeral/determiner agreement and for modal verb maybe-constructions give good results for both precision and recall. precision and recall of the modal verb constructions are as good as 98%. we are aware that this still needs to be tested on an independent corpus. the quality is measured using basic precision, recall and f 1 scores, such that recall @xmath0, precision @xmath1: f 1 = 2 p ×r p +r , where t p is a count of true positives, f p false positives, t n true negatives and f n false negatives. we also ran a test run of the automatic evaluation on the marked-up gold corpus of lule sámi, to see if the grammar checker finds true errors and also to improve the error mark-up of grammatical errors in the corpus, keeping in mind that the corpus had been originally marked up for predominantly spelling errors. a lot of errors found by the grammar checker are true positives. many of them were either not marked up or -more frequently -marked up with a different scope. since the start of marking up the corpora for spelling errors, the mark-up guidelines have been developed further in connection with gramdivvun, the north sámi grammar checker, and adapted to automatic evaluation, where the grammar checker output is tested against the corpus mark-up. there are examples of when the grammar checker actually found grammatical errors that the human proof-reader missed out. thirdly, there are examples where the original marking is not consistent with the newer guidelines for how much the scope of the error should be with regard to how much the grammar checker actually marks up. example (12) is one of the cases where an error in relative pronoun agreement has been identified correctly by the grammar checker. this error type a manually proofread lule sámi corpus used for a new text to speech (tts) tool. the grammar checker did find errors that the proofreader had missed and was therefore useful in a project where we want the text to be perfect. most of the responses from the grammar checker on this corpus were however false positives, with the grammar checker marking correct forms as errors. these 'bad' results were in turn used to improve and fine tune the grammar checker rules. we find this a very beneficial way of working -using our tools to double-check a proofread corpus, and at the same time using the results of the corpus to improve our tools. when running the grammar checker on a university level thesis, the grammar checker found many real errors. it was interesting that some highly frequent repeated errors were due to changes in the language norm. the overall results show us that the grammar checker actually finds real errors, but the main challenge with making it usable to users is to restrict the rules. at this point there is too much noise with more false positives than true positives. ## conclusion we have shown that by using a related language grammar checker as a starting point, we were able to create a basic level grammar checker for lule sámi, categorise a fair amount of frequent error types and collect regression tests for each of them in a reasonable amount of time (120 hours between two linguists, one of them a native speaker). the importance for language revitalisation cannot be measured before integrating the tools in the respective text processing programs for the language community to use. but we know from experience with the spell checker, that the tools have a wide group of users, and their importance can usually be felt in the number of complaints that are sent when something is wrong with the distribution or other technical issues. in the future, we want to offer a high-performance tool for the most common error types to the lule sámi users. we aim to release a beta version together with the commonly distributed spellchecker in 2022. 5 from the developer side we aim at regression tests of at least 100 examples per error type with at least 90 % precision and 70 % recall, so that the tool will be useful for a wider language community, be used in