Towards automatic essay scoring of B asque language texts from a rule-based approach based on curriculum-aware systems (2023)

Abstract: Although the Basque Education Law mentions that students must finish secondary compulsory education at B2 Basque level and their undergraduate studies at the C1 level, there are no objective tests or tools that can discriminate between these levels. This work presents the first rule-based method to grade written Basque learner texts. We adapt the adult Basque learner curriculum based on the CEFR to create a rule-based grammar for Basque. This paper summarises the results obtained in different classification tasks by combining information formalised through CG3 and different machine learning algorithms used in text classification. Besides, we perform a manual evaluation of the grammar. Finally, we discuss the informa- tiveness of these rules and some ways to further improve assisted text grading and combine rule-based approaches with other approaches based on readability and complexity measures.

## introduction text classification of writing and reading materials is laborious and sometimes hard to do manually. teachers that do not have a linguistic background do not feel confident in this task, but in some languages, researchers can use automatic text classification tools to point to some objective measures (type token ratio, pos-based measures...). however, this automatic task is difficult to address for low resourced-languages. the classifi-cation of essays is worthy of interest because even if basque education law mentions that students must finish compulsory secondary education at the cefr b2 level and their undergraduate studies at the c1 level, there are no objective tests or tools that can discriminate between these levels. using deep learning-based methods could be difficult for teachers as these do not follow the language curriculum or the learning stage of the student. if automated systems could describe the curriculum or the learning stage of the student in a way that the teachers can understand or employ, this would be very useful, and teachers would have an additional source of information where they could offer more adapted materials and teaching. this work aims to explore rule-based models to classify written learner texts. the motivation of this work is to lighten the burden on teachers in the correction task. we want to contribute to the area of tools or applications to carry out objective tests automatically to fundamentally discriminate between levels b2 and c1. in this line, previous work @xcite emphasises the role of automatic systems to help teachers: automated essay evaluation represents a practical solution to a time-consuming, labour-intensive and expensive activity of manual grading of student's essays. furthermore, this approach could help to define language-based classification criteria that follow heoc, the adult basque learner curriculum @xcite . one of the difficulties of classifying and grading essays is represented by the perceived subjectivity of the grading process. this issue may be faced through the adoption of automated assessment tools for essays @xcite . there have been other studies of automatic classification for the basque language but from different approaches @xcite @xcite @xcite . we expand on these works and study how automatic text classifiers can benefit from basque curricular grammar, a formalisation of the linguistic expressions described in the basque curriculum. other similar works include a system for the arabian language @xcite as well as feature-based machine learning approaches for estonian @xcite ) and monolingual, cross-lingual, and multilingual classification with three languages: german, czech and italian @xcite . for estonian, the best model reported by @xcite reaches a prediction accuracy of 79%. regarding the automated essay-scoring task, @xcite conducted an automatic assessment using automatic essay scoring systems. @xcite , in their work, showed that early approaches were rule-based, but later systems relied on probabilistic models based on natural language processing methods that exploit the corpus of learners. their method exploits machine learning algorithms to classify learner writings with many metrics, including specificallydesigned microsystem metrics. microsystems are composed of several competing constructions (for instance the use of the article) grouped according to functional proximity. they can be defined as families of competing constructions in a unique paradigm. results on internal data show that different microsystems help to classify essays from b1 to c2 levels (82% accuracy). we follow a language-and curriculum-based approach: we formalise the expressions and linguistic phenomena described in the heoc using cg3 @xcite , creating a level-informed grammar for basque. the basque cg3 grammar contains 296 add rules that add language-level information. these rules are based on the linguistic indicators described for each level in the heoc. we apply this grammar to the habe-ixa basque learner corpus @xcite , annotating the phenomena described by the curriculum. we use the information provided by the grammar to classify texts in binary and multi-class experiments and analyse which rules are relevant to discriminate different cefr levels. the paper is organised as follows: in section 1, some background information on the basque curriculum and text classification task is provided. then our method to support essay classifying is described in section 2. in this work, we evaluate and compare the results obtained using the cg3 grammar features with different algorithms. detailed figures shall be shown in section 3. after that, in the discussion, we propose some future lines of work, in section 4. finally, in section 5 we sum up the main conclusions. ## method we adapted the adult basque learner curriculum based on the cefr to create a rule-based grammar for basque. first, we identified and defined phenomena and linguistic structures collected in heoc that will be formalised for each level. the result of this task is what we call basque cg3 grammar. the employed corpus contains essays written in official habe (basque government department for language certification) exams. it contains 480 texts (146,465 tokens) from the b1, b2, c1 and c2 cefr levels. the corpus is balanced, it contains 120 essays of each level. these essays have been evaluated by at least two language expert testers. following habe's evaluation criteria, some of the texts have not obtained a passing grade for the exam, others have passed by a small margin and others have passed with good grades (see we apply our grammar in the morphologically annotated habe-ixa corpus. then, the results are filtered by removing linguistic instances that are either too common (the absolutive case, common nouns) or appear scarcely in this corpus (less than 10 total tags). the number of times each rule has been applied can be seen in figure 1 . it must be mentioned that the length of the essays in the corpus depends on its cefr level, with b1 texts being the shortest and c2 the longest. to evaluate this set of rules we have followed an intrinsic manual evaluation method checking whether the labels were applied correctly and an extrinsic automatic evaluation method where we use the annotation data to automatically classify texts depending on their cefr level. for the latter, we want to see if the expressions identified by the cg3 rules encode relevant information about a text's level and complexity. we will perform the tasks of classification in two experiments: ## results we evaluate the results through a detailed analysis consisting of an intrinsic and extrinsic evaluation. regarding the intrinsic manual evaluation method, we check manually if the labelled features were properly annotated using cg3 rules. the results in during this evaluation, we realise that some rules are too general, so they are not informative, such as the use of common nouns, the use of absolutive case, the use of certain verbs tenses, and so on. therefore, certain linguistic features of the heoc are common at all levels and are very basic features that will have a greater presence in the texts, but from a qualitative point of view, they do not represent linguistic structures that help to determine a specific level. these common features will therefore be discarded in a future version because we will obtain this data by other means, for example using the basque version of ctap. regarding the extrinsic evaluation method, we proposed a classification task using only the cg3 rules: we use the data generated by our rule set to classify essays into a cefr level. we want to see i) if the curriculum indicators that we have formalised as a ruleset allow us to determine the cefr of an essay, and ii) which rules are most relevant for this classification. we encode each text of the habe-ixa corpus as a feature vector containing the extracted data of each rule as shown in figure 2 . we eliminate redundant information by filtering all the rule results that have a pearson correlation higher than 0.95. then, to avoid classification based on text length, we normalise each feature vector with the number of tokens in the text. finally, we rescale the features to the same range (from 0 to 1) using a min-max scaler. we split our data into training and evaluation sets (see ## discussion in this section, we explain some issues that may help to better understand the results: the size of the corpus and the application/design of the cg3 rules. the size of our corpus is one of the characteristics to take into account. the corpus is made up of 480 texts, 120 texts for each level (from b1 to c2). in total, there are 146,465 tokens in the corpus. compared to corpora with similar characteristics, our corpus is a bit smaller. for example, thewissen (2013) uses 223 texts with 150,000 tokens to study the evolution of errors through the different levels. chen and baker (2016) study lexical bundles in learner essays. the corpus used for that reason is bigger: it is made up of 585 essays and 202,154 tokens. on the other hand, lahuerta (2018) examines the texts of 100 spanish efl learners. the total number of tokens is 31,900. the corpus is used to study the accuracy and grammatical complexity. @xcite want to predict proficiency levels in learner writing. to do this, they use two datasets: i) the learner output corpus (320 texts and 140,949 tokens) and ii) the expert input corpus (818 texts and 289,312 tokens). as it has been observed, the size of our corpus is small compared to other similar works. con-sequently, we have been able to find fewer errors, and this limits the accuracy of the results. apart from the size of the corpus, we think that it should be noted that the labels introduced by the rules indicate the level at which a given linguistic structure corresponds, this does not imply that they are only applied in texts corresponding to that level. for example, the labels of the most basic levels (a1, a2), such as common nouns, declension, verb tense and so on, also apply in texts of higher levels such as b2, c1, basque c2... we can say that the grammar is coherent with the heoc curriculum on which it is based. that means that each level meets all the features and linguistic phenomena of the levels below it. finally, from the grammarian perspective, the typology of these rules is varied (general phenomena and rules for specific constructions or words) and there have also been small differences in the way the labels are designed (for instance, for connectors we have general rule vs fine-grained rules). therefore, it would be convenient to unify the criteria for creating rules for the next version. ## conclusion and future work in this paper, we present the first cg3 version of the basque grammar based on heoc to grade written basque students. the classification tasks performed on the experiments based on this first version show that the information provided by our rules is useful for discriminating different cefr levels. there is a correlation between the greater the number of labels of different types, the higher the level of the text. our experimental results suggest that our approach has promising results, advancing the construction of automatic tools to test and discriminate between b2 and c1. a more detailed analysis of the results shows also that the informativeness of these rules should be improved in the future. in that sense, we think that a redefinition of the principles for writing grammar benefits the explainability of the linguistic information added by the rules. finally, we believe that we will improve in assisted text grading by combining rule-based ap-proaches with other approaches based on readability and complexity measures. supplementary material basque rule translation to english 1 a1. sintaxia. aditz trinkoa a1. syntax. synthetic verb 2 b2. zenbatzaileak b2. quantifiers 3 a2. sintaxia. aditz-izena a2. syntax. verbal noun 4 a2. sintaxia. indikatiboko orainaldia a2. syntax. present indicative 5 c1. adt c1. synthetic verb 6 b1. juntadura b1. coordination 7 b1. aditza b1. verb 8 b1. zehaztugabeak b1. indeterminates 9 a2. sintaxia. aspektu burutua a2. syntax. perfective aspect 10 b2. izena b2. nouns 11 b2. izenordainak b2. pronouns 12 a1. sintaxia. indikatibo orainaldia a1. syntax. past indicative 13 a1. izena. biziduna a1. noun. animate 14 a1. izena. berezia a1. proper noun 15 a1. aditz iragankorra a1. transitive verb 16 a1. sintaxia. gertakizuna a1. syntax. future 17 a1. izen funtziozko menderakuntza a1. noun subordinate clauses 18 a2. deklnabidea. soziatiboa a2. sociative declension 19 b2. galdetzaileak b2. interrogatives 20 a1. elkartuak. aurkaritzakoa a1. adversative 21 b1. aditz-izena b1. verbal noun 22 a2. sintaxia. bakuna, baiezkoa a2. simple sentence, affirmative 23 a1. deklinabidea. instrumentala a1. instrumental declension 24 a1. galdetzailea. zergatik a1. interrogative why 25 a2. sintaxia. aspektu ezburutua a2. syntax. imperfective aspect. 26 a1. elkartuak. hautakaria a1. disjunctive 27 a1. galdetzailea. zer a1. interrogative what 28 a2. deklinabidea. adlatiboa a2. adlative declension 29 c1. graduatzaileak c1. grade particles 30 b2. adberbioak b2. adverbs 31 a2. deklinabidea. inesiboa a1. inessive declension 32 b1. menderakuntza. zehar galdera b1. subordination. indirect/reported question 33 a2. deklinabidea. partitiboa a2. partitive declension 34 c1. adl c1. auxiliary verb 35 c1. deklinabidea c1. declension 36 a1. deklinabidea. ergatiboa a1. ergative declension 37 a2. deklinabidea. genitibo edutezkoa a2. possessive genitive declension 38 a1. elkartuak. baldintza a1. conditional 39 b2. indartuak b2. strengthened forms 40 a2. sintaxia. al partikula a2. syntax. al particle 41 b1. adberbioak b1. adverbs 42 a2. deklinabidea. genitibo leku-denborazkoa a2. declension. spatiotemporal genitive 43 c1. postposizioak c1. postpositions 44 b2. postposizioak b2. postpositions 45 a1. deklinabidea. ablatiboa a1. ablative declension 46 c2. modalizazioa c2. modalization 47 b2. partikulak b2. particles 48 b1. puntua laburduretan b1. dot in abbreviations 49 c1. juntagailuak c1. conjunctions 50 c2. testu-markatzaileak c2. text markers 51 b1. plural hurbila b1. close plural 52 c1. determinatzaile zehaztugabea c1. indefinite determiner 53 c1. indartuak c1. strengthened forms 54 a2. deklinabidea. destinatiboa a2. destination declension 55 c1. aditzak c2. verbs 56 b1. deiktiko pertsonalak b1. personal deictics 57 c1 moduzkoak1 c1. modal clauses1 58 c1. moduzkoak2 c1. modal clauses2 59 b1. elkarkariak b1. reciprocality 60 c1. moduzkoak3 c1. modal clauses3 61 c2. operatzaile argudiozkoak c2. argumentative operators 62 c1. testu-antolatzaileak c1. discourse markers 63 c1. helburuzkoak c1. final clauses 64 c1. kontzetsiboak c1. concessive 65 a1. sintaxia. ahalera a1. syntax. potential 66 c2. berbaldi markatzaileak c2. discourse markers 67 b2. menderakuntza. galde-perpausa b2. subordination. question sentence 68 c1. aditzondoak c1. mood adverbs 69 b2. aditz lokuzioak b2. verbal locution 70 c1. mendekoak c1. subordination