A Prompt-independent and Interpretable Automated Essay Scoring Method for C hinese Second Language Writing (2021)

Abstract: With the increasing popularity of learning Chinese as a second language (L2) the development of an automatic essay scoring (AES) method specially for Chinese L2 essays has become animportant task. To build a robust model that could easily adapt to prompt changes we propose 90linguistic features with consideration of both language complexity and correctness and introducethe Ordinal Logistic Regression model that explicitly combines these linguistic features and low-level textual representations. Our model obtains a high QWK of 0.714 a low RMSE of 1.516 anda considerable Pearson correlation of 0.734. With a simple linear model we further analyze the contribution of the linguistic features to score prediction revealing the model’s interpretability and its potential to give writing feedback to users. This work provides insights and establishes asolid baseline for Chinese L2 AES studies.

## introduction automatic essay scoring(aes) is one of the most important natural language processing (nlp) applications in the field of education @xcite , and has been widely used in standardized language tests @xcite . however, existing works mainly focus on the scoring of english essays @xcite @xcite or chinese essays by native speakers and minority learners @xcite @xcite . although chinese second language (l2) acquisition has enjoyed an increasing boom in recent decades, the aes system designed for chinese l2 writing has received much less attention. meanwhile, existing aes methods face two important challenges. firstly, the scoring models are mostly built in a prompt-dependent style, i.e. training and testing for each specific prompt. it requires to collect prompt-specific data, yielding great costs in dataset construction @xcite . besides, the built models are of weak generalization capabilities and cannot be used to score essays of other prompts. secondly, although neural network methods have achieved great success in nlp tasks, the gains in neural aes systems are far from being satisfactory. for example, @xcite find that fine-tuning bert produces similar performance to classical models at significant additional cost. apart from the costs, the deep neural models are also weak in interpretability of the results. however, it is a very important property for aes users who expect to get feedback on the writing @xcite @xcite , not just a score. to solve the above problems, this paper proposes a prompt-independent and interpretable aes method for chinese l2 writing. specifically, we build prompt-independent models that could make full use of l2 writing data, and make predictions without the prompt limitations. for interpretability considerations, we extract 90 linguistic indices on accounting of the usage of characters, words, clauses, collocations, dependency structures, syntactic constructions which are emphasized in chinese l2 acquisition, and 5 indices that address different types of writing errors. furthermore, we integrate these linguistic and correctness indices into text representations, and introduce the ordinal logistic regression (olr) model to the aes task for chinese second language writing. our model achieves a high quadratic weighted kappa (qwk) score of 0.714, a low root mean square error (rmse) of 1.516, and a high pearson coefficient of 0.734, performing much better than the classical machine learning models and neural network baselines. the contribution of this paper is two-fold. (1) instead of building prompt-specific essay scoring models, it presents a generic model that could make full use of writing data, and score general narrative and argumentative essays. ## conclusion and future work in this paper, we propose a prompt-independent and interpretable aes method for chinese l2 writing. we build explainable representations of both the linguistic and text features, and the threshold-based ordinal logistic regression model is introduced to our chinese l2 aes task. the result on olr-at model under ling+err+text setting obtains a high qwk score of 0.714, a low rmse of 1.516, and a high pearson coefficient of 0.734. further, we find that with our method and the feature set, the model is explainable and has the potential to offer users feedback on the writing. this work provides insights and a solid baseline for aes studies of chinese l2 writing. at present, our method has integrated linguistic complexity, writing correctness and text features of essays. it still needs further study on developing prompt-essay relevance measures, as well as the discourse level indices e.g. cohesion and coherence. just as noteworthy, in