C hinese Content Scoring: Open-Access Datasets and Features on Different Segmentation Levels (2020)

Abstract: In this paper, we analyse the challenges of Chinese content scoring in comparison to English. As a review of prior work for Chinese content scoring shows a lack of open-access data in the field, we present two short-answer data sets for Chinese. The Chinese Educational Short Answers data set (CESA) contains 1800 student answers for five science-related questions. As a second data set, we collected ASAP-ZH with 942 answers by re-using three existing prompts from the ASAP data set. We adapt a state-of-the-art content scoring system for Chinese and evaluate it in several settings on these data sets. Results show that features on lower segmentation levels such as character n-grams tend to have better performance than features on token level.

## introduction short answer questions are a type of educational assessment that requires respondents to give natural language answers in response to a question or some reading material @xcite . the applications used to automatically score such questions are usually thought of as content scoring systems, because content (and not linguistic form) is taken into consideration for automatic scoring @xcite . while there is a large research body for english content scoring, there is less research for chinese. 1 the largest obstacle for more research on chinese is the lack of publicly available data sets of chinese short answer questions. working with chinese poses substantially different challenges than work on english data. unlike english, which uses spaces as natural separators between words, segmentation of chinese texts into tokens is challenging @xcite . furthermore, there are more options on which level to segment chinese text. apart from tokenization and segmentation into characters, which are two options also available and often used for english, segmentation into components, radicals and even individual strokes are additionally possible for chinese. ## challenges in chinese content scoring in this section, we highlight the main challenges when processing chinese learner data in comparison to english data sets. we first focus on segmentation, as tokenization is more difficult in chinese than in english and there are more linguistic levels on which to segment a chinese text compared to english. next, we discuss variance in learner answers, which is a challenge for content scoring in any language but manifests itself in chinese differently than in english. ## prior work on chinese content scoring as shown in for segmentation, most prior work uses the jieba tokenizer 2 for pre-processing. however, 2 https://github.com/fxsjy/jieba the performance of the tokenization is rarely discussed. we also notice that no related work uses segmentation on character or component level. @xcite perform stop word removal, but they do not mention if it included some kind of removal of grammatical markers. ## chinese scoring data sets in this section, we review existing chinese content scoring data sets. they are not publicly available, which is a major obstacle to reproducibility in the field. we thus produce two new chinese data sets (see detailed description in section 4.2), which are available online foot_1 to foster future research . ## experimental setup in this section, we adapt a state-of-the-art content scoring system to chinese. we evaluate it in six settings with different feature sets on the data sets described above in order to investigate different options for segmentation of chinese text. ## results and discussion although the performance of different feature sets is quite close to each other, we see a slight but significant advantage across data sets of component and character features over the token baseline.