A i M : Taking Answers in Mind to Correct C hinese Cloze Tests in Educational Applications (2022)

Abstract: To automatically correct handwritten assignments, the traditional approach is to use an OCR model to recognize characters and compare them to answers. The OCR model easily gets confused on recognizing handwritten Chinese characters, and the textual information of the answers is missing during the model inference. However, teachers always have these answers in mind to review and correct assignments. In this paper, we focus on the Chinese cloze tests correction and propose a multimodal approach(named AiM). The encoded representations of answers interact with the visual information of students’ handwriting. Instead of predicting ‘right’ or ‘wrong’, we perform the sequence labeling on the answer text to infer which answer character differs from the handwritten content in a fine-grained way. We take samples of OCR datasets as the positive samples for this task, and develop a negative sample augmentation method to scale up the training data. Experimental results show that AiM outperforms OCR-based methods by a large margin. Extensive studies demonstrate the effectiveness of our multimodal approach.

## introduction the growing number of students has brought much pressure on teachers to correct assignments manually in the educational field. recently, optical characters recognition (ocr) based methods are widely used in several applications 2 to automatically complete this task. the ocr model first recognizes the text in the image, and then the ocr output is compared with the correct answer to feedback correction results. in this pipeline figure 1 : task examples and correction methods. the ocr-based method misunderstands student's intent due to the width between "车" and "可" in question 2. aim can tackle this problem since it takes both the handwriting image and the textual answer into consideration. method, the post-comparison mainly relies on the pre-recognition. without prior knowledge of answers, the ocr model easily gets confused when recognizing handwritten chinese characters, especially for various handwriting styles, ligatures, and shape-similar characters. however, most of them can be distinguished by human teachers, because they take answers in mind at first. in this paper, we focus on the chinese cloze correction (ccc) and propose an answer-in-mind correction model (aim) to tackle the above prob-lem. figure 1 shows examples of the ccc task and the comparison of the ocr-based method with our method. we look at question 2. obviously, the student knows the correct answer and writes down "轲", but it is recognized as "车可" by the ocr model because of the slightly large width of the hand-written content. consequently, the correction result is 'wrong'. we hypothesize that the information of correct answers can help the neural model to understand student handwriting. as shown in figure 1 , aim is a multimodal model, which takes the image and the answer text as input. through the interaction of two modality information, our model understands the handwritten content and feeds back the 'right' correct result. in the aim model, the image is encoded to sequential feature representations through resnet @xcite , where each of them represents a fixedwidth pixels block. the answer text is encoded by word embedding. then transformer @xcite self-attention is adopted to compute contextual representations for each modality. in order to fuse them, we develop a cross-modal attention. it renders the textual representations to interact with the visual information of students' handwriting. on the top of aim, instead of predicting 'right' or 'wrong', our model performs sequence labeling on the answer text to infer which character differs from the handwritten content. to train aim, we collect einkcc, a dataset containing about 5k handwriting images, answers, and correction results of cloze questions, from our educational application. teachers distribute cloze tests in our app, and students practice on the electronic paper hardware. in addition to einkcc, ocr datasets can be used for this task. we take samples of ocr datasets @xcite as the positive samples, and construct negative samples by replacing the label characters with shape-similar ones derived from an open-sourced confusion set @xcite . the same method also augments einkcc to scale up the training set. we pretrain the image encoder in aim to get better visual representations, and further train aim with the correction objective. experimental results show that compared with ocr-based methods @xcite , aim achieves 11% accuracy improvements. extensive analyses verify that with the interactions between two modalities through our attention mechanism, aim can understand student handwriting and ligatures, and more handwritten characters confused by ocr can be predicted well by aim. the main contributions are summarized as follows: i) we propose aim, a multimodal model for chinese cloze correction, to make up for shortages of ocr-based methods. ii) we extend ocr datasets using a negative sample augmentation method to fit this task. iii) comprehensive experiments show that aim achieves better performance compared with ocr-based methods, and it's effective and necessary to use a multimodal approach to correct chinese cloze tests. ## method we first introduce the sequence labeling conversion for the ccc task in section 3.1. model architecture of aim is shown in section 3.2 and figure 2 . finally, we describe the data augmentation and pretraining methods in section 3.3. ## experiment in this section, we first introduce datasets, baselines, and other details of our experiments. then we show experimental results and perform analyses from different views. ## conclusion in this paper, we propose a multimodal model aim to effectively correct chinese cloze tests. aim employs cross-modal attention to understand the correlation between modalities. we collect data from different sources and develop a data augmentation method. experiments show that aim outperforms traditional ocr-based methods with over 11% accuracy improvements.