Automated Assessment of Noisy Crowdsourced Free-text Answers for H indi in Low Resource Setting (2020)

Abstract: The requirement of performing assessments continually on a larger scale necessitates the implementation of automated systems for evaluation of the learners’ responses to free-text questions. We target children of age group 8-14 years and use an ASR integrated assessment app to crowdsource learners’ responses to free text questions in Hindi. The app helped collect 39641 user answers to 35 different questions of Science topics. Since the users are young children from rural India and may not be well-equipped with technology, it brings in various noise types in the answers. We describe these noise types and propose a preprocessing pipeline to denoise user’s answers. We showcase the performance of different similarity metrics on the noisy and denoised versions of user and model answers. Our findings have large-scale applications for automated answer assessment for school children in India in low resource settings.

## introduction posing and assessing open-ended descriptive questions to children is crucial to evaluating their learning levels and improving their understanding of concepts. unfortunately, this puts an enormous load on the classroom teacher who is faced with assessing and providing feedback to every child. as a result, teachers are not able to give writing assignments or do oral evaluation as often as they would wish. the problem is even more important in rural areas where availability of teachers is low, and children seldom get access to quality assessments. this highlights the importance of developing applications that automate assessments for children. free-text questions allow a respondent to answer in open text format such that they can answer based on their complete knowledge and understanding. this means that response to this question is not limited to a set of options. though the evaluation of multiple-choice questions is straightforward and can be scaled, we need robust systems to assess the free text questions as well. this presents an interesting challenge for automated assessments as there are multiple versions of correct answers for the same question in free-text format. we show some examples in this paper presents our experience with automated assessment of free-text answers by developing an automated assessment system for one of the low resource languages -hindi which is the medium of instruction in government schools in rajasthan and uttar pradesh states of india. the children were of age group 8-14 years from rural areas of india, and hence were not well equipped with technology like their urban counterparts. we developed an android assessment app for these children to give assessments at any time they wanted to. one assessment has a mix of question types including ## previous work research in the area of evaluation of descriptive free text answers has been in progress since a decade and a half. @xcite did a comprehensive review of automatic short answer grading (asag) research and systems according to history and components. their historical analysis identifies 35 asag systems within 5 temporal themes that mark advancement in methodology or evaluation. butcher et al., (2010) compared the marking accuracy of three separate computerized systems, one system (intelligent assessment technologies freetext author) is based on computational linguistics whilst two (regular expressions and openmark) are based on the algorithmic manipulation of keywords. @xcite experimented with training a naive bayes classifier based on three parameters: keywords, grammar and question specific things. they proposed a system where students will have a certain degree of freedom while writing the answer as the system checks for the presence of keywords, synonyms, right word context and coverage of all concepts. but the experiment was conducted with only 20 students and 3 questions to each student. @xcite presented a comparative evaluation between bleu-inspired algorithm and a system based on latent semantic analysis and proposed a combination schema. despite the simplicity of these shallow nlp methods, they achieved state-of-the-art correlations to the teachers' scores while keeping the languageindependence and without requiring any domain specific knowledge @xcite proposed multiple data augmentation strategies for improving performance on automatic short answer scoring. they combined it with the latest fine-tuned bert model for the short answer scoring task, and show significant gain. @xcite discuss a new question paraphrase retrieval (qpr) system that can be used to understand and answer rare and noisy reformulations of common questions by mapping them to a set of canonical forms. to the best of our knowledge, this is the first study which handles noises in user answers while evaluating the short answers typed in by the children in indian language. also, focus is in a method with less computing and memory needs, so that it can be used for large scale implementation in android devices. ## data and attributes we conducted our assessment in the hybrid learning program of pratham which reaches more than 1000 villages and over 109,560 children. every group of 5-6 children have one tablet provided by pratham with digital learning content. an android assessment app (figure 1 ) was developed and loaded in these tablets for children to give assessments anytime they wanted to. the assessment app helped collect 39641 user answers to 35 different questions of science topics. each answer was then manually evaluated as correct/incorrect by two different raters and their agreement score as calculated by cohen's kappa κ score is 0.74. total average number of unique correct answers per question is 34. ## descriptive answer assessment while including questions with descriptive answers is an important aspect of learning, it brings unique challenges in terms of assessment. in this section we describe our methodology for assessment of such descriptive free-text answers and the challenges that are unique to it, thus emphasizing the importance of more research in this area. one major challenge as described previously is the presence of various noise types in the crowdsourced answers data. in this section we describe a preprocessing pipeline to denoise the answers. we also provide description of the semantic similarity measures we use to assess the user answers at scale in low resource setting. ## results and analysis we now compare the performance of various similarity scores on the human evaluated ground truth on answer assessment. we study the performance of these similarity measures before and after denoising and observe that denoising leads to considerable improvements in their performance. ## conclusion in this paper we have described various noise types and proposed a preprocessing pipeline to denoise crowdsourced free-text answers provided by children for grade 8 level science topics in hindi. this work is intended to facilitate research in automatic assessment of student's free text answers in regional india languages in low resource settings. we have compared the performance of various semantic similarity scores using human evaluated ground truth on their original noisy and denoised versions. we see that denoising helps jaccard similarity outperform semantic similarity measures thus presenting a strong case for feasibility of automated assessment in low resource settings. in the next phase of this work, we will fine tune the existing embeddings on education domain. with pratham's reach into 22 states and up to 15 million children in india, we can scale our crowdsourcing easily to include more responses in other regional languages. we hope that our findings mutually benefit the research community working in the area of descriptive answer assessments for regional languages meanwhile solving a very practical problem for society at scale.