S c AA : A Dataset for Automated Short Answer Grading of Children’s free-text Answers in H indi and M arathi (2020)

Abstract: Automatic short answer grading (ASAG) techniques are designed to automatically assess short answers written in natural language. Apart from MCQs, evaluating free text answer is essential to assess the knowledge and understanding of children in the subject. But assessing descriptive answers in low resource languages in a linguistically diverse country like India poses significant hurdles. To solve this assessment problem and advance NLP research in regional Indian languages, we present the Science Answer Assessment (ScAA) dataset of children’s answers in the age group of 8-14. ScAA dataset is a 2-way (correct/incorrect) labeled dataset and contains 10,988 and 1,955 pairs of natural answers along with model answers for Hindi and Marathi respectively for 32 questions. We benchmark various state-of-the-art ASAG methods, and show the data presents a strong challenge for future research.

## introduction answer assessment is a key component of teaching and learning process and automating it has many advantages including speed, availability, consistency and fairness of assessments. though the evaluation of multiplechoice questions is straightforward and can be scaled, there is need for systems to assess free text answers as well. prior research has shown that recognition questions (like mcq) are deficient as they do not capture multiple aspects of acquired knowledge such as reasoning and self-explanation @xcite . the free text answers are important as they help measure the understanding of the student with respect to a particular concept. grading responses to short answer questions is considered difficult as it requires deep understanding of natural language. there can be multiple versions in which a correct answer can be articulated for the same question (examples highlighted in an android app 2 for assessment was developed by our team at pratham and was piloted in our hybrid learning program. the hybrid learning program of pratham is spread across 3 states of india -rajasthan, uttar pradesh and maharashtra. the android assessment app enabled collection of free text answers to questions in hindi and marathi essential for building systems for asag in indian languages. these free-text answers are 2-way labeled as correct/incorrect by human annotators. the main contributions of this paper are: in hindi and 1,955 in marathi to 32 questions in 8 different topics of science ( § 3) • benchmark of various state-of-the-art methods for automated assessment of free-text answers by children ( § 4) ## prior art there are numerous standard asag datasets publicly available for the research community to experiment with: beetle and scientsbank(seb)released as part of sra corpus @xcite , csd @xcite , x-csd @xcite , powergrading (pg) @xcite and asap @xcite . the total no of prompts in these datasets are in the range of 10 (asap) to 135 (seb). while some annotated dataset (pg; beetle) have 2-way labels (correct/incorrect), a few (csd; x-csd; asap) have scores on an ordinal scale within a range, e.g. 0-5, seb has a more complex 5-way labels. though there are numerous asag datasets available, all these are in english and none are available for indian languages. to the best of our knowledge, this is the first comprehensive corpus with reasonable size for hindi and marathi. numerous approaches have been tried before for asag. @xcite and @xcite do a comprehensive review of asag systems. @xcite show that answers can be accurately graded by using semantic measures. @xcite propose word matching between (user answer, model answer) pair. @xcite propose an unsupervised asag technique using sequential pattern mining. @xcite train a supervised model based on semantic similarity features. supervised methods include neural architectures by @xcite where performance and optimal parameter settings vary across prompts, a joint-multidomain deep learning architecture by @xcite which learns generic and domain-specific aspects. @xcite introduce data augmentation strategies and show that this combined with latest bert model brings significant gain. question in hindi question in marathi english translation (etl) correct answers in hindi english translation correct answers in marathi english translation एक वय&क मनु *य के कं काल का वजन लगभग 2कतना होता है ? एका 7ौढ :य;ती=या सां गा?याचे वजन अं दाजे 2कती असते ? what is the approximate weight of an adult human skeleton? एक वे &ट मनु *य के कं काल का वजन लगभग 10 2कलोdाम होता है the skeleton of a n adult human we ighs about 10 kil ogram 10 2कdॅ 10 kg १० 2कdा 10 kilogram दहा 2कलो dॅ म ten kilogram लगभग 10 2कलोdाम approximately 10 kilogram चाँ द पर कोई 2कसी कl आवाज ;यn नहoं सु न सकते ? अं तराळवीर चं qावर एकमे कां चे आवाज का ऐकू शकत नाहoत? why can't anyone hear someone's voice on the moon? वायु मं डल नहoं है no atmosphere वातावरण न सते there is no atmosphere ;यn2क वहां पर वायु नहoं है because there is no air wतथे हवा नाहo there is no air वायु मं डल का अनु पि&थती के कारण because of the absence of atmosphere कारण wतथे हवा नाहo because there is no air वायु मणडल ना होने से कारण due to lack of air ## scaa data creation we curate science answer assessment (scaa) dataset in hindi and marathi language comprising of 8 science topics with 4 questions per topic, i.e. a total of 32 parallel question-model answer pairs in hindi and marathi. the dataset is created via three stages: question and model answer curation, user answer collection, user answer evaluation. the questions were selected from grade 8 level science topics: adaptation, circulatory system, eye and vision, heat, simple machine, skeletal system, sound, water chemistry. the users here are children in the age group of 8-14 years from 3 states of india: uttar pradesh, rajasthan and maharashtra. ## automated short answer assessment we model the assessment of user answers against reference answers as a similarity task. each (user answer, model answer) pair is assigned a similarity score using the various state-of-the-art methods for assessment of free-text answers described in this section. we use random score assignment as baseline. user and model answers are tokenized into their constituent words using indicnlp tokenizer @xcite ). ## results and analysis we now benchmark the asag methods described earlier on scaa taking only the unique answers for evaluation ( note that while on full scaa with repeating answers patternmatch-repeat is comparable to s-bert, its use is suitable in batch mode to extract answer patterns. it therefore renders itself unusable in apps, where the requirement is for real time evaluation for single (user answer, model answer) pair. ## conclusion in this paper we present scaa, a dataset of children's free-text answers to 32 questions of grade 8 level science topics in hindi and marathi along with their user answers. this dataset is intended to facilitate research in automatic assessment of short answers in indian languages. we benchmark the performance of various stateof-the-art asag methods on scaa and observe that even though bert based model performs best, it makes errors in assessment that can affect students' learning engagement, thus leaving scope for improvement before such techniques can be deployed in real world and presenting a strong case for more research in this area. we believe that this dataset will be useful for the research community working on automated short answer assessment for indian languages and aid in solving a very practical problem for society at scale.