Automated Answer Validation using Text Similarity (2023)

Abstract: Automated answer validation can help improve learning outcomes by providing appropriate feedback to learners, and by making question answering systems and online learning solutions more widely available. There have been some works in science question answering which show that information retrieval methods outperform neural methods, especially in the multiple choice version of this problem. We implement Siamese neural network models and produce a generalised solution to this problem. We compare our supervised model with other text similarity based solutions.

## introduction reading comprehension based question answering, where a passage is given and models have to answer questions based on the passage, has been studied extensively in the past decade. the science question answering problem is relatively harder given the it's a narrower domain. in particular, science question answering in the presence of distractors (known generally as multiple choice answers) remains an active area of research. sciq dataset introduced by @xcite ) has questions, multiple choices, correct answer, and a supporting passage. text similarity solutions are a reasonable choice to solve this problem. comparing the answers to ground truth in closed-book qa is the most generalized version of the problem. having multiple options makes the problem slightly harder since distractors can have lots of words in common with the correct answer. the particular problem that we want to address in this work is automated answer validation. given the question and the answer provided by students, we need to evaluate if the answer is correct, using figure 1 : an example question, options, answer and support text from sciq dataset. our task is to validate the user answer using the support text. the question and the supported text. we need to generate a pair between the student answer and the correct answers, where the correct answers are determined from the supporting text. consider the example in the sciq dataset in figure 1 . given the question and the supporting text, we can determine the correct answer is mesophilic organisms. now we can validate whether the student answer is correct based on text similarity. this is a relatively simple example, where the phrase meosphilic organisms occurs verbatim in the supporting text, but this may not always be the case. a typical information retrieval based solution to this problem will search the multiple options in the supporting passage. we want to investigate if a neural network model can perform well on this task. further, if possible, we would like to compare such a supervised model with large language model based solutions. we'll describe our solution in more detail in section 3. the automated answer validation system for science question answering will benefit educators and students in science-related subjects. it will streamline the assessment process, reduce manual effort, and ensure consistent evaluations, leading to improved learning outcomes in the science domain. improving learning outcomes in science education was also the motivation for the sciq project at allen institute for ai. in the indian context, solving this problem could help reduce the disparities in learning opportunities among different income levels and location of the students. in particular, our solution could help create a scalable platform for students appearing in competitive exams to take mock tests without having to pay for such services in coaching centers. ## related work approaches to multiple-choice science exam question answering (qa) differ in terms of their reasoning architectures and training processes. @xcite present a collection of sub-issues and methods for solving them. @xcite approach assesses how well a scenario, built from the question and augmented with additional knowledge base (kb) data, holds together. on the other hand, @xcite develop a model focused on entailment that uses a max-margin ranker to find the right answer based on background knowledge. models that employ probabilistic reasoning include techniques such as markov logic networks @xcite and an integer linear program-guided approach that constructs proof chains using organized knowledge @xcite . the aristo ensemble @xcite combines various reasoning approaches with basic statistical methods that rely on lexical relationships and information retrieval (ir). these simpler methods alone offer surprisingly effective starting points. there's been limited effort to apply neural networks to this challenge, mainly due to the scarcity of training data. this paper aims to tackle that problem by creating a substantially larger dataset than what existed before, and we showcase the outcomes of tests conducted using cutting-edge reading comprehension methodologies on our new datasets. siamese networks were introduced by @xcite who used the method for the task of determining whether two input signatures originated from the same person. siamese networks feature two or more identical subnetworks that accept distinct inputs but share the same parameters and weights. these subnetworks are then joined at some point, and the network learns a similarity function over its inputs. the architecture is particularly useful for tasks that involve comparing two different but similar pieces of data, such as in the case of signature or face verification. over the years, siamese networks have been adapted and extended for various tasks beyond signature verification, including face recognition, object tracking, and more. they have become a foundational architecture in the area of learning similarity metrics and are widely used in various applications today. @xcite describes the implementation of siamese network models for text and image similarity. @xcite describe an hybrid approach that combines bert extraction with deep siamese bi-lstm model for text similarity. @xcite focus on interpretability of bert based text similarity. (gollapalli and ng, 2022) proposed using a text similarity measure for question generation. @xcite propose using text similarity for question answering in a particular field of medical domain, namely rheumatoid arthritis questionanswering system. @xcite discuss the limitations of pearson correlation for validating text similarity metrics. ## our approach our solution for automated answer validation is as shown in figure 2 . given a question, answer, multiple options and supporting text, we use different text similarity methods to predict the correct answer. this answer prediction can be evaluated using ground truth (correct answer) available in the dataset. we implement a siamese text similarity model and compare with other methods like few shot inference and retrieval augmented generation using large language models. after implementing the above model, we compare our solution with an unsupervised method, where sbert with cosine similarity provides the text similarity score. we also discuss how our solution compares with large language models in section 6.3. while using llms, there are couple of approaches that we could try. retrieval augmented generation is closer to the information retrieval based solutions in this problem. we can ask the llm to predict if the selected answer is correct, given the question and the supporting text. as a separate exercise, we could also evaluate llms on closed book question answering. in the rag approach, we can try to extract a relevant portion of the supporting text, to help the llm make the correct prediction. in another approach, we can use few shot prompting, where we give the model few examples of the question, multiple choices and correct answers. an instruction fine tuned model like flan-t5 or gpt3.5 could use the samples to validate the answers. we discuss some of these in section 6.3 but do not implement in this work. ## siamese neural networks a siamese neural network @xcite , often referred to as a twin neural network, is a type of artificial neural network designed for comparing the similarity between two different input vectors by utilizing shared weights in parallel. one of the output vectors is typically precomputed, establishing a baseline for comparison against the other output vector. this concept can be likened to fingerprint comparison but is better described as a distance function used in locality-sensitive hashing. for the text similarity tast, the siamese network looks like the architecture in figure 3 . mathematically, the goal is to minimize the distance metric for similar objects while maximizing it for dissimilar ones. the most commonly used distance metric is the euclidean distance, represented as: here, x and y are vectors of dimension n, and d(x, y) calculates the euclidean distance between them. ## discussion in this section, we'll discuss ablations, deployment considerations and future extensions of this work.