Implicit Phenomena in Short-answer Scoring Data (2021)

Abstract: Short-answer scoring is the task of assessing the correctness of a short text given as response to a question that can come from a variety of educational scenarios. As only content, not form, is important, the exact wording including the explicitness of an answer should not matter. However, many state-of-the-art scoring models heavily rely on lexical information, be it word embeddings in a neural network or n-grams in an SVM. Thus, the exact wording of an answer might very well make a difference. We therefore quantify to what extent implicit language phenomena occur in short answer datasets and examine the influence they have on automatic scoring performance. We find that the level of implicitness depends on the individual question, and that some phenomena are very frequent. Resolving implicit wording to explicit formulations indeed tends to improve automatic scoring performance.

## introduction automatic short answer scoring is an application area of natural language processing where short free-form answers written by students in an educational context are automatically scored based on the correctness of their content. they occur for example in science education @xcite , but also in foreign language learning to measure reading @xcite or listening comprehension @xcite . in such a scoring task, answers are graded based on their content alone -in comparison to essay scoring @xcite where also linguistic form is taken into consideration. thus, judging whether an answer is correct or not may require the resolution of a number of implicit language phenomena as a form of normalization. figure 1 implicit: 3 is the perfect amount, 2 is not enough, 3 is too many. explicit: 3 scoops is the perfect amount of fertilizer, because 2 scoops is not enough, but 3 scoops is too many. shows two answers that express the same content, but with differing levels of explicitness. how the content is expressed on the surface does not matter for the score. in fact, the two answers in the example should be treated in the same way regardless of their explicitness. the only relevant criterion should be whether they convey the right content and thus show that the learner understood the concepts. while humans often effortlessly resolve implicit phenomena, automatic resolution of many of these phenomena is not trivial. however, we argue that resolution of implicitness is a kind of normalization step that can help to improve automatic scoring performance. most work on automatic short-answer scoring does not actively resolve most implicit phenomena. however, the c-rater system performs pronoun resolution @xcite , but they do not report the impact of that single component. @xcite perform implicit resolution of coreferences between entities in learner answers and entities in the question and similarly target ellipses resolution, where part of the question is implied in the learner answer, both by aligning concepts from the learner answer to the question. they report a positive influence on overall scoring performance. another notable exception is information structure, i.e. whether the answer repeats parts of the question as researched through focus annotations by @xcite . they report only a minor effect on automatic scoring performance. in this paper, we analyse which implicit phenomena occur in short answer scoring datasets. we then analyze the impact of implicit language on automatic scoring performance. ## implicit language in learner answers there are a number of linguistic phenomena that pertain to the implicitness of language and are especially relevant for learner answers. in the following, we describe the ones we considered as candidates for our analysis. coreference coreference describes the phenomenon that the same entity is referred to several times throughout a text, often using different referring expressions (see @xcite ). the most prototypical example of pronominal reference is shown in example 1, where they at the beginning of the second sentence refers to the same entity as pandas in the first sentence. ## implicitness annotations short answer-scoring datasets can include very different prompts, i.e. an (optional) reading text and some question the student has to answer, coming from domains such as sciences, biology, or english language arts. to cover a range of different learner answers, we select prompts from two short answer datasets and annotate occurrences of the implicit phenomena within the learner answers given in response to these prompts. this procedure has three goals: first, we want to assess the frequency of these phenomena in learner data. second, we want to evaluate the effect of implicitness on the final score an answer receives, i.e. we ask whether implicit answers are on average scored higher or lower than explicit ones by teachers. and finally, we want to know the effect of implicitness on automatic scoring performance. we investigate this third question by extracting explicit versions of the answers regarding the different phenomena from the implicit versions. ## impact of implicit language on automatic scoring as we have seen in our dataset analysis, there is a large variance whether learners use implicit or explicit language. however, as in content scoring, only the meaning and not the form of an answer is important, both variants should be scored by an automatic scoring model in completely the same way. many state of the art models heavily rely on lexical information, be it word embeddings in a neural network or n-grams in an svm. thus, the exact wording of an answer might very well make a difference, especially if one variant is much more frequent than the other and therefore only rarely seen in the training data. to asses the extent of the influence of implicitness, we perform in this section automatic scoring experiments that control for the implicitness of our annotated phenomena in the data. ## conclusions we find that implicit language does occur frequently in short answer data and that the phenomena we focused our analysis on can reliably be annotated in learner answers, thus showing that such data is a promising source for implicit language in a relatively controlled setting. we will publish our set of annotated answers. as we find that making the answers more explicit improves their automatic scoring, a next step would be to automatically resolve implicit language into explicit, to enable examining this effect on a larger scale. subsequent analyses will also widen the experiments to include more different implicit phenomena and resolve more than one phenomenon in the same set of answers.