Towards Automatic Short Answer Assessment for F innish as a Paraphrase Retrieval Task (2022)

Abstract: Automatic grouping of textual answers has the potential of allowing batch grading, but is challenging because the answers, especially longer essays, have many claims. To explore the feasibility of grouping together answers based on their semantic meaning, this paper investigates the grouping of short textual answers, proxies of single claims. This is approached as a paraphrase identification task, where neural and non-neural sentence embeddings and a paraphrase identification model are tested. These methods are evaluated on a dataset consisting of over 4000 short textual answers from various disciplines. The results map out the suitable question types for the paraphrase identification model and those for the neural and non-neural methods.

## introduction computer-assisted assessment brings about the promise of alleviating the workload of teachers, allowing them to concentrate manual efforts towards more creative pedagogical tasks. not all assessment types, however, have widely adopted fully automated or computer-assisted grading methods. essays, for example, are a common way to evaluate student knowledge, but are resource-demanding to grade. an angle to automatic essay evaluation is to group together similar essays for batch grading, but this is complicated by the complex structure of essays. short answers, on the other hand, often consist of only one or a few claims, and thus represent a desirable starting point for textual answer clustering. in addition to being a simplified target for studying textual answer clustering, short answers are also a common form of assessment; very short answer questions have been shown to have desirable traits of reliable assessments, such as the scores showing a fair and balanced distribution @xcite . automated short answer assessment is used in this paper as an umbrella term to refer to compu-tationally assisting the evaluation of short textual answers, while there is no unified definition for short textual answers @xcite . whereas some impose only length restrictions on the textual answers (e.g. one phrase to one paragraph), others have additional criteria such as the answer being a natural language response, or the focus of the assessment being knowledge content instead of grammar @xcite . in practice, the definition for short textual answers depends on the actual application, and the answers vary in terms of textual length, topic, assessment criteria, educational level of students, etc. these variations have fueled the long ongoing research on automated assessment of short textual answers. @xcite survey computer-assisted assessment techniques developed in the years 2000-2015 targeting short answers ranging from a sentence long to a maximum of 100 words. they suggest a matchmaking framework to guide the choice of appropriate techniques for practitioners and call for computer-assisted assessment methods that do not rely on model answers, as automated short answer grading (asag) systems usually do. one such alternative method is to group together semantically similar short textual answers for batch grading. this is a less explored research area but has been shown to effectively reduce the number of manual actions required for grading @xcite . the essence of both asag and short answer grouping is how the texts are represented, and thus their research methods are influenced by the advances in semantic textual similarity (sts) and paraphrase research. here, a typical asag system would measure the similarity between teachersupplied model answer(s) and student answers, whereas short answer grouping measures and groups student answers among themselves. apart from traditional string-based and corpus statisticsbased methods, dense vector representation meth-ods based on deep learning are naturally highly applicable to the task. a typical example of such methods are sentence-transformers @xcite that adapt the bert model to sentence representation by explicitly optimizing the similarity of dense-vector representation for pairs of sentences known to carry the same meaning. such models can be applied to answer grouping in a straightforward manner by comparing the dense representations of sentences across different answers. in a different line of work, @xcite approach paraphrase detection as a form of semantic search by training a question-answering type of a model to detect a paraphrase of a query from a given context document. this methodology can be seen as highly relevant to examining answer grouping: given an answer, or a part of an answer constituting a single claim, such model can then identify answers containing the same claim or its paraphrase among other students' answers. such an approach would, in theory, then allow the grading teacher to retrieve all such answers and carry out a common grading action. while not eliminating manual grading work, this approach could potentially significantly reduce the need, if paired with an appropriate interface and workflow. in this paper, we pursue this direction, approaching answer grouping from an information retrieval (ir) perspective, i.e. given an answer, or a claim from one answer, the task is to identify other answers containing the same claim or its paraphrase, not relying on the availability of model answers. the objective here is to retrieve similar answers for a given query to support e.g. batch grading. while we do not want to limit our methods to short answer assessment only, full long essays are likely too long as retrieval candidates. rather than retrieving on essay level, the natural unit for the retrieval would be to do it on the claim level, looking for similar claims inside the essays. however, for the time being we lack any manual annotation for individual claims posed in the essays, making the evaluation of such claim-level retrieval methods difficult. therefore, we approach the problem by using short answers only, where the answer typically includes only one or a few claims. the overall score assigned for the answer can then be used as a proxy of claim similarity, as all answers with high scores can be assumed to contain similar claims, even if using different wordings. we therefore formulate the overall task setup as such: given one claim as a query (in the form of a short answer), how well the experimented models are able to retrieve a similar claim among all candidates answering the same prompt (here "prompt" refers to the question posed by the teacher to which the students are answering) when judging the similarity based on the scores assigned to the answers. we use a dataset of over 4,000 teacher-graded short answers from actual university examinations of 24 distinct courses. we test non-neural and neural sentence embedding methods as well as the above-mentioned question answering -based paraphrase retrieval model, and map which types of questions are suitable for what types of answer grouping methods. ## related work the most researched direction for automated evaluation of short textual answers is automatic short answer grading (asag). this research field has seen the application of rule-based, machine learning, and deep learning methods @xcite @xcite . asag is typically modelled as a supervised learning task and seen as either a classification or a regression task, where a student answer is compared to a model answer, and the output label or score is based on their similarity. consequently, model answers are usually required for these systems. @xcite test the performance of various transformer-based @xcite language models on the semeval-2013 dataset @xcite , one of the most common asag dataset. they find that a robustly optimized bert pretraining approach (roberta)-large model @xcite fine-tuned on the multi-genre natural language inference (mnli) dataset @xcite performs best. short answer grouping is a less explored research direction, where short textual answers are grouped together based on their similarity. @xcite use a feature-based similarity metric to group short textual answers into hierarchical clusters. their features include i.a. difference in length, fraction of words with matching base forms, and cosinesimilarity of tfidf vectors. they show that such clustering can effectively reduce the number of actions required for grading. @xcite use the hyperlink-induced topic search (hits) algorithm @xcite ) to cluster open-ended questionnaire answers from students. applying this method to both english and finnish datasets, they obtain satisfactory results on the english dataset but less ideal results on the finnish dataset, potentially due to the finnish answers being longer in length. both the study of @xcite and @xcite predate the era of deep neural network-based methods of meaning representation. ## data our experiments are based on a large scale dataset of over 261k anonymized textual answers from different university-level examinations. however, for the purpose of this study, the dataset is heavily filtered in order to obtain a subset including only examples considered as short answers suitable for the study. we aim to find prompts looking for concise fact-based descriptions, which are likely to contain only a single claim and therefore have an increased likelihood that two answers with a high score are likely to be paraphrases of each other (although that naturally cannot be guaranteed without manual annotation). such suitable prompts ask for example term definitions, listings of the components of certain concepts, explanation of the workings of a process or device, explanations why e.g. a german noun is of a certain gender, or basically anything targeting to a short semanticfocused answer. in addition to the proper answer content, we also need the prompts to fit to our retrieval task setting, meaning that for each unique prompt, we need to have several student answers as retrieval candidates. one such example prompt together with few graded student answers for it is given in the original dataset is a collection foot_0 of 261k student answers gathered across various disciplines in the university of turku, finland. together with the textual answers, the data include the course identifier, question prompt, assigned score, and possible score range for each answer. the textual answers are written by mainly undergraduate students, and the most common languages are finnish and english. figure 1 illustrates the data filtering process. the filtering criteria for identifying a suitable short answer subset for this study are as follows: the prompt length must be under 10 tokens and the answer length under 30 tokens as determined based on the finbert model tokenizer 2 , and the lan-prompt: digital legacy score answer 1.0 a digital legacy is all the files and data about a person that remain on the internet or the digital world after the death of the person. 1.0 the trace that we leave behind digitally when we die (e.g. files, digital photos and usernames). 1.0 all the digital material that remains of a person after death. digital legacies include for examples passwords, usernames and photos of the deceased person. 0.5 traces left by the user of a computer or other technological device. what websites they have visited and what software they have on their device. 0.5 any data a person leaves behind on the internet or other computer systems. 0.0 all the things that were born in the digital form. 0.0 digital legacy means electronic waste, often exported to the third world. 0.0 the evolutionary trajectory of digital devices. as the query, should have the same grade. consequently, a method which is better at the retrieval task should, on average, be more likely to retrieve answers with the same score as the query than a method which is worse at the retrieval task. as we are mostly interested in relative method performance, we measure and report the success of the retrieval by top-1 accuracy and r-precision. the relevance of the retrieval is binary, meaning that retrieval with matching grade to the query is counted as "correct", and otherwise "incorrect". ## results the top 1 accuracy and r-precision of the methods across the 24 courses are shown in tables 4 and 5 respectively. courses numbers 8 and 17 have values of 1.0 on both metrics for all methods because both of them have 10 answers to a prompt where all students answer correctly. excluding these two courses, the span detection model scores the best or equally the best with another method on 11 and 12 courses respectively on top 1 accuracy and r-precision, outperforming the other methods. sbert-finn performs well in terms of r-precision on the life sciences discipline, performing the best on 8 out of 9 courses. the numerical differences of the accuracy values among these four methods are oftentimes minimal, and we investigate the ones with bigger differences to establish whether certain kinds of prompts are particularly suitable for a given method. we observe that the neural representation is advantageous when the prompts are challenging, which leads to the students inventing plausible answers using the keywords. an example of a query from a prompt where the tfidf method underperforms the neural method by a large margin (0.4 vs. 0.7) is shown in ## discussion in this paper, the span detection model is forced to only predict the probabilities of whole documents being paraphrases of the query. if this restriction is removed, the span detection model is capable of predicting arbitrary spans as the paraphrases of the query. this becomes relevant when obtaining the full score requires mentioning of multiple claims. for example, if a prompt asks students to explain abbreviations, a full scoring answer requires the student to provide the full form of an abbreviation and explain what it means. in our initial experiments, we observe that the span detection model can retrieve a span out of the full answer which is semantically equivalent to a partial answer. the evaluation of such retrievals, however, is not possible given our current data without manual annotations because a full scoring answer has a different score than a partial answer, nor is there a way to attribute which sub-spans of the full answer contribute how much to the final score. the exploration of how the span detection model can be applied to answers consisting of multiple claims may pave the way to eventually automatically evaluating essays. a potential way is to combine the answers of related prompts as queries and documents. we leave this to future work. a challenge for experimental design is the selection of suitable metrics. top 1 accuracy has the advantage of being easily understandable and interpretable, but its calculation ignores the expected value of random retrievals. r-precision mitigates the randomness to some extend, since it takes into account the top r retrievals where r is the number of relevant documents. when all the documents are relevant, r-precision is always 1 and it is not immediately obvious if the model performs meaningful prediction, though this can arguable be regarded as unsuitable data for retrieval, or, from a practical point of view, the retrievals will always be relevant. the design of r-precision is not completely compatible with the nature of the span detection model, as the model predicts null, which has to be taken into account if it ranks among the top r. the null prediction can either be regarded as an irrelevant prediction, or ignored altogether as we have done so in this paper. the use of binary relevance means a retrieved document is either relevant if it has the same score as the query, or irrelevant if it does not. this does not take advantage of some of the scores being of higher granularity. for example, if the query scores 1 and model a retrieves a document scoring 0.7 and model b a document scoring 0.3, the retrieval of model a is likely better than that of model b. an ideal metric would thus take into account the numerical difference between the scores of the query and the retrieval, as well as the informativeness of the set of documents available for retrieval. a class of metrics we have explored but did not eventually use is normalized discounted cumulative gain (ndcg). ndcg is a class of commonly used ir metrics, where the discounted cumulative gain, which sums the relevance of the query and retrieval (which can be graded instead of binary) discounted by the ranked position, is normalized by the ideal the theory is that the more information you process, the better it is remembered. the quality of processing is more important than the duration. xlm-r← 0.5 the most important thing in information processing is quality, not duration. sbert-para span detection 0.5 the most important thing in information processing is quality, not duration. the multilingual sentence embedding model does not outperform the non-neural baseline. this is somewhat surprising, as some of the short answers contain code-switching, such as the examples in figure 2 . this shows that language-specific sentence embeddings and models are still more suitable for this task. the task setup is only an approximation. the same grade does not imply the query and document being paraphrases, not for high grades nor for low grades, unless the grading criteria is semantically stringent, in the cases of e.g. translation studies. however, the hope is that the noise can be mitigated by using a large dataset and some signals can be seen as to whether the models are able to retrieve semantically documents. our results show that they indeed can. ## conclusion in this work, we explored several methods for grouping student answers to exam prompt. in addition to the standard setup whereby whole short answers are represented as either sparse (tfidf) or dense (transformer) vectors and compared to one another, we also tested a more retrieval-style approach, whereby we formed documents by concatenating a number of answers to the same prompt and testing to what extent the model is able to retrieve similar answers from such documents. this approach models the case of matching individual claims in longer answers. unsurprisingly, we find that the dense representations are more suitable to the task. interestingly, we find that a span detection model trained on finnish paraphrase data performs better than sentence-level embedding comparison methods. it might therefore be fruitful to pursue models which are not restricted to apriori given sentence boundaries, and which are capable of finding individual claims in collections of potentially longer essay-style answers. while the study is based on real exam answers from a number of courses, the data lacks manual annotation of the semantic equivalence of answers, which is challenging to produce. further, to be able to use the grades as a proxy to retrieval evaluation, we had to restrict ourselves to short, fact-checking questions, only using a small portion of the over 200,000 answers we have at our disposal. a natural further study would expand the use of the retrieval model to longer answers and employ teachers to evaluate the retrievals provided by the model and establish the overall benefit of such approach.