‘Meet me at the ribary’ – Acceptability of spelling variants in free-text answers to listening comprehension prompts (2022)

Abstract: When listening comprehension is tested as a free-text production task, a challenge for scoring the answers is the resulting wide range of spelling variants. When judging whether a variant is acceptable or not, human raters perform a complex holistic decision. In this paper, we present a corpus study in which we analyze human acceptability decisions in a high stakes test for German. We show that for human experts, spelling variants are harder to score consistently than other answer variants. Furthermore, we examine how the decision can be operationalized using features that could be applied by an automatic scoring system. We show that simple measures like edit distance and phonetic similarity between a given answer and the target answer can model the human acceptability decisions with the same inter-annotator agreement as humans, and discuss implications of the remaining inconsistencies.

## introduction imagine a listening comprehension task where a student listens to two people scheduling a meeting at the library. the student is then supposed to answer the question 'where do they want to meet?' and writes 'ribary' instead of 'library'. is this answer acceptable or not? the answer to this question is not an easy one. human experts perform a complex holistic decision in such a case, primarily based on whether they assume that the learner understood the right answer (see section 2). the aim of this paper is to get a deeper understanding on which factors influence the acceptability of a spelling variant and ultimately how to model this decision automatically. thereby, we aim at a model that is transparent and uses features which allow to explain under which conditions the system accepts a variant and under which not. to this end, we conduct a corpus study based on real learner answers and human ratings in a high stakes test of german as a foreign language and explore different operationalizations of spelling variant acceptability. we show that our classifier does not yet reach an adjudicated gold standard, but the human decisions can be approximated up to the same level as human-human agreement. finally, we discuss possible reasons and implications of the remaining inconsistencies. the remainder of the paper is structured as follows: in section 2, we give some background about listening comprehension tasks and the role of orthography. in section 3, we introduce the data set and in section 4, we analyze the distribution of spelling variants and the human acceptability decisions. section 5 examines different features that could be used to operationalize the holistic human acceptability decisions. ## background in many high stakes language tests, listening comprehension is tested with a free-text production task (e.g. dalf foot_0 for french, goethe certificate foot_1 and testdaf foot_2 for german, cambridge certificate foot_3 for english). this means that the test takers have to listen to an audio prompt and formulate an answer in their own words. this gives rise to variance in the answers, e.g. synonyms or different syntactic or orthographic variants @xcite , which makes the automatic scoring of such answers a challenging nlp task. while variance at the level of wording or syntax is a topic extensively covered both by short-answerscoring in general @xcite as well as computational semantic similarity @xcite , the implications of orthographic variance are an understudied topic in automatic scoring. in e.g. reading comprehension tasks, where test takers can often copy material from the prompt, spelling errors are usually ignored @xcite . in listening comprehension tasks, however, the assessment of orthographic variants (e.g. ribary or librarie for library), plays a much more central role, as we will briefly outline. receptive skills like listening comprehension can only be measured indirectly, i.e. by inferring comprehension from the performance in a derived task @xcite , e.g. multiple-choice or true/false questions or free-text production tasks. all these tasks require skills that go beyond pure listening comprehension @xcite , e.g. reading comprehension for answering multiple-choice items and writing skills for free-text answers. test designers have to carefully decide whether such a skill is considered to be relevant for the construct to be tested or not. in the context of academic listening, for example, note-taking is an important skill and therefore considered to be construct-relevant @xcite . orthography, in contrast, is considered a construct-irrelevant skill for the task and should thus be ignored for scoring. this means that if the test-taker had the right answer in mind without being able to express it in an orthographically correct way, the answer should be marked as correct (see e.g. @xcite , @xcite ). the crucial difficulty hereby is that the spelling of the word interferes with the assessment whether the test-taker had the right answer in mind. if the test-taker, for example, just produces some vague encoding of the relevant phonetic string, this likewise leads to a spelling variant of the correct answer but it should be marked as incorrect. hence, the acceptability of a spelling variant is based on a complex holistic decision that an automatic scoring system is not straightforwardly able to make in the same way. nevertheless, an operationalization has to be found which leads to ratings that match the human ratings as closely as possible. furthermore, in a high stakes test it is crucial that the decisions of the automatic scoring system are transparent and understandable to human experts. ## data set in this paper, we experiment with data from the digital testdaf. it is a high stakes test designed for students planning to apply for studying at a german university. it assesses test-takers' language abilities at the testdaf levels 3, 4 or 5, corresponding to the cefr levels b2 to c1. the listening comprehension section consists of seven different task types, including selectedresponse item formats like multiple-choice questions, as well as three constructed response tasks where test-takers are asked to write short answers, between single words and a few sentences in length. in this paper, we concentrate on the task that elicits very short answers of a maximum of five words per prompt. this task is particularly suitable to study the role of spelling variants because other sources of variation are limited compared to longer textual answers. in this task, test-takers listen to a pre-recorded conversation between two or three native speakers in a situation typical for everyday student life, e.g. a conversation between a student and a professor. test-takers are presented a table, form or chart related to the content of the listening text with five blanks that are to be filled while listening to the input text. see figure 1 for an example. while testtakers can type in a maximum of five words per blank, all blanks can be answered correctly with one or two words. for the analyses in this paper, we extracted all answers from 17 different prompts where each prompt corresponds to one blank in the task described above. ## human ratings of spelling variants in the following, we will focus on spelling variants in the data set. target answers are given in blue. we see a timetable for a job fair with the days as columns and morning and afternoon activities (was?) and the corresponding locations (wo?) as rows. the upper left gap, e.g., prompts the test-taker to complete the entry presentation about the topic "a career in " with the target answer being public service. ## operationalizing acceptability decisions in the following, we will analyze criteria for the acceptability ratings of spelling variants which could be used by an automatic system. we base our analyses on the set of different spelling variant types. thereby, we always use the adjudicated labels as the gold standard. ## conclusion and future work we presented an analysis of the rating of spelling variants in a listening comprehension task from the testdaf test. we found that spelling variants are more challenging to score for human experts than other types of variants. furthermore, we ex-