Complementary Systems for Off-Topic Spoken Response Detection (2020)

Abstract: Increased demand to learn English for business and education has led to growing interest in automatic spoken language assessment and teaching systems. With this shift to automated approaches it is important that systems reliably assess all aspects of a candidate’s responses. This paper examines one form of spoken language assessment; whether the response from the candidate is relevant to the prompt provided. This will be referred to as off-topic spoken response detection. Two forms of previously proposed approaches are examined in this work: the hierarchical attention-based topic model (HATM); and the similarity grid model (SGM). The work focuses on the scenario when the prompt, and associated responses, have not been seen in the training data, enabling the system to be applied to new test scripts without the need to collect data or retrain the model. To improve the performance of the systems for unseen prompts, data augmentation based on easy data augmentation (EDA) and translation based approaches are applied. Additionally for the HATM, a form of prompt dropout is described. The systems were evaluated on both seen and unseen prompts from Linguaskill Business and General English tests. For unseen data the performance of the HATM was improved using data augmentation, in contrast to the SGM where no gains were obtained. The two approaches were found to be complementary to one another, yielding a combined F0.5 score of 0.814 for off-topic response detection where the prompts have not been seen in training.

## introduction spoken language assessment of english is on the rise as english is the chosen language of discourse for many situations. businesses and academic institutes demand rigorous assessment methods to ensure prospective employees and students exceed a baseline standard for english proficiency so they can succeed and contribute in their new environment. standardised assessments such as ielts @xcite , pearson test of english academic @xcite and toefl @xcite include "free speaking" tasks where the candidate speaks spontaneously in response to a prompted question to ensure their speaking skills are fully assessed. a candidate might attempt to achieve a higher grade by speaking a pre-prepared response, irrelevant to the prompt. for scoring validity it is important that measures are taken to detect any off-topic responses so they do not influence the final grade. this is particularly true for automatic assessment systems which are increasingly being deployed to cope with the growing demand for examinations and may see increased cheating if candidates are aware that a computerised system is responsible for grading them (e.g. @xcite ). these systems are more susceptible to inaccurate scoring due to emphasis given to criteria such as fluency, pronunciation and language use, over topic relevance @xcite . automatic off-topic spoken response detection systems based on attention @xcite and similarity grid @xcite models have shown good performance for prompts seen in training. for operational reasons it would be cost and time effective to be able to use the same systems on responses to new prompts, unseen in training i.e. removing the need to collect new data and retrain models prior to deployment. @xcite has had some success with handling unseen prompts. this is still a challenging research problem, however, with significant degradation observed on even the best performing hierarchical attention-based topic model (hatm) @xcite , and with no assessment to date of @xcite 's similarity grid model (sgm) approach. this paper therefore focuses on investigating how to improve performance on unseen prompts for these models. it presents extensions to the hatm and sgm with the goal of learning robust representations of seen prompts for effective generalisation to unseen prompts. the resulting systems are shown to have complementary detection characteristics, yielding improved off-topic response detection when combined. the remainder of the paper is structured as follows: section 2 presents related work; section 3 details the components of the hatm and sgm, proposing modifications to each (universal regularisation and multi-channel cosine-similarity, respectively) to make them more robust; data augmentation is proposed in section 4 to overcome limited training data; the experimental set-up and structure of the data is described in section 5; section 6 presents the experimental results and analysis; conclusions are given in section 7. ## related work initial off-topic spoken response detection systems were based on vector space models, measuring the relevance between the spoken response and test prompts inspired by systems for written essays @xcite . @xcite 's approach using speech confidence derived features is unsuited to general free speaking tasks, whereas yoon and xie (2014) required hundreds of example responses for each prompt from highly proficient speakers. using word embeddings and deep neural networks to measure sentence similarity has since become dominant. @xcite generated sentence-level relevance scores for written essays by using various similarity metrics based on word embeddings. for spoken responses, @xcite proposed a topic-adapted recurrent neural network language model (rnnlm) to rank prompt-response pairs. this handles sequential information but cannot handle prompts unseen in training so @xcite introduced an attention-based topic model (atm) which can. the deep learning atm architecture uses an attention mechanism to attend over response word embeddings with the prompt sentence embedding as the key. a hierarchical variant of the atm (hatm) is proposed in @xcite where an additional (prompt) attention mechanism is incorporated to attend over sentence embeddings of a set of seen prompts with the test prompt embedding acting as the key. hence, an unseen prompt is able to lock onto the vector representation of a combination of the seen prompts. the hatm assumes the set of seen prompts are sufficiently diverse to capture aspects of all possible unseen prompts. @xcite observed lower performance on unseen prompts due to a lack of diversity. a radically different approach was proposed by @xcite based on initial work in @xcite and @xcite . very deep cnns are employed for automatic detection of offtopic spoken responses in which a prompt-response pair is represented as a similarity grid that can be interpreted as an image (this model will be referred to here as sgm). similarity measurements are made based on word embeddings (or other distance metrics) of the prompt/response content words. the training data used in @xcite was from exams with generally short prompts which resulted in a limited number of content words to use. it was not assessed on unseen prompts. ## off-topic spoken response detection this paper builds upon the hierarchical attentionbased topic model (hatm) @xcite (section 3.1) and the similarity grid in cnns model (sgm) @xcite (section 3.2) for detection of off-topic spoken responses to prompted questions. the candidate can answer freely so automatic speech recognition (asr) is needed to determine the words in their response in each case. both systems assign a probability that the asr obtained response, ẑ, is relevant to the prompt, x. the performance of the hatm and sgm models on responses to prompts seen in training is high, with f 0.5 scores above 0.9. a key issue, however, in the practical deployment of off-topic response detection systems is handling responses to prompts that were unseen in training so that new examination questions can be asked without requiring example responses to be collected and the detection system retrained. although @xcite improved off-topic detection on unseen prompts compared to earlier work, the performance is still quite far below that of seen, and the sgm has not been evaluated in the unseen prompt scenario. this section presents two approaches to potentially improve performance on unseen prompts: universal regularisation in an attention mechanism as a structural modification to the hatm in order to en-courage generalisation; multi-channel sgm based on cosine distance (msgm). data augmentation strategies to increase the number of prompts available for training are presented in section 4. the hierarchical attention-based topic model (hatm) @xcite is depicted in figure 1 . the system uses an encoding of the prompt word sequence, h p , as the key for an attention mechanism over an embedding of the response to yield a fixed length vector, h, that is used to predict the probability, y, that the response was relevant to the prompt. to improve the robustness of the estimate of the prompt embedding an additional attention mechanism is run over all n t embeddings of the prompts seen in training, h(1) p , . . . , h(n t ) p . this attention mechanism uses an embedding of the test prompt, ĥp , as the key to yield h p . this additional attention mechanism over the prompts was found to improve the performance of the system when the prompt had not been seen in the training data @xcite . the same network configuration as that used in @xcite was implemented in this work: ## data augmentation in general, the performance of off-topic response systems is limited by insufficient unique prompts being available for training. data augmentation, where the training data is modified in some way to create new examples, is regularly applied on low resource tasks in areas such as speech recognition (e.g. @xcite ) and computer vision (e.g. @xcite ) and has had some success in nlp (e.g. @xcite @xcite ). this motivates investigating if augmentation of the training prompts such that multiple versions of each prompt are generated can help improve robustness foot_4 . prompt augmentation will permit the model to explore the region around each unique prompt rather than being restricted to a discrete point in the highdimensional prompt-space. both structured and unstructured data augmentation techniques are considered here. augmentation of prompts is performed on-the-fly during training. note, the hierarchy of seen prompts of the hatm in the prompt attention mechanism (figure 1 ) does not include the additional augmented prompts because the expectation is that augmented prompts will not dramatically differ from the original unique prompts. easy data augmentation (eda) techniques were trialled by @xcite . they proposed that different variants of any textual data can be generated using a combination of synonym replacement, random insertion, random swap and random deletion of words. a single hyper-parameter, α, controls the fraction of words that are modified in the original text. using the default value of α = 0.1, prompts are augmented using the above techniques to replace, insert, swap or delete 10% of the words randomly in the original prompt. this structured augmentation approach should enable the model to learn a more robust representation of each unique prompt. back-translation is employed as an unstructured method to augment the amount of available training data. a machine translation model is employed to translate a given training prompt into a foreign language by taking the maximum likelihood output. then a reverse machine translation model takes the prompt in the foreign language and translates it back into english. the expectation is that the original and final pair of english prompts will be very similar in meaning but will have a different ordering and choice of specific words. therefore, the back-translated prompt can be treated as a new prompt which can be paired with the original prompt's response to generate a new promptresponse pair. the use of several different languages permits the creation of several variants of the same prompt. translation can be achieved using standard machine translation packages. ## data and experimental set-up the hatm and sgm models and the proposed extensions were assessed on their ability to detect off-topic responses to prompts in free speaking tests where the candidates can talk for up to one minute in answering the question. ## conclusion this paper addresses the issue of off-topic detection in the context of unconstrained spoken responses to prompts. in particular, the problem of robustness to prompts unseen in training is considered. the hierarchical attention-based topic model (hatm) @xcite and similarity grid model @xcite are compared and extended. universal regularisation and data augmentation, from structured techniques and machine translation, increased the hatm f 0.5 by 19% relative to 0.782 on the unseen evaluation set. this contrasts with a three channel sgm (msgm) based on cosine distances between prompt and response embeddings which yielded f 0.5 of 0.800. the extended hatm and msgm are shown to have very different precision-recall characteristics on unseen prompts, with the hatm having a very high precision at low recall but with a fairly sharp drop-off whilst the sgm's precision does not reach quite the same level but degrades at a much more gradual rate. the best individual systems are found to be complementary, with system combination boosting off-topic response detection on both unseen and seen prompts, achieving the best performance of f 0.5 of 0.814 on unseen and 0.935 on seen prompts. this combined system closely follows, and slightly enhances, the envelope of the best precision-recall path across the two individual systems. finally the distance between a test prompt and the closest training is shown to predict the system performance, indicating which prompts may require additional training data to be collected.