Gated Convolutional Bidirectional Attention-based Model for Off-topic Spoken Response Detection (2020)

Abstract: Off-topic spoken response detection, the task aiming at predicting whether a response is off-topic for the corresponding prompt, is important for an automated speaking assessment system. In many real-world educational applications, off-topic spoken response detectors are required to achieve high recall for off-topic responses not only on seen prompts but also on prompts that are unseen during training. In this paper, we propose a novel approach for off-topic spoken response detection with high off-topic recall on both seen and unseen prompts. We introduce a new model, Gated Convolutional Bidirectional Attention-based Model (GCBiA), which applies bi-attention mechanism and convolutions to extract topic words of prompts and key-phrases of responses, and introduces gated unit and residual connections between major layers to better represent the relevance of responses and prompts. Moreover, a new negative sampling method is proposed to augment training data. Experiment results demonstrate that our novel approach can achieve significant improvements in detecting off-topic responses with extremely high on-topic recall, for both seen and unseen prompts.

## introduction off-topic spoken response detection is a crucial task in an automated assessment system. the task is to predict whether the response is off-topic for the corresponding question prompt. off-topic examples in human-rated data is often too sparse to train an automated scoring system to reject off-topic responses. consequently, automated scoring systems tend to be more vulnerable than human raters to scoring inaccurately due to off-topic responses @xcite . to ensure the validity of speaking assessment scores, it is necessary to have a mechanism to flag off-topic responses before scores are reported @xcite . in our educational application, we use the automated speaking assessment system to help l2 learners prepare for the ielts speaking test. we do see a higher rate of off-topic responses in freemium features as some users just play with the system. in such a scenario, accurate off-topic detection is extremely important for building trust and converting trial users to paid customers. prompt: what kind of flowers do you like? on-topic: i like iris and it has different meaning of it a wide is the white and um and the size of a as a ride is means the ride means love but i can not speak. off-topic: sometimes i would like to invite my friends to my home and we can play the chinese chess dishes this is my favorite games at what i was child. initially, many researchers used vector space model (vsm) @xcite @xcite to assess the semantic similarity between responses and prompts. in recent years, with the blooming of deep neural networks (dnn) in natural language processing (nlp), many dnn-based approaches were applied to detect off-topic responses. @xcite used the topic adapted recurrent neural network language model (rnn-lm) to rank the topic-conditional probabilities of a response sentence. a limitation of this approach is that the model can not detect off-topic responses for new question prompt which was not seen in training data (unseen prompt). later, off-topic response detection was considered as a binary classification task using end-to-end dnn models. @xcite proposed the first end-to-end dnn method, attention-based rnn (att-rnn) model, on off-topic response detection task. they used a bi-lstm embedding of the prompt combined with an attention mechanism to attend over the response to model the relevance. cnns may perform better than rnns in some nlp tasks which require key-phrase recognition as in some sentiment detection and question-answer matching issues @xcite . @xcite proposed a siamese cnn to learn semantic differences between on-topic response-questions and off-topic response-questions. @xcite proposed an approach based on similarity grids and deep cnn. however, the cold-start problem of off-topic response detection has not been handled well by the aforementioned approaches. it is not until enough training data of unseen prompts are accumulated that good performance could be achieved. besides, these methods draw little attention to the vital ontopic false-alarm problem for a production system. i.e., extremely high recall of on-topic responses is also required to make real-user-facing systems applicable. in this paper, to address the issues mentioned above, a novel approach named gated convolutional bidirectional attention-based model (gcbia) and a negative sampling method to augment training data are proposed. the key motivation behind our model gcbia is as follows: convolution structure captures the key information, like salient n-gram features @xcite of the prompt and the response, while the bi-attention mechanism provides complementary interaction information between prompts and responses. following r-net @xcite in machine comprehension, we add the gated unit as a relevance layer to filter out the important part of a response regarding the prompt. these modules contribute to obtaining better semantic matching representation between prompts and responses, which is beneficial for both seen and unseen prompts. additionally, we add residual connections @xcite in our model to keep the original information of each major layer. to alleviate the cold-start problem on unseen prompts, a new negative sampling data augmentation method is considered. we compare our approach with att-rnn model and g-att-rnn (our strong baseline model based on att-rnn). experiment results show that gcbia outperforms these methods both on seen and un-seen prompts benchmark conditioned on extremely high on-topic response recall (0.999). moreover, the model trained with negative sampling augmented data achieves 88.2 average off-topic recall on seen prompts and 69.1 average off-topic recall on unseen prompts, respectively. in summary, the contribution of this paper is as follows: ## conclusion in this paper, we conducted a series of work around the task of off-topic response detection. first of all, a model framework of five major layers was proposed, within which bi-attention mechanism and convolutions were used to well capture the topic words of prompts and key-phrase of responses, and gated unit as relevance layer was applied to better obtaining semantic matching representation, as well as residual connections with each major layer. moreover, the visualization analysis of the off-topic model was given to study the essence of the model. finally, a novel negative sampling augmentation method was introduced to augment off-topic training data. we verified the effectiveness of our approach and achieved significant improvements on both seen and unseen test data.