Investigating the effect of auxiliary objectives for the automated grading of learner E nglish speech transcriptions (2020)

Abstract: We address the task of automatically grading the language proficiency of spontaneous speech based on textual features from automatic speech recognition transcripts. Motivated by recent advances in multi-task learning, we develop neural networks trained in a multi-task fashion that learn to predict the proficiency level of non-native English speakers by taking advantage of inductive transfer between the main task (grading) and auxiliary prediction tasks: morpho-syntactic labeling, language modeling, and native language identification (L1). We encode the transcriptions with both bi-directional recurrent neural networks and with bi-directional representations from transformers, compare against a feature-rich baseline, and analyse performance at different proficiency levels and with transcriptions of varying error rates. Our best performance comes from a transformer encoder with L1 prediction as an auxiliary task. We discuss areas for improvement and potential applications for text-only speech scoring.

## introduction the growing demand for the ability to communicate in english means that both academic and commercial efforts are increasing to provide automated tutoring and assessment systems. these educational systems address the increasing need for online resources to help students learn and to map users to the validated proficiency scales which play a critical role in securing education and work opportunities @xcite . language learning applications delivered through smart speakers such as amazon alexa and google home are a novel form of educational technology. these offer obvious benefits to users in terms of immediacy, interaction and * currently at google u.k. to assess spontaneous speech, automated grading systems tend to use a combination of features extracted from the audio recording and the transcription resulting from asr. for instance, speechrater tm by the educational testing service uses text-based features based on frequency counts and lexical unigrams -among others, the number of word tokens per second, the length of interpausal units in words, the vocabulary size normalized by recording duration -and score predictions are made using linear regression @xcite @xcite . however, without the audio recordings, proficiency scoring must be performed based on the text alone. thus robust methods for text-only speech scoring need to be developed to ensure the reliability and validity of educational applications in scenarios such as smart speakers. relatively few automated speech graders use neural approaches that incorporate text-based features from transcripts. @xcite used a linear regression model on the concatenated high-level representation outputs of two separate rnns for sequential audio and text inputs; @xcite use a bi-directional rnn which uses word embeddings concatenated with an encoding of the given prompt and an attention mechanism over all tokens to predict grades. in this work, we address the task of automatically grading the language proficiency of spontaneous speech based on asr transcriptions only, and seek to investigate the extent to which current state-ofthe-art neural approaches to language assessment are effective for the task at hand. specifically, we make the following contributions: ## related work automated grading of student responses to exam questions until recently tended to adopt featurebased approaches to score prediction, for instance using distinctive word or part-of-speech n-grams @xcite @xcite , as well as grammatical errors and phrase-structure rules @xcite . more recently, word and character embeddings have served as input to deep neural network models, with a final regression layer predicting the score @xcite @xcite . the advantage of the latter approach is the relative ease of data pre-processing since text representations are learned through distributional methods rather than hand-crafted features. the field of nlp has seen advances recently thanks to a shift from fixed word embeddings to contextualized representations such as elmo @xcite and those which can be obtained from large transformer models such as bert @xcite . similarly in text scoring, some have incorporated contextualized word embeddings to improve performance @xcite . we now apply such approaches to the grading of spoken transcriptions in a scenario where the audio, or information derived from it, is not available. in other words the task is analogous to essay scoring except for the presence of characteristic speech features such as false starts, repetitions and filled pauses @xcite . this poses a particular challenge as most models used in data pre-processing and representation learning have been trained on written not spoken texts @xcite . furthermore, most existing approaches to speech grading do have access to audio features, and indeed extract a large number of prosodic or duration-based features @xcite @xcite . prosodic and phonological features extracted from the audio and asr model are undoubtedly useful for human assessment of speech proficiency and for providing feedback. on the other hand, previous work suggests that models trained solely on asr text-based features are competitive with those using only acoustic features or a combination of the two @xcite . their interpretation of these results was that the transcription offers some proxy information for prosodic and phonological performance -for instance the presence of hesitation and silence markers, the number of word tokens in the transcription, and the transcription errors which might arise from mispronunciations. we instead allow our models to learn from auxiliary (morpho-syntactic and other) tasks: multi-task learning has been shown to help in automated essay scoring @xcite and grammatical error detection of learner english essays @xcite , whilst information about a learner's native language has been shown to help in error detection for english and the grading of norwegian essays @xcite . furthermore, multi-task learning objectives can allow the model to learn more general features of language and composition, and a much richer set of representations @xcite , without relying on the availability of any external linguistic tools or annotations at inference time. ## data we train our models using spoken responses collected from candidates taking cambridge assessment's bulats examination 2 . the spoken section of the bulats exam tests candidates' proficiency in business english through monologue responses to a series of prompts. the candidate may speak for up to one minute in each response and we include only the prompts which invite spontaneous responses (we exclude the prompts which require reading aloud of given sentences, and prompts asking for personal information about the candidates). there are seven such prompts in each exam. fortysix unique versions of the bulats exam are represented in the training and test sets, meaning that there are 322 unique prompts (7 * 46). each response has been assigned a score between 0 and 6 by expert human examiners, with scoring increments of .5 available and with each whole integer mapping to a proficiency level on the common european framework of reference for languages (cefr): a fail (score of 0), beginner (scores of 1, 2: a1 and a2); intermediate (scores 3, 4: b1 and b2); advanced (scores 5, 6: c1 and c2). examiners are required to consider five attributes of each candidate's speaking proficiency: pronun-2 https://www.cambridgeenglish.org/ exams-and-tests/bulats ; now discontinued and replaced by the linguaskill business exam. ciation, hesitation, language resource, coherence and task achievement. in the transcription-only scenario, we cannot assess the first component, have only a proxy for the second in terms of filled pause occurrence ('umm', 'err', etc), but still have access to the other three components through the asr transcriptions. our data comes from 1213 exam candidates with six first languages in approximately uniform distribution: arabic, dutch, french, polish, thai and vietnamese. the distribution of candidates over proficiency levels is approximately normal, with a peak over the intermediate scores (figure 1 ). the train/validation/test split across candidates is roughly 55 : 25 : 20 as detailed by each candidate's recordings are transcribed by a teacher-student asr system with a latticefree maximum-mutual-information acoustic model @xcite . the teacher-student training procedure uses kullback-leibler divergence between the word sequence posteriors from the student model and a teacher ensemble as the loss function @xcite . the result is a computationally efficient asr system, as the student is able to decode in a single run to a similar level of performance as an ensemble decoder requiring multiple runs @xcite . there is more information about the asr system in @xcite . we also evaluate performance on manual transcriptions of the test set, in order to assess the impact of asr errors on our models. a native speaker of english was asked to transcribe the recordings as faithfully as possible to include hesitations, disfluencies and partial words. a subset of 230 recordings were transcribed by a second native speaker: inter-annotator agreement on this subset is high (cohen's κ = .898). compared against the annotator's manual transcriptions, the word error rate of the asr is 19.5% overall, but with variance from 32% for speakers with a score of 1, to 15% for speakers with scores 5 and 6. to be able to predict morpho-syntactic labels, we parse the data using udpipe (wijffels, 2018), trained on the universal dependencies (ud) english web treebank 2.4 made up of 255k words and 16.6k sentences from weblogs, newsgroups, emails, reviews, and yahoo! answers @xcite . we use udpipe to automatically generate penn treebank part of speech (pos) tags @xcite and uds @xcite for our training data. filled pauses were excluded before parsing, so that they would not affect the parse of other words in the transcription, but were then re-inserted with null parse values, in case they serve as a useful signal to the language proficiency models. transcriptions were parsed as whole units: we did not attempt to delimit speech-units. for the most part this results in fairly lengthy, but not impractically long, word sequences. the asr transcriptions are on average 44 word tokens long (σ = 33.0), with a minimum of 2 tokens, a maximum of 179, and 50% of the texts being between 23 and 54 tokens long. as seen in figure 2 , the distribution of transcription length differs according to proficiency level: the failing grades tend to be very short responses, the beginner level responses are a little longer, and the bulk of intermediate responses are between 25 and 50 tokens long (recordings are between 20 and 60 seconds duration). ## model architecture the speech grader foot_1 takes a sequence of token embeddings [x 1 , . . . , x n ] as input and predicts a proficiency level score. tokens are first converted to vector representations x t , and then passed through an encoder. we trial two different encoders: a bidirectional lstm @xcite and bert @xcite . the encoding is passed through the prediction head, a series of linear layers and activation functions, where the final activation function is bound to the scoring scale (0-6). the model uses mean squared error (mse) as the loss function e score for the main task. lstm encoder the bi-directional lstm encoder uses the word-level tokenization provided by ud-pipe. for each token, the hidden states of the two lstms are concatenated, creating a context-aware hidden state h the hidden layers that are formed at the final timesteps of the bidirectional lstm (h 1 , h n ) are concatenated for the scoring prediction head. bert encoder the bert encoder uses a pretrained model checkpoint and tokenizer, specifically bert-base-uncased, provided by the hug-gingface transformer library @xcite . bert's tokenizer uses the wordpiece model @xcite , resulting in a much larger vocabulary than the lstm encoder. bert embeddings are extracted from a transformer trained with a masked lm objective: a percentage of input tokens are masked and then the network learns to predict the masked tokens. bert is also trained with a second objective: given two input sequences, it predicts whether one sequence directly follows another. a sequence level embedding is produced by pooling the hidden states of the special first token, @xcite , resulting in a 768 dimensional embedding. ## results we ran a series of experiments to analyze the impact that data pre-processing and encoder design have on the performance of our automated speech grader. all results presented are computed over 10 repetitions, include filled pause information and use an asr system with a wer of 19.5% (see section 3) unless otherwise stated. ## conclusion we presented an effective approach to grading spontaneous speech based on asr transcriptions only, without direct access to the audio recording or features derived from it. our best performing model involves a bert encoder with first language prediction as an auxiliary task. we showed that this model improves on alternative lstm-based models, and over a feature-rich baseline, by better predicting scores at the edges of the proficiency scale, while also offering (smaller) gains at the central points on the scale. its error is on average less than 1, and 76% of its predictions are within 1 grade of the examiners' gold scores. we recognise that without the audio signal, some information is lost that would be useful for speech assessment -namely prosodic and phonemic features -but that assessment on transcriptions alone has a use case in educational technology for home assistants. furthermore such applications may become increasingly relevant as organisations reduce the types of data they collect from the end user due to privacy concerns. further work should be undertaken in terms of scoring validity and the robustness of such an approach, before such models are applied to any 'high stakes' (i.e. exam) scenario, as opposed to the kind of at-home practice apps we have discussed in this paper. we also showed that the models improve as they are trained on increasingly accurate asr transcriptions, though performance deteriorates when they are evaluated on manual transcriptions. we surmise that this is because of stylistic differences in the machine and human transcriptions, and that adaptation of the models to manual transcriptions will help mitigate the drop in performance. additional experiments indicated that the removal of filled pauses from the transcriptions was beneficial to the scoring models, and that scoring performance is best for the middle grades of the scoring range. further research is needed to improve machine assessment at the upper and lower ends of the scoring scale, although these are the scores for which the least training data exists. therefore future work could include different sampling methods, generation of synthetic data, or training objectives which reward models which are less conservatively drawn to the middle of the scoring scale. finally, we acknowledge that speaking proficiency in a second language is a multi-faceted construct made up of more than the features which can be drawn from transcriptions @xcite . for instance, the speaker's prosody, pronunciations and disfluencies are also contributing factors. however, given the text-only constraints faced by third-party application developers for home assistants, the proficiency assessment models we present in this work allow for progress in providing low-stakes assessment and continuous practice for language learners, with the caveat that fuller speaking skills should be taught and assessed with the complete construct in mind.