Grammatical error detection in transcriptions of spoken E nglish (2020)

Abstract: We describe the collection of transcription corrections and grammatical error annotations for the CrowdED Corpus of spoken English monologues on business topics. The corpus recordings were crowdsourced from native speakers of English and learners of English with German as their first language. The new transcriptions and annotations are obtained from different crowdworkers: we analyse the 1108 new crowdworker submissions and propose that they can be used for automatic transcription post-editing and grammatical error correction for speech. To further explore the data we train grammatical error detection models with various configurations including pre-trained and contextual word representations as input, additional features and auxiliary objectives, and extra training data from written error-annotated corpora. We find that a model concatenating pre-trained and contextual word representations as input performs best, and that additional information does not lead to further performance gains.

## introduction we introduce a new resource for speech-centric natural language processing (speech nlp) -more than a thousand transcriptions and error annotations for 383 distinct recordings from the crowded corpus @xcite . crowded is a crowdsourced english corpus of short monologues on business topics, recorded by both native and non-native speakers. it was created in response to the lack of speech corpora freely available for research use, and the lack of appropriate native speaker reference corpora with which language learners' exam monologues can be compared. in this new project, crowdworkers were asked to first correct existing speech transcriptions and then to edit the resulting transcriptions to make them more fluent. these new annotations enable both post-editing of noisy speech transcriptions -such as might come from automatic speech recognisers -and also grammatical error correction for spoken english. there has been a marked increase in openly-available nlp resources in recent years, especially for english, but these have on the whole been sourced from written texts. public resources for speech nlp, on the other hand, are relatively scarce, even for english -though english is again by far the best served in this respect. obtaining linguistic data from large, distributed, online workers ('crowdsourcing') has become a well-established practice. with the contribution of these new annotations, we note that the crowded corpus now features a thousand recordings which have all been transcribed, and of which almost 40% now have improved transcriptions and error annotations thanks to this work. the entire corpus has been collated through crowdsourcing means. in this paper we describe the method for collecting the new data, analyse the annotations received, and report on some initial grammatical error detection experiments (ged). in the ged experiments we trial various configurations which have been successful in ged for written texts. we found that we can identify errors in the transcriptions fairly reliably using a publicly-available sequence labeller adapted to take contextual word representations as additional input, similar to previous work on ged in written corpora @xcite . all new transcriptions and annotations referred to in this paper are made available on the corpus website foot_0 . ## related work compared to the relatively abundant amounts of written text corpora, there are fewer speech corpora in which both transcriptions and audio files have been made available, even for the english language. some well-known exceptions include the british national corpus @xcite , the switchboard corpus @xcite , and ami corpus @xcite . however, it is now almost thirty years since the bnc and switchboard recordings were made, switchboard is only available for a fee, and all three contain spontaneous dialogues -either on general (bnc and switchboard) or business topics (ami) -whereas we have a research interest in assessing spoken monologues in language exams. in terms of non-native speaker english, one of the few freely available corpora is the nict japanese learner english corpus (jle; @xcite ). again, this corpus contains transcriptions of conversations about general topics, which are certainly of interest but not the type of data we need to work with relating to our other work on speech assessment @xcite : namely, short monologue responses by learners of english. one reason why speech corpora are rare is that their preparation is a labour-intensive process: human effort is required to obtain recordings, transcriptions and optionally annotations. crowdsourcing has been widely used for corpus annotation, whether word senses (lopez de lacalle and agirre, 2015), system evaluation @xcite , grammatical errors @xcite , as well as speech transcription @xcite . in the case of the crowded corpus, crowdsourcing was used for the whole data collection process, from recording to transcription and annotation, and was found to be cost-effective @xcite . even rarer are freely-available speech corpora with grammatical error annotations: indeed it is the first one we know of, because nict jle does offer error-annotated transcriptions but does not make the audio files available. grammatical error annotations can be used to train and evaluate machine learning models in the ged task, in which classifiers are required to identify grammatical errors in word sequences. most of the previous work on ged has tended to focus on written corpora @xcite @xcite @xcite . there has been less ged research with spoken corpora but that is more a reflection of the scarcity of appropriate data than a lack of interest. there has been work to assess grammatical accuracy by spoken call systems ('computer-assisted language learning') where such judgements might be rule-based @xcite or compared to a reference if the task is suitably constrained (de @xcite . there have been several shared tasks requiring binary acceptability judgements (correct/incorrect) at the utterance level @xcite @xcite . treating spoken ged as a word classification task is a more recent approach @xcite , one which applies the sequence labelling approach for written ged of @xcite and @xcite to speech transcriptions. it involves pre-training models on large written datasets -e.g. the fce corpus @xcite -and then fine-tuning to the target spoken corpus. ## the crowded corpus we use the crowded corpus for our experiments @xcite . the first release of the crowded corpus included one thousand short recordings in german and english from eighty people, transcribed into thirty-four thousand word tokens. all speech recordings and transcriptions were collected through crowdsourcing platforms (crowdee foot_1 and crowdflower foot_2 respectively), and it was demonstrated that crowdsourcing is a fast, feasible and cost-effective method of corpus collation. the speakers responded to 20 questions designed to prompt spontaneous monologues of up to one minute about imagined business scenarios. for instance, in the scenario of being asked to give advice about setting up a new shop, speakers were asked to respond to prompts such as the following: what do you think is the best location for a retail business? what are the most effective ways to advertise a new shop? why is it important to find good suppliers? there were 4 different scenarios: setting up a new shop, preparing for business visitors from a foreign country, running a taxi company, and the pros and cons of sponsoring sports events. each scenario had 5 associated questions. speakers who identified themselves as speakers of both languages were prompted to answer 10 questions in english and 10 in german; in the monolingual setting, speakers were asked to respond to all 20 questions in english. there was a quality control filter on the collected recordings, flagging and removing any with poor audio quality or in which the speaker failed to respond appropriately. approved recordings were passed to crowdflower workers for transcription, but only the english transcriptions were returned from crowd-flower with acceptable quality: the german recordings were not successfully transcribed, either because crowdworkers falsely claimed to know the language, or the job settings requiring german competence did not properly filter the worker pool. therefore we focus on the english section of the corpus in the remainder of the paper, and intend to revisit the german data in future work. in what follows we describe the subset of recordings which have been error annotated since the initial release of the corpus, and report on our experiments to automatically detect grammatical errors in the transcriptions. ## grammatical error detection automatic grammatical error detection in natural language is a well-developed area of research which tends to involve one of several established datasets: for example, the fce corpus @xcite @xcite , jfleg @xcite , and the write & improve corpus (w&i; @xcite ) among others. these corpora all contain written essays, whether written for language exams (fce, jfleg) or for practice and learning (conll-2014, w&i). thus techniques for ged are advanced and tuned to the written domain. we evaluate how well these methods transfer to spoken language. ged is usually treated as a separate task to grammatical error correction (gec) -which involves proposing edits to the original text. gec could be an area for future work with the crowded corpus, but at first we wish to explore ged for this speech dataset, anticipating that performance will be quite different to ged on written texts in which error types more often relate to word forms, punctuation and spelling ( the state-of-the-art approach to ged involves sequence labelling word tokens as correct or incorrect with a bi-directional lstm: the original model @xcite has evolved to include forwards and backwards language modelling objectives @xcite and contextual word representations concatenated to pre-trained word-level representations @xcite . in addition, multi-task learning has proven effective for ged, with auxiliary predictions of error types, part-of-speech tags and grammatical relations aiding model performance @xcite . we take these insights forward to ged in the crowded corpus, running a series of experiments with modifications to the publicly available sequence labeller released with @xcite 5 . note that f 0.5 has been the standard evaluation metric for ged since @xcite , and it weights precision twice as much as recall. ## conclusion in this paper we have presented a new resource for speech nlp: 1108 separate corrected transcriptions and grammatical error annotations for 383 distinct english recordings from the crowded corpus. these are available for research use foot_5 and complement the existing crowded audio files and original transcriptions available in the same place. these data enable further research into automatic post-editing of speech transcriptions for readability (inserting full-stops and orthographic correction), which we have not explored here but could do in future work. in addition the error annotations allow experiments in grammatical error detection and correction (ged and gec). we undertook ged experiments in this work, using methods shown to be highly effective on written corpora. we find that a combination of contextual and pre-trained word representations as inputs to a bi-directional lstm lead to good performance in sequence labelling word tokens in speech transcriptions as correct or incorrect. performance is still some way off that for written ged, which perhaps may be accounted for by the small size of the dataset compared to written ones, the fact that the word representations are trained on written rather than spoken data, and the fairly different composition of error types found in crowded compared to equivalent written corpora. these factors, along with the possible need for extra pre-processing of the transcriptions to handle characteristic features of spoken language such as disfluencies, ellipsis and sections of unclear speech, may be explored in future work. another future improvement will be to augment the crowded corpus with new data: new tasks, new languages, new recordings and annotation. note that there are hundreds of crowded recordings which do not have the transcription updates and error annotation described here, so if funding allows there is unrealised potential in the existing data, besides adding to it with new data. further insight may come from adapting the error typology to the spoken domain, in particular subdividing the majority 'other' error type into new types specific to spoken language, and then learning to better detect and correct these. we will then be able to improve upon the best glove 300 +bert base model with techniques for ged which are tailored to the kinds of errors found in spoken language.