On Assessing and Developing Spoken ’Grammatical Error Correction’ Systems (2022)

Abstract: Spoken ‘grammatical error correction’ (SGEC) is an important process to provide feedback for second language learning. Due to a lack of end-to-end training data, SGEC is often implemented as a cascaded, modular system, consisting of speech recognition, disfluency removal, and grammatical error correction (GEC). This cascaded structure enables efficient use of training data for each module. It is, however, difficult to compare and evaluate the performance of individual modules as preceeding modules may introduce errors. For example the GEC module input depends on the output of non-native speech recognition and disfluency detection, both challenging tasks for learner data. This paper focuses on the assessment and development of SGEC systems. We first discuss metrics for evaluating SGEC, both individual modules and the overall system. The system-level metrics enable tuning for optimal system performance. A known issue in cascaded systems is error propagation between modules. To mitigate this problem semi-supervised approaches and self-distillation are investigated. Lastly, when SGEC system gets deployed it is important to give accurate feedback to users. Thus, we apply filtering to remove edits with low-confidence, aiming to improve overall feedback precision. The performance metrics are examined on a Linguaskill multi-level data set, which includes the original non-native speech, manual transcriptions and reference grammatical error corrections, to enable system analysis and development.

## introduction grammatical construction is one of the key elements in second language acquisition, and textbased grammatical error correction (gec) has been widely studied over the past decade @xcite @xcite . with speaking skills playing a big part in language learning, it has become increasingly important to analyse spoken grammars. previous works have investigated grammatical error detection (ged) on spoken language transcriptions @xcite , and tighter integration of disfluency removal and grammar correction on spontaneous learner speech @xcite . this paper focuses on the spoken grammatical error correction (sgec) task. there are several challenges facing sgec: running automatic speech recognition (asr) on learner english is harder than native speech due to potential pronunciation and grammatical errors; spoken language often comes with disfluent speech events such as repetitions and false starts, which are disruptive to downstream tasks; there is very little endto-end speech to correction data that can be used for training. in this paper, sgec adopts a cascaded structure: an asr module produces transcriptions; a disfluency detection (dd) @xcite module recovers a fluent text flow; and a conventional machine translation style gec @xcite module produces error corrections. several metrics have been developed to assess text-based gec. gleu @xcite score adopts bleu @xcite based n-gram precision over the reference. it rewards wordlevel corrections, as well as correctly preserved source text. maxmatch m 2 @xcite captures phrase-level edits, and calculates f 0.5 scores accordingly. it assesses performance in terms of edits, which suits well with feedback oriented applications. for spoken gec assessment, however, it is not straight forward to apply those standard metrics. a common problem facing cascaded style spoken language applications is that it is difficult to compare across models when upstream modules (e.g. asr) are different. for example, input text to gec module varies when upstream asr and dd models are changed. if standard gleu and m 2 f 0.5 are to be applied, these metrics mean differently every time asr transcriptions change, and thus results are incomparable across systems. not only for cascaded systems, evaluation metrics is not clearly defined for endto-end trained spoken systems. it is difficult to migrate text-based metrics to spoken tasks, since end-to-end models do not provide any intermediate variables for assessment. this paper first discusses metrics to assess cascaded sgec systems. when evaluating individual modules, standard metrics can be used. however, these metrics are not suitable for system-level assessment, since they sometimes take into account module inputs. when downstream module inputs change with its upstream modules, results can become incomparable across systems. to make systems comparable, we use edit distance based metrics instead and focus on the output quality. a common issue in cascaded systems is error propagation, since individual modules are trained separately. to mitigate this issue, semi-supervised fine-tuning is conducted. it aims to tune dd and gec modules for optimal system performance with non-native asr transcriptions. self-distillation is also investigated, which learns from a rich distribution of semi-supervised references. both fine-tuning experiments are conducted on learner english without readily available annotations. for system development purposes, we focus on optimising output quality; and for assisting language learning, we shift the emphasis to give high quality feedback. we first remove ambiguous corrections, and further filter out low-confidence edits to improve feedback precision as well as the overall quality. ## evaluation metrics cascaded spoken grammatical error correction (sgec) consists of three modules, namely speech recognition (asr), disfluency detection (dd) and grammatical error correction (gec). it converts disfluent, grammatically incorrect audio sequences into fluent, grammatically correct text. variables are notated as such: x for input audio, w for speech transcriptions, t for disfluency tags, w f for transcriptions with disfluencies removed, and y for grammatically correct outputs. n.b.: bold letters are used to represent sequences, with subscripts omitted, e.g. x short for x 1:t . when evaluating individual modules, standard metrics can be used. word error rate (wer) is used for asr to compute word-level edit distance. dd is modeled as a sequence tagging task, and f 1 score is used to indicate the mean of precision and recall (use hat, e.g. â, to indicate hypothesises): for gec module, a standard evaluation is to compare reference and hypothesised edits, and use f 0.5 score to reflect a weighted precision and recall: for cascaded systems, it is also important to look at system-level evaluation, which assesses a combination of modules. when evaluating asr and dd combined, the standard dd metric f 1 score no longer apply. the reference tags t have a one-toone correspondence with input word tokens w, and we need a different set of reference t for different asr transcriptions. it is therefore not feasible to compare across systems that have different asr transcriptions if f 1 is used. thus, we use wer instead, to directly analyse the output quality from disfluency removal: when evaluating asr, dd and gec modules combined i.e. the sgec system, standard gec metric m 2 f 0.5 cannot be used, since it does not allow comparison across systems. it requires input sequences w f to be given for edit extraction, and changes in upstream asr and dd modules will lead to a different set of reference edits e. therefore the focus is laid on the quality of outputs. we adopt sentence error rate (ser) to analyse sentencelevel matches between references and hypotheses. to achieve greater granularity, we also adopt translation edit rate (ter) @xcite to assess word-level distance from references. individual module evaluation s asr , s dd and s gec helps develop each module separately. system-level metrics s asr+dd and s asr+dd+gec both emphasise output quality, which enables comparison across systems even when upstream modules change. they also help guide further tuning and development of the sgec system as a whole. ## module error mitigation each module in the cascaded sgec system is trained individually. dd is trained on a native spoken corpus, and gec is trained on written text that is processed to be like speech transcripts (details in 5). individual training allows efficient use of data on one hand, yet on the other hand, it suffers from error propagation due to mismatches between training and evaluation. for example, dd and gec modules have not seen any asr transcriptions during training, and thus any asr error at evaluation time would potentially disrupt their performance. ideally, fine-tuning on a non-native spoken corpus would most effectively mitigate error propagation, but similar to many other speech to text tasks, there is no readily available data for training. therefore in this section, we adopt semi-supervised approaches to fine-tune the sgec system on a spoken learner corpus without manual annotations. here we use the asr training corpus, which is comparatively abundant and less costly to obtain, compared to end-to-end sgec annotation. it consists of audio sequences x and manual transcriptions w. fine-tuning aims to train the system to be more robust against error propagation. the asr training set provides non-native audio sequences and their corresponding manual transcriptions, yet lacking references for grammatical error corrections. we generate pseudo references by feeding manual transcriptions through the baseline system, and hypotheses by feeding through audio sequences. the pseudo references are not impacted by asr errors, and therefore minimising distance between references and hypotheses should help mitigate asr error propagation. figure 1 shows the fine-tuning pipeline. when fine-tuning the dd module, references and hypotheses are generated as such: ## feedback and confidence filtering section 2, 3 mainly focus on evaluating and optimising output quality. another important aspect for language learning applications is feedback to learners, since feedback quality directly impacts learner's progression in language learning. this section first describes how feedback is extracted and assessed for sgec systems, then introduces confidence-based filtering that aims to improve feedback precision. for gec tasks, feedback usually suggests where the error is and how to correct it. in written gec, feedback edits are extracted using eqn. 5 by comparing input and output sequences. its quality is analysed using f 0.5 (eqn. 6) by comparing hypotheses against reference edits. this is not applicable to spoken gec, since reference edits change with upstream asr transcriptions, are consequently f 0.5 scores are not comparable across systems. here we modify the definition of reference and hypothesised edits as such: f 0.5 can be calculated as before. reference edits e are generated using manual fluent transcripts w f as source sequences. with reference e defined independent from asr or dd module, feedback f 0.5 can be compared across systems. hypothesised edits ê use hypothesised fluent transcriptions ŵf as source sequences. therefore ê account for errors from all three modules, and reflects the true feedback given to users when the system is deployed. note that such mismatched source sequences in e and ê put extra penalty on f 0.5 . to given an example: when system output ŷ matches with reference y i.e. the system output is perfectly correct, differences in w f and ŵf will still result in differences in ê from e, leading to degraded f 0.5 score. sgec is a very challenging task due to potential errors coming from transcriptions, disfluencies as well as the correction process. to avoid giving erroneous feedback to language learners, we do not want to give feedback on edits that our models have little confidence in, assuming lower confidence indicates lower accuracy. to conduct confidence filtering, we need to define a confidence measure. in the cascaded sgec pipeline (eqn. 1), each module produces a token-level confidence score associated with its prediction. we first define sentence-level confidence for each module as the lowest token probability over the entire sentence. sentence-level filtering can be conducted by rejecting sentences with low confidence. we also explore the option of using edit-level confidence, i.e. confidences are calculated over each edit instead of sentence. note that for asr module, we always use the lowest over sentence, to mitigate a known issue of asr error propagation. the overall confidence is calculated using a weighted sum of all thee modules: log @xmath0, p dd and p gec are sentence/edit-level confidence of each module. ## conclusions this paper focuses on assessing and developing cascaded sgec systems. we discuss standard metrics for individual module assessment, as well as edit distance based metrics for system output evaluation. to mitigate module error propagation in cascaded systems, we experimented with semi-supervised fine-tuning and self-distillation approaches, aiming to improve system output quality. lastly, confidence-based filtering is investigated, and it proves to be effective in improving feedback precision as well as the overall quality. for future work, we plan to experiment with the state-of-the-art end-to-end asr systems, which potentially gives lower wer and further improves the sgec performances. another research direction is to investigate tighter integration of modular sgec systems, which allows a richer information flow cross module connections.