Improving Grammatical Error Correction with Multimodal Feature Integration (2023)

Abstract: Grammatical error correction (GEC) is a promising task aimed at correcting errors in a text. Many methods have been proposed to facilitate this task with remarkable results. However, most of them only focus on enhancing textual feature extraction without exploring the usage of other modalities’ information (e.g., speech), which can also provide valuable knowledge to help the model detect grammatical errors. To shore up this deficiency, we propose a novel framework that integrates both speech and text features to enhance GEC. In detail, we create new multimodal GEC datasets for English and German by generating audio from text using the advanced text-to-speech models. Subsequently, we extract acoustic and textual representations by a multimodal encoder that consists of a speech and a text encoder. A mixture-of-experts (MoE) layer is employed to selectively align representations from the two modalities, and then a dot attention mechanism is used to fuse them as final multimodal representations. Experimental results on CoNLL14, BEA19 English, and Falko-MERLIN German show that our multimodal GEC models achieve significant improvements over strong baselines and achieve a new state-of-the-art result on the Falko-MERLIN test set.

## introduction grammatical error correction (gec) is one of the promising applications in natural language processing (nlp), aiming to correct sentences containing grammatical errors. gec has attracted substantial attention in the past few decades owing to its importance in writing assistance for language learners @xcite @xcite @xcite @xcite ## data construction owing to the lack of speech data in the gec task, we need to construct multimodal gec datasets for multimodal gec tasks. therefore, in this section, we give the details of dataset construction. speech processing has achieved promising improvement over the past few decades, including converting sentences in the text into utterances @xcite . therefore, we employ the advanced speech synthesis system to convert each piece of source side of gec data (i.e., the ungrammatical side) into audio data to construct gec multimodal data. as a result, each example in the gec dataset is expanded into a triplet consisting of the ungrammatical sentence, the audio generated from the corresponding ungrammatical sentence, and the grammatical sentence. ## analyses 5.1 ablation study to explore the effectiveness of our proposed method, we conduct the ablation studies with the following settings: a) removing the moe layer (-moe) and retaining the dot attention module to fuse acoustic and textual features. b) removing the speech encoder (-speech enc.), which degenerates our multimodal gec model into a text-only unimodal gec model. as shown in ## related work grammatical error correction (gec) is the task of automatically identifying and correcting grammatical errors in a text @xcite . previous research in this field has primarily focused on strengthening the representations of text data through data augmentation techniques, such as using the back-translation method @xcite for the gec task @xcite @xcite , and injecting noise with specific rules into grammatical sentences @xcite @xcite . more recently, pre-trained language models (plms) have been demonstrated to be effective in improving the performance of gec tasks. studies such as @xcite have leveraged sequential transfer learning to adapt pre-trained transformer models to the gec domain. @xcite initialized an encoder-decoder gec model with pre-trained bert weights to enhance gec performance. @xcite utilized the pre-trained bart model as a generic pre-trained encoder-decoder model for gec, and rothe et al. (2021) adopted a pre-trained t5 model to distill gec corpus and used the pre-trained structure as part of the network for distilled gec training, achieving promising results. however, to date, no previous work has attempted to incorporate multimodal information (e.g., speech modality) into the gec task. our work is the first to explore the use of multimodal information for gec. multimodal many studies have demonstrated the potential of incorporating multimodal information in improving the performance of single-modal tasks in the nlp domain. for example, @xcite integrated image modality into the twitter sarcasm detection task and found that incorporating image information can enhance the performance of this text-only task. @xcite proposed to integrate radiology images and textual findings to improve impression generation. additionally, @xcite fused acoustic and text encoding to jointly learn a unified representation, thereby improving speech-to-text translation tasks. @xcite demonstrated that fusing speech modality can enhance the readability of text summarization tasks. @xcite studied a joint speech-text embedding space through a semantic matching objective, achieving improved results in downstream tasks. @xcite proposed a method for learning the cross-modality interaction between acoustic and textual information, which outperformed the unimodal models in emotion classification. in this work, we are the first to attempt to fuse acoustic and text to improve the gec task. ## conclusion this paper presents a novel approach to the task of multimodal gec that integrates speech and text features to improve grammatical error correction. due to the scarcity of speech data in gec, we expand the original gec data to create new multimodal gec datasets for english and german, where each sample in our datasets is a triple (grammatically incorrect text, audio, and corrected text). our approach utilizes a speech and text encoder to extract acoustic and textual features from the speech and input text, respectively. then, we employ an moe approach to selectively extract audio features that align with the textual features and use a dot attention layer to fuse the features from different modalities as the final representation. this fused representation is input to the decoder to generate the corrected sentence. our experimental results on widely-used benchmarks demonstrate the effectiveness of our proposed model, achieving significant improvements compared to existing studies.