Efficient and Interpretable Grammatical Error Correction with Mixture of Experts (2024)

Abstract: Error type information has been widely used to improve the performance of grammatical error correction (GEC) models, whether for generating corrections, re-ranking them, or combining GEC models. Combining GEC models that have complementary strengths in correcting different error types is very effective in producing better corrections. However, system combination incurs a high computational cost due to the need to run inference on the base systems before running the combination method itself. Therefore, it would be more efficient to have a single model with multiple sub-networks that specialize in correcting different error types. In this paper, we propose a mixture-of-experts model, MoECE, for grammatical error correction. Our model successfully achieves the performance of T5-XL with three times fewer effective parameters. Additionally, our model produces interpretable corrections by also identifying the error type during inference.

## introduction grammatical error correction (gec) is a task that aims to detect and correct any errors in a given text. through scientific progress over the past decades, grammatical error correction has outgrown its name. gec not only deals with grammatical errors but also includes the correction of misspellings, orthographic errors, semantic errors, and more @xcite . a gec model receives a possibly erroneous text and should produce a corrected version of the text with minimal modification. an example is given in even after adopting end-to-end approaches using neural network models, the error type information remains useful. some recent models approach the problem by separating the error detection and correction models @xcite , training a single model for both detection and correction @xcite , or training a model to predict the edit/transformation tag, which can include the error type @xcite . error type information is useful not only for building the grammar correction model but also for reranking the outputs of gec models @xcite and combining outputs of gec models @xcite @xcite . gec system combination has been very effective in improving the state of the art, and it has been repeatedly demonstrated that it works best when the base systems have complementary strengths @xcite , such as which error types they can correct more accurately. however, system combination is computationally expensive because it needs to run inference on each base system first before running the combination method itself. it would be desirable to just have one model with multiple sub-networks that specialize in different aspects, such as which error types it can correct more accurately. the capability of a neural network to correct one error type is not necessarily transferable to correcting other error types, as @xcite report that state-of-the-art gec models are often less accurate in correcting certain error types than weaker gec models. this could be caused by task interference during the training of the model. when fine-tuning t5-v1.1-base for grammatical error correction, we observe that the model becomes less accurate at correcting punctuation (punct) and preposition (prep) error types when it has the best overall (all) performance (figure 1 ). we also notice that the model's accuracy source the rich people will buy a car but the poor people always need to use a bus or taxi . correction rich people will buy a car , but poor people always need to use a bus or taxi . edits (0, 2, 'rich', det), (7, 7, ',', punct), (8, 9, ", det) ## mixture of experts mixture of experts (moe) is a learning procedure for a system that can be decomposed into multiple separate networks, with each network representing an expert at a particular task @xcite . mixture of experts has been reported to perform well on tasks that can be broken down into different sub-tasks, such as multilingual machine translation @xcite , and single tasks that have multiple objectives, such as video recommendation @xcite . when used with the transformer architecture, moe is typically applied by replacing the feedforward layer f (x) in a transformer block with an moe layer foot_1 f ′ (x). moe is applied independently to some or all transformer blocks in the network. an moe layer consists of a set of experts {e 1 , e 2 , . . . , e m } with parameters ϕ i , @xmath0, . . . , m }, a router or a gating function r that selects a subset of k experts, and optionally an aggregation function g i that produces the weight for expert i. the aggregation function can also be the same as the router. let x denote the vector representation of a token from the previous layer. we formalize the moe layer in equation 1 below: one problem that may arise in moe is one expert getting chosen more often than others, making it receive more gradient updates and subsequently making it more preferred by the router, a self-reinforcing problem. to avoid this issue, the number of tokens in a sequence @xmath1, x 2 , . . . , x n } that an expert can process is often limited to a threshold called expert capacity, and a load balancing loss is used to encourage a more balanced expert allocation. if an expert already reaches the maximum capacity, subsequent tokens that are assigned to that expert do not go through the expert layer. the load balancing losses are calculated based on the fraction of the input that goes into expert i, represented by w i . @xcite propose to penalize the model based on the mean square of w i , but the top-k operation to select the experts is not differentiable. as such, it approximates the loss function by multiplying the average routing score for expert i on all tokens (v i ) with the fraction of tokens that are allocated to it (w i ), then takes the average of that score from all experts. the loss function is formalized in equation 2 below. ## method in this section, we explain how we build moece (mixture of error correction experts), a mixtureof-experts model for grammatical error correction. ## results we report the scores of our models and the comparable dense model in by 2.44 points and 0.94 points respectively, while moece-st-base improves the f 0.5 score on the same test sets by 2.59 points and 1.07 points. our method still improves the f 0.5 scores on all test sets when we scale up the models to moece-gs-large and moece-st-large, and the improvements are statistically significant with @xmath2.01. compared to previous work (figure 3 ), the performance of our moece-gs-large model is comparable to t5-xl @xcite despite having vastly fewer effective parameters. t5-xl with 3b effective parameters has an f 0.5 score of 67.65 5 on the conll-2014 test set and 73.92 on the bea-2019 test set, while moece-gs-large that has three times fewer effective parameters has an f 0.5 score of 67.79 on the conll-2014 test set and 74.07 on the bea-2019 test set ( ## conclusion and future work in this paper, we present a new grammatical error correction model that is more efficient by utilizing a mixture of experts, called moece. our experiments show that our model can improve the f 0.5 scores of the comparable dense model by up to 0.55 points on the conll-2014 test set and 1.32 points on the bea-2019 test set. with the same model and hyper-parameters, the model can improve the f 0.5 score on out-of-domain test sets by up to 2.59 points on cweb-g and 1.07 points on cweb-s. the larger variant of our model, moece-gs-large, successfully reaches performance slightly better than a model, t5-xl, that has three times its effective parameter count and almost double its total parameter count. our proposed error type loss makes our model interpretable by producing corrections with error types. our analysis shows that the error type loss helps in routing the input token to the appropriate expert based on its error type. in addition, we find that our error type loss can be an alternative to existing load balancing loss. we believe that interpretable and explainable grammatical error correction models are needed to help language learners with their study and we hope more research explores this direction.