Beyond the Repo: A Case Study on Open Source Integration with GECT o R (2023)

Abstract: We present a case study describing our efforts to integrate the open source GECToR code and models into our production NLP pipeline that powers many of Educational Testing Service’s products and prototypes. The paper’s contributions includes a discussion of the issues we encountered during integration and our solutions, the overarching lessons we learned about integrating open source projects, and, last but not least, the open source contributions we made as part of the journey.

## introduction gector (grammatical error correction tag, not rewrite) 1 is a set of deep learning models developed by grammarly for the task of grammatical error correction or gec @xcite . gector achieves state-of-the-art results for the gec task and its inference speed is up to 10 times as fast as that of equivalent transformer-based sequence-to-sequence (seq2seq) gec systems. the most commonly proposed systems for the gec task leverage seq2seq neural machine translation (nmt) models to "translate" from errorful text to corrected text. however, such systems generally suffer from slow inference and require a large amount of data. to deal with these issues, gector simplified the task from sequence generation to sequence tagging. to train this tagging system, gec-tor utilizes three training stages: pre-training on synthetic data, fine-tuning on parallel datasets that contain both errorful and corrected texts, and further fine-tuning on a combination of high-quality, parallel datasets containing both errorful/corrected as well as error-free texts. however, one of the largest benefits of gector to the nlp community is that it was open sourced under a commerciallyunrestricted apache 2.0 license. gector provides three already trained english gec models based on bert, roberta, and xl-net. it also has scripts for training and inference. since our pipeline is english-only for now, the existing gec models perfectly fit our needs. therefore, our integration efforts focus entirely on the inference side. ## related work adopting open source software in commercial environments has been a topic of much interest. there are several barriers to open source use: lack of knowledge, inability to incorporate it into existing legacy systems, too many forks created by different groups, technological immaturity, et cetera @xcite . one formal approach that can be used to assess an open source project before putting it into production is the open source maturity model (osmm) which assesses factors like the product, support, training, documentation, product integration and professional services, and gives them a weighted score. however, this model is only a first step in identifying which projects are worth a more in-depth evaluation @xcite . in this paper, we hope to provide a detailed case study with illustrative, concrete steps for using open source projects. there are major forks and re-implementations of the original gector project available on github. fast-gector foot_2 claims to be a faster and simpler implementation of the original project leveraging amp foot_3 and deepspeed @xcite . however, we were unable to reproduce the same results as the original model in our experiments with fast-gector. gector-large @xcite focuses on improving gector by upgrading the transformer encoders, and using an ensemble model for spanlevel edits. according to the paper, while the larger encoders do yield better performance, they do so at the cost of speed with inference being 2.3-2.5x slower. ## integration in this section, we discuss the challenges we faced when integrating gector into our pipeline and our solutions. ## lessons learned while bolstering gector, we learned a few general lessons about open source development and integration that we would like to share. ## future work in the future, we plan to take our own advice from §5.4 and transition our fork of gector away from allennlp. we plan to replicate the functionality we need by using actively maintained open source projects from huggingface such as transformers @xcite , datasets @xcite , and accelerate foot_9 directly. ## conclusion we presented a case study on integrating an open source project into a commercial, production nlp pipeline. while we consider the primary contributions of this paper to be a clear and concise description of the issues we faced (and solved) as well as the larger lessons we learned, we also hope that the nlp community will benefit from our fork of gector that is actively maintained, more modern, more robustly tested, and easier to use for inference.