Multi-teacher Distillation for Multilingual Spelling Correction (2023)

Abstract: Accurate spelling correction is a critical step in modern search interfaces, especially in an era of mobile devices and speech-to-text interfaces. For services that are deployed around the world, this poses a significant challenge for multilingual NLP: spelling errors need to be caught and corrected in all languages, and even in queries that use multiple languages. In this paper, we tackle this challenge using multi-teacher distillation. On our approach, a monolingual teacher model is trained for each language/locale, and these individual models are distilled into a single multilingual student model intended to serve all languages/locales. In experiments using open-source data as well as customer data from a worldwide search service, we show that this leads to highly effective spelling correction models that can meet the tight latency requirements of deployed services.

## introduction spelling correction is vital to the modern search experience. users expect it, mobile devices and speech-to-text interfaces make it more crucial than ever, and uncaught spelling errors can lead to urgent problems of security and trust if problematic search results are shown to users. for services with a global reach, this poses a substantial challenge for multilingual nlp: spelling errors must be caught and corrected in any language, and even in queries using multiple languages. the promise of multilingual language models is that we may be able to meet these challenges with a single spelling correction model serving all languages/locales. in the present paper, we develop and motivate such a multilingual approach relying crucially on multi-teacher distillation. on our approach, an individual teacher model is trained for each language/locale, and these individual models are distilled into a single multilingual student model intended to serve all languages/locales. our distillation objective is a purely behavioral one: the multilingual student is trained to mimic the input-output behavior of the individual teachers. this brings a number of key advantages in our setting. first, we can customize the individual teacher models to specific languages/locales, which proves especially useful in the area of tokenization. second, when we want to add a new language/locale l, we train just two models: the new teacher for l and the new multilingual model distilled from the input-output pairs generated by all the teacher models. third, the individual teacher models are themselves assets that can be distilled into student models; where these are superior (common for data-rich languages), they can be used. we motivate our approach with a wide range of experiments using open-source data as well as proprietary user data from a worldwide search service. overall, we find that our multi-teacher distillation approach leads to superior models compared to both individual monolingual student models and multilingual student models distilled from a single multilingual teacher. in addition, we show that we can efficiently add new languages and easily meet the tight latency requirements imposed by industrial search engines. overall, we suggest that this is a promising modeling approach not only for spelling correction but also for the other services needing to serve numerous languages and locales. ## related work spelling correction is a widely studied problem @xcite . earlier work relied on lexical rules @xcite or language models plus linguistic features @xcite . in more recent work, spelling correction is cast as an encoder-decoder problem @xcite @xcite , theoretically making it easier to scale to multilingual settings. however, methods that are efficient and scalable across numerous languages have been much less explored. spelling correction is highly sensitive to different tokenization schemes, since it involves manipulating characters and other subword units. subword tokenization schemes provide the right balance between operating on subword units and being efficient for training and inference. popular methods for subword tokenization include byte pair encoding (bpe; @xcite ), byte-level bpe (bbpe; @xcite ), sentence-piece @xcite , and unigram language model @xcite . bpe splits words into subword units based on their statistical properties and is extensively employed by various transformer models such as gpt @xcite , roberta @xcite , and bart @xcite . bbpe operates at the byte-level, efficiently enabling the encoding and decoding of texts across different languages with non-overlapping character sets. in this paper, we explore bpe and bbpe tokenization schemes in our experiments. increasing model size and amount of compute used for training will generally improve the performance of neural language models @xcite , but the costs might be prohibitive. in model distillation @xcite , a large teacher model is used to guide the training of a smaller student model. this is a viable solution for developing deployable models with strict production constraints. these approaches have proven highly successful for seq2seq problems in general @xcite . distillation approaches vary in the degree to which they presuppose access to the teacher model during student model training @xcite . at one extreme, the teacher and student models are trained together (i.e., co-distillation; @xcite . at the other extreme, the teacher is simply used to generate output labels for the training data, based on which the student is trained (e.g., sequence-level knowledge distillation (seq-kd); kim and rush 2016). in our multi-teacher distillation, we aim to decouple the teacher and student training regimes in order to train the best model for each language, and so we use seq-kd. there are existing studies about multiple teacher distillation. for example, you et al. combined multiple teacher networks by averaging the softened output targets and selecting layers in student and teacher networks @xcite . yuan et al. selected soft labels from a collection of teacher models, based on the reward signal from performance of distilled student model @xcite . both studies use multiple teachers to generate variant candidates and distill knowledge to build a robust and accurate student. in this paper, we apply multi-teachers for multilingual problem, where each teacher specializes in one language and they work together to guide the learning of a multilingual student. ## methods we aim to create a high-performing model that can serve multiple languages while satisfying latency restrictions. we propose a multi-teacher distillation method. the main idea is to train teacher models with optimal configurations for individual languages and then build a single student model based on the multi-teacher inference. figure 1 provides a high-level overview of our multi-teacher distillation approach, which we describe in this section. ## experiments on user data in this section, we report on experiments with user data from a large, global search service, and the user data are their search queries. in section 5, we report on experiments with open source data that are natural language sentences. with the opensource data, we can be completely open about the findings, with some costs in terms of realism. with the user data, we are required to conceal some details, but the findings themselves still provide a clear picture of how our approach fares in the real world. ## experiments on open-source data to supplement our experiments on user data, we also conducted experiments with open-source data for which we can supply absolute performance numbers. ## conclusion we developed and motivated a multi-teacher distillation approach for multilingual spelling correction. on our approach, teacher models for individual languages are used to distill a single multilingual student model. by focusing on improving the performance of teacher models for specific languages, we can enhance the overall performance of the student model. additionally, our approach allows for the inclusion of new monolingual teacher model inference data into the distillation process, enabling the expansion of the multilingual student model without the need to retrain the entire set of teacher models for all languages. we believe that this modeling approach holds promise not only for spelling correction services but also for other services needing to serve numerous languages and locales.