Search Query Spell Correction with Weak Supervision in E -commerce (2023)

Abstract: Misspelled search queries in e-commerce can lead to empty or irrelevant products. Besides inadvertent typing mistakes, most spell mistakes occur because the user does not know the correct spelling, hence typing it as it is pronounced colloquially. This colloquial typing creates countless misspelling patterns for a single correct query. In this paper, we first systematically analyze and group different spell errors into error classes and then leverage the state-of-the-art Transformer model for contextual spell correction. We overcome the constraint of limited human labelled data by proposing novel synthetic data generation techniques for voluminous generation of training pairs needed by data hungry Transformers, without any human intervention. We further utilize weakly supervised data coupled with curriculum learning strategies to improve on tough spell mistakes without regressing on the easier ones. We show significant improvements from our model on human labeled data and online A/B experiments against multiple state-of-art models.

## introduction an incorrectly spelt search query can return irrelevant products in e-commerce which hurts both the business and experience for a user who is unable to find the intended product. as per the latest english proficiency index report of written test data from 100 countries, foot_0 only 29% countries are proficient in english. our platform operates in asian countries like india which are ranked low in this survey. as per the latest indian census only ∼10% of the indian population is versed in the english language thus causing high spell errors in the user queries. spelling errors are generally classified as -typographic and cognitive @xcite . typographic errors emanate from the typing mistakes on the keyboard which worsens on mobiles due to smaller keypads. cognitive errors happen when a user does not know how to correctly spell a word. this leads to phonetic errors like "cenityjer" for "sanitizer". edit-distance foot_1 based spell corrections at run-time are computationally expensive for higher edit distances at web-scale. also, this approach typically works at an individual word level @xcite @xcite is not able to identify contextual mistakes like "greeting cart". human labeling being expensive and time-consuming, these methods use a large amount of user query reformulations as training data for learning. most of the web search work relies on this for generating correct-incorrect word pairs @xcite . query reformulations alone fail to cover all kinds of errors like phonetic (cognitive) ones -"metras" vs "mattress" or edit/phonetic+word-compounding like "ball pen" vs "bolpan" when the user herself does not know the correct spelling. fig. 1 shows the distribution of different types of spell errors on our platform over a sample of ∼ 23k queries as classified by human judges. inspired by the low-resource machine translation research, the latest spell systems create synthetic data to learn neural @xcite and statistical @xcite ) models for spell correction. to solve at internet scale, 3 billion search queries a month from over 450 million users, we too create a large amount of synthetic and user feedback data. given significant colloquial phonetic mistakes (1), we develop a novel way of generating phonetic mistakes besides edit-distance and compounding ones. along with the user query reformulations, the user clicks on the spell-corrected queries also provides another source of noisy labeled data. curriculum learning @xcite has shown to generate robust models which show the improvements in their learning ability when they first learn on simpler tasks followed by tougher tasks. we devise a few simple curriculums to improve accuracy on tougher mistakes. using transformers, the main contributions of our work are: (i) generating large synthetic and weakly supervised labelled data for different spell mistakes including novel deep learning models for phonetic mistakes. we report the effectiveness of our approach reparos (reparo-spell) through offline evaluation as well as online a/b experiment on user queries with significant improvements. in the following sections, we discuss related work followed by approach, the experiments, and finally the results. ## related work context-free word-based spell checkers in web search have been commonly implemented in two ways: edit distance @xcite and statistical noisy channel. damerau-levenshtein edit distance finds the correct words by editing 'k' characters of the input word. on the other hand, statistical noisy channel methods @xcite ) assume that the user inadvertently introduces some noise through keyboard errors. brill and moore improves the standard noisy channel model. rulebased systems like soundex @xcite and metaphone @xcite generate a phoneme sequence for a given word to match against the phoneme sequence of the correct word. contextaware spell checkers @xcite incorporate the surrounding words to improve the correction. however, this approach still corrects each word individually and is not able to address word-compounding spell mistakes. machine translation ( @xcite ) based approaches treat a correct query as a translation of the misspelled query. zhou et al. use an rnn with encoder-decoder architecture that improves upon the statistical approaches. to avoid expensive human-labeled data, it's become common to generate copious amounts of synthetic labeled data @xcite for model training. neuspell @xcite formulates spell correction as a sequential labeling problem where a correct word is labelled as itself and a misspelled token is labeled as its correction. these approaches suffer due to synthetic data being mainly edit-distance based errors. simple phonetic based corrections have been explored as well @xcite , @xcite . neuspell also ignores the compounding errors which form more than 25% of the spell errors. our approach overcomes these by using state-of-art context-aware transformer models with curriculum learning, user feedback-based data and synthetic data sets for different types of spell mistakes. in the following sections we describe data generation methods for training, model details followed by experiments and discussion. ## model and training details we formulate the problem of query correction as a neural machine translation (nmt) problem @xcite , where for a source sequence (incorrectly spelt query) the task is to predict the spell corrected query. an nmt model learns the conditional probability p(y 1 , .., y t ′ |x 1 , .., x t ) where (x 1 , .., x t ) is the input query and (y 1 , .., y t ′ ) is the target correct query. note that the input query length t may differ from the output length t ′ due to splitting mistakes (space as a character) by users. we use transformer @xcite neural architecture that is superior (as shown in results section) due to bidirectional self-attention, multi-head attention and uses best neural tricks of batch normalization, resnets with encoder and decoder layer. our choice to use transformer was also because we found pre-trained (on in-domain data) models like bart (et al, 2019) (combining bert @xcite and gpt) was slightly inferior to simple transformer model which is most likely due to bart/bert kind of models requiring longer contexts. hyperparameter and latency details are included in the appendix. ## results and discussion we conduct both offline and online evaluation of our models. offline evaluation: online evaluation: in production, we adopt a 2-step architecture by adding an ml ranker (yang, 2022) that does the final candidate selection (from multiple candidates from the reparos/googleswbs). this multi-stage setup empirically produced better results than just finetuning the nmt model since top-10 accuracy of reparos-c2 is 76.31% on improvement set and 98.08% on regression set. this helped all the models (and reparos more due to their better top-k accuracy) and is removed from results discussion for brevity. in both the a/b tests, 20% of the users were randomly assigned to each bucket (control and treatment) and the experiments were run for 3 weeks each to achieve statistical and practical significance. reparos-c2 is the currently deployed model serving the entire search traffic of our app for more than 9 months. at our scale, 0.01% is quite a significant change in pagectr. both reparos-base and reparos-c2 show a reduction in click backs while increasing spell system coverage as well as the search results page ctr. reparos-c2 also reduced the null searches drastically while reparos-base improved the number of cart adds by the users. these results demonstrate the effectiveness of both the versions of reparos across multiple user and business metrics. ## conclusion we presented the generation of large synthetic and weakly supervised labeled data for different types of user spell mistakes including a creative deep learning model to generate the phonetic mistakes. we then presented a sequence-to-sequence deep transformer based spell model with curriculum learning on tough spell mistakes and user feedback data that demonstrates superior performance than state-of-the-art statistical and neural spelling correction models. our solution is currently deployed on an india's e-commerce platform and serves over billions of queries per month from over 450 million users across 100% zip codes of india. with new users with varying literacy rates joining our platform every day, spell correction remains a tough problem to solve. we found that deeper nmt models result in better performance but are impractical (in the current state due to higher latencies) to deploy in production to serve real-time user requests. hence our future work is to focus on model pruning, quantization, and knowledge distillation and continual learning to reduce latencies for deployment and improve upon our current system. casey whitelaw, ben hutchinson, grace y. chung, and gerard ellis. 2009. using the web for language independent spellchecking and autocorrection. in proceedings of the 2009 conference on empirical methods in natural language processing: volume 2 -volume 2, emnlp '09, page 890-899, usa. association for computational linguistics. fan et al yang. 2022. spelling correction using phonetics in e-commerce search. in proceedings of the fifth workshop on e-commerce and nlp (ecnlp 5), dublin, ireland. association for computational linguistics. yingbo zhou, utkarsh porwal, and roberto konow. 2017. spelling correction as a foreign language. arxiv preprint arxiv:1705.07371. canxiang zhu, zhiming chen, yang liu, juan hu, shu qiong sun, bixiao cheng, zhen-dong, and yang. 2019. automatic query correction for poi retrieval using deep and statistical collaborative model.