Domain-specific transformer models for query translation (2023)

Abstract: Due to the democratization of e-commerce, many product companies are listing their goods for online shopping. For periodic buying within a domain such as Grocery, consumers are generally inclined to buy certain brands of products. Due to a large non-English speaking population in India, we observe a significant percentage of code-mix Hinglish search queries e.g., sasta atta. An intuitive approach to dealing with code-mix queries is to train an encoder-decoder model to translate the query to English to perform the search. However, the problem becomes non-trivial when the brand names themselves have Hinglish names and possibly have a literal English translation. In such queries, only the context (non-brand name) Hinglish words needs to be translated. In this paper, we propose a simple yet effective modification to the transformer training to preserve/correct Grocery brand names in the output while selectively translating the context words. To achieve this, we use an additional dataset of popular Grocery brand names. Brand names are added as tokens to the model vocabulary, and the token embeddings are randomly initialized. Further, we introduce a Brand loss in training the translation model. Brand loss is a cross entropy loss computed using a denoising auto-encoder objective with brand name data. We warm-start the training from a public pre-trained checkpoint (such as BART/T5) and further adapt it for query translation using the domain data. The proposed model is generic and can be used with English as well as code-mix Hinglish queries alleviating the need for language detection. To reduce the latency of the model for the production deployment, we use knowledge distillation and quantization. Experimental evaluation indicates that the proposed approach improves translation results by preserving/correcting English/Hinglish brand names. After positive results with A/B testing, the model is currently deployed in production.

## introduction due to the democratization of e-commerce, online shopping has evolved in recent times, where most customers choose to shop online. as an effect, the majority of product companies are keen on making their products available for online shopping. when it comes to domains such as grocery, where users have to shop periodically, they typically have a preference for buying products of certain brands. hence, for grocery, it was observed that a significant portion of search queries contain brand names. due to a large non-englishspeaking population in india, we observe a significant percentage of code-mix hinglish search queries. a hinglish query is where one or more hindi words are written in english, e.g., sasta atta. since there are no standard spellings, we observe a large variation in the hinglish words. we also observe many queries where brand names are misspelled. an intuitive approach to deal with code-mix queries is to train an encoder-decoder model to translate the query to english and use an english search api to retrieve the products @xcite . however, the problem becomes more challenging when the brand names themselves are hinglish words and possibly have a valid english translation. we observe that in the grocery domain, many brand names have hinglish names, e.g. aashirvaad, gowardhan, veer, navratna etc. in such queries, only the context (non-brand name) hinglish words need to be translated, and brand names (though hinglish) must not be altered in the translation. e.g. for the query, 'sasta dabur lal tel', a literal translation would be 'cheap dabur red oil'. however, the expected translation is 'cheap dabur lal oil' since 'dabur lal' is a brand name. although most of the words in the query are hinglish, only the first and last words need to be translated. if a brand name gets altered during the translation, it will lead to non-ideal search results. in some cases, the query does not need a translation even though it contains a hinglish brand name, e.g., veer brand oil. if an english/hinglish brand name is misspelled, it needs to be corrected in the translation. in general, the seq2seq model should be able to handle the following scenarios. to ensure such behavior, one would need large manually labeled data inclusive of many brand names. in this paper, we propose a simple yet effective modification to the transformer training to preserve/correct brand names in the output while selectively translating the context words. to achieve this, we use an additional dataset of high-demand grocery brand names provided by the product team. first, to output brand names as a whole, we add them as tokens to the model vocabulary and randomly initialize the corresponding token embeddings. further, we introduce a brand loss for training the translation model. brand loss is a cross entropy loss computed using a denoising auto-encoder objective with brand name data. we warm-start the training from a generic pre-trained checkpoint (such as bart/t5) and further adapt it for query translation using the domain data. results indicate that introducing brand loss significantly improves accuracy by preserving/correcting brand names in the translation. we also verify that introducing brand information as the loss is more effective than introducing it as the training data. the model is generic and can be used with english as well as code-mix hinglish queries, alleviating the need for language detection. further, to reduce the latency of the model for the production use-case, we use knowledge distillation and quantization. using a large model as the teacher, we obtain pseudo-labels for a large set of unlabeled queries. we then train a small student opennmt @xcite model on this dataset. we are able to achieve more than 28x reduction in the latency with a slight drop in accuracy. experimental results demonstrate the efficacy of the proposed approach. ## related works transformers @xcite is the current state-of-the-art model for translation. large-scale self-supervised pre-training of encoder-decoder models followed by domainspecific fine-tuning can significantly improve the translation quality with a limited labeled set @xcite . search query translation is essential for cross-lingual information retrieval (clir). bhattacharya et al. @xcite use word vector emebedding and clustering to find groups of words representing the same concept from different languages. these multilingual word clusters are then used to perform query translation for clir between english, hindi and bengali. @xcite proposes an approach to perform vernacular query translation without using any parallel corpus. authors only utilize unlabeled query corpus from two languages, a pre-trained multilingual translation model, and train it with cross-language training to translate vernacular search queries to english. for code mix query translation, multilingual and english pre-trained encoder-decoder models have been explored @xcite ) @xcite . @xcite explored statistical and neural ma-query ground truth without brand loss with brand loss asribad ata aashirvaad atta ashirwad atta aashirvaad atta dabber lal tel dabur lal oil dabber lal oil dabur lal oil emni rice brand oil emami rice bran oil emni rice bran oil emami rice bran oil daadis peanut khakra daadi's peanut khakhra grapes peanut seeds daadi's peanut khakra goverdhan desi ghee gowardhan desi ghee goverdhan desi ghee gowardhan desi ghee detol original dettol original original detol dettol original farnely all product farmley all product free all product farmley all product veer brand oil veer brand oil mustard oil veer brand oil all out mosquito refill in contrast to these approaches, we enforce the model to copy brand names using an additional loss component computed on the brand name data. the model still has a default generate ability which helps in correcting misspelled brand names. ## proposed approach in the following sections, we provide details of the dataset and training methods. ## results ## knowledge distillation for improved latency the search query translation models are userfacing and need to have low latency to support high throughput. though the bart-base model with query pre-training and fine-tuning provided good accuracy on the test set, it was not sufficient for production deployment due to the latency constraints. we observed that the p95 latency of the bart-base model with pytorch implementation was ~200 ms, which is not acceptable for the production use-case. to reduce the latency of the model, we use knowledge distillation with open-nmt @xcite framework. open-nmt provides a ctranslate wrapper for faster inference, making it a good choice for low latency use-cases. our approach is to train a small open-nmt student model using grocery bart-base model as the teacher model. since the student model resides in another programming framework, we use a pseudo-labeling approach to transfer knowledge from the teacher to the student. to create a parallel corpus for open-nmt model training, we obtain translation labels on ~38m query set using the teacher model. we then train the open-nmt model on this large parallel corpus and the manually tagged set. we use a single layer open-nmt model with a vocab size of 18k and a hidden dimension of 384. the model has ~23m trainable parameters. for open-nmt model as well, we add the brand name tokens to the vocab. we use weight quantization during model inference. we performed a/b testing of the open-nmt student model w.r. ## conclusion in this paper, we proposed a simple yet effective approach for domain-specific query translation. for the grocery domain, it was noticed that a significant percentage of queries contained brand names due to user preferences and periodic buying. we also observed a significant percentage of code-mix hinglish queries and queries with grammatical errors. since some grocery brand names are themselves hinglish words, we wanted a brand-aware query translation model. to better preserve brand names in translation, we added brand name tokens to the model vocab and introduced an additional brand loss in transformer training. the modification improved translation accuracy by depicting desired brand name preserving effect. to reduce the latency of the model for the production deployment, we used knowledge distillation with the open-nmt student. using a large model as a teacher and with pseudo labeling, we trained a single layer open-nmt student model. we could obtain more than a 28x reduction in latency with a slight drop in accuracy. after positive results with a/b testing, the model was deployed in production.