Machine translation impact in E -commerce multilingual search (2022)

Abstract: Previous work suggests that performance of cross-lingual information retrieval correlates highly with the quality of Machine Translation. However, there may be a threshold beyond which improving query translation quality yields little or no benefit to further improve the retrieval performance. This threshold may depend upon multiple factors including the source and target languages, the existing MT system quality and the search pipeline. In order to identify the benefit of improving an MT system for a given search pipeline, we investigate the sensitivity of retrieval quality to the presence of different levels of MT quality using experimental datasets collected from actual traffic. We systematically improve the performance of our MT systems quality on language pairs as measured by MT evaluation metrics including Bleu and Chrf to determine their impact on search precision metrics and extract signals that help to guide the improvement strategies. Using this information we develop techniques to compare query translations for multiple language pairs and identify the most promising language pairs to invest and improve.

## introduction multilingual search capability is essential for modern e-commerce product discovery @xcite . localization of e-commerce sites have led users to expect search engines to handle multilingual queries. recent proposals of cross-lingual information retrieval handle multilingual queries, and language-agnostic crossborders product indexing has gained traction with neural search engines @xcite @xcite @xcite , but legacy e-commerce search indices are still built on monolingual product information and support for multilingual search is bridged using query translation @xcite @xcite @xcite . query translation allows users to look up information represented in documents written in a languages different from the language of the query. it takes as input the query typed in source or query language and returns a translated query to the search engine to retrieve documents in the target language. it follows that query translation plays a key role and its output significantly affects the retrieval results. previous studies have demonstrated performance of clir (cross-lingual information retrieval) correlates highly with the quality of the machine translation (mt), and improving the quality of mt improves retrieval quality @xcite . however, these evaluations are done separately for each task. this leaves a large gap in understanding the impact of improving mt quality iteratively on clir performance in a real time industrial setting. since machine translation is used here as interim application, the objectives of the retrieval task may have varying levels of tolerance to the inherent translation quality. information retrieval evaluation usually involves human-annotated relevance labels of search results candidates. in an industry setting, annotating a representative sample is a time consuming and expensive task, particularly during iterative improvement of mt for the search use case. additionally, a general-purpose mt evaluation metric may not necessarily adapt to the query evaluation for downstream retrieval task. to address these above concerns, we propose an mt evaluation framework to build an e-commerce specific clir test set. it exploits behavioural signals from search retrieval results to evaluate mt quality for a given query. in order to identify the benefit of improving an mt system, we further investigate the sensitivity of retrieval quality to the presence of different levels of mt quality as measured by bleu, and chrf using experimental datasets collected from actual traffic. based on these experiments, we recommend the pairs that are worth continued investment in improving mt systems for search. our main contributions are: this paper is organized as following: we propose a rank-based evaluation framework in section 2. we propose two mt impact rates, mt launching impact rate and mt improvement mt rate respectively in section 3. section 4 is the experiment with 12 language pairs from 6 stores. section 5 is the results and analysis. we defer related work to section 6 where we compare it with our proposed work. we draw a conclusion in section 7. ## cross-lingual information retrieval (clir) evaluation framework for e-commerce product search different from static test sets in academia, industrial search applications are dynamic as user queries and behavioral signals change with world trends. moreover, product inventory is dynamic, changes often and quickly. a previous study @xcite proposes the traditional normalized discounted cumulative gain (ndcg) for clir using all search results from the reference translation as relevance ground truth to compute ndcg for mt translation (aka ndcg-mt). however, their approach imposes a strong assumption that the top-k search results from reference translation are all relevant to the query and relevance is inversely scaled by the ranking of the results. although behavioral signals from users' clicks and purchases are useful proxy @xcite to expensive human relevance annotations, these are dynamic and change according to the product life cycle and seasonal business trends. these behavioral signals need to be updated at regular cadence to accurately represent relevance information needed to compute search metrics. we introduce a ranking-based evaluation framework through search ranking metrics using behavioral signals as a proxy to relevance information without any human annotation; to the best of our knowledge, there is no systematic study on crosslingual information retrieval for e-commerce search that neither requires ground-truth click/purchase information nor human annotated relevance data. figure 1 illustrates the test sets creation workflow for mt evaluation in e-commerce clir: we refer to these human translated queries as q src . we propose the following evaluation framework with the test sets created above to evaluate machine translation in the context of clir for e-commerce queries. ## results and analysis figure 5 and 6 in the appendix are the plots for all other language pairs. a/b testing: we have also conducted parallel online a/b testing for the following language pairs: enus-jajp, ptpt-eses, frca-enca, mlin-enin, nlnl-dede, engb-dede. for each language pair, we have deployed two fine-tuned mt systems and integrated them into the search pipeline for the designated store, and the mt system with the comprehensive improvement has higher off-line mt metrics ( +5 bleu points on average) than the baseline model. the a/b testing lasted for 4 weeks on average for all the experiments. for the high impact language pairs, the improved mt systems have seen large increases in business metrics, such as, order product sales (ops), composite contribution profit (ccp), compared to the baseline model, and have much larger positive impact on the search result quality. for the low impact language pairs, we observe much smaller or even no impact at all. overall, the a/b testing results are consistent with the mt launching impact rate results we have computed. moreover, for ptpt-eses and nlnl-dede, we also conducted another round a/b testing with the same experiment setup except using mt with specific improvement to compare with the baseline models. those two improved mt enhanced the terminology translation of 3-5% of query traffic. the results are consistent with our hypothesis that the mt with si improvement has much more impact than the mt with ci improvement. ## related work machine translation is necessary to bridge the gap between query translation and cross-lingual information retrieval @xcite . query translation a key component in large e-commerce stores, previous studies have demonstrated that better translation quality improves retrieval accuracy @xcite . queries are naturally short and search engines usually have preferred word choices and collocations based on users' query patterns @xcite . this complicates the evaluation of machine translation for cross-lingual information retrieval in the context of 'fitting in well to the search index'. while machine translation evaluation is well-studied, translation evaluation in downstream task requires more attention especially in the e-commerce cross-lingual information retrieval. traditionally, information retrieval evaluation relies on behavioral signals as ground truth to measure relevance of search results; mean reciprocal ranking (mrr), mean average precision (map), normalized discounted cumulative gain (ndcg) @xcite @xcite . previous studies in cross-lingual information retrieval (clir) evaluation relies on pre-annotated datasets that are relatively small and specific to domains outside of e-commerce; for example, the clef ehealth test sets @xcite @xcite and wikipedia cross-lingual test set @xcite . although @xcite proposed the ndcg-mt metric that leveraged on the reference translation to measure search results relevance, reliance on the ground truth data is still necessary. in pursuit of a more effective approach, we integrate clir and mt more closely and evaluate them in an endto-end task. our proposed method allows us to fully-automate the evaluation and study the impact of improving mt on clir by collecting organic queries in the target language of the e-commerce service and use reference results of these queries as a proxy to human annotation. ## conclusion in this paper, we propose an evaluation framework for mt in the e-commerce multilingual product search through ranking-based search metrics using behavioral signals as proxy relevance information without any human notation, which makes it practical to iteratively improve mt models for the search use case and evaluate them frequently offline. this framework can also be used to create cross-lingual information retrieval (clir) test sets for e-commerce at scale. we also propose a method to measure off-line the mt launching impact and and improvement impact rate on search. the former can identify the the high-impact language pairs can be prioritized with more investment in the mt improvement. these experiments can help select the most promising improvement strategy either comprehensive or specific improvement or combination of both to bring a larger impact on the search performance of a given language pair. we have experimented with the proposed evaluation framework and mt impact measuring method on 12 language pairs from 6 stores, and identified the high language pairs of different impact on search and assigned potential improvement strategies. the results are consistent with on-line a/b testing.