Evaluating Machine Translation in Cross-lingual E -Commerce Search (2022)

Abstract: Multilingual query localization is integral to modern e-commerce. While machine translation is widely used to translate e-commerce queries, evaluation of query translation in the context of the down-stream search task is overlooked. This study proposes a search ranking-based evaluation framework with an edit-distance based search metric to evaluate machine translation impact on cross-lingual information retrieval for e-commerce search query translation, The framework demonstrate evaluation of machine translation for e-commerce search at scale and the proposed metric is strongly associated with traditional machine translation and traditional search relevance-based metrics.

## introduction multilingual search capability is essential for modern e-commerce product discovery @xcite . localization of e-commerce sites have led users to expect search engines to handle multilingual queries. recent proposals of cross-lingual information retrieval handles multilingual queries and language-agnostic cross-borders product indexing have gained traction with neural search engines @xcite @xcite @xcite , but legacy e-commerce search indices are still built on monolingual product information and support for multilingual search is bridged using machine translation @xcite @xcite @xcite . machine translation (mt) is notoriously hard to evaluate manually; human evaluation is slow, expensive and inconsistent @xcite @xcite @xcite . automated machine translation evaluation metrics have evolved from simple word error rates @xcite to modern string based metrics that usually ignores the source inputs and uses a single reference @xcite @xcite . neural evaluation metrics @xcite @xcite have gain recent popularity as they attempt to incorporate human annotations and multi-references through supervised learning. although neural metrics have shown to agree more with human evaluation, they are built off language models that introduces new biases @xcite . despite the usefulness of evaluation metrics, machine translation is often used as interim application and objectives of the downstream tasks could have varying levels of tolerance of the inherent translation quality. keeping the actual utility of machine translation in mind, extrinsic task-based evaluations were developed for spoken-language systems @xcite @xcite @xcite , information extraction @xcite , automatic post-editing @xcite and domain-specific translation that requires different fidelity requirements @xcite @xcite . information retrieval evaluation usually involves human-annotated relevance labels of search results candidates. industrially, the scale of annotating a representative sample is impractical and can only serve as anecdotal evidence of search quality. as a proxy for human annotations, it is common to use behavioral signals from clicks and purchases @xcite . however, these behavioral signals pose a cold-start problem where such information is unavailable for newly established marketplaces. in this paper, we examine the evaluation of machine translation of search query in the context of cross-lingual e-commerce search. we propose: the rest of the paper is structured as follows. section 2 gives an overview of the proposed ranking-based evaluation framework and edit-distance based evaluation framework for e-commerce cross-lingual information retrieval (clir). section 3 describes the experiment setup on the test set used to evaluate machine translation (mt) models tuned on search data and the edit-distance metric hyper-parameters. section 4 presents our experiment results and analysis of the association between mt metric, traditional ndcg and the levenshtein editdistance based metric proposed in this paper. section 5 presents related work and section 6 concludes the paper. ## cross-lingual information retrieval (clir) evaluation framework for e-commerce search different from static test sets in academia, industrial search applications are dynamic as user queries and behavioral signals change with world trends. moreover, product inventory is dynamic, changes often and quickly. previous study @xcite proposes the traditional normalized discounted cumulative gain (ndcg) for clir using all search results from the reference translation as relevance ground truth to compute ndcg for mt translation (aka ndcg-mt). however, their approach imposes a strong assumption that the top-k search results from reference translation are all relevant to the query and relevance is inversely scaled by the ranking of the results. although behavioral signals from users' clicks and purchases are useful proxy @xcite to expensive human relevance annotations, these are dynamic and change according to product life cycle and seasonal business trends. these behavioral signals need to be updated at regular cadence to accurately represent relevance information needed to compute search metrics. we introduce a ranking-based evaluation framework through search ranking metrics using behavioral signals as a proxy to relevance information without any human annotation; and a novel edit-distance based framework to measure the difference in search candidate ranks between the human and machine translated queries without the need for relevance information. to our best knowledge, there is no systematic study on cross-lingual information retrieval study for e-commerce search that neither requires ground-truth click/purchase information nor human annotated relevance data. ## results and analysis as an upper-bound reference, spanish to english (es-en) and hebrew to english (he-en) achieve an ndcg@16 score of 0.60 when using the reference translation that produces the r ref search results. likewise, english to german (en-de) and polish to german models (pl-de), they achieve an ndcg@16 score of 0.62 for their respective r ref . we can use these upper-bounds to expect the possible improvements that can be made to the machine translation in the cross-lingual ir setting. for example, es-en language pair has an 0.53 ndcg score while he-en in the same marketplace scores at 0.47, we can expect that there is more room for improvement for the he-en, given that the reference translation r ref achieved a score of 0.60. juxtaposing the generic and search mt models, we expect the search models to perform better given the domain-specific tuning. the difference in machine translation performance as measured by bleu in ## related work machine translation is necessary to bridge the gap between query translation and cross-lingual information retrieval @xcite . query translation a key component in large e-commerce stores, previous studies have demonstrated that better translation quality improves retrieval accuracy @xcite . queries are naturally short and search engines usually have preferred word choices and collocations based on users' query patterns @xcite . this complicates the evaluation of machine translation for cross-lingual information retrieval in the context of 'fitting in well to the search index'. while machine translation evaluation is well-studied, translation evaluation in downstream task requires more attention esp. in the e-commerce cross-lingual information retrieval. traditionally, information retrieval evaluation relies on behavioral signals as ground truth to measure relevance of search results; mean reciprocal ranking (mrr), mean average precision (map), normalized discounted cumulative gain (ndcg) @xcite @xcite . previous studies in cross-lingual information retrieval (clir) evaluation relies on preannotated datasets that are relatively small and specific to domains outside of e-commerce; for example, the clef ehealth test sets @xcite ; @xcite ; @xcite and wikipedia cross-lingual test set @xcite . ## conclusion in this study, we introduce a framework which provides a recipe to evaluate machine translation in the context of cross-lingual e-commerce search at scale. additionally, we proposed an editdistance based metric lev@k to evaluate mt quality that bypasses the reliance on behavioral signals and/or expensive and slow relevance annotations from human. the proposed metric has shown correlations with traditional relevance-based search metric and it is also strongly associated with the classic machine translation evaluation metric. the difference between a machine translation system as measured by bleu can be demonstrated with the proposed edit-distance based metric in the context of cross-lingual search. we suggest the use of the lev@k metric in future clir researches in addition to the traditional search metrics, especially when relevance information is unavailable.