Improve MT for Search with Selected Translation Memory using Search Signals (2022)

Abstract: Multilingual search is indispensable for a seamless e-commerce experience. E-commerce search engines typically support multilingual search by cascading a machine translation step before searching the index in its primary language. In practice, search query translation usually involves a translation memory matching step before machine translation. A translation memory (TM) can (i) effectively enforce terminologies for specific brands or products (ii) reduce the computation footprint and latency for synchronous translation and, (iii) fix machine translation issues that cannot be resolved easily or quickly without retraining/tuning the machine translation engine in production. In this abstract, we will propose (1) a method of improving MT query translation using such TM entries when the TM entries are only sub-strings of a customer search query, and (2) an approach to selecting TM entries using search signals that can contribute to better search results.

## introduction localization of e-commerce sites has led users to expect search engines to handle multilingual queries and return product information in customers' preferred language. multilingual product search capability is essential for modern e-commerce product discovery @xcite . recent proposals of cross-lingual information retrieval that handle multilingual queries and language-agnostic cross-border product indexing have gained traction with neural search engines @xcite @xcite @xcite , but legacy e-commerce search indices are still built on monolingual product information and support for multilingual search is bridged using machine translation (search mt) @xcite @xcite @xcite . in practice, a translation memory matching step is usually arranged before machine translation systems for search query translation. a translation memory (tm) is a database which stores the source text and its corresponding translation in language pairs that have been previously translated. for example, @xmath0, kinder @xmath1. the translation memory is usually activated when a run-time query exactly matches an entry in the memory. therefore, a translation memory can (i) reduce the computation footprint and latency for synchronous translation (ii) effectively enforce terminologies for specific brands or products. although such issues can be mitigated through terminology constraint mechanism in the machine translation model ( dinu et al. (2019); post and vilar (2018); susanto et al. (2020); wang et al. (2021); ailem et al. (2021) ), the turnover time to fix the translation would be unacceptable to the users and companies that expect an instant fix and, (iii) fix machine translation issues that cannot be resolved easily or quickly without retraining/tuning the machine translation engine in production @xcite ; @xcite ; @xcite ; @xcite ). we have also observed that many translation memory (tm) entries can partially match a large percentage of queries at run-time. it is necessary to integrate translation memory (tm) to the machine translation systems and enable a run-time query to partially match an entry in the memory, that way one query translation can come from both translation memory and machine translation systems. unlike exact string matching, one tm entry can only impact one run-time query, partial matching can allow one tm entry to impact a large number of queries, so it is crucial only to select tm entries that can bring a positive impact to the customers' shopping experience. therefore, in this paper we propose: the rest of the paper is organized as following: we will propose the method of integrating translation memory to machine translation systems enabling sub-string matching in section 2; in section 3 we will propose an approach to selecting an optimal translation memory subset using search signal; section 4 is the experiment setup and section 5 is result and analysis. we draw the conclusion in section 6. ## machine translation with selected translation memory in production this approach includes a sub-string partial matching feature that enables neural machine translation (nmt) models at run-time to recognize the longest tm entry as a sub-string, then use the sub-string tm translation to replace the mt output of that sub-string. figure 1 .illustrates this approach using a query translation example from german to english: ## translation memory (tm) subset selection using search signal partial matching enables one tm entry to impact a larger number of queries, so it is crucial only to select tm entries that can bring a positive impact to the customers' shopping experience. the search results matter from the customer's perspective and mt query translations are used as intermediate artifacts for search. we rely on customer purchasing behavior as a signal for relevance judgments to automatically estimate the search performance of mt query translations, and a tm entry is selected if the mt query translation with the tm sub-string substitution has better search performance than the default mt query translation. figure 2 illustrates our proposed translation memory subset selection workflow. for a given tm entry e, we first sample queries in the source language q src from the historical traffic data that can partially match to the entry. we then also sample hundreds of thousands of the purchased product ids p and their frequencies f associated with each source query. we will use the top two most frequent source queries q src (q @xmath2: for each source query q src , we will use mt to generate two versions of query translation, one t mt is returned from the mt and other t mt+tm is returned from mt with translation memory (tm). we will retrieve two sets of search results r mt and r mt+tm for these two versions of query translations proceedings of the 15th biennial conference of the association for machine translation in the americas orlando, usa, september 12-16, 2022. volume 2: users and providers track and government track t mt and t mt+tm respectively; then we will use the purchased product ids p q associated the source query q src as a proxy to the relevant products and logarithm of their frequencies f q as the scaled relevance for the search rank-based metrics s computation of these two sets of search results s mt and s mt+tm . the tm entry is selected if the search rank-based metric of the mt query translation with tm s mt+tm is higher than the search rank-based metric of default query mt translation s mt for both source queries. we also observe some tm entries are terminologies such as brands, and they also overlap with the common vocabulary of the source language that usually needs to be translated. for example, take the entry kinder. this word is both a brand and a common word in german meaning children; when it refers to the brand it is expected to be preserved in the german-to-english translation. therefore, if such entries exist in the tm, we suggest creating a frequent collocation kinder schokolade alone based on the query log and adding the new entry pair kinder schokolade -kinder chocolate to the translation memory before the subset selection. ## experiment we conduct both offline and online experiments for the proposed approaches for portuguese queries on amazon.es, german queries on amazon.com (us), and dutch queries on amazon.de. ## results and analysis a/b testing: we have also conducted parallel online a/b testing for the three language pairs. for each language pair, we have deployed a baseline mt and an improved mt model with selected translation memory (tm). both are integrated into the search pipeline for the designated store. the a/b testing lasted for 4 weeks on average for all the experiments. the improved mt with tm of three language pairs impacted 3-5% of query traffic, and have seen large increases in business metrics, such as, order product sales (ops), composite contribution profit (ccp), compared to the baseline mt models. moreover, they all have much larger positive impact on the search result quality, which indicates that our approach has the overall user's multilingual search experiences have received larger improvements. ## conclusion in this paper, we have proposed a method of improving mt query translation using such translation memory (tm) entries when the tm entries are only sub-strings of a customer search query, and an approach to selecting tm entries using search signals that can contribute to better search results. we have conducted both offline and online experiments for improving mt with the selected tm subset using the search signal for portuguese queries on amazon.es, german queries on amazon.com (us), and dutch queries on amazon.de. both off-line and on-line results have shown our approach can improve search query translation and have seen increased order product sales and improved user experience in the multilingual e-commerce search.