Leveraging Latent Topic Information to Improve Product Machine Translation (2023)

Abstract: Meeting the expectations of e-commerce customers involves offering a seamless online shopping experience in their preferred language. To achieve this, modern e-commerce platforms rely on machine translation systems to provide multilingual product information on a large scale. However, maintaining high-quality machine translation that can keep up with the ever-expanding volume of product data remains an open challenge for industrial machine translation systems. In this context, topical clustering emerges as a valuable approach, leveraging latent signals and interpretable textual patterns to potentially enhance translation quality and facilitate industry-scale translation data discovery. This paper proposes two innovative methods: topic-based data selection and topic-signal augmentation, both utilizing latent topic clusters to improve the quality of machine translation in e-commerce. Furthermore, we present a data discovery workflow that utilizes topic clusters to effectively manage the growing multilingual product catalogs, addressing the challenges posed by their expansion.

## introduction with the advent of localized e-commerce sites, customers can now shop in their preferred language. modern e-commerce platforms provide multi-lingual product discovery @xcite @xcite @xcite with machine translated product information, titles, descriptions, and bullet points @xcite @xcite . as e-commerce product catalogs continue to expand, the task of maintaining up-to-date machine translation systems poses significant challenges. when the vast amount of product information is sourced from various sellers or suppliers, each can present the data differently. consequently, this can lead to inconsistencies in the source data and, in turn, translation inaccuracies. validating such a substantial volume of data for mt training at scale becomes increasingly difficult and time-consuming, demanding significant resources for manual review and error correction to guarantee the accurate interpretation of product information. from data usage perspective, topic words extracted from the latent topic clusters in the training data can be mapped to topics that can help with word sense disambiguation and improve overall performance of mt. therefore, in this study, we propose two approaches to leverage latent topical clusters to improve machine translation quality: (i) topic-based data selection and (ii) topic-signal augmentation. both approaches use dirichlet mixture model (dmm) @xcite with collapsed gibbs sampling (cogs) @xcite to cluster large volumes of data efficiently, and the number of the clusters can be inferred automatically. the topic-based data selection approach first distinguishes between clusters of clean desirable data and those of noisy undesirable data based on the inspection of textual patterns, then selects training data from the desirable clusters for mt training. the topic-signal augmentation approach extends training data with extracted latent topic words prefixed to the source input and augments mt training with additional contextual information. additionally, we propose a data discovery workflow to cluster training data and generates cluster summary and data visualization to uncover the latent topics and textual patterns, it can also identify new noisy data patterns so that strategies can be devised to prevent the occurrence of such data in the future. ## related work previous studies have successfully used topic models to improve statistical machine translation @xcite @xcite and neural machine translation @xcite . @xcite integrates topic models as feature functions in the phrase-tables to improve statistical machine translation for e-commerce domain adaption. @xcite presents an approach using topic models to increase the likelihood of word selection from the same topic as the source context. instead of explicitly affecting the parameters or vocabulary selection, in this paper, we utilize a topical cluster model for data selection, and context augmentation implicitly adapting the mt model to the latent topic information. ## topical clustering we use dirichlet multinomial mixture (dmm) @xcite and collapsed gibbs sampling (cogs) @xcite for topical clustering. dmm and cogs are efficient clustering algorithms capitalizing on symbolic text representation, making them ideal to cluster industry scale e-commerce data based on textual patterns. moreover, the number of topic clusters is automatically inferred to adequately capture both frequent and rare textual patterns. we use the dmm model to label each document (input text) with one topic tag. dmm is a probabilistic generative model for documents and embodies two assumptions about the generative process: first, the documents are generated by a mixture model; second, there is oneto-one correspondence between mixture components and clusters. when generating document d, dmm first selects a mixture component (topic cluster) k according to the mixture weights (weights of clusters) p (@xmath0. then document d is generated by the selected mixture component (cluster) from distribution p (d|@xmath1. we can characterize the likelihood of document d with the sum of the total probability over all mixture components: ## topic-based data selection as figure 1 shown, the data selection approach first clusters large volume of the training data. empirically, larger clusters can capture the major topical and textual patterns so they are usually the clean desirable data whereas the smaller clusters can capture smaller and rare textual patterns so they are likely to be the noisy undesirable data. additionally, we can also distinguish between desirable and undesirable data based on the data inspection of the clusters, we will further discuss the data discovery and inspection process in section 8. finally, only clusters of desirable data are chosen for training to improve mt. data providers are also informed of the undesirable data patterns for future data quality control. 5 topic-signal augmentation figure 2 presents the topic-signal augmentation approach. we first cluster the data, then extract the most frequent top-k content words as the topic words for each cluster. then, we choose the larger clusters and prefix the source texts with the top-k topic words as topic signal as shown in figure 4 , we choose larger clusters are usually have more clear and interpret-able topic words, which can provide clear topic signals. finally we extend the original training data with the training data with topic signal before training the mt model in figure 3 . figure 3: use training data extended with topic words to further augment the mt training figure 4: extend the source text of the training data with top-k (@xmath2 we experiment on four language pairs, english-chinese (enus-zhcn), english-arabic (enus-arae), english-german (engb-dede), spanish-english (eses-engb). we train the models on a large volume of in-house generic training data and a subset of productinformation data (product titles, descriptions and bulletpoints) for domain adaptation. we use the transformer-based architecture @xcite with 20 encoder and 2 decoder layers with the sockeye mt toolkit @xcite to train a generic mt using generic data and domain-specific data, then fine-tune the model on the domain-specific product information data for domain adaptation. for each language pair, we have three test data sets for product titles, descriptions and bulletpoints respectively. each test data set has 2000 test segments and evaluate the models using bleu foot_1 and chrf @xcite to assess the translation quality. for the topic clusters, the source text is lower-cased, tokenized and stemmed using nltk toolkit @xcite , stemmed tokens with document frequency less than or equals to 2 are removed in the preprocessing steps. for all 4 language pairs, the initial upper-bound number of topical clusters is set to 500 for enus-zhcn and enus-arae, and 1000 for engb-dede and eses-engb. the number of the topic clusters is inferred automatically during the collapsed gibbs sampling process. the number of iterations is set to 30, and both hyper-parameters α and β are set to 0.1. we create 2-d plots using jensen-shannon distance @xcite and multi-dimensional scaling technique @xcite with ldavis @xcite to easily visualize the size and relations of the topic clusters returned from the algorithm, and to inspect the topic words extracted from the clusters. 1 shows the statistics of the resulting clusters for each language pair. the number of the total clusters is automatically inferred by the algorithm. each segments in the training data is assigned cluster ids, and data selection uses a surprisingly simple heuristic-based human inspection of the data clusters. we distinguish between major and minor clusters by the number of segments assigned to each cluster. for example, enus-arae clusters that contain more than 1000 segments are considered as major and selected as the training data for the model training; 1.32% of the training data with less than 1000 segments per cluster were dropped from the training data after the selection process. to yield a similar size data to the other three language pairs, we lower the major cluster threshold to 50 for the enus-zhcn. the other language pairs have their major cluster threshold set at 1000. by removing minor clusters, 0.07%-1.32% data from the training data are removed. based on our inspection, we observe that the major clusters contain mostly the desirable data that captures the e-commerce themes, and the top-k words in the major clusters are intuitively good topic signals to improve machine translation. meanwhile, we observe that the minor clusters capture undesirable various textual patterns that include noisy data. therefore, we select only the data from the major clusters for our experiments. section 8 will discuss further details on the data inspection process with an analysis for enus-arae translations.