x PQA : Cross-Lingual Product Question Answering in 12 Languages (2023)

Abstract: Product Question Answering (PQA) systems are key in e-commerce applications as they provide responses to customers’ questions as they shop for products. While existing work on PQA focuses mainly on English, in practice there is need to support multiple customer languages while leveraging product information available in English. To study this practical industrial task, we present xPQA, a large-scale annotated cross-lingual PQA dataset in 12 languages, and report results in (1) candidate ranking, to select the best English candidate containing the information to answer a non-English question; and (2) answer generation, to generate a natural-sounding non-English answer based on the selected English candidate. We evaluate various approaches involving machine translation at runtime or offline, leveraging multilingual pre-trained LMs, and including or excluding xPQA training data. We find that in-domain data is essential as cross-lingual rankers trained on other domains perform poorly on the PQA task, and that translation-based approaches are most effective for candidate ranking while multilingual finetuning works best for answer generation. Still, there remains a significant performance gap between the English and the cross-lingual test sets.

## introduction product question answering (pqa) is a key technology in e-commerce applications. given a question about a product, a pqa system searches the product webpage and provides an instant answer, so that customers do not need to traverse the page by themselves or seek help from humans @xcite या कपड़ा 'सला ह ु आ 'मले गा (will the cloth be available stitched) oomph! women's unstitched georgette salwar suit dupatta material -navy blue नह1ं । यह कपडा 'सला ह ु आ नह1ं 'मले गा। (no. this cloth will not be available stitched.) answer: figure 1 : cross-lingual pqa: the user asks questions about a product in their language (such as hindi), then the system searches for product information in english and generates an answer in the same language as the question. to address this, we present xpqa, the first largescale dataset for cross-lingual pqa enabling non-english questions to be answered from english content. most comprehensive product information is usually available in a majority language such as english. therefore, searching for relevant information in english often has a better chance of finding an answer. 2 this paper explores how to effectively train systems that retrieve information from english and generate answers in the question language to allow users to ask questions in any language. fig 1 shows an example. most existing multilingual qa datasets are created by translating english questions, introduc-ing translation artifacts and discrepencies from native speakers' real information-seeking behaviors @xcite . instead, we collect questions from the original market places as written by native speakers, hire bilingual annotators to check the relevant product information and write the final answers in their target languages. this eliminates the need for translations and ensures that the information-seeking behaviors of native speakers are accurately represented. based on the collected dataset, we report baseline results on two subtasks: (a) candidate ranking, which selects the best english candidate that contains the information to answer the non-english question; (b) answer generation, which generates a natural-sounding non-english answer to present to the user based on the selected english candidate. we find that applying a cross-lingual ranker trained on a wikipedia-based qa dataset generalizes poorly to the product domain. the performance is even worse than training a multilingual ranker on the english in-domain data, suggesting that domain transferability is even more crucial than language transferability. the translation-based approach is the most effective for candidate ranking while the multilingual-finetuning works the best for answer generation. nonetheless, on both tasks, there is a substantial gap between the english-based and cross-lingual performances. in the following, we first elaborate on the problem formulation for the cross-lingual pqa task ( §2), then explain the xpqa data collection process ( §3), and present experiment results ( §5.2) and conclusions ( §6). ## problem formulation task there are two important tasks for a crosslingual pqa system: candidate ranking and answer generation. in candidate ranking, given a question in a target language and a list of candidates in english, the ranker predicts a relevance score for every candidate and selects the top one. candidate ranking is necessary because a given product webpage may contain hundreds of information pieces about the product, so as a practical matter we select the top candidate to use in generation. after getting the top candidate, an answer generator takes it as input together with the question and produces an answer in the question language. this step is crucial in order to deploy a user-friendly pqa system since the candidate is neither in the user language nor written specifically to answer the question. ## xpqa dataset collection to train and evaluate our two tasks, the xpqa dataset contains annotations for (1) questioncandidate relevance to label whether every candidate is relevant to the question or not, and (2) answers where a natural-sounding answer is manually written if the candidate contains enough information to address the question. the collection process follows the steps below: ## approaches for each task, we experiment with three types of baseline approaches: translate-test, translatetrain, and multilingual @xcite . fig 2 provides a summary of these approaches. translate-test the essential idea here is to rely exclusively on english-centric models and datasets. in the zero-shot scenario, models are trained on the epqa dataset. in the fine-tuned scenario, we must translate questions and answers in the xpqa dataset into english as this is an english-centric model. this translated dataset, termed xpqa_mt is used to further fine-tune the zero-shot models. at runtime, we use an external machine translation model to translate the question into english and apply the ranker to select the best candidate. afterwards, an english-based generator produces an answer in english, which is then post-translated to the target language. translate-test is a common approach in industry as it uses well-trained english-based models and off-the-shelf translation tools without further modifications. however, such a pipelined process introduces runtime latency and can lead to error propagation if translation quality is not perfect. translate-train in contrast to the above, here we apply all translation processes in training, or offline, so that no additional latency is added at runtime. in the zero-shot scenario, we machine-translate all questions and answers in the epqa dataset into each of the 12 languages we consider. the resulting dataset, termed epqa_mt, is used to train a multilingual model. in the fine-tuned scenario, we further finetune the model on the xpqa dataset. as the model is defined to be multilingual, it can directly take input questions in their original languages and output answers in the target languages without any translation process. multilingual finally, this approach is similar to the translate-train one in that both use multilingual models rather than an english-only model, but the difference is that the multilingual approach requires no translations at training time. in the zeroshot scenario, it trains a multilingual pretrained model directly on the english-only epqa dataset and relies only on its own pretrained multilingual knowledge to adapt to other languages. in the finetuned scenario, we further fine-tune the model on the xpqa dataset. note that this approach still requires runtime post-translation of the generated english answer into the target language. this is because we find that multilingual models can only generate english answers when trained only on english datasets. although special decoding constraints could be use to restrict output vocabulary to that of the target language, zero-shot multilingual adaptation in generation tasks is still an open challenge @xcite . it is worth mentioning that the three types of approaches can be combined. for example, we could follow the translate-train approach to train the candidate ranker and follow the multilingual approach to train the answer generator. details of the model implementation are in appendix c. ## conclusion this paper presents xpqa, a dataset for crosslingual pqa supporting non-english questions to be answered from english product content. we report baseline results and findings for three approaches: translate-test, multilingual, and translatetrain. experiments show that the translate-test approach performs the best for the candidate ranking task while the translate-train approach performs the best for the answer generation task. however, there remains significant room for improvement relative to an english-based monolingual pqa system. we hope that future research can benefit from our work to improve cross-lingual pqa systems.