Unlocking Markets: A Multilingual Benchmark to Cross-Market Question Answering (2024)

Abstract: Users post numerous product-related questions on e-commerce platforms, affecting their purchase decisions. Product-related question answering (PQA) entails utilizing product-related resources to provide precise responses to users. We propose a novel task of Multilingual Cross-market Product-based Question Answering (MCPQA) and define the task as providing answers to product-related questions in a main marketplace by utilizing information from another resource-rich auxiliary marketplace in a multilingual context. We introduce a large-scale dataset comprising over 7 million questions from 17 marketplaces across 11 languages. We then perform automatic translation on the Electronics category of our dataset, naming it as McMarket. We focus on two subtasks: review-based answer generation and product-related question ranking. For each subtask, we label a subset of McMarket using an LLM and further evaluate the quality of the annotations via human assessment. We then conduct experiments to benchmark our dataset, using models ranging from traditional lexical models to LLMs in both single-market and cross-market scenarios across McMarket and the corresponding LLM subset. Results show that incorporating cross-market information significantly enhances performance in both tasks.

## introduction online shoppers on platforms such as amazon post numerous questions related to specific products every day @xcite . productrelated question answering (pqa) involves providing accurate and informative responses to these questions. by leveraging product-related information, such as reviews and product meta information, responses to product-related questions can be expanded, offering enhanced depth and authenticity for potential customers @xcite . the recent success in cross-market pqa underscores the capability to effectively leverage rele-figure 1 : an example of enhancing product-related qa using cross-market data. ① depicts generating answers with cross-market reviews. ② depicts ranking-related cross-market questions to find the answer. we, therefore, propose a novel task of multilingual cross-market product-based question answering (mcpqa). we define this task as producing the answer to a product-related question in an original marketplace, using information sourced from an auxiliary marketplace with richer resources, within a multilingual setting. to this end, our initial goal is to address the following research question rq1: in a multilingual context, how can we utilize an auxiliary marketplace to enhance question-answering in the main marketplace by leveraging product-related resources (i.e., questions, reviews)? to answer rq1, we propose the first large-scale mcpqa dataset, covering 17 different marketplaces (including the us auxiliary marketplace and 16 main marketplaces) across 11 different languages from real amazon product qa sources. specifically, our dataset consists of over 7 million product-related questions with a total of 52 million product reviews. different from existing pqa datasets, more diverse information is provided in the dataset, exploring the possible answers with both questions and reviews available. additionally, we perform automatic translation on the electronics category of the dataset, naming it mcmarket. we then perform comprehensive data analysis on mcmarket to address rq1. we demonstrate a notable increase in the percentage of reviewanswerable questions across all marketplaces, with support from the auxiliary us marketplace. given the recent success of large language models (llms) in nlp tasks @xcite , their potential application to the mcpqa task prompts our second research question rq2: can llms benefit the dataset construction in the mcpqa task? delving into rq2, on mcmarket, we create a subset by randomly selecting some questions from each marketplace and perform gpt-4 auto-labeling. specifically, we focus on two widely-studied pqa subtasks under the multilingual cross-market settings, including reviewbased answer generation (ag) @xcite and product-related question ranking (qr) @xcite . for ag, we ask llms to judge whether a question is answerable from associated reviews and provide its corresponding answer. we denote the subset as mcmarket r . for qr, given two qa pairs, we ask llms to judge if one helps answer the other and denote the subset as mcmarket q . with these two subsets, we conduct human assessment to analyze llm-generated results from multiple perspectives. surprisingly, in mcmarket r , 61.8% llm-generated answers are assumed 'better' than the human ground truth. finally, we are interested in answering the research question rq3: in the multilingual context, how can we effectively leverage the unique features of cross-market information to enhance product-related question answering? to this end, we perform experiments of models on ag and qr subtasks. for each task, we report the performance of state-of-the-art methods under singleand cross-market scenarios on both mcmarket and the corresponding llm-labeled subset. we benchmark methods ranging from traditional lexical models (i.e., bm25) to llm-based approaches (i.e., llama-2, flan-t5). we demonstrate the superiority of cross-market methods against their singlemarket counterparts on both subtasks. in conclusion, our contributions are as follows: we also collect two llm-annotated subsets and adopt human assessment to analyze their characteristics. cross-domain and cross-lingual qa. our work can be seen as a special format of cross-domain qa in e-commerce, which involves addressing questions that span different domains or fields of knowledge @xcite @xcite @xcite . for instance, @xcite propose a general framework that effectively applies the shared knowledge from a domain with abundant resources to a domain with limited resources. also, cross-domain qa is often in close connection to cross-lingual qa in the sense that both involve transferring knowledge and understanding from one domain or language to another @xcite @xcite . @xcite expand the scope of open-retrieval question answering to a cross-lingual setting, allowing questions in one language to be answered using contents from another language. most recently, @xcite introduce a multilingual pqa dataset called xpqa where cross-market information is also leveraged to aid the product-based question answering. ## problem formulation we investigate two subtasks of the mcpqa task, review-based answer generation (ag) and productrelated question ranking (qr), where answers to a product question are obtained by a generative or ranking way, respectively. review-based answer generation. in this task, we assume that the answer can be obtained from the reviews of the product (or similar products). based on the setting in @xcite , we define this task in a multilingual cross-market scenario. given a question q in the main marketplace m t , we first retrieve and rank all the related reviews from similar items within both m t and auxiliary marketplace m a . given the retrieved review set ω = {r 1 , ..., r k }, we predict if q is answerable from it by assigning a tag t. if yes, a generative function γ is learned: @xmath0, ω), so that answer a is generated with both q and ω as input. product-related question ranking. following the problem setting in @xcite , we assume that there are similar questions already asked about the product or similar products in other marketplaces. therefore, given a main marketplace in language l m , denoted as m t , which usually suffers from resource scarcity of the number of knowledgeable users answers, m t consists of several items {i 1 , ..., i m }, where each i k contains a set of question answering pairs {qa k1 , ...qa kn }. besides, there also exists a high-resource marketplace m a , denoted as the auxiliary marketplace (the us marketplace in our case) in language l a (note that in some cases l a can be the same as l m ). similarly, m a also includes several items {i ′ 1 , ..., i ′ z }, where we can assume @xmath1. the task is defined as, for a given question q in the main marketplace m t , in a multilingual setting, we rank the questions from both m t and m a to take the corresponding answers of the top ranks as the possible answer to q. ## data collection & analysis we describe how we collect our dataset and perform several analysis to answer rq1 and rq2. ## conclusions we propose the task of multilingual cross-market product-based question answering (mcpqa). we hypothesize that product-related information from a resource-rich marketplace can be leveraged to enhance the qa in a resource-scarce marketplace. to facilitate the research, we then propose a largescale dataset, covering over 7 million questions across 17 marketplaces and 11 languages. additionally, we perform automatic translation for the electronics category, labeling it as mcmarket. we also provide llm-labeled subsets on mcmarket for each of the two tasks, namely mcmarket r and mcmarket q . specifically, we focus on two different tasks: ag and qr. we conduct experiments to compare the performance of models under single/cross-market scenarios on both datasets.