Generate-then-Retrieve: Intent-Aware FAQ Retrieval in Product Search (2023)

Abstract: Frequently Asked Question (FAQ) retrieval aims at retrieving question-answer pairs for a given a user query. Integrating FAQ retrieval with product search can not only empower users to make more informed purchase decisions, but also enhance user retention through efficient post-purchase support. Providing FAQ content without disrupting user’s shopping experience poses challenges on deciding when and how to show FAQ results. Our proposed intent-aware FAQ retrieval consists of (1) an intent classifier that predicts whether the query is looking for an FAQ; (2) a reformulation model that rewrites query into a natural question. Offline evaluation demonstrates that our approach improves 12% in Hit@1 on retrieving ground-truth FAQs, while reducing latency by 95% compared to baseline systems. These improvements are further validated by real user feedback, where more than 99% of users consider FAQs displayed on top of product search results is helpful. Overall, our findings show promising directions for integrating FAQ retrieval into product search at scale.

## introduction product search engines help users find relevant products across large product catalogues and generate sales revenue for e-commerce companies @xcite . while such engines are primarily designed to handle keyword searches for products, customer behavior has been changing with an increase in users asking information-seeking service or product related questions @xcite . however, most product search engines are not effective at handling non-product search related queries (e.g., "return a package"). providing correct answers to these frequently asked questions (faqs) @xcite is essential to provide a positive pre-and post-purchase experience, which can lead to improved user retention and trust. product search and faq retrieval are typically powered by independent retrieval systems. this separation is often due to the challenges in combining multiple answering sources (e.g. product details and faqs) into a holistic retrieval application @xcite . furthermore, determining what answering source can satisfy the user's information need is challenging to perform at scale. hence, e-commerce websites tend to isolate faq search functionality from product search. for example, apple offers vertical search where users are required to navigate among different tabs (e.g., product search, support, store location). 1 such designs require users to navigate multiple links, which can lead to increased user effort and unsatisfactory shopping experiences @xcite @xcite . therefore, we propose to integrate faq retrieval into a product search engine, so that users can search products and access faqs seamlessly from an unified search interface. a potential solution is aggregated search, which refers to the task of searching and assembling information from a variety of sources and presenting them in a single unified interface @xcite @xcite . the main challenge here lies in determining when and how information from multiple verticals should be presented effectively and efficiently. when to show faq results? query intent is inherently ambiguous @xcite @xcite . figure 1 illustrates an example where the users can use the same query "apple tv bluetooth" to retrieve products, or find information about bluetooth connectivity. in the latter case, the query is intended to express the question, "how do i connect a bluetooth device to my apple tv?". we define a query that can be answered by an faq entry as having question intent. it is important to note that a query with question intent may also have product search intent, as queries can be inherently ambiguous. determining when a user query can be answered by either product or faq search is tightly coupled with predicting the user's information need. due to query ambiguity, displaying answers from faq sources for all searches causes high false positive rates, due to the lack of question intent by the user. our analysis (cf. §5.1) from a leading ecommerce site shows that if faqs would be shown to all queries, 98% of the faqs would be irrelevant to user's needs. as users mostly use product search for shopping, injecting faq results that are irrelevant or not needed causes significant friction in the user experience. furthermore, performing faq retrieval for every query is inefficient @xcite since only a small portion of traffic has question intent @xcite . 2 while we cannot disclose the intent distribution of our data for reasons of confidentiality, question intent represent a minor portion of the overall query traffic. to address the problems above, we train an intent classifier that distinguishes when a query has question intent, and thus, can be answered by an faq source. in mixed retrieval scenarios, this allows us to trigger faq retrieval and show faq results only for queries with question intent, causing less friction for users. in terms of efficiency, running faq retrieval only on question intent queries significantly improves latency. our experiments validate that deploying intent classifier brings a 95% latency reduction compared to baselines without the intent classifier. lastly, we demonstrate that existing techniques such as upsampling are enough to achieve satisfactory performance in classifying question intent in imbalanced traffic. how to show faq results? when a search query has question intent, an interface that jointly displays product search and faq retrieval results is required. as in prior work on aggregating web search results @xcite , we integrate the top-1 faq result alongside product search results, as illustrated in figure 1 , for the following two reasons. first, since product search is the core functionality of e-commerce search engines, majority of the space is dedicated to the ranked product list. if we consider additional modalities such as mobile search, space constraints are even greater. displaying more faq results comes at the cost of reducing the number of product results, which can lead to reduced revenue @xcite . second, compared to product search, where users are required to compare multiple options, question intents generally require less exploration since users already have a specific request in their mind. given the above reasons, we need to optimize the faq retrieval system for high precision at the top ranks (i.e., hit@1). queries are usually short and consists of several keywords. to achieve a high precision for faq retrieval, we propose to rewrite a query with question intent into a more specific natural language question. this rewriting process aims to make the queries semantically and syntactically more similar to the questions found in faqs than the original queries, inspired by previous studies of query reformulation @xcite @xcite . our experiments validate that through query reformulation we can achieve significantly higher accuracy in retrieving ground-truth question at first rank (hit@1) with more than 13% improvement when compared to using original queries for faq retrieval. contributions we summarize our contributions in this paper as follows: inspired by earlier keyword-to-question generation approaches, we utilize the state-of-the-art generation models to reformulate keyword queries into questions for faq retrieval. experiments show that our reformulation model trained from human annotated query-question pairs significantly improves hit@1 by 13% compared to using original query. ## method our intent-aware faq retrieval approach, shown in figure 2 , consists of two main components: (1) an intent classifier that takes a user query as input and determines whether it has question intent and can be answered by an faq; (2) a query reformulation model which reformulates queries with question intent into a natural language question that is used for faq retrieval. regardless of query intent, the product search is always performed. foot_2 ## experimental setup in this section, we first discuss datasets used, followed by implementation details of retrieval baselines, experimental settings and evaluation metrics. ## experiments and results we study the following research questions: rq1: how much does using the question intent classifier as a filtering step benefit the integrated faq retrieval pipeline? rq2: how effective is query-to-question reformulation for faq retrieval? rq3: how efficient is our intent-aware faq retrieval approach when integrated with a product search engine? ## online deployment our intent-aware faq retrieval solution is deployed and integrated into the product search interface of 9 https://aws.amazon.com/ec2/instance-types/ 10 the raw inference time cannot be disclosed. a leading global e-commerce website. the live version uses a much larger faq corpus than the ones used here. when faq results are displayed, we collect optional explicit user feedback on whether the answer is helpful or not, as demonstrated in figure 1 . over one month of traffic was collected from the us marketplace, showing that 71% of the rendered faq results received explicit positive customer feedback. the feedback results were aggregated at the query level, allowing each query to receive multiple positive feedback responses. these findings demonstrate that our practical solution is not only effective in offline evaluation, but also helpful to real users. ## conclusion we proposed to integrate faq retrieval with product search to address challenges in aggregated search from an e-commerce perspective. our approach first classifies queries with question intent, which then reformulates into natural language questions that are used to retrieve faqs. offline experimental results show that on the best-performed faq retrieval system (i.e., bert-rerank (top 10)), the proposed intent classifier saves a substantial amount of inference costs (95%) and also improved retrieval performance through query reformulation by 13% on hit@1. these improvements are also reflected in online evaluation: over one month of user feedback demonstrated that about 71% of the rendered faq results were considered to be helpful. overall, the findings in this work suggest promising directions for e-commerce platforms to support faq retrieval at scale without disrupting customer's shopping experience.