TRAVEL : Tag-Aware Conversational FAQ Retrieval via Reinforcement Learning (2023)

Abstract: Efficiently retrieving FAQ questions that match users’ intent is essential for online customer service. Existing methods aim to fully utilize the dynamic conversation context to enhance the semantic association between the user query and FAQ questions. However, the conversation context contains noise, e.g., users may click questions they don’t like, leading to inaccurate semantics modeling. To tackle this, we introduce tags of FAQ questions, which can help us eliminate irrelevant information. We later integrate them into a reinforcement learning framework and minimize the negative impact of irrelevant information in the dynamic conversation context. We experimentally demonstrate our efficiency and effectiveness on conversational FAQ retrieval compared to other baselines.

## introduction retrieving faq questions that match users' intent during user-system conversations is critical for online customer service in large companies (e.g., alibaba, amazon). in this paper, we call this scenario conversational faq retrieval. normally, it employs an ai assistant to model user behaviors (e.g., queries and clicks) and iteratively retrieve faq questions until meeting users' intent @xcite . to satisfy user experience @xcite , it is needed to design an efficient retrieval strategy to find faq questions that match user intent in minimal turns. current methods focus on modeling the semantic information in the conversation context. they model the semantic similarity to perform faq retrieval by concatenating user queries and clicking questions @xcite or applying attention mechanisms @xcite . although these methods show promising results, they assume that users' behaviors strictly † corresponding author. taking @xcite as an example, the user's target question is "how to cancel the automatic repayment of credit cards?". due to domain unfamiliarity, in figure.1 (c), the user clicks the question "how to cancel the automatic payment of bank cards", because it contains the same tag ("cancel") with the user's intent. unfortunately, this introduces irrelevant tags such as "automatic payment" and "bank card". as a result, the system retrieves irrelevant faq questions in subsequent turns (e.g., "how to cancel the automatic payment of the credit cards?" in the next turn). therefore, such irrelevant information makes existing systems require more turns to hit the faq question that matches the user's intent. we believe that the key to finding the right faq question in minimal turns is to reduce the impact of irrelevant tags in clicked questions. thus, we need to estimate whether a tag is irrelevant to user intent. accordingly, as shown in figure .1 (b), we assume that a tag has a high probability of being irrelevant if the user seldom clicks faq questions containing this tag. besides, as shown in figure .1 (e), the probability of a tag being irrelevant can be gradually estimated along with the dynamics of the conversation. this motivates us to utilize reinforcement learning (rl) to model the dynamic changes of the tags' irrelevance estimation. by maximizing the cumulative reward based on the estimation, the rl model learns an optimized strategy to reduce the impact of irrelevant tags and obtain the user intent in minimal turns. more specifically, we convert the conversation context into a representation that models the dynamic tags' irrelevance estimation. when taking the representation as the state, we punish the rl system when it retrieves faq questions containing irrelevant tags, and reward it when it retrieves users' desired faq questions in minimal turns. by doing so, the rl system can dynamically adjust its strategy to avoid retrieving questions that contain tags with high estimated irrelevance and achieve a successful retrieval as quickly as possible. in this way, we can effectively decrease the negative impact of irrelevant information in the conversation context, and retrieve users' desired faq questions in minimal turns. we call our method the tag-aware conversational faq retrieval via reinforcement learning (travel). to sum up, we have the following contributions: (1) for the first time, we point out the significance of reducing the impact of irrelevant information introduced by noisy user behaviors in conversational faq retrieval. (2) we propose a tag-aware reinforcement learning strategy that models the dynamic changes of the tags' irrelevance to achieve successful faq retrieval in the minimal turn. ## framework notation. we denote the collection of questionanswer pairs as f @xmath0, a 1 ), ..., (q n , a n )}, where q i is a faq question with a i as its answer. each question q i is categorized into a set of tags for a user u i , there is a question q i that matches his/her intent, which needs to be retrieved. at the turn t of a conversation, the system performs retrieval based on the conversation context. the conversation context at the turn t consists of: h t , which records the user's queries; q t click , which records the questions that user clicked; q t rej , which records the questions that user ignore; the conversation ends if the system retrieves the question q i . otherwise, it continues to retrieve until reaching the maximum turns t . ## conclusion in this paper, we focus on proposing a strategy for retrieving faq questions that match user intent in minimal turns. to achieve this goal, we introduce a tag-aware conversational faq retrieval framework via reinforcement learning called travel. this framework is designed to eliminate the detrimental impact of irrelevant information in the conversation context. it contains two main components: tag-level state representation and conversational retrieval strategy optimization. travel mitigates the negative impact of irrelevant information in the context by estimating its degree of irrelevance and employs a reinforcement learning strategy for performing faq retrieval. to verify our ideas, we create a dataset and develop a user simulator. through extensive experiments, we justify the effectiveness of the travel framework, offering valuable insights into achieving successful faq retrieval in the fewest possible turns.