Unsupervised FAQ Retrieval with Question Generation and BERT (2020)

Abstract: We focus on the task of Frequently Asked Questions (FAQ) retrieval. A given user query can be matched against the questions and/or the answers in the FAQ. We present a fully unsupervised method that exploits the FAQ pairs to train two BERT models. The two models match user queries to FAQ answers and questions, respectively. We alleviate the missing labeled data of the latter by automatically generating high-quality question paraphrases. We show that our model is on par and even outperforms supervised models on existing datasets.

## introduction many websites and online communities publish faq to help their users find relevant answers to common questions. an faq consists of pairs of questions and answers {(q, a)}. the faq retrieval task involves ranking {(q, a)} pairs for a given user query q. 1 searching over faq can leverage multifield indexing and retrieval @xcite . hence, a user query q may be matched with either the question field q, the answer field a or the concatenated field q+a @xcite . the association of questions to answers in the faq pairs, can be utilized as weak supervision, for training neural models to predict the similarity between user queries and answers (i.e., q-to-a matching) @xcite @xcite . however, faq pairs by themselves do not provide the required labeled data for training a model to predict the association between user queries and faq questions (i.e., q-to-q matching). thus, a labeled dataset with user queries q and their matching {(q, a)} pairs is required for supervised learning @xcite @xcite . such a dataset is usually manually generated or obtained from query-log mining. yet, the construction of such a dataset either requires domain expertise (e.g., enriching the dataset with manually generated question paraphrases @xcite ) or assumes the availability of query-logs @xcite . whenever such a dataset is unavailable, one must resort to utilizing unsupervised retrieval models for q-to-q matching. previous unsupervised faq retrieval models @xcite @xcite @xcite have utilized so far "traditional" information retrieval techniques, such as lexical and semantic text matching, query expansion, etc. in this paper we overcome the aforementioned unsupervised gap, by using distant supervision to train neural models. our method is composed of a combination of three unsupervised methods. each method is utilized for re-ranking an initial pool of faq pairs obtained by a simple bm25 retrieval @xcite . the first method applies a focused-retrieval approach, utilizing passages for answer re-ranking @xcite . each one of the two other methods fine-tunes a bert model @xcite , one for matching q-to-a and one for matching q-to-q. to overcome the lack of training data in the latter's case, we further implement a novel weaksupervision approach using automatically generated question paraphrases, coupled with smart filtering to ensure high-quality paraphrases. we then combine the outcome of the three methods using an unsupervised late-fusion method. overall, we show that our unsupervised faq retrieval approach is on par and sometimes even outperforms state-ofthe-art supervised models. ## related work several previous works have also utilized deep neural networks (dnn) for faq retrieval. @xcite used convolution neural networks (cnn) for matching user queries to faq. @xcite used combinations of long short-term memory (lstm) to capture qto-q and q-to-a similarities. yet, those works are supervised and use user queries (q) for training. following the success of bert @xcite in nlp tasks, @xcite have recently used a search engine for q-to-q matching and then combined its results with a supervised bert model for q-to-a matching. we use a similar bert model for q-to-a matching, but differently from @xcite , we use it in an unsupervised way, and we further introduce a second unsupervised bert model for q-to-q matching. a somewhat related area of research is community question answering (cqa) @xcite and the related trec tracks. 23 while cqa shares some common features to faq retrieval, in cqa there are additional signals such as votes on questions and answers, or the association of user-answer and user-question. clearly, in a pure faq retrieval setting, such auxiliary data is unavailable. hence, we refrain from comparing with such works. ## unsupervised faq retrieval approach our proposed faq retrieval approach uses distant supervision to train neural models and is based on an initial candidates retrieval followed by a reranking step. recall that, the faq dataset is composed of {(q, a)} pairs. the initial candidate retrieval is based on indexing {(q, a)} pairs into a search engine index (section 3.1) and searching against the index. the re-ranking step combines three unsupervised re-rankers. the first one (section 3.2) is based on a focused-retrieval approach, utilizing passages for answer re-scoring. the two other rerankers fine-tune two independent bert models. the first bert model (section 3.3), inspired by @xcite , is fine-tuned to match questions (q) to answers (a). at run time, given a user query q, this model re-ranks top-k {(q, a)} candidate pairs by matching the user query q to the answers (a) only. the second bert model (section 3.4) is designed to match user queries to faq questions. here, we utilize weak-supervision for generating high quality question paraphrases from the faq pairs. the bert model is fine-tuned on the questions and their generated paraphrases. at run time, given a user query q, this model gets the topk {(q, a)} candidate pairs and re-ranks them by matching the user query q to the questions (q) only. the final re-ranking is obtained by combining the three re-rankers using an unsupervised latefusion step (section 3.5). the components of our method are described in the rest of this section. ## summary and conclusions we presented a fully unsupervised method for faq retrieval. the method is based on an initial retrieval of faq candidates followed by three rerankers. the first one is based on an ir passage retrieval approach, and the others two are independent bert models that are fine-tuned to predict query-to-answer and query-to-question matching. we showed that we can overcome the "unsupervised gap" by generating high-quality question paraphrases and use them to fine-tune the query-toquestion bert model. we experimentally showed that our unsupervised method is on par and sometimes even outperforms existing supervised methods.