MFAQ : a Multilingual FAQ Dataset (2021)

Abstract: In this paper, we present the first multilingual FAQ dataset publicly available. We collected around 6M FAQ pairs from the web, in 21 different languages. Although this is significantly larger than existing FAQ retrieval datasets, it comes with its own challenges: duplication of content and uneven distribution of topics. We adopt a similar setup as Dense Passage Retrieval (DPR) and test various bi-encoders on this dataset. Our experiments reveal that a multilingual model based on XLM-RoBERTa achieves the best results, except for English. Lower resources languages seem to learn from one another as a multilingual model achieves a higher MRR than language-specific ones. Our qualitative analysis reveals the brittleness of the model on simple word changes. We publicly release our dataset, model, and training script.

## introduction organizations create frequently asked questions (faq) pages on their website to provide a better service to their users. faqs are also useful to automatically answer the most frequent questions on different communication channels: email, chatbot, or search bar. faq retrieval is the task of locating the right answer within a collection of candidate question and answer pairs. it is closely related to the tasks of non-factoid qa and community qa, although it has its own specificities. the total number of possible answers is generally small (the average faq page on the web has 6 answers), and only one is correct. retrieval systems cannot rely on named entities, as they are typically shared by many possible answers. for example, three out of four answers in since faq-finder @xcite , researchers applied different methods to the task of faq retrieval @xcite @xcite @xcite . however, since the advent of deep-learning and transformers, the interest has somewhat faded compared to other areas of qa @xcite . one possible explanation is the lack of a dedicated large-scale dataset. the ones available are mostly limited to english, and domain-specific. on the other hand, the task of factoid question answering received the attention of many researchers. recently, transformers encoders such as dense passage retrieval (dpr) @xcite have been successfully applied to the retrieval part of factoid qa, overcoming strong baselines such as tf-idf and bm25. however, we show that dpr's performance on passage retrieval is not directly transferable to faq retrieval. @xcite recently released paq, a dataset of 65m pairs of probably asked questions. however, answers are typically short in paq (a few words), which differs from faqs where answers are longer than questions. another way to answer users' questions is to use knowledge grounded conversation models as it does not require the pre-generation of all possible pairs of questions and answers @xcite . however, at the time of writing these models can hallucinate knowledge @xcite , which limits their attractiveness in a corporate environment. in this paper, we provide the first multilingual dataset of faqs. we collected around 6m faq pairs from the web in 21 different languages. this is significantly larger than existing datasets. however, collecting data from the web brings its own challenges: duplication of content and uneven distribution of topics. we also provide the first multilingual faq retriever. we show that models trained on all languages at once outperform monolingual models (except for english). the remainder of the paper is organized as follows. we first review the existing models and datasets available for the task of faq retrieval. we then present our own dataset and apply different models to it. we finally perform some analysis on the results and conclude. our dataset and model are available on the huggingface hub 4 . ## related work in this section, we review the existing literature on faq retrieval. we first start by reviewing available models and then look at the available datasets. ## multilingual faq dataset in this section, we introduce our new multilingual faq dataset. ## models in this section, we describe the faq retrieval models used in our experiments. let p be the set of all user queries and @xmath0, a 1 ), ..., (q n , a n )} be the set of all faq pairs for a given domain. an faq retrieval model takes as input a user's query p @xmath1, and outputs a relevance score h(p i , f j ) for f j with respect to p i . however, our dataset does not contain live user queries (or paraphrases) p , we thus use questions q as queries @xmath2, ..., q n } and restrict the faq set to the answers @xmath3, ..., a n }. the task becomes to rank the answers a according to the questions q. ## experiments in this section, we evaluate the retrieval performance of our model on mfaq. in all our experiments, we use three metrics to evaluate the performance: precision-at-one (p@1), mean reciprocal rank (mrr), and recall-at-5 (r@5). for space reasons, we only report on mrr in the main text, the full results are available in the annex. we used the same parameters for all experiments unless mentioned otherwise. 15 we insert a special @xmath4. answers are size reaches the desired size, we start over with the remaining pairs. 15 we used a batch size of 800, sequences were limited to 128 tokens (capturing the entirety of 90% of the dataset), an adam optimizer with a learning rate of 0,0001 (warmup of 1000 steps). dropout of 25%. respectively prepended @xmath5. all of our experiments use a subset of the training set: only one page per domain as this technique achieves higher results. refer to section 5.3 for more information. we start by studying the performance of multilingual models, then compare it against monolingual models. ## qualitative analysis in this section, we dive into the model's predictions and try to understand why and where it goes wrong. we do so by focusing on a single faq page from the admission center of the tepper school of business. 16 the faqs are displayed in ## future work important non-indo-european languages such as chinese, hindi, or japanese are missing from this dataset. future work is needed to improve data collection in these languages. second, we did not evaluate the model on a real-life faq retrieval dataset (with user queries). future work is needed to see if our model can perform question-to-question retrieval, or if it needs further training to do so. a linguistic study could analyze the model's strengths and weaknesses by studying the model's performance by type of questions, answers, and entities. ## conclusion in this work, we presented the first multilingual dataset of faqs publicly available. its size and breadth of languages are significantly larger than other datasets available. while language-specific bert-like models can be applied to the task of faq retrieval, we showed it is beneficial to use a multilingual model and train on all languages at once. this method of training outperforms all monolingual models, except for english. our qualitative analysis reveals our model is overly reliant on keywords to match questions and answers.