SSP : Self-Supervised Post-training for Conversational Search (2023)

Abstract: Conversational search has been regarded as the next-generation search paradigm. Constrained by data scarcity, most existing methods distill the well-trained ad-hoc retriever to the conversational retriever. However, these methods, which usually initialize parameters by query reformulation to discover contextualized dependency, have trouble in understanding the dialogue structure information and struggle with contextual semantic vanishing. In this paper, we propose {pasted macro ‘FULLMODEL’} ({pasted macro ‘MODEL’}) which is a new post-training paradigm with three self-supervised tasks to efficiently initialize the conversational search model to enhance the dialogue structure and contextual semantic understanding. Furthermore, the {pasted macro ‘MODEL’} can be plugged into most of the existing conversational models to boost their performance. To verify the effectiveness of our proposed method, we apply the conversational encoder post-trained by {pasted macro ‘MODEL’} on the conversational search task using two benchmark datasets: CAsT-19 and CAsT-20.Extensive experiments that our {pasted macro ‘MODEL’} can boost the performance of several existing conversational search methods. Our source code is available at https://github.com/morecry/SSP .

## introduction the past years have witnessed the fast progress of the ad-hoc search @xcite @xcite . however, when it confronts more complicated information needs, the traditional ad-hoc search seems to be less competent. recently, researchers proposes conversational search which is the combination of the search engine and the conversational assistant @xcite @xcite @xcite . different from the keyword-based search: tell me about throat cancer. search: tell me about lung cancer. ## related work conversational search has become a hop research topic in recent years. trec conversational assistant track (cast) competition @xcite , which holds the benchmark largely promotes the progress of conversational search. in the beginning, researchers simply view conversational search as the query reformulation problem. they suppose that if a context-dependent query could be rewritten to a de-contextualized query based on historical queries, then it directly uses the well-trained ad-hoc retriever to obtain retrieval results. transformer++ @xcite fine-tunes the gpt-2 on query reformulation dataset canard @xcite to rewrite query. queryrewriter @xcite exploits large amounts of ad-hoc search sessions to build a weak-supervision query reformulation data generator, then these automatically generated data is used to fine-tune the language model. however, these methods underestimate the value of context, which contains various latent search intentions and topic information. after that, the conversational dense retriever is proposed. it straightly encodes full conversation whose last query denotes the user's real search intention to dense representation. convdr @xcite forces the contextual representation to mimic the reformulation query representation based on the teacher-student framework, which slightly deals with the conversational search data scarcity problem. further, coted @xcite points out that not all queries in context are useful and devises a curriculum denoising method to inhibit the influence of unnecessary contextual queries. these dense methods additionally perform the warm-up on the other domain dataset to initialize the parameters based on their own objective. however, their warm-up ignore the conversation structure information, which is crucial for capturing the relationship between utterances and understanding the search intention of the user. in this respect, we devise a novel self-supervised post-training (ssp) to replace the warm-up as figure 2 . ad-hoc retriever pre-training ssp (ours) conv. encoder finetune warm-up hardly capturing the dialogue structure. warm-up simply initializing parameter. ## problem formulation we assume that there is a multi-turn search conversation @xmath0, q 2 , . . . , q n }, where q @xmath1,1 , x i,2 , . . . , x i,l i } represents the i-th question in the conversation and x i,j is the j-th token in q i . the last query q n is the user's real search intention. we insert special tokens in this section, we propose our self-supervised post-training, abbreviated as ssp. an overview of ssp is shown in figure 3 , which consists of three self-supervised tasks: after jointly training the conversational encoder using these three self-supervised tasks, we finetune the encoder to the conversational search downstream task using the existing conversational search methods. ## conclusion in this work, we propose a novel self-supervised post-training framework ssp for conversational search, which could easily be applied to existing methods and boost their performance. different from the conventional warm-up method, our proposed ssp introduces three self-supervised tasks to