Unsupervised Text Representation Learning via Instruction-Tuning for Zero-Shot Dense Retrieval (2024)

Abstract: Dense retrieval systems are commonly used for information retrieval (IR). They rely on learning text representations through an encoder and usually require supervised modeling via labelled data which can be costly to obtain or simply unavailable. In this study, we introduce a novel unsupervised text representation learning technique via instruction-tuning the pre-trained encoder-decoder large language model (LLM) under the dual-encoder retrieval framework. We demonstrate on multiple languages that the corpus representation can be augmented by the representations of relevant synthetic queries generated by the instruct-tuned LLM founded on the Rao-Blackwell theorem. Furthermore, we effectively align the query and corpus text representation with self-instruct tuning. We evaluate our proposed method under low-resource settings on three English, two German and one Portuguese retrieval datasets measuring NDCG@10, MRR@100, Recall@100. We significantly improve the average zero-shot retrieval performance on all metrics, increasing out-of-box FLAN-T5 model variations by [4.73%, 6.15%] in absolute NDCG@10 and exceeding four supervised dense retrievers.

## introduction dense retrieval systems typically employ dualencoder retrieval models which use two separate encoders, either symmetric or asymmetric, to represent the query and corpus in any languages @xcite @xcite . the corpora are indexed with representation and will be retrieved in response to each query based on the relevance scores. the scores are usually calculated based on embedding similarity, such as dot product or cosine similarity. although dense retrieval systems have developed rapidly, the model performance largely depends supervised text representation learning and relevancy capturing between the query and corpus @xcite . yet, it remains to be a major challenge to properly retrieve when lacking labeled modeling data. existing work @xcite leveraged pre-trained large encoders (specifically t5 models, @xcite ) to alleviate the data thirst. however, their proposals still required annotated datasets either by web mining or manual annotation for fine-tuning in order to improve the generalization ability of dual-encoder retrieval models, for example, dealing with out-of-domain data. an alternative solution is to train a dense retrieval on synthetic query-corpus relevance pairs. @xcite trains a question generation system on general domain data and applies it to the targeted domain to construct synthetic questionpassage data. to save the effort of training a taskspecific generation model on general data, like natural questions @xcite or msmarco @xcite , promptagator @xcite proposes to use pre-trained large language models (llms), like flan @xcite , as a few-shot query generator to build the data for training the dual-encoder. however, the synthetic queries are not directly leveraged at inference, potentially causing gaps between training and inference of dense retrievers @xcite . earlier work, e.g., doc2query @xcite , concatenates the generated queries with the corresponding corpus, aiming to enrich the corpus representation with questions that the corpus can potentially answer. an improved version, doctttttquery @xcite leverages pre-trained t5 models as the expansion model, leading to more relevant synthetic queries and better retrieval performance. different from the previous work, we demonstrate directly on the embedding level instead of the text level, that the synthetically generated queries' embeddings can effectively augment the corpus representation (figure 1 ). here, we propose an unsupervised representation learning approach through self-instructed-tuning leveraging the embedding generation and sequence generation capability of an encoder-decoder llm. this approach consists of two steps, i.e., self-instructed-learning and rao-blackwellization. in the first step, we design two instruction tasks, namely question generation and keyword summarization, to generate synthetic questions and keywords for each given corpus via prompting a pre-trained llm. next, we apply filters to gate the synthetic data quality and instruction-tune the pre-trained llm (and its variant versions) on the filtered output (step one in figure 2 ). in the second step, we use the instruct-tuned llm to generate better synthetic questions and keywords following the same instruction prompts as in training. we then obtain the embeddings of the newly generated synthetic questions and keywords and that of corpus from the instruct-tuned encoder, and take the weighted average as our augmented corpus representation (step two in figure 2 ). we consider the corpus representation learning task as a problem of query embedding expectation estimation. based on the rao-blackwell theorem, the crude estimator, corpus embedding, can be improved by taking the conditional expectation given the sufficient statistics, i.e., sample mean of the embedding of their (synthetic) relevant queries and keywords. thus, we expect combining the raw corpus embedding and synthetic query embedding to achieve better corpus representation. besides, by aligning instruction-tuning and synthetic query generation, the retrieval model is directly optimized on corpus representation during training. to assess the effectiveness of our proposed method, we compare retrieval method of corpus only embedding with our augmented corpus representation, models with and without instruction-tuning and evaluate against four competitive dense retrievers (i.e., mdpr @xcite , mbart @xcite , t-systems (t-systems, 2020), albertina-pt @xcite ). our main contributions are as follows: ## method we propose an unsupervised text representation learning approach through self-instructed-tuning a pre-trained encoder-decoder llm. first, we generate instruction following responses from an llm and instruction-tune the llm itself with filtered quality (natural language instruction, response) pairs. next, we compute the augmented corpus embedding weighing in synthetic queries' (e.g., questions, keywords) embeddings. figure 2 presents the overall flow of our approach. ## conclusion in our research, we propose the unsupervised text representation learning technique through selfinstructed-tuning encoder-decoder llms. based on the rao-blackwell theorem, we leverage the embeddings of synthetically generated queries (i.e., questions and keywords) to augment the corpus representation for the dual-encoder retrieval framework. in zero-shot experiments, our proposed corpus representation consistently improves the performance over encoder-only corpus representation. even if the open-box llm was not pre-trained on retrieval task and there is no labeled modeling data, after fine-tuning with our approach it exceeds the sota models across different datasets, presenting the high effectiveness and data efficiency of our method in retrieval tasks. in future work, we plan to explore our proposed method on separate encoder and decoder models.