INTERS : Unlocking the Power of Large Language Models in Search with Instruction Tuning (2024)

Abstract: Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Despite this, their application to information retrieval (IR) tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. While prompt-based methods can provide task descriptions to LLMs, they often fall short in facilitating a comprehensive understanding and execution of IR tasks, thereby limiting LLMs’ applicability. To address this gap, in this work, we explore the potential of instruction tuning to enhance LLMs’ proficiency in IR tasks. We introduce a novel instruction tuning dataset, INTERS, encompassing 20 tasks across three fundamental IR categories: query understanding, document understanding, and query-document relationship understanding. The data are derived from 43 distinct datasets with manually written templates. Our empirical results reveal that INTERS significantly boosts the performance of various publicly available LLMs, such as LLaMA, Mistral, and Falcon, in IR tasks. Furthermore, we conduct extensive experiments to analyze the effects of instruction design, template diversity, few-shot demonstrations, and the volume of instructions on performance. We make our dataset and the fine-tuned models publicly accessible at https://github.com/DaoD/INTERS.

## introduction large language models (llms) have shown remarkable capabilities across various natural language processing (nlp) tasks. while these models have learned vast knowledge from large text corpora, their (pre-)training objective is not aligned with human's objective: the latter requires models to "follow human instructions and perform tasks" rather than "predict the next token". to address this mismatch, instruction tuning is proposed, serving as an effective technique to align llms with human tasks and preferences @xcite @xcite @xcite . after instruction tuning, llms can better understand users' intent and show impressive generalization to new tasks. in the area of information retrieval (ir), the introduction of llms has also led to notable developments @xcite @xcite . due to the high cost of fine-tuning, many existing studies leverage prompting methods to apply llms in ir tasks. however, some of them have reported that llms cannot consistently outperform fine-tuned smaller models in this manner @xcite . for example, rankgpt @xcite based on gpt-3.5-turbo underperforms monobert with 340m parameters on passage ranking tasks. this discrepancy may stem from the complexity of irspecific concepts like queries, relevance, and search intent, which are infrequently encountered in pretraining corpora and are inherently challenging to comprehend. to fill the gap, in this work, we build a novel instruction tuning dataset for search (inters). 1 this dataset is designed to specifically enhance the search capabilities of llms. since the search process involves various tasks, we choose to focus on three pivotal aspects: query understanding, document understanding, and the understanding of the relationship between queries and documents. we collect 43 datasets covering 20 distinct tasks, ensuring the dataset's comprehensive coverage and richness. in order to improve diversity and broaden applicability, we manually craft 12 unique templates for each dataset and consider both zero-shot and few-shot examples in data generation. to further improve the models' generalizability, we also manually write a detailed task description for each task, which serves as a bridge to connect each dataset under the same task. finally, we plan to release all data, templates, model checkpoints, experimental results, and source codes. we wish the openness of our dataset could support more future research and development in this field. we conduct experiments by fine-tuning several open-sourced llms using the inters dataset. experimental results show that inters consistently enhances the performance of llms of different sizes across a spectrum of search tasks. notably, this improvement is observed not only in tasks that are directly learned in the training data (in-domain) but also in tasks that are unseen in the training set @xcite . our further experiments highlight several key insights: (1) customized templates and task descriptions effectively improve model performance; (2) the diversity of templates can enhance model generalizability; (3) instruction tuning specifically tailored for search tasks addresses the existing gap of nlp instructions for such tasks; (4) combining instruction tuning and few-shot prompting can further improve performance; and (5) the substantial data volume can benefit the efficacy of instruction tuning. different from existing studies, our research focuses on using instruction tuning to improve the overall performance of llms on various search tasks. this involves enhancing the models' abilities to interpret and respond to search-related instructions more effectively, thereby improving their utility in complex ir scenarios. instruction tuning for llms instruction tuning (it) aims at fine-tuning pre-trained llms on a collection of formatted instances in the form of natural language @xcite @xcite . after it, llms can better follow instructions and perform human tasks. this approach bears a close resemblance to supervised fine-tuning @xcite and multitask prompt training @xcite . instruction tuning's efficacy lies in its ability to not only enhance llms' performance on tasks they have been directly trained on but also to equip them with the ability to generalize to new, unseen tasks @xcite . in this work, we leverage it to specifically enhance llms' performance on search-related tasks. our dataset is designed with a deep understanding of the task characteristics. experiments will show that it is also an effective way to improve llms' overall performance on search tasks. ## instruction tuning for search instruction tuning has proven to be effective for llms in responding to instructions. this method essentially involves training llms through supervised learning to execute particular tasks based on provided instructions. a notable benefit of this approach is that, after fine-tuning, llms can compre- ## experiments we fine-tune several open-sourced llms on our inters, and evaluate their performance in different settings. our experiments will investigate the following research questions: (1) can the model obtain the capability to solve search tasks through instruction tuning on inters? ( §4.2) (2) is this capability generalizable? ( §4.3) (3) is our instruction design effective? ( §4.4.1) (4) are there any advantages compared to existing instruction sets? ( §4.4.2) (5) is the model still effective with fewshot demonstrations? ( §4.4.3) (6) what is the impact of data volume? ( §4.4.4) (7) what are the effects of different ranking strategies? ( §4.4.5) ## conclusion in this paper, we investigated the application of instruction tuning to augment the capabilities of llms in performing search tasks. our instruction tuning dataset inters demonstrated its effectiveness in consistently enhancing the performance of various open-sourced llms across both in-domain and out-of-domain settings. our extensive experiments delved into several critical aspects, including the structure and design of instructions, the effects of few-shot learning, and the significance of data volumes in instruction tuning. it is our aspiration that this paper will serve as a catalyst for further research in the realm of llms, particularly in their application to ir tasks, and will encourage continued exploration into the optimization of instructionbased methods for enhancing the performance of these models.