MAIR : A Massive Benchmark for Evaluating Instructed Retrieval (2024)

Abstract: Recent information retrieval (IR) models are pre-trained and instruction-tuned on massive datasets and tasks, enabling them to perform well on a wide range of tasks and potentially generalize to unseen tasks with instructions. However, existing IR benchmarks focus on a limited scope of tasks, making them insufficient for evaluating the latest IR models. In this paper, we propose MAIR (Massive Instructed Retrieval Benchmark), a heterogeneous IR benchmark that includes 126 distinct IR tasks across 6 domains, collected from existing datasets. We benchmark state-of-the-art instruction-tuned text embedding models and re-ranking models. Our experiments reveal that instruction-tuned models generally achieve superior performance compared to non-instruction-tuned models on MAIR Additionally, our results suggest that current instruction-tuned text embedding models and re-ranking models still lack effectiveness in specific long-tail tasks. MAIR is publicly available at https://github.com/sunnweiwei/Mair.

## introduction large language models (llms) have demonstrated impressive capabilities in performing a wide range of natural language processing (nlp) tasks by being first pre-trained on large-scale corpora and then instruction-tuned on numerous downstream tasks @xcite @xcite . this advancement has garnered significant attention in other fields, including information retrieval (ir) @xcite @xcite @xcite @xcite . ir techniques aim to retrieve a set of relevant candidates from a large corpus based on semantic relevance or other query-specific criteria @xcite . these techniques are critical components of many ai applications, from web search @xcite to various retrieval-augmented tasks @xcite . however, most traditional ir models, typically trained on a single task, often exhibit poor generalization to other ir tasks or domains @xcite . inspired by the success of llms, recent research explores training models for the general-purpose ir @xcite @xcite . these models, instruction-tuned on multiple retrieval tasks, show significant improvements in aligning with the user intent across different ir tasks. to evaluate the generalization capabilities of newly emerged ir models, several benchmarks such as beir @xcite , kilt @xcite , and mteb @xcite have been established recently, compiling a variety of ir tasks. however, as shown in to fill this gap, this paper introduces mair (massive instructed retrieval benchmark), a large-scale ir benchmark consisting of 126 diverse retrieval tasks with 805 distinct instructions to evaluate model generalization on unseen tasks. in mair, we collect tasks from existing ir datasets, such as those found in (i) sigir resource track papers, (ii) tasks in existing benchmarks, (iii) publicly accessible shared tasks in trec (trec.nist.gov), and (iv) recent llm benchmarks. tasks in mair are elaborately selected to cover various information retrieval requirements in practice, including diverse types of queries and document types, as well as specific relevance criteria. as a result, a mair beir kilt followir instructir (this work) thakur et al. ( 2021) petroni et al. (2020) weller et al. (2024) oh et al. (2024 total of 126 unique tasks are selected to constitute the benchmark mair. subsequently, we clean and merge the data of these tasks, and then sample them, resulting in a dataset that includes 10,038 queries and 4,274,916 documents. we perform the sampling mainly due to the data distribution imbalance. we validate that the sampled benchmark produces results highly correlated with full-scale testing, achieving a balance between evaluation accuracy and cost. finally, we manually annotate retrieval instructions based on the queries and corresponding documents for these ir tasks. following previous work @xcite , these instructions specify the types of queries, documents, and relevance criteria. we ultimately annotate 805 distinct instructions, representing 805 different querydocument-relevance combinations. some tasks include query-level instructions, meaning each query has a different relevance criterion. these annotations make mair particularly challenging and suitable for evaluating instruction-tuned retrievers in terms of their ability to follow instructions in completing unseen tasks. based on mair, we benchmark various different types of retrieval models, including (i) sparse retriever, (ii) single-task text embedding models @xcite , (iii) non-instruction-tuned multi-task text embedding models @xcite @xcite , (iv) instruction-tuned embedding models @xcite , and (v) re-ranking models @xcite . we evaluate on both no instruction and with instruction, and found instruction-tuned embedding models show clear improvement when instructions are added. gritlm-7b achieves the best overall score, with an average ndcg@10 of 55.20. furthermore, both e5-mistral-7b-instruct and gritlm-7b show notable improvement when instructions are added. ## data construction this section illustrates the process of data collection and benchmark construction. as aforementioned, the benchmark for instruction-tuned ir models should be capable of evaluating ir baselines on a variety of tasks across different domains and using as realistic instruction as possible. to this end, we construct our benchmark based on the following criteria: (a) task and domain variety: to assess the generalization of different ir baselines on various tasks from different domains, we collect data from a large range of domains and tasks. specifically, the data is collected from 126 distinct ir tasks across 6 domains. each task is manually filtered to avoid duplication. (b) instruction diversity: for each task, we annotate and review a large number of detailed instructions. these instructions can assist in thoroughly evaluating models' instruction-following capabilities when searching queries. ## dataset analysis the task correlation to measure the task diversity of mair, we calculate the similarity of two tasks using the pearson correlation coefficient of the different models' performance on two tasks. based on the results of all baseline models on the mair, we calculated the correlation between all tasks, and got a correlation matrix in @xmath0, m[i, j] denote correlation between task i and j. we plot this matrix in figure 7 . to better visualize the matrix, we use t-sne to visualize the task correlation matrix in 2d. figure 2 presents the t-sne visualization of the correlation matrix, where each task is represented as a point, and the distance between two tasks reflects their correlation. we have annotated the tasks from beir and observed that most of these tasks cluster together in the orange group, indicating that the performance of different models on these tasks is highly correlated. in contrast, tasks in mair show greater diversity, covering a larger area and thus providing a more comprehensive evaluation of the models. the specific pairwise correlations are displayed in the heatmap in figure 7 , revealing that many tasks exhibit negative correlations. this suggests that these tasks have significantly different definitions, leading models that perform well on one task to perform poorly on another. ## conclusion in this paper, we introduce a novel massive instructed ir benchmark called mair for evaluating instruction-tuned retrieval models. compared with existing related benchmarks, mair has a more comprehensive coverage of various ir tasks, with different types of queries, documents and relevance criteria. specifically, mair comprises 126 distinct ir tasks, with 805 newly annotated instructions. all tasks and instructions are manually selected and annotated to ensure the benchmark can assess the instruction-following capabilities of different ir models in practice. based on mair, we benchmark various retrieval models, including both instruction-tuned and noninstruction-tuned models. our results demonstrate that mair poses greater challenges than existing benchmarks, and provides a more comprehensive evaluation.