One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2023)

Abstract: We introduce INSTRUCTOR, a new method for computing text embeddings given task instructions: every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). Unlike encoders from prior work that are more specialized, INSTRUCTOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. We first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture with a contrastive loss. We evaluate INSTRUCTOR on 70 embedding evaluation tasks (66 of which are unseen during training), ranging from classification and information retrieval to semantic textual similarity and text generation evaluation. INSTRUCTOR, while having an order of magnitude fewer parameters than the previous best model, achieves state-of-the-art performance, with an average improvement of 3.4% compared to the previous best results on the 70 diverse datasets. Our analysis suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets. Our model, code, and data are available at https://instructor-embedding.github.io .

## introduction text embeddings represent discrete text inputs (e.g., sentences, documents, and code) as fixed-sized vectors that can be used in many downstream tasks. these tasks include semantic textual similarity @xcite @xcite , information retrieval @xcite @xcite , automatic text evaluation @xcite @xcite , prompt retrieval for in-context learning @xcite @xcite , and beyond. recently, we have seen dramatic advances in learning text embeddings @xcite @xcite @xcite @xcite that perform well on their intended tasks or datasets. however, most existing embeddings can have significantly degraded performance when applied to new tasks or domains @xcite . for example, dpr @xcite is stronger for retrieval than text similarity tasks, and vice versa for simcse @xcite . moreover, existing embeddings usually perform poorly when applied to the same type of task but in different domains such as medicine and finance. a common method to address this issue is to further finetune the embeddings on datasets in downstream tasks and domains, which often requires a lot of annotated data (guru- ## instructor instructor encodes inputs together with task instructions, thereby providing task-specific representations that can be used for many downstream language tasks, without any additional training. here we introduce the architecture of instruc-tor ( §2.1), present how we perform multitask instruction-based finetuning ( §2.2), and describe how we collect and annotate the medi training data ( §2.3). by default, we refer "task" to a dataset, and use them interchangeably throughout the paper, while a "task category", such as retrieval, includes many tasks. ## experiments we train instructor on the medi data and evaluate it on a wide range of 70 downstream tasks. specifically, we use the mteb benchmark from recent work @xcite , which consists of 56 datasets over 7 diverse task categories, such as classification, reranking, and information retrieval. we then further apply instructor to prompt retrieval for in-context learning and text generation evaluation. in all three settings, in-structor achieves the state-of-the-art performance. see appendix §a and §b for our detailed settings. ## analysis and ablations we demonstrate instructor enables universal text embeddings for many diverse tasks. here we analyze our results from various perspectives: the importance of instructions ( §4.1), instruction robustness ( §4.2) and complexity ( §4.3), model sizes ( §4.4), domain shifts ( §4.5), and qualitative analysis ( §4.6). by default, we report average performance across all categories. ## conclusion we introduced instructor, a single model that creates broadly-applicable text embeddings using natural language instructions. we constructed medi, a collection of diverse datasets, to finetune instructor with instructions. our extensive experiments showed that instructor achieves state-of-the-art performance on text embedding benchmarks, as well as prompt retrieval for few-shot in-context learning. we hope that researchers and practitioners will benefit from our embeddings or our datasets for tasks of their interest.