Answer is All You Need: Instruction-following Text Embedding via Answering the Question (2024)

Abstract: This work aims to build a text embedder that can capture characteristics of texts specified by user instructions clarifying the similarity criterion. While previous methods improve general task awareness by injecting the instruction information into encoding, they fail to be sensitive to clearer criteria like “evaluate similarity based on emotion”. We instead propose a different viewpoint, which treats the instruction as a “question” about the input text and encodes the expected answers to obtain the representation accordingly. Intuitively, texts with the same (implicit) semantics would share similar answers following the instruction, thus leading to more similar representations. Specifically, we propose InBedder that instantiates this learning-to-answer idea by only fine-tuning language models via abstractive question answering tasks. Despite its simplicity, InBedder demonstrates significantly improved instruction-following capabilities according to our proposed instruction awareness tests and instruction robustness tests, when applied to language models with large language models (LLMs) (e.g., llama-2-7b) and smaller encoder-based LMs (e.g., roberta-large). Additionally, our qualitative analysis of clustering outcomes, achieved by applying diverse instructions to the same unlabeled corpus, demonstrates a high degree of interpretability in the clusters formed.

## introduction text embedders play a crucial role in large-scale textual data analysis and management. while existing models @xcite @xcite @xcite demonstrate strong effectiveness in representing texts in general, they lack the ability to address user-specific objectives. this limitation hinders their application in more intricate scenarios where the embedding task requires the model to represent particular characteristics of the texts @xcite . consider figure 1 , where a single set of reviews is required to be clustered in three distinct manners to derive meaningful insights. in response, we attempt to equip the text embedders with instruction-following capability in this paper. one straightforward solution is to embed the concatenated instruction and input. nonetheless, generic textual embeddings represent the texts in a form that can be used in textual similarity tasks, search and clustering, etc, rather than following instructions. even for those that are trained with multi-task contrastive objective @xcite , there are no guarantee to generalize to new instructions due to the inevitably restricted diversity of training instructions written by humans. we offer a novel viewpoint, which treats the instruction as a question about the input text and encodes the expected answers. specifically, using the instructed input as the prompt to generative language models, we argue that the generated answers can be natively utilized to model semantic similarity under different instructions. for instance, given the sentences "i love cats" and "i love dogs", the instruction "do they love animals?" will lead to a uniform response of "yes/certainly/..."; conversely, distinct answers would be generated in response to "what animals do they love?". therefore, we believe that the expectation of answer representations given the prompt can serve as an instructionfollowing embedding. we support this hypothesis by our empirical observations in section 4.2 on existing instruction-tuned llms @xcite @xcite foot_0 which have demonstrated that hidden states corresponding to the generated answers show considerably better instruction-awareness compared to those derived from the prompt. our observations indicate that function words and phrases in the answers do not contribute to better embedding quality. for instance, the introduc-what types of feedbacks are there? how do customers like this product? what kind of customers do we have? as a student, i'm disappointed with the poor quality of this product, which failed to meet my expectations and needs. fast delivery -a perfect choice for the white-collar professional. ## methodology in this section, we introduce inbedder that is derived from observations on llms. we first define several ways to acquire sentence embeddings from llms in section 4.1. subsequently, we illustrate early observations in section 4.2. finally, we introduce a framework that fine-tunes an llm to an instruction-following embedder, inbedder, in section 4.4. ## experiments we introduce experimental setup in section 5.1 and section 5.2. we then present results on proposed instruction awareness tests and instruction robustness tests in section 3.2 and section 5.4. finally, we show a comparison of generic embedding tasks in section 5.5. ## embedder clustering interpretation interpreting neural embeddings has long been an aspiration in numerous research endeavors @xcite . we show in this section that inbedder naturally possesses interpretability due to its instruction-following training objective. specifically, we propose a method to "extract answers" from semantic clusters produced by the embedder. ## conclusions and future work our work addresses a novel problem, text embedding with instruction-following. we propose inbedder to produce desirable embeddings from llms via generating expected answers. the method is inspired by observations on existing llms. our text embedder model llama-2-7b-inbedder outperforms both traditional sentence transformers and aggregated embeddings from llms on instruction-awareness tests, and instruction robustness tests and achieves close performance on traditional generic tasks. we also show that inbedder inherently is applicable for embedding cluster explanation which will significantly facilitate user-oriented dataset analysis. we encourage future works to investigate more efficient solutions which is important in large-scale retrieval systems.