Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix (2023)

Abstract: Many real-world applications require making multiple predictions from the same text. Fine-tuning a large pre-trained language model for each downstream task causes computational burdens in the inference time due to several times of forward passes. To amortize the computational cost, freezing the language model and building lightweight models for downstream tasks based on fixed text representations are common solutions. Accordingly, how to learn fixed but general text representations that can generalize well to unseen downstream tasks becomes a challenge. Previous works have shown that the generalizability of representations can be improved by fine-tuning the pre-trained language model with some source tasks in a multi-tasking way. In this work, we propose a prefix-based method to learn the fixed text representations with source tasks. We learn a task-specific prefix for each source task independently and combine them to get the final representations. Our experimental results show that prefix-based training performs better than multi-tasking training and can update the text representations at a smaller computational cost than multi-tasking training.

## introduction fine-tuning large pre-trained language models for downstream tasks has become a popular solution in natural language processing @xcite . although effective, fine-tuning the whole language model might cause some computational burdens, especially for those applications that require making multiple predictions from the same text. for example, given a facebook post, we may want to know its topic, predict its sentiment, extract events, and decide if it contains offensive words, etc. if we train a separate model for each task, we may have a latency during inference time due to several times of forward passes, * work done while interning at meta ai. to amortize the computational cost, several works @xcite consider freezing the language model as the text encoder. they use the frozen language model to obtain the fixed representations for a text and build a lightweight model for each downstream task on top of such fixed representations. they show that by fine-tuning a pre-trained language model with some source tasks in a multi-tasking way, the generated fixed representations capture general information and generalize well for unseen target tasks. in this work, we consider the same goal and make the inference computationally efficient. we aim to learn fixed representations from some source tasks that can generalize to unseen target tasks. instead of multi-tasking training, we propose a new method based on the prefix tuning (li and @xcite to learn the fixed representations. specifically, we learn a task-specific prefix for each source task independently. during inference time, all the task-specific prefixes are combined together to produce the final fixed representations. since those prefixes carry task-specific information, the generated fixed representations capture enough information to generalize to unseen target tasks. compared to multi-tasking training, the advantage of prefix-based training is that the fixed text representations can be easily updated at a small computational cost. for example, if we want to add source tasks, we can simply train new prefixes for those tasks without re-training the whole model. similarly, if we want to remove source tasks, we can directly disable the corresponding prefixes during inference time. in contrast, multi-tasking training requires re-training the whole model, which is less flexible and computationally expensive. all the results suggest that prefix-based training can be a promising approach to this research direction. general purpose text representations. large pre-trained language models are usually used for extracting text representations @xcite @xcite . several approaches consider self-supervised learning to improve the quality of representations @xcite @xcite . to improve the generalization for unseen tasks, some works consider additional source tasks to learn the representations, such as natural language inference corpus @xcite @xcite and paraphrase pairs @xcite . recently, frozen text representations have caught attention for amortizing the computational cost in real-world applications @xcite . prefix tuning and prompt tuning. recently, prefix tuning and prompt tuning become popular ways to learn parameter-efficient models. early works usually consider discrete prompts, where the prompts consist of real words @xcite @xcite . later works study soft prompts, where the prompt words are learnable @xcite @xcite @xcite . several studies have shown that prefixes and prompts can effectively capture the key information about the tasks @xcite @xcite . our work is motivated by the fact that those learnable parameters can be viewed as embeddings for transferring knowledge or representing tasks @xcite @xcite . ## experiments we conduct experiments to show the potential of prefix-based training and its flexibility for updating text representations. ## conclusion we focus on learning fixed text representations with source tasks that can generalize well to unseen target tasks. we propose a prefix-based training method that learns a task-specific prefix for each source task. the fixed text representations are com-puted by combining all task-specific prefixes together. our experimental results show that prefixbased training performs better than multi-tasking training and can update representations at a smaller computational cost than multi-tasking training.