PANDA : Preference Adaptation for Enhancing Domain-Specific Abilities of LLM s (2024)

Abstract: While Large language models (LLMs) have demonstrated considerable capabilities across various natural language tasks, they often fall short of the performance achieved by domain-specific state-of-the-art models. One potential approach to enhance domain-specific capabilities of LLMs involves fine-tuning them using corresponding datasets. However, this method can be both resource and time-intensive, and not applicable to closed-source commercial LLMs. In this paper, we propose Preference Adaptation for Enhancing Domain-specific Abilities of LLMs (PANDA), a method designed to augment the domain-specific capabilities of LLMs by leveraging insights from the response preference of expert models without requiring fine-tuning. Our experimental results reveal that PANDA significantly enhances the domain-specific ability of LLMs on text classification and interactive decision tasks. Moreover, LLM with PANDA even outperforms the expert model that being learned on 4 tasks of ScienceWorld. This finding highlights the potential of exploring tuning-free approaches to achieve weak-to-strong generalization.

## introduction large language models (llms) have shown exceptional performance across a broad spectrum of tasks via prompting @xcite @xcite @xcite , suggesting they possess advanced general-purpose capabilities. despite this, when tested on specific tasks, the effectiveness of these general-purpose llms often falls short of specialized state-of-the-art models @xcite @xcite . therefore, improving the domain-specific abilities of generalpurpose llms remains a critical challenge. ## experiments to validate effectiveness of panda, we evaluate it on interactive decision making and text classification tasks. in all experiments, we utilize sentencebert @xcite to calculate sentence embeddings and retrieve relevant insights base on cosine similarity. we set the number of retrieved insights to 1 in scienceworld and 6 in tweeteval. all of the experiments take gpt-3.5-turbo-1106 as the llm in panda. ## conclusion we propose panda, a tuning-free method that aims to improve the task-specific capability of llms. in panda, we leverage llms to learn from preferences of expert model and form an insight pool during learning stage. at inference time, panda firstly retrieved relevant insights to current query and boost the performance of llms by adapting its preference to align with the expert model with the guidance of relevant insights. through extensive experiments on 20 tasks in sci-enceworld and 11 tasks in tweeteval, we verify that panda improve llms in most cases. notably, panda achieves the weak-to-strong generalization on 4 tasks in scienceworld based on react, which highlights the potential of exploring tuning-free approaches to achieve weak-to-strong generalization.