O rchestra LLM : Efficient Orchestration of Language Models for Dialogue State Tracking (2024)

Abstract: Large language models (LLMs) have revolutionized the landscape of Natural Language Processing, but are computationally expensive. To reduce the cost without sacrificing performance, previous studies have explored various approaches to harness the potential of Smaller Language Models (SLMs) as cost-effective alternatives to their larger counterparts. Driven by findings that SLMs and LLMs exhibit complementary strengths in a structured knowledge extraction task, this work presents a novel SLM/LLM routing framework designed to improve computational efficiency and enhance task performance. In dialogue state tracking tasks, the proposed routing framework enhances performance substantially compared to relying solely on LLMs, while reducing the computational costs by over 50%.

## introduction large language models (llms) have become versatile tools capable of tackling a wide range of tasks with only a few training examples. however, their expanding sizes have brought escalating computational demands. in contrast, more efficient smaller language models (slms) often require a substantial amount of fine-tuning data to become truly effective. this work addresses scenarios where only limited task-specific data is available, making fine-tuned slms less dependable. our objective is to develop a routing framework that orchestrates slms and llms, enhancing task performance while reducing computational costs. task-oriented dialogue is crucial for efficient human-computer interaction, enabling systems to understand and assist with specific tasks like booking flights or scheduling meetings. task-oriented dialogues involving structured data typically rely on dialogue state tracking (dst), where user intent is extracted from the dialogue history between a user and the agent in the form of slot values associated with a predefined schema. fine-tuned slms have been used in dst for a few years, including both autoregressive lms @xcite @xcite and sequence-to-sequence lms @xcite @xcite @xcite . llms have been used for few-shot in-context learning in dst @xcite @xcite where llms are prompted with human-authored task descriptions or in-context exemplars. in our work, we seek to take advantage of the effectiveness of llms with a small amount of training data but reduce the cost. strategies that leverage both slms and llms have been developed to mitigate the computational demands of llms. cascade-based approaches direct a query to an llm when it cannot be resolved by an slm @xcite . these approaches introduce latency and computational redundancy since they consistently query slms. other approaches use binary classifiers to predict the most appropriate lm to utilize @xcite . a limitation of the classifier-based approaches is the necessity for retraining when introducing new models. in this work, we propose a dynamic routing framework, orchestrallm (illustrated in figure 1 ), that leverages small (fine-tuned) and large lm experts. hypothesizing that examples with similar semantic embeddings are of the same difficulty level, we select an appropriate expert based on embedding distances between the testing instance and instances in expert pools. the expert pools contain examples representing the types of dialogue contexts where the different lms provide more reliable answers. after retrieving the top k nearest examples, an expert is selected based on the majority vote. unlike cascade-based and classifierbased approaches, the proposed framework eliminates the need for router training, though handlabeled data is needed for creating the expert pools. in addition, the retriever can be fine-tuned with target task labels or expert information to achieve more efficient and accurate routing. in summary, the key contribution of this work is the introduction of a novel switching model designed to reduce the computational costs associated with llms while simultaneously enhancing performance. experimental results on two different multi-domain dst benchmarks (multi-woz @xcite and sgd @xcite ) demonstrate that orchestrallm capitalizes on the proficiencies of different experts, outperforming llm systems while also achieving a substantial reduction of over 50% in computational costs. ## dialogue state tracking in this work, we focus on combining generalpurpose llms and task-specific slms to achieve better efficiency for dialogue state tracking (dst). in the following, we first provide the necessary task setups and then detail the two representative dst models using llms and slms respectively. a task-oriented dialogue (tod) consists of a sequence of exchanges between two parties, each of which is initialized by the user and followed by a response from the system. here, we denote each exchange as a turn leading to a sequence, u 1 , a 1 , ..., u t , a t , where u t and a t represent the user utterance and the system response, respectively. for the t-th turn, the user provides a new utterance u t , and the system agent responds with utterance a t . at turn t, the corresponding dialogue context is c @xmath0, a 1 , . . . , a t-1 , u t }, which excludes the latest system response a t . the goal of dst is to extract task-relevant information as structured representations (dialogue states) from user-system utterances so that the user requests can be fulfilled accordingly. to facilitate this, there is typically a task-specific schema provided. in a multi-domain scenario considered in this paper, the schema contains m domains @xmath1, . . . , d m } and n slots @xmath2, . . . , s n } to track. dst t , the dialogue state at turn t, defines the current mappings from pairs of (d m , s n ) into a value v based on dialogue context c t . specifically, in the literature, task-specific slm-based dst models are typically fine-tuned with full-parameter updates while dst models using llms are realized via few-shot in-context learning. we discuss the two different dst models considered below. llm dst. ic-dst @xcite is an incontext learning (icl) framework that enables fewshot dst with llms. the prediction is the change in each turn pair instead of the accumulated dialogue states. to obtain the accumulated dialogue states, the turn changes are aggregated across turns. dialogue states of previous turns are used as a summary of the context history and it allows to fit in more exemplars which is crucial for icl performance. concretely, given the schema table, k incontext exemplars, dialogue states of the previous turn, and the input instance (most recent agent-user utterance pair), the llm outputs ## routing approach here, we present our approach for routing with orchestrallm applied to the dst task. the overall framework is illustrated in figure 1 . we denote different dst models as experts. given a new input instance (the triplet (dst t-1 , a t-1 , u t )), orchestrallm first computes its semantic embedding, compares it with exemplar embeddings of triplets from each expert pool using a cosine distance, and retrieves the top-k exemplars. the router assigns the input to an expert based on majority vote. while our approach draws inspiration from the work of @xcite , it is important to note that their approach primarily focuses on optimizing task performance in zero-shot task transfer scenarios, whereas our emphasis lies in improving computational efficiency within the fewshot learning settings. ## conclusion we introduce orchestrallm, a routing framework that seamlessly integrates a slm and a llm, orchestrated by a retrieval-based router. during inference, a dynamic router guides instances to either lm based on their semantic embedding distances with the retrieved lm exemplars, leveraging the expertise of both slm and llm. our evaluation on dst demonstrates that orchestrallm outperforms llm-based systems while also achieving computational cost savings of over 50%. this research represents a significant step towards efficient collaboration of language models, particularly in a multi-turn human-computer interaction system such as task-oriented dialogue.