Frugal Prompting for Dialog Models (2023)

Abstract: The use of large language models (LLMs) in natural language processing (NLP) tasks is rapidly increasing, leading to changes in how researchers approach problems in the field. To fully utilize these models’ abilities, a better understanding of their behavior for different input protocols is required. With LLMs, users can directly interact with the models through a text-based interface to define and solve various tasks. Hence, understanding the conversational abilities of these LLMs, which may not have been specifically trained for dialog modeling, is also important. This study examines different approaches for building dialog systems using LLMs by considering various aspects of the prompt. As part of prompt tuning, we experiment with various ways of providing instructions, exemplars, current query and additional context. The research also analyzes the representations of dialog history that have the optimal usable-information density. Based on the findings, the paper suggests more compact ways of providing dialog history information while ensuring good performance and reducing model’s inference-API costs. The research contributes to a better understanding of how LLMs can be effectively used for building interactive systems.

## introduction large language models (llms) have rapidly transformed the landscape of natural language processing (nlp) research through emergent capabilities like prompt-based learning, in-context learning (icl), and conversational capabilities @xcite . while these novel approaches are being applied to various domains and tasks with an unrealistic speed and effectiveness, many dimensions of llms remain unexplored. for instance, since we do not need to follow a fixed schema for the textual inputs anymore (like standard supervised learning for text), the ways in which input-text can be presented, and its impact on task performance is an essential aspect that needs to be investigated. additionally, as various llm-inference apis are becoming available for a price, trade-off between performance gain and prompting (inference) cost is another dimension that requires attention. while efforts have been made to reduce inferencing costs of transformer @xcite models, these contributions have mostly been at the architecture level and require access to the model weights and source codes @xcite . as many models like gpt-3 @xcite , codex @xcite , lamda @xcite , and palm @xcite are now closed source, it is not possible for the end-user to optimize the model's costs using these approaches. in recent prompt engineering literature, the focus has been on optimizing the prompt to improve downstream task accuracy @xcite , with the majority of past efforts targeting single-turn tasks (e.g., classification, reading comprehension, question answering, etc.). however, for longer inputs, another critical factor is the inferencing api cost, which has largely been ignored in prior works. this is especially true for interactive or dialog tasks. this paper explores the trade-off between cost and performance for llms in a prompt-based/incontext learning (icl) setup. we propose the idea of frugal prompting in the context of dialog models, which involves input optimization methods to maintain performance gains while minimizing costs. to compare effectiveness of input representations for in-context learning based methods while considering both cost and task performance, we introduce a new metric called usable information density (uid). using this metric, we gain insights into the capabilities of various icl model families for understanding and accessing information from different input representations. overall, we make the following contributions in this paper. (1) we explore the effectiveness of various icl models and input formats for dialog modeling. (2) we propose a new metric, uid, that captures the tradeoff between accuracy and length for various (input format, icl model) combinations. (3) extensive experiments on two benchmark dialog datasets (msc and tc) and four icl models show that (a) adding more context as part of the input does not necessarily improve uid by similar amounts across all icl models. (b) for most icl models, using the most semantically related utterance from dialog history is more cost effective compared to using full history, summarized dialog history or most recent utterance. ## prompting methods for dialog systems we first present the necessary ingredients of a prompt for dialog systems. next, we discuss recipes for manual and algorithmically optimized prompts. lastly, we present ways of effectively including context information as part of prompts. ## conclusion in conclusion, this paper has explored the tradeoff between model performance and cost in interactive tasks where dialog history plays a crucial role. since recent large language models tend to produce longer dialog responses, using this long dialog history as context for next utterance prediction becomes more expensive. however, the experiments conducted in this study have demonstrated that compressing dialog history can improve model performance without significantly increasing cost. our findings suggest that the optimal representation of dialog history is one that provides the highest amount of usable information per token. summaries of dialog history are better than using full history itself. recent utterance or best semantically similar utterance are both better than summaries. one best semantically similar utterance is the best from both accuracy as well as usable information perspective. overall, our results highlight the importance of carefully balancing model performance and cost in interactive tasks that rely on dialog history.