TEL e R : A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks (2023)

Abstract: While LLMs have shown great success in understanding and generating text in traditional conversational settings, their potential for performing ill-defined complex tasks is largely under-studied and yet to be benchmarked. However, conducting such benchmarking studies is challenging because of the large variations in LLMs’ performance when different prompt types/styles are used and different degrees of detail are provided in the prompts. To address this issue, this paper proposes a general taxonomy that can be used to design prompts with specific properties in order to perform a wide range of complex tasks. This taxonomy will allow future benchmarking studies to report the specific categories of prompts used as part of the study, enabling meaningful comparisons across different studies. Also, by establishing a common standard through this taxonomy, researchers will be able to draw more accurate conclusions about LLMs’ performance on a specific complex task.

## introduction recently, conversational large language models (llms) such as gpt-3 @xcite , bard @xcite , llama @xcite , bloom @xcite , palm @xcite , etc. have demonstrated exceptional performance in a wide range of popular natural language processing (nlp) tasks @xcite @xcite . prompt, as a stimulator, refers to a textual input provided to the llms with the intention of guiding its output toward a specific task. unsurprisingly, the quality and effectiveness of the prompt can greatly influence the performance of the llms for a particular task, and therefore, designing appropriate prompts with the right amount of detail has become more important than ever @xcite . in recent years, researchers have spent a significant amount of effort proposing different ways of designing "appropriate" prompts. for example, @xcite showed a standard prompting technique with question-answer pairs that can result in a few-shot effect. researchers also explored other prompt design techniques such as chain-ofthought (cot) @xcite , reasoning and acting (react) @xcite , and other techniques @xcite @xcite in terms of improving the reasoning and acting of llms in solving question-answering tasks. meanwhile, @xcite proposed a prompting scheme where the agent recursively criticizes and improves its output (rci) to solve a task. however, these experiments primarily emphasized the utilization of diverse prompts to evaluate the ability of llms to perform "well-defined" nlp tasks, while studies with diverse prompts for ill-defined complex tasks are still rare, if not nonexistent. while conducting multiple benchmarking studies with various llms for complex tasks seems interesting and compelling, conducting such studies is challenging because of the large variations in llms' performance when different prompt types/styles are used and different degrees of detail are provided in the prompts, especially in case of complex tasks. in this paper, we exclusively focus on understanding llms' potential for performing complex tasks that are mostly: 1) ill-defined, 2) abstract goal-oriented, 3) highly dependent on subjective interpretation, and 4) very hard to evaluate quantitatively @xcite . these complex tasks often involve multiple steps/sub-tasks, and designing "appropriate" prompts for such tasks is indeed challenging as there is no single rule book to follow in these cases @xcite . a further complication arises if we want to compare two independent benchmarking studies targeted towards the same goal (complex) task. such a complication arises because, for a given complex task and a particular llm, the performance of the llm can drastically vary when different types/styles of prompts are fed to it. indeed, the exact details included in the prompt play a big role in how llms will perform in solving the goal complex task. this indeed creates a problem for evaluation and benchmarking purposes if an apple-to-apple comparison is not made in terms of the prompts that are provided to the llms. in other words, just reporting accuracy numbers for llms without specifying the finer details of the prompts used in the experiments makes comparisons across llms meaningless. unfortunately, every complex task is different, and so are the prompts users can try to perform the task; therefore, a general taxonomy that can categorize these diverse kinds of prompts using a single standard/taxonomy has now become a pressing need. the main contribution of this paper is to introduce one such general taxonomy (we name it teler) that can be used by any benchmarking study that leverages llms to perform some complex task. the major benefit of adopting our proposed teler taxonomy is that it will facilitate more meaningful comparisons among multiple llms in terms of their performances across various complex tasks reported by multiple independent groups of researchers/developers and, thus, help derive more accurate conclusions. teler will achieve this goal by grounding different types of prompts into a common standard and allowing an apple-to-apple comparison across different prompt categories using the same standard. as such, this taxonomy will serve as an important tool to establish a common consensus around the state-ofthe-art llm performance for performing complex tasks. prompt engineering becomes even more critical and challenging in case of performing complex tasks @xcite with llms, as complex tasks usually involve multiple steps/sub-tasks requiring higher levels of semantic understanding, planning, reasoning, and generation of natural language @xcite . these tasks often require the model to go beyond simple pattern recognition or retrieval of information and involve simulating more advanced cognitive abilities. in fact, differences in prompts along several key factors can have a significant impact on the accuracy and performance of llms in complex tasks. below, we list those key factors of prompt designing. ## proposed teler taxonomy our proposed taxonomy is based on the key factors we discussed in section 2, whose variations can lead to different outcomes while using llms to perform complex tasks. to begin with, we represent each prompt as a combination of directive and data. assuming data is fixed for a given goal task, the difference between two prompts essen-tially originates from the specifics/details of directives they include. more specifically, we propose to categorize llm prompts for complex tasks along the following four dimensions. because we used the following four factors, i.e., turn, expression, level of details and role, to define our taxonomy, we name it as teler. the overall taxonomy is pictorially depicted in figure 1 . ## two example use cases in this section, we present two interesting use cases of the proposed teler taxonomy that involve llms for performing a complex task: 1) generating a meta-review from peer-reviewer comments on a scholarly work, and 2) combining multiple alternative narratives into a single braided one. ## final words in this paper, we emphasize the importance of a standardized taxonomy for llm prompts targeted towards solving complex tasks and, subsequently, propose such a general taxonomy, i.e., teler, which can serve as a unified standard for comparing and benchmarking llms' performances reported by multiple independent research studies. we urge the community to use the teler taxonomy for designing prompts in their future work and report the specific categories of prompts they experimented with in their manuscripts. this will enable more meaningful comparisons among llms and, thereby, help to derive more accurate conclusions from multiple independent studies. this, in turn, will help the community to reach a consensus on state-of-the-art llm performances more accurately and faster than otherwise. ## limitations the proposed teler taxonomy is exclusively applicable to llm prompts targeted toward solving a complex task. this taxonomy, especially the seven levels, does not apply to simple tasks. as such, teler taxonomy will be most useful to researchers and developers who are conducting applied llm research and development that is focused on performing complex tasks. also, the teler taxonomy should not be considered as an ultimate taxonomy for llm prompts, and further extensions of this taxonomy are certainly possible and actually desired. having said that, the teler taxonomy is actually very general and can be easily extended by adding more dimensions for categorization as deemed necessary by the target application.