Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptions (2025)

Abstract: With the help of in-context learning (ICL), large language models (LLMs) have achieved impressive performance across various tasks. However, the function of descriptive instructions during ICL remains under-explored. In this work, we propose an ensemble prompt framework to describe the selection criteria of multiple in-context examples, and preliminary experiments on machine translation (MT) across six translation directions confirm that this framework boosts ICL performance. But to our surprise, LLMs might not care what the descriptions actually say, and the performance gain is primarily caused by the ensemble format, since it could lead to improvement even with random descriptive nouns. We further apply this new ensemble framework on a range of commonsense, math, logical reasoning and hallucination tasks with three LLMs and achieve promising results, suggesting again that designing a proper prompt format would be much more effective and efficient than paying effort into specific descriptions.

## introduction in-context learning (icl) boosts the performance of large language models (llms) across numerous natural language processing (nlp) tasks, where llms are presented with in-context examples containing input and ground truth output @xcite . many works have verified the vital role of in-context examples in icl @xcite . however, @xcite find that ground truth labels might not be the key to icl performance on classification tasks. the selection of in-context examples has been proven significant to the performance of icl @xcite and there have been various works on in-context example selection @xcite 2023; @xcite . besides diverse approaches of selecting examples, no existing work has tried to explicitly tell llms in what way those specific examples are selected. we hypothesize that if llms are prompted with instructions describing the properties of selected in-context examples, they might learn better from these examples, since instruction following is one of llms' most important qualities nowadays @xcite @xcite . @xcite prompt llms with examples selected based on both word-level and syntax-level criteria for machine translation (mt) for better icl performance. this inspires us to tell llms where different in-context examples come from when they are selected by multiple methods. in our experiments on mt, we first select incontext examples based on lexical and syntactic similarity for each test input separately. then we combine both to construct the complete set of examples, with half word-level examples and half syntaxlevel examples. further, we devise a novel ensemble prompt framework (as shown in the left part "prompt template" of figure 1 ), adding examplelevel instructions to describe that the following examples are with similar words or similar syntax. experimental results on mt demonstrate that adding such ensemble prompt framework does improve llms' performance over conventional prompts. meanwhile, we find that when the example-level descriptions do not correspond to the source of in-context examples or are completely nonsense, llms still benefit from the prompt. these surprising results indicate that in fact llms might not care what the descriptions say and are more sensitive to the prompt format. in other words, a proper format can be much more effective than well-designed descriptions in icl. to further verify the superiority of the ensemble framework, we present empirical evaluations on commonsense, math, logical reasoning and hallucination benchmarks (including nine datasets in total) across three small-scale llms (alpaca, llama3 and mistral) and one large-scale llm (gpt-3.5). the novel prompt framework is able to achieve promising results even with the descriptive nouns in the prompt being random nouns, further suggesting that a proper prompt format would be much more effective and efficient compared with laborious design of detailed and specific descriptions. there are a few studies very related to our work. @xcite find that the labels of in-context examples do not need to be correct for classification tasks. @xcite find that larger language models learn from in-context examples even when the labels are flipped or unrelated. @xcite demonstrate that optimizing examples is less effective in some tasks given a high-quality task instruction. our work is different from the above in that we focus on the meaning of descriptions rather than labels or examples in icl and our finding is that the format of prompts is more important than carefully designed descriptions. our contributions can be summarized as follows: our code is available at https://github.com/ jamydon/format-beats-descriptions . ## prompting llms for mt primarily, we focus on mt, a typical generation task. recently, various approaches of selecting in-context examples have been proposed for mt @xcite @xcite . however, no existing work has tried to make llms aware of in what way those specific in-context examples are selected. we assume that llms would perform better when they are told the reasons for selecting those examples. @xcite select examples based on a combination of word-level and syntax-level criteria, which inspires us to present an ensemble prompt framework to make llms clearly know the reasons behind example selection. in addition, to have a comprehensive idea of whether llms really know what is said in the descriptions, we design some prompt variants that are less meaningful or completely nonsense. ## generalizing the new ensemble prompt framework to more tasks to further verify our conclusion obtained from mt that our proposed ensemble framework improves icl even when the example-level descriptions are incorrect or meaningless, we perform the comparison between vanilla and ensemble (random + random), which we would refer to as err, on more types of tasks across different language models. ## related work in-context example selection @xcite suggest that llms' icl performance strongly depends on the selection of in-context examples. in consequence, many works have been trying to explore ways of selecting better in-context examples in recent years. @xcite train a unified in-context example retriever across a wide range of tasks. @xcite select examples based on both relevance and diversity, with the help of determinantal point processes. @xcite ensure n-gram coverage to select better examples for mt. @xcite train an in-context example scorer for mt based on several features. @xcite combine both word-level and syntax-level coverage when selecting examples for mt. ## conclusion in this work, we analyze the effect of descriptive instructions in prompts during icl and propose an ensemble prompt framework describing the properties of in-context examples selected by different methods. experimental results on mt indicate that while llms are sensitive to prompt formats, they might not care the actual meaning of the descriptions and the framework improves llms' performance even with meaningless descriptions compared with the conventional prompt. we further apply the ensemble framework to four other nlp tasks and find that it achieves promising results, especially on small-scale models. these results suggest that rather than working hard on well-designed descriptions, making use of a proper prompt format would be more effective and efficient.