Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection (2025)

Abstract: Recent advances on instruction fine-tuning have led to the development of various prompting techniques for large language models, such as explicit reasoning steps. However, the success of techniques depends on various parameters, such as the task, language model, and context provided. Finding an effective prompt is, therefore, often a trial-and-error process. Most existing approaches to automatic prompting aim to optimize individual techniques instead of compositions of techniques and their dependence on the input. To fill this gap, we propose an adaptive prompting approach that predicts the optimal prompt composition ad-hoc for a given input. We apply our approach to social bias detection, a highly context-dependent task that requires semantic understanding. We evaluate it with three large language models on three datasets, comparing compositions to individual techniques and other baselines. The results underline the importance of finding an effective prompt composition. Our approach robustly ensures high detection performance, and is best in several settings. Moreover, first experiments on other tasks support its generalizability.

## introduction the development of instruction-tuned large language models (llms) has led to an increased interest in prompting techniques to augment inputs @xcite . whereas prompts for earlier generative language models consisted only of the input text and special tokens @xcite , they may now encompass complex and explicit task instructions with ancillary information. among others, successful prompting techniques include personas @xcite , in-context demonstrations @xcite , and reasoning steps @xcite . the success and applicability of prompting techniques does, however, depend on several parameters, including the target task, the language model figure 1 : exemplary excerpt of a prompt composition for social bias detection. while certain techniques might benefit detection performance (say, those with green squares), others might not (red squares). some parts always need to be present (blue squares). a full prompt example is shown in figure 10 . and its size, and the context provided @xcite @xcite @xcite . finding an effective prompt is, therefore, often still a timeconsuming process that needs re-evaluation should any of the parameters change. recent automated prompting methods either address the lexical aspect by finding the best formulation @xcite or manipulate the latent space @xcite . most automatic methods, however, focus on optimizing a single technique and do not consider a composition of techniques that could take advantage of their individual strengths. the example in figure 1 visualizes a prompt composition of several techniques. while a definition provides a theoretical background, in-context demonstrations clarify its application. as recent works indicate that llms cannot attend equally to all available information @xcite , simply using more techniques may add noise and reduce performance. finding an optimal prompt composition is, therefore, desirable. to this end, we propose an adaptive prompting approach to predict the optimal composition of discrete prompt techniques ad-hoc, i.e., for each input instance. first, an encoder model learns to predict optimal compositions based on a pool of individual prompting techniques. consecutively, the approach predicts the best composition for each instance and prompts the llm accordingly. we evaluate adaptive prompting for five techniques and their compositions for the task of social bias detection. the task requires semantic understanding and world knowledge @xcite , likely benefitting from using multiple prompting techniques, making it a good candidate to evaluate prompt compositions. to better understand the importance of each technique and their second-order interactions, we further conduct a shapley interaction analysis @xcite . we test our adaptive prompting approach on three social bias datasets for three open-weight llms, namely mistral (7b) @xcite , command-r (35b) @xcite , and llama 3 (70b) @xcite . we compare to baselines within and across datasets, including a fine-tuned model and a composition ensemble. the results suggest that our approach robustly ensures high classification performance; in many cases, it even outperforms all baselines and fixed compositions. moreover, follow-up experiments on three other nlp tasks stress the generalizability of our approach beyond social bias. to summarize, our main contributions are: among existing approaches to automatic prompt optimization, continuous prompt optimization learns to adjust the latent space @xcite , whereas several studies generate discrete optimized instructions from task examples, to easier adapt to unseen data @xcite @xcite or new llms @xcite . other works focus on iteratively optimizing prompts @xcite @xcite or predicting the suitability of prompts @xcite . instead of optimizing the latent space or single techniques, we propose to automatically find optimal prompt compositions for unseen inputs. related to the idea of prompt compositions, @xcite propose a framework to optimize the use of multiple techniques by training a parameterized model that automatically optimizes the prompt. we do not optimize techniques but learn to predict the optimal composition of techniques for a given setting. we further analyze the importance of each technique using shapley values. while several other studies aim to detect social bias in text corpora @xcite @xcite , we aim to identify bias in single text instances, similar to @xcite ; @xcite ; @xcite , by optimizing prompt compositions. detecting social bias reliably requires understanding of social language and pragmatics to interpret the implications of text @xcite . @xcite figure 2 : the three steps of our adaptive prompting approach: (1) bias labels are collected for all considered prompt compositions. (2) a model is trained on the collected labels to predict the optimal composition for any given text. ## approach we evaluate instruction-tuned llms to identify social bias using compositions of prompting techniques. specifically, we propose a three-step approach to predict the optimal composition ad-hoc per input text. the process is illustrated in figure 2 . the first step is to create prompt compositions that represent order-sensitive combinations of one or more prompting techniques. we prompt an llm with each of these compositions to collect social bias classification labels. in the second step, we use the collected labels to train an adaptive prompting model that finds the optimal prompt composition for a given text instance. aside from the main task, we also conduct a shapley value analysis to determine the importance of prompting techniques. we then apply adaptive prompting to unknown texts to find the optimal prompt composition for each. ## experiments in this section, we detail the adaptive prompting experiments that we carried out for the task of social bias detection on three datasets with three stateof-the-art instruction-tuned llms. we selected five common prompting techniques that fit the task, which we combine to create prompt compositions. we evaluate our approach against own and relatedwork baselines, both within and across datasets. ## results and discussion figure 3 shows the results of the prompt composition evaluation on stereoset (see appendix d for results on sbic and cobraframes). in the following, we highlight and discuss findings indicating the benefit of prompt compositions. furthermore, we find that instance-specific compositions can perform better than any single composition alone. command-r (35b) llama 3 (70b) 0 0.2 0.4 0.6 0.8 1 mistral (7b) macro f1-score (over all seeds) stereo set majority baseline (.400) random baseline (.497) deberta fine-tuned (.781) worst test (.483) base (.575) oracle (.998) best val (.806) ensemble (.785) best test (.817) ours stereoset (.853) ours cobra (.774) ours sbic (.684) worst test (.629) ours cobra (.737) base (.711) ours stereoset (.809) ours sbic (.675) ensemble (.737) best test (.800) best val (.791) worst test (.407) base (.462) ours stereoset (.781) ours cobra (.632) oracle (.966) oracle (.960) ensemble (.629) ours sbic (.603) best test (.706) best val (.706) self-diagnosis baseline (.286) figure 3: social bias detection results on stereoset (others in appendix d: figures 8-9): macro f 1 -score of all prompt compositions with each llm (baselines shown as vertical lines). our adaptive prompting approach (ours stereoset) outperforms all fixed compositions. ours sbic and ours cobra are trained on other datasets. the variance over all compositions (shown as box plots) indicates the llms' sensitivity to the prompt. ## conclusion in this paper, we have introduced the notion of prompt compositions, that is, combining multiple prompting techniques to improve llm performance. we have further proposed an adaptive prompting model that learns to predict optimal prompt compositions ad-hoc, based on the input instance in the context of social bias detection. through extensive experiments and a shapley analysis, we have provided insights into the utility and importance of several prompting techniques for the given task. we find that the benefit of each technique and composition notably depends on the input and the llm used, highlighting the need for automated systems to optimize prompt. we show that our adaptive prompting approach can improve upon single compositions on selected datasets. in future work, we seek to work on techniqueand task-agnostic approaches to find optimal prompt compositions and do so more efficiently. we hope that our work contributes towards fairer nlp through better social bias detection systems and enables research on using llms more efficiently through better prompting techniques.