C onstitutional E xperts: Training a Mixture of Principle-based Prompts (2024)

Abstract: Large language models (LLMs) are highly capable at a variety of tasks given the right prompt, but writing one is still a difficult and tedious process. In this work, we introduce ConstitutionalExperts, a method for learning a prompt consisting of constitutional principles (i.e. rules), given a training dataset. Unlike prior methods that optimize the prompt as a single entity, our method incrementally improves the prompt by surgically editing individual principles. We also show that we can improve overall performance by learning unique prompts for different semantic regions of the training data and using a mixture-of-experts (MoE) architecture to route inputs at inference time. We compare our method to other state of the art prompt-optimization techniques across six benchmark datasets. We also investigate whether MoE improves these other techniques. Our results suggest that ConstitutionalExperts outperforms other prompt optimization techniques by 10.9% (F1) and that mixture-of-experts improves all techniques, suggesting its broad applicability.

## introduction large language models (llms) are highly capable at a variety of nlp tasks when prompted with appropriate natural language instructions @xcite . however, writing an llm prompt remains a difficult and ambiguous task, often involving significant experimentation and effort @xcite . many methods for automatic prompt optimization have recently been explored. some rely on access to model parameters and gradients to optimize discrete @xcite or continuous @xcite prompts given task-specific training data. others involve revising the task-prompt with discrete manipulations, figure 1 : training loop for a single constitution-alexpert. our method samples incorrect predictions from a training dataset, then uses two separate llms to mutate (llm-o) the prompt given these observed mistakes and then evaluate (llm-s) these mutated prompts on a validation set, to determine which of these new candidate experts survive for the next iteration. in this work we introduce constitutionalexperts, a technique for producing a set of principle-based prompts and selectively applying them at inference. our approach is inspired by the constitu-tionalai workflow @xcite used to create fine-tuning datasets for llms. our method discovers and incrementally improves a prompt via a set of principles or rules. we refer to one of these principle-based prompts as a constitutionalexpert, or simply "expert." similar to prior techniques, our figure 2 : hard-routing the constitutionalexperts at inference. each constitutionalexpert is learned from a cluster in the training data. to then hard-route a constitutionalexpert at inference, we compute the similarity between the test sample and each cluster's centroid (a), and then route the sample to the most similar expert @xcite . to evaluate constitutionalexperts, we compare it to state-of-the-art prompt optimizing baselines, including protegi @xcite and promptbreeder @xcite , across six nlp tasks. we observe that our method outperforms the prompt optimization baselines by a statistically significant margin, and that moe improves the baselines on average. we finish by discussing the limitations of our method and future work. ## constitutionalexperts similar to protegi @xcite , our method optimizes discrete prompts with natural language using a training dataset. however con-stitutionalexperts differs in key ways from pro-tegi and other natural language prompt optimization techniques: firstly, prompts ("experts") are trained via structured rather than free-form mutations, where a single principle is either added, removed, or revised. this constraint of using a small set of interpretable principles introduces significant inductive bias, which we hypothesize will improve the method's generalizability. secondly, we employ a mixture-of-experts @xcite architecture by training a unique expert for each semantic cluster of the training data, and use embedding similarity to route individual examples at inference time (figure 2 ). clustering. to cluster the training dataset, we calculate the embeddings of each training sample with the palm-based text-embedding-gecko@001 model, and then cluster with k-means. we set k to be either 2 or 3, selecting the setting with the higher silhouette score (see training the experts. for each cluster, we train an expert consisting of a set of principles p that are used to instruct a scoring model (llm-s) (figure 1 ). our method for training an expert is to initialize p (initial prompts can be found in for each, ask an optimizer model (llm-o) to explain why the prediction is incorrect. we use beam search to better explore the prompt space. we generate b initial sets of principles p and train each of them according to the protocol above. to evaluate candidates, we use the "ucb bandit" selection procedure proposed by @xcite , using llm-s and the validation set to approximate and select the top b candidates (as measured by f1 in our experiments) for the following iteration. we repeat this process j times. routing at inference. we employ a "hard routing" approach during prediction by first embedding the input sample v test and measuring its cosine similarity to each cluster centroid {v 1 , v 2 , . . . , v k } (fig. 2a ). we then route prediction to the expert corresponding to the nearest centroid: 2b ). ## conclusion we propose constitutionalexperts, a method for learning and applying a mixture of principle-based prompts ("experts"). building on prior work, we introduce a novel method for mutating each expert, which involves (1) determining what edits to make to the expert's principles and (2) applying these targeted edits. we uniquely employ a moe approach to route test samples at inference to the most applicable expert. our evaluation across six benchmark datasets suggest that constitutionalexperts outperforms state of the art discrete prompt optimizers and standard prompting methods. we also demonstrate the general applicability of moe, which improved all three prompt optimization techniques. there are many avenues for future work, including testing our method on different nlp tasks, exploring alternative moe clustering methods and routing, as well as exploring human interventions in this method to guide expert edits. principle diversity. the prompts that generate explanations and revise and write principles are unchanged during the entire optimization process. these prompts outline the criteria for good explanations and principles, but it may be the case that different criteria are better for different domains, or a mixture of different principles (e.g. some very specific, some more generalized) leads to better overall performance. to expand the search space, the optimization prompts could be dynamic (or mutated like in @xcite ) in order to increase the diversity of principles generated (and thus classifiers tested). alternatively, using a human-in-the-loop that incorporates real-time feedback to generate principles such as @xcite might provide more efficient learning of principles or higher overall performance. principle generalizability and overfitting. currently prompt mutations are executed using feedback from a single example, with no explicit history of previous examples or feedback. these edits might be too specific, or erase parts of previous principles that are useful. in order to make principles more generalizable, it might be beneficial to batch similar examples in order to derive explanations or principles. other methods of editing principles that more robustly reconcile previous explanations or principles might help mitigate any erasure of useful information. positional bias. llms have demonstrated bias in the classification domain with respect to giving a higher value or importance to the first option presented @xcite , which we also observed during experimentation. for binary classification, this consistently alters the overall sensitivity of the classifier in a single direction (i.e. if the positive class is first, we would expect higher recall). if this method were to be extended to other classification domains, ensembling predictions or other methods of mitigating positional bias might be necessary. additionally, there might be other steps in our method (e.g. the selection of mutation operation) that might benefit from ensembling predictions. prompt format. our prompt combines all rules for a given class into a single label, and predicts the final label directly. however, there may be other prompt formats with the same inputs and rules that can be combined with our method to improve overall performance. for example, chain-of-thought reasoning @xcite has increased performance in other domains, and might provide additional improvements to the method. sampling multiple times to generate self-consistent reasoning @xcite might provide additional boosts to performance. duplicate or contradictory principles. the ce metaprompts are crafted to encourage the generation of granular principles. however candidate constitutional experts may nevertheless include duplicate principles, or principles at different levels of resolution (for example where one principle implies another). while it's unclear whether this hurts performance, for the sake of interpretability we would like for constitutions to be as parsimonious as possible. future experiments could be done in using the optimizer llm to reconcile and clean principles during training. clustering and routing. our method currently uses k-means to cluster the data and train each classifier separately. at inference time, individual predictions are routed to the classifier with the closest corresponding centroid. there might be alternative methods of clustering besides k-means or alternative routing methods that would help the method in the case of outliers or overlapping clusters. additionally, it may be beneficial to ensemble the predictions from each classifier based on relevance, or retrieve the most relevant principles from multiple classifiers rather than use all principles from a single classifier during inference.