Learning to Correct for QA Reasoning with Black-box LLM s (2024)

Abstract: An open challenge in recent machine learning is about how to improve the reasoning capability of large language models (LLMs) in a black-box setting, i.e., without access to detailed information such as output token probabilities. Existing approaches either rely on accessibility (which is often unrealistic) or involve significantly increased train- and inference-time costs. This paper addresses those limitations or shortcomings by proposing a novel approach, namely CoBB (Correct for improving QA reasoning of Black-Box LLMs). It uses a trained adaptation model to perform a seq2seq mapping from the often-imperfect reasonings of the original black-box LLM to the correct or improved reasonings. Specifically, the adaptation model is initialized with a relatively small open-source LLM and adapted over a collection of sub-sampled training pairs. To select the representative pairs of correct and incorrect reasonings, we formulated the dataset construction as an optimization problem that minimizes the statistical divergence between the sampled subset and the entire collection, and solved it via a genetic algorithm. We then train the adaptation model over the sampled pairs by contrasting the likelihoods of correct and incorrect reasonings. Our experimental results demonstrate that CoBB significantly improves reasoning accuracy across various QA benchmarks, compared to the best-performing adaptation baselines.

## introduction large language models (llms) have achieved significant advancements in various nlp tasks, demonstrating exceptional capabilities in understanding and generating text @xcite @xcite . nevertheless, llms still present notable limitations, such as biased opinions toward specific groups @xcite or inaccurate predictions for infrequent topics @xcite , primarily due to the imperfections in the knowledge acquired during pre-training @xcite . consequently, it is essential to control and adapt the responses of llms to achieve optimal performance for specific use cases. representative methods include fine-tuning on supervised training datasets @xcite and input-level optimization through prompt engineering and retrieval augmentation @xcite . however, these approaches require huge training costs or exhibit limited adaptation performance, respectively. to address these challenges, prior works have focused on training relatively smaller models using responses from llms and human supervision, then generating adapted responses while assuming that the llm parameters are fixed or inaccessible (i.e., black-box). one approach assumes that the output token probabilities are available @xcite , but this is often unrealistic. although @xcite recently proposed training a verifier and employing beam search to obtain adapted responses without this assumption, this method results in computationally expensive training and inference pipelines. alternatively, @xcite introduced a straightforward seq2seq learning framework to enhance the alignment of black-box llms. however, extending this framework to other tasks is challenging, particularly in terms of constructing the training dataset and ensuring the effectiveness of the training method. in this paper, we propose a simple yet efficient framework, learning to correct for qa reasoning of black-box llms (cobb). our key idea is to learn a seq2seq mapping from the original reasoning of black-box llm to correct and improved reasoning, by training an adaptation model initialized with a relatively small open-source llm. after training, the adaptation model can be easily deployed during inference as a single additional module, as illustrated in figure 1 . specifically, we firstly sample multiple chainof-thought reasonings from black-box llm and label their correctness using ground-truth human labels. then, from all possible pairs of correct and incorrect reasonings, we subsample a few representative pairs that preserve the characteristics of the entire set. to identify such a subset, we formulate an optimization problem that minimizes the statistical divergence between the subset and the entire set, solving it via a genetic algorithm. finally, using this optimized subset, we train the adaptation model to simultaneously increase the likelihood of correct reasoning and decrease the likelihood of incorrect reasoning for the given input and reasoning. an overview of cobb is presented in figure 2 . we demonstrate the effectiveness of cobb in improving qa reasoning with black-box llms through extensive evaluations on four different qa datasets. for instance, cobb achieved average accuracy improvements of 6.2% and 2.2%, compared to the original black-box gpt-3.5-turbo and previous state-of-the-art adaptation methods, respectively. furthermore, we found that the adaptation model trained for a specific black-box llm could generalize to adapt other llms, including both api-based and open-source models, which is crucial for efficient deployment in practice. additionally, our in-depth analyses reveal how cobb improves and corrects the reasoning from the blackbox llms. we hope our work provides valuable insights into llm adaptation research, which is becoming increasingly important for the future success of llms in real-world applications. ## cobb: learning to correct for qa reasoning with black-box llms in this section, we introduce our framework for learning to correct for improving qa reasoning with black-box llms (cobb). we begin with an overview of the problem setup in section 3.1. next, in section 3.2, we present how to construct an effective dataset for training the adaptation model. this dataset is created by solving an optimization problem using a genetic algorithm, to preserve the characteristics of the entire set of correct and incorrect reasoning pairs from black-box llm. finally, we describe a training scheme in section 3.3, where the adaptation model is trained by contrasting the likelihoods of positive and negative reasonings. the full procedure of cobb is outlined in algorithm 1, and an overview is provided in figure 2 . ## conclusion in this paper, we proposed cobb, a simple yet effective framework for learning to correct qa reasoning of black-box llm. we propose to learn a seq2seq mapping from the original reasoning of black-box llm to correct and improved reasoning, by training a relatively small adaptation model with the newly proposed dataset construction and training objective. our experiments demonstrate the effectiveness of cobb across various qa tasks and llms. therefore, we believe our framework can contribute to various real-world applications that require the adaptation of black-box llms.