ARES : Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback (2024)

Abstract: Large Multimodal Models (LMMs) excel at comprehending human instructions and demonstrate remarkable results across a broad spectrum of tasks. Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF) further refine LLMs by aligning them with specific preferences. These methods primarily use ranking-based feedback for entire generations. With advanced AI models (Teacher), such as GPT-4 and Claude 3 Opus, we can request various types of detailed feedback that are expensive for humans to provide. We propose a two-stage algorithm ARES that Alternates REinforcement Learning (RL) and Supervised Fine-Tuning (SFT). First, we ask the Teacher to score how much each sentence contributes to solving the problem in a Chain-of-Thought (CoT). This sentence-level feedback allows us to consider individual valuable segments, providing more granular rewards for the RL procedure. Second, we ask the Teacher to correct wrong reasoning after the RL stage. The RL procedure requires substantial hyperparameter tuning and often generates errors such as repetitive words and incomplete sentences. With correction feedback, we stabilize the RL fine-tuned model through SFT. We conduct experiments on the multi-modal datasets ScienceQA and A-OKVQA to demonstrate the effectiveness of our proposal. The ARES rationale achieves around 70% win rate compared to baseline models judged by GPT-4o. Additionally, we observe that the improved rationale reasoning leads to a 2.5% increase in inference answer accuracy on average for the multi-modal datasets.

## introduction large language models (llms) and large multimodal models demonstrate remarkable performance across diverse language and multi-modal tasks @xcite @xcite @xcite . however, these large models (lms) often generate toxic and biased content @xcite because lms are primarily trained to predict the next token based on extensive corpus datasets. to align lm behavior more closely with user preferences, previous works @xcite fine-tune their models using reinforcement learning from human feedback (rlhf). furthermore, with advancements in lms, advanced lm feedback can replace costly human feedback, yielding reinforcement learning from ai feedback (rlaif) @xcite @xcite . however, rlhf and rlaif encounter two significant challenges. first, both methods often utilize ranking-based feedback @xcite , which orders the generated samples by preference. for instance, if sample a is preferred over sample b, the model is fine-tuned to generate more outputs like a and fewer like b. however, if b contains certain valuable parts, they may be discarded. to alleviate these issues, @xcite ; @xcite propose sentence-level feedback, applying it solely to the reward model without reinforcement learning (rl), called the processsupervised reward model (prm). it demonstrates its potential through search algorithms on the prm, such as best-of-n or monte carlo tree search (mcts). furthermore, @xcite demonstrate the effectiveness of rl with sentence-level feedback by heuristically scoring each sentence in math problems, where evaluating the predicted answer is straightforward. thus, sentence-level feedback exhibits significant promise compared to existing ranking-based feedback. nonetheless, acquiring sentence-level feedback is more costly than ranking-based feedback. second, the rl process is inherently unstable and requires extensive hyperparameter tuning @xcite . this instability often results in the generation of repetitive words and truncated sentences. hyperparameter tuning becomes an enormous burden as the model size increases, making exhaustive tuning seemingly impossible, especially for individuals. the existing rlhf method @xcite recycles the dataset used in pretraining within the loss function with proximal policy optimization (ppo) @xcite to mitigate this degeneration problem. however, this approach prevents the model from fully maximizing the sum of rewards through rl and may limit the opportunity for diverse improvements, which differ from the pretraining distribution. in this work, we aim to address the two challenges mentioned above through various types of feedback using an advanced ai model as a teacher. many advanced ai models, including gpt-4 and claude 3 opus, are already used as evaluators for many tasks and generate reliable human-level answers @xcite . 1) we request a score from the teacher for each sentence, ranging from 0.0 to 1.0. each score indicates how much a sentence contributes to solving the problem. this provides detailed reward feedback to the training model and can be applied to both mathematical and more general multi-modal chain-of-thought (cot) (or rationale reasoning) problems. 2) we ask the teacher to identify and correct minor errors in the rl results, such as incorrect or cut-off parts. with this corrected dataset, we fine-tune the model using supervised fine-tuning (sft). this stage allows the model to maximize the rewards while properly deviating from the pretraining distribution. in summary, we propose a hybrid algorithm ares that alternates reinforcement learning and supervised fine-tuning. to evaluate how much rationale reasoning can be improved through the ares framework, we use the scienceqa and a-okvqa datasets, which are large-scale, multi-modal datasets that include rationale reasoning data. we use multimodal-cot (mm-cot) @xcite as the baseline. mm-cot employs two separate models: one model is responsible for generating rationale, and the other model, an inference model, processes the concatenated input (problem and generated rationale). this distinct framework enhances performance, even with relatively smaller models like flan-alpaca base @xcite (251m) and flan-alpaca large (790m) with vit feature @xcite . we perform ares on the rationale reasoning model of mm-cot. we compare ares rationale reasoning with that of mm-cot through gpt-4o, determining which rationale is better and computing the win rate. additionally, we check whether the improved rationale reasoning leads to better answer accuracy. our results show that our rationale reasoning outperforms the baselines with around 70% win rate and demonstrates 2.5% increase in inference answer accuracy on average for the different model sizes and the multi-modal tasks. in summary, our key contributions are: ## methodology this section briefly introduces the preliminaries in section 2.1 and presents our two-stage hybrid algorithm ares that alternates reinforcement learning and supervised fine-tuning. 1) we request a score for each sentence in the chain-of-thought (cot) from the advanced ai model (teacher) to determine how much it contributes to solving the problem (section 2.2). we perform reinforcement learning (rl) with the score feedback on our training model. ## experimental setup data: we first evaluate our proposed method on the scienceqa @xcite ) dataset, a large-scale, multi-modal science dataset designed to assess multi-hop reasoning abilities. we choose scienceqa because it contains reasoning chains to derive the answer. each problem consists of a question, multiple options, multi-modal contexts, a correct answer, and an annotated lecture or solution chain (note that around 9.5% lack the solution chain). in addition, we conduct experiments on a-okvqa @xcite , a knowledge-based multi-modal benchmark with a diverse set of challenging questions paired with rationales, demanding non-trivial commonsense knowledge (see appendix b). baselines: we mainly compare our method with multimodal-cot (mm-cot) @xcite as the baseline because it utilizes reasoning chains to solve multi-modal tasks. mm-cot leverages two distinct models: the first generates a rationale for a given problem, and the second, an inference model, takes the concatenated input (problem and generated rationale). this separated framework shows improved performance, even for relatively small models such as flan-alpaca base (chia et al., 2023) (251m) and flan-alpaca large (790m). we use the rationale model provided by mm-cot for scienceqa and retrain the rationale model ourselves for a-okvqa because there is no provided model. prompts for feedback: since our proposed ares requests different types of feedback for each stage, a corresponding prompt exists separately. we use claude 3 haiku for all training to get feedback because it is approximately 20 times cheaper than the top competing models, yet still demonstrates decent performance. we first request scores ranging from 0.0 to 1.0 for each sentence in cot to proceed with the rl stage. to obtain reasonable scores, we let haiku consider the starting point of thought, the process of elimination, or true statements. (see in order to collect the corrected dataset for the sft stage, we let haiku refer to the given problem and correct the answer as the prompt. we ask haiku to maintain the format of the existing rationale chains as much as possible and correct only the parts that require correction. the rl stage often makes the training model generate repetitive sentences. this repetition is not easily removed even by gpt-4 when the repetitive sentence exists in the middle of rationale reasoning. to reduce the burden of feedback, we simply hard-code the removal of repetitive sentences before adding the generated rationale to the prompt. (see appendix c.2.) training details: for the ares base rl stage, we use a learning rate of 2e-5 and 10 epochs for ppo with a batch size of 8 for both scienceqa and a-okvqa. the learning rate for ares large is 2e-5 with 5 epochs for ppo and a batch size of 2 for both tasks. we proceed with 2 rounds of our pipeline for ares base and 2 rounds for ares large for scienceqa. for a-okvqa, we proceed with 1 round for both model sizes. for the sft stage for correction, we follow the hyperparameters used in mm-cot for both model sizes. additionally, we replace mm-cot's inference model, which is the same size as the rationale model, with the low-rank adaptation (lora) @xcite added to the rationale model (figure 4 ). the lora adapter effectively utilizes the rationale model's features with a small number of weights, enabling 2x-14x faster inference compared to mm-cot, which introduces a separate inference model (see the time comparison in ## experimental results in this section, we evaluate our pipeline ares that alternates reinforcement learning and supervised fine-tuning by requesting diverse types of feedback for an advanced ai model (teacher) (claude 3 haiku). the goal of ares is to improve the rationale reasoning quality. we demonstrate how ares enhances rationale reasoning in the following sections. ## related work chain-of-thought (cot) is a multi-step reasoning method for problem-solving that encourages llms to consider the intermediate reasoning steps. zero-shot-cot @xcite promotes cot by using prompts such as "let's think step by step" for llms. for few-shot-cot @xcite , a few examples with reasoning processes are provided, allowing the model to refer to these examples and understand how to perform cot. @xcite reveal that this cot the first sentence is the compound sentence. it is made up of two simple sentences joined by a comma and the conjunction so. the camera isn't working, so the battery must be dead. ## conclusion we propose a hybrid algorithm, ares, which alternates reinforcement learning (rl) and supervised fine-tuning (sft) to enhance multimodal rationale reasoning for scienceqa and a-okvqa. ares leverages two types of feedback: