Automatically Correcting Large Language Models: Surveying the Landscape of Diverse Automated Correction Strategies (2024)

Abstract: While large language models (LLMs) have shown remarkable effectiveness in various NLP tasks, they are still prone to issues such as hallucination, unfaithful reasoning, and toxicity. A promising approach to rectify these flaws is correcting LLMs with feedback, where the LLM itself is prompted or guided with feedback to fix problems in its own output. Techniques leveraging automated feedback—either produced by the LLM itself (self-correction) or some external system—are of particular interest as they make LLM-based solutions more practical and deployable with minimal human intervention. This paper provides an exhaustive review of the recent advances in correcting LLMs with automated feedback, categorizing them into training-time, generation-time, and post-hoc approaches. We also identify potential challenges and future directions in this emerging field.

## introduction recent years have seen striking empirical successes of large language models (llms), as they consistently obtain impressive results across a diverse range of nlp benchmarks @xcite @xcite , while also showcasing surprising abilities of language understanding @xcite , generation @xcite @xcite , and reasoning @xcite @xcite . however, these models are not without their flaws. llms are observed to intermittently display undesired and inconsistent behaviors such as producing seemingly convincing but inaccurate ''hallucinations'' @xcite @xcite , conducting unfaithful reasoning @xcite @xcite , generating inappropriate or harmful content @xcite @xcite , and failing to trustfully follow rules and constraints @xcite . such flawed behaviors hamper the trust in llms and pose hurdles to their real-world applications @xcite . a prevailing strategy to rectify these undesired behaviors of llms is learning from feedback, mirroring a typical human learning strategy where individuals actively refine their behaviors through a cycle of trial, error, and correction. humans, when making mistakes, often gather feedback either from others or through self-reflection @xcite @xcite @xcite . such feedback offers valuable insights for humans to correct mistakes and modify their behavior accordingly. inspired by this natural learning mechanism, extensive research @xcite @xcite has been undertaken to improve llms through the paradigm of learning from both internal and external feedback. one popular line of research involves the use of human feedback to evaluate and refine models, as encapsulated in the survey by @xcite . these methods typically involve direct optimization of llms against human feedback on their outputs @xcite @xcite , where human evaluations of output quality serve as a reward signal to improve model performance. however, this approach has two primary drawbacks: it can be costly due to the manual labor involved, and it lacks real-time capabilities as humans cannot provide instant feedback. to minimize the need for human intervention, another strategy is correcting llms with automated feedback. as illustrated by the conceptual figure 1 : a conceptual framework for correcting llms with automated feedback. we identify three parties involved in the prototypical correction pipeline that are analogous to a patient, doctor, and treatment in medicine, respectively: a language model produces initial output, a critic model analyzes the output and provides feedback, and a refine model provides treatment to either the output or the language model. we taxonomize existing works using this conceptualization along five key aspects: the problem to be corrected, the source and format of the feedback, and the strategy and learning method of the refine model. framework in figure 1 , the language model (iteratively) learns from automatically generated feedback signals to understand the consequences of its actions and adapts its behaviors. the source of automated feedback can be multifaceted, spanning from the llm itself acting as the feedback model @xcite , a separately trained feedback model @xcite , readily available external tools @xcite , to external knowledge sources such as wikipedia or the internet @xcite . various strategies of correction have been proposed, including self-training @xcite , generate-then-rank @xcite , feedback-guided decoding @xcite , iterative post-hoc revision @xcite , etc. recently, the incorporation of such strategies has demonstrated their effectiveness across a myriad of tasks, from question answering @xcite and reasoning @xcite to code generation @xcite and toxicity detection @xcite . in light of these advancements, our paper aims to provide a comprehensive survey. we start by establishing the concept of correcting llms with automated feedback and creating a taxonomy of the different methods ( § 2). we then discuss the major techniques ( § 3), categorized as trainingtime, generation-time, and post-hoc correction. finally, we discuss the connection to earlier works ( § 4) and five potential future directions ( § 5). ## conceptual framework for clean exposition, we first present a conceptual framework outlining the overall process of correcting llms with feedback in figure 1 , using an analogy of medical treatment in our daily life. three parties are involved in this process: ## methodologies in this section, we delve into a detailed review of various correction methodologies. depending on the time that the correction happens, we categorize them as training-time correction, generation-time correction, and post-hoc correction. ## conclusion in this paper, we present a comprehensive survey of self-correcting large language models with automated feedback. we categorize and analyze various self-correction strategies, including training-time, generation-time, and post-hoc corrections. we also connect recent work with prior research and discuss the applicable scenarios for automated correction. finally, we outline five potential future directions and associated challenges in this field. our goal with this paper is to provide a comprehensive and useful resource for readers interested in the development of this rapidly evolving domain. to aid in this effort, we create a continually updated reading list in a github repository as follows: https://github.com /teacherpeterpan/self-correction-llm -papers .