LLMC rit: Teaching Large Language Models to Use Criteria (2024)

Abstract: Humans follow criteria when they execute tasks, and these criteria are directly used to assess the quality of task completion. Therefore, having models learn to use criteria to provide feedback can help humans or models to perform tasks better. However, current research in this area tends to consider only a limited number of criteria, or only a limited number of quality assessment aspects. To fill this gap, we propose a general framework that enables large language models (LLMs) to use comprehensive criteria for a task in delivering natural language feedback on task execution. In particular, we present a model-in-the-loop framework that semi-automatically derives criteria from collected guidelines for different writing tasks and constructs in-context demonstrations for each criterion. We choose three tasks from real-world scenarios to operationalize this idea: paper introduction writing, Python code writing, and Reddit post writing, and evaluate our feedback generation framework using different LLMs. The results reveal the fine-grained effects of adding criteria and demonstrations and provide valuable guidance on how to teach LLMs to use criteria more effectively.

## introduction a criterion is a principle or standard by which something may be judged or decided @xcite ). an example criterion for a writing task could be that "the text should not contain grammar errors". in the fields of cognitive science and psychology, it is widely recognized that humans, when engaging in a task, invariably follow certain criteria @xcite which are often directly used for judging the quality of task completion. despite the importance of criteria, the current field of research has relatively neglected this piece of human wisdom, with works mainly focusing on using a limited number of criteria @xcite @xcite or using please provide feedback on this code: ```...``t he code is too long and could be refactored into smaller functions. ## related work augmented llms augmenting llms with additional context has become a powerful tool for continual learning as well as combating hallucinations. these augmentations include learned rule library @xcite to perform deductive reasoning, retrievers to retrieve relevant documents from knowledge bases to answer fact-related queries @xcite or various tools and apis to perform specialized tasks. @xcite . we continue this line of work by proposing to augment llms with criteria extracted from the writing task guidelines so that models can generate more helpful feedback for a piece of writing for that task. automated critique generation it is commonplace to use llms as evaluators @xcite @xcite -even replacing human evaluations for training (cui et al., 2023) -which can be seen as a basic way of critique. having models criticize their own output and then rewriting is another popular use of synthetic data creation @xcite @xcite . departing from that, our focus is on teaching llms to learn criteria so that they can generate more comprehensive feedback that is aligned with human values. ## llmcrit framework our proposed framework is shown in fig. 2 . there are four steps within this framework. ## data and evaluation foundation to demonstrate the effectiveness of llmcrit, we conduct experiments on three application scenarios: paper introduction writing, python code writing, and reddit post writing. for data preparation including criteria extraction, demonstration creation, and test data construction, we used claude2 (anthropic, 2023) prompting with the chain-of-thought (cot) technique @xcite . ## further analysis we explore two questions further: (1) how does adding criteria and demonstrations affect the generation of critiques for individual criteria? ( 2 ) how providing criteria at different granularities affects the quality of model-generated feedback? ## conclusion in this paper, we present a general framework for teaching llms to use criteria for feedback generation that takes the guidelines as a starting point, semi-automatically extracts criteria from them, and constructs in-context demonstrations. we then apply these criteria and in-context demonstrations to guide the feedback generation process of llms. in order to provide a more comprehensive assessment of the quality of feedback, we propose a layered evaluation methodology to measure the quality of generated feedback from different perspectives. experiments conducted on three writing tasks and seven different llms provide insights on the most effective way of teaching llms to use criteria. we also analyzed the effectiveness of applying criteria at different levels of granularity and found that providing criteria at the granularity of an aspect is more effective than providing criteria at the granularity of a single criterion.