KIWI : A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions (2024)

Abstract: Large language models (LLMs) adapted to follow user instructions are now widely deployed as conversational agents. In this work, we examine one increasingly common instruction-following task: providing writing assistance to compose a long-form answer. To evaluate the capabilities of current LLMs on this task, we construct KIWI, a dataset of knowledge-intensive writing instructions in the scientific domain. Given a research question, an initial model-generated answer and a set of relevant papers, an expert annotator iteratively issues instructions for the model to revise and improve its answer. We collect 1,260 interaction turns from 234 interaction sessions with three state-of-the-art LLMs. Each turn includes a user instruction, a model response, and a human evaluation of the model response. Through a detailed analysis of the collected responses, we find that all models struggle to incorporate new information into an existing answer, and to perform precise and unambiguous edits. Further, we find that models struggle to judge whether their outputs successfully followed user instructions, with accuracy at least 10 points short of human agreement. Our findings indicate that KIWI will be a valuable resource to measure progress and improve LLMs’ instruction-following capabilities for knowledge intensive writing tasks.

## introduction as llm-powered conversational agents @xcite @xcite have gained widespread adoption, users have turned to these systems for assistance on a wide range of real-world tasks. in particular, recent works studying llm interactions "in the wild" @xcite have discovered that 10-20% of user queries contain requests for writing assistance, i.e. using an llm to create, revise, or organize a piece of written text. however, we currently lack an understanding of llms' capabilities as writing assistants, especially in knowledgeintensive settings. what types of writing instructions do users issue to llms, and how well can llms follow different types of instructions? we present kiwi, a dataset of expert-written knowledge-intensive writing instructions to better understand llms' instruction-following abilities as writing assistants. to collect kiwi, we set up an interactive interface between a researcher and an llm (figure 1 ). we first prompt an llm to generate a long-form answer @xcite to an nlp research question, based on a set of passages from relevant research papers. a researcher then iteratively issues instructions for the model to revise the answer and evaluates the model-generated revision. the interaction session continues until the user is satisfied with the final answer or a maximum number of turns is reached. kiwi contains instructions collected from 234 interaction sessions with three state-of-the-art language models (gpt-4,gpt-3.5-turbo and llama-2-70b-chat), consisting of a total of 1,260 unique instruction instances. each instance contains (1) a user instruction, (2) the model's previous and (3) revised answers, and (4) a human annotator's judgment (both categorical rating and free-form explanation) on whether the revised answer followed the instruction. using kiwi, we conduct an in-depth analysis to characterize the types of instructions issued by researchers, and to measure how well models can follow different types of instructions. we find that llms do not excel at this task yet, with the best model (gpt-4) achieving success for only 59% of the instructions. specifically, llms fail to precisely follow user's instructions (such as satisfying location and length constraints), often cause answer quality to degrade when integrating new information into a previous answer, and struggle to avoid making changes to answers that are not requested. in-context learning works due to … research has shown that in some cases, in-context learning can be understood as the implicit implementation of known learning algorithms, including gradient descent can you gather more explanations for why in-context learning works including data mixtures, lm scale, emergence, etc. in-context learning … one explanation is the lm's ability to leverage previously learned latent concepts and infer the task from examples in the prompt it didn't really include any more papers except for maybe 1. in-context learning … multiple works have provided evidence for the effectiveness of incontext learning and have demonstrated its capabilities. one paper … it's still not finding more explanations and only sticking with a few high level points. in-context learning … one explanation is the lm's ability to leverage previously learned latent concepts and infer the task from examples in the prompt figure 1 : an example interaction session between a user and our system. given a question q and a set of documents d, the model first generates an initial answer a 0 . at each turn t, the user issues an instruction i t , for which an updated answer a t is generated. the user provides a rating r t and a comment c t for the generated answer, and optionally edits the answer (such as a ′ 3 in turn 3) such that the edited answer completely follows the instruction. finally, we examine the ability of the strongest llm (gpt-4) to evaluate whether a response follows an instruction, comparing its judgement to the user ratings in kiwi. unlike prior works which found that llms can judge outputs as reliably as humans for open-ended instruction following @xcite , our experiments show that gpt-4 cannot reliably evaluate responses for instructions in kiwi, which are often specific and precise. while the best performing model which retrieves and prepends 10-shot examples improves upon zero-shot performance by 5% accuracy, it still lags behind human agreement by 12% accuracy. given these findings, we believe that kiwi will be a valuable resource for future research in instruction following and evaluation. ## dataset collection we first describe our approach to curate highquality (q, d) pairs ( §3.1), followed by our annotation protocol for interaction collection ( §3.2). ## analysis of kiwi we first present results on overall model performance ( §4.1). then we conduct a fine-grained analysis to understand the different types of instructions ( §4.2) and errors made by llms ( §4.3). ## experiments: automatically evaluating instruction following our analysis so far demonstrates that current llms often fail to follow users' instructions to revise long-form answers. next, we examine whether they can serve as an evaluator to assess whether a model-generated answer followed the instructions @xcite @xcite @xcite . setting and data formally, the task is to predict the user rating r @xmath0, i t , a t-1 , a t ), introduced in §2.2. to simplify the task, we collapse {neutral, ## related work instruction following a number of recent efforts have collected instruction following datasets to train and evaluate llms. some datasets contain crowd-sourced interactions in the wild @xcite @xcite @xcite , while others target specific tasks such as summarization @xcite and controllable generation @xcite . our work focuses specifically on instruction-following for writing assistance and text editing. a number of prior works have curated instruction-based editing benchmarks @xcite @xcite . however, these efforts consist of post-hoc synthetic instructions derived from previous datasets, consisting of a limited set of edit instructions. our data consists of diverse instructions issued by expert users interacting with llms. ## conclusion we present kiwi, a dataset with expert-issued writing instructions to revise a long-form answer to a research question using relevant documents. our analysis and experiments show that current llms (including gpt-4) cannot perform this task reliably yet and identify common failure patterns. we are optimistic that kiwi will be a useful resource for several research directions. first, the model revisions that were judged as successful by humans, and answers edited manually by humans when models failed, can be used as training data to improve models' instruction following abilities. second, human judgments of model revision quality can be used to develop more accurate reward models for writing assistance. finally, the instructions in kiwi can be used as inputs to evaluate the performance of future models; either through human judgments, or by future reward models. author contributions: fangyuan xu led the project and performed all the technical contributions, including designing and building the annotation interface, dataset collection, running the experiments and performing the data analysis. fangyuan also contributed to project scoping and ideation and led the writing of the paper. bailey kuehl helped with collecting the interaction data, including piloting the task, giving feedback and hiring and managing the upwork annotators. david wadden, kyle lo and luca soldaini were mentors of the project during and after fangyuan's internship, and contributed to project scoping, experiments design, ideation and direction throughout the project. david wadden and kyle lo contributed to paper writing and luca soldaini contributed to annotation interface development. eunsol choi provided mentorship throughout the project, including project scoping, ideation, experiement design and paper writing. fangyuan, bailey and david contributed to the question annotation.