S uri: Multi-constraint Instruction Following in Long-form Text Generation (2024)

Abstract: Existing research on instruction following largely focuses on tasks with simple instructions and short responses. In this work, we explore multi-constraint instruction following for generating long-form text. We create Suri, a dataset with 20K human-written long-form texts paired with LLM-generated backtranslated instructions that contain multiple complex constraints. Because of prohibitive challenges associated with collecting human preference judgments on long-form texts, preference-tuning algorithms such as DPO are infeasible in our setting; thus, we propose Instructional ORPO (I-ORPO), an alignment method based on the ORPO algorithm. Instead of receiving negative feedback from dispreferred responses, I-ORPO obtains negative feedback from synthetically corrupted instructions generated by an LLM. Using Suri, we perform supervised and I-ORPO fine-tuning on Mistral-7b-Instruct-v0.2. The resulting models, Suri-SFT and Suri-I-ORPO, generate significantly longer texts (5K tokens) than base models without significant quality deterioration. Our human evaluation shows that while both SFT and I-ORPO models satisfy most constraints, Suri-I-ORPO generations are generally preferred for their coherent and informative incorporation of the constraints.

## introduction improving the instruction-following abilities of modern large language models (llms) is critical to increasing their effectiveness and generalizability for many practical applications. however, most existing instruction-following datasets (e.g., alpaca) contain only simple instructions that can be solved by short model generations @xcite @xcite . what about tasks with complex, multi-constraint instructions that can only be satisfied with long-form outputs (i.e., thousands of tokens), such as creating detailed technical reports or writing engaging fictional narratives? we explore this question by conducting the first in-depth study of long-form instruction following with multi-constraint instructions. to facilitate our experiments, we create a new dataset, suri, foot_1 using instruction backtranslation @xcite . this process involves feeding a human-written long-form text (e.g., chapters from a novel) into an llm to generate instructions that could have been followed to create the text. the resulting dataset, suri, consists of 20k texts paired with llm-generated instructions, each containing ≈10 semantic and stylistic constraints (figure 1 ). how can we use suri to improve an llm's longform instruction following abilities? while supervised fine-tuning (sft) has been quite effective for short-form datasets @xcite @xcite @xcite , we observe that fine-tuned suri models often generate texts that are incoherent and fail to satisfy constraints towards the end in the instructions. preference tuning methods such as dpo @xcite and rlhf @xcite are challenging to use in this setting due to difficulties and cost in obtaining preference judgments on long-form texts @xcite . specifically, when annotating preferences for long texts, human annotators may struggle to determine if different sections of the text are faithful to the instructions while simultaneously considering multiple aspects of the text, such as coherence and informativeness. motivated by this, we devise an alignment method that relies on synthetically corrupted instructions. specifically, we take the backtranslated suri-i-orpo suri-sft alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do: once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, "and what is the use of a book," thought alice "without pictures or conversation?" figure 1 : our work consists of two stages. first, we construct the suri dataset using gold responses sampled from three existing datasets that include creative writing and open web text, along with backtranslated instruction x w and corrupted instruction x l . second, we fine-tune mistral-7b-instruct-v0.2 on suri, resulting in two variations: suri-i-orpo (via i-orpo) and suri-sft (via supervised fine-tuning). instruction x w and corrupt its constraints using an llm such that the gold response does not satisfy the corrupted constraints (for example, see x l in figure 1 ). we then develop a variant of the odds ratio preference optimization objective @xcite to use these corrupted instructions as negative feedback. we refer to this alignment method as instructional orpo, or i-orpo for short. we conduct a series of automatic and human evaluations on generations from sft and i-orpotuned models to validate our method. compared to the base model, mistral-7b-instruct-v0.2 @xcite , both sft and i-orpo significantly increase the generation length from 1k to 5k tokens. our fine-tuned models also improve the ability to differentiate between correct and corrupted instructions by at least 10% while maintaining low levels of n-gram repetitions in the text. we find that llm judges, such as gpt-4o (ope-nai, 2024), gemini-1.5-pro @xcite , and claude-3.5-sonnet (anthropic, 2024), cannot reliably evaluate long-form responses, which makes human evaluation crucial for assessing the constraint-following capabilities of our generations. annotators note that our fine-tuned models effectively follow given constraints, with i-orpo being preferred to our sft model for its ability to incorporate constraints coherently, informatively, and enjoyably. ## the suri dataset we focus on the task of long-form writing, both fictional and non-fictional, under multiple constraints. when using an llm for a complex writing task, users might have many constraints in mind and expect lengthy, detailed responses in the form of books, blog posts, etc. this task is particularly challenging for current llms, which struggle with generating coherent long-form outputs @xcite , and this difficulty can be amplified when multiple constraints are involved. recent instruction-following datasets have featured multi-constraint instructions @xcite and long-form responses @xcite , but none has integrated these two elements ( we collect human-written english text samples, such as books, religious texts, and blog posts, to serve as gold responses (y). since gathering humanwritten instructions for such lengthy responses is difficult and expensive, we turn to instruction backtranslation @xcite , in which an llm is provided with a human-written text (e.g., a short story) and prompted to generate instructions (x w ) that could have been followed to create that text. we further corrupt the constraints in x w to obtain synthetically corrupted instructions (x l ) for our i-orpo alignment method. in total, suri contains 20k single-turn examples, each consisting of a backtranslated instruction x w , corrupted instruction x l , and a human-written response y. in this section, we detail our approach to selecting high-quality text samples ( §2.1) and creating backtranslated instructions ( §2.2). we also validate our generated instructions ( §2.3) and analyze the resulting dataset ( §2.4). ## aligning language models with suri our goal is to assess whether suri helps improve the instruction-following capabilities of mistral-7b-instruct-v0.2 for long-form text generation. we experiment with two methods of fine-tuning mistral-7b-instruct-v0.2 on suri: supervised finetuning (sft) using (x w , y) pairs and a modified orpo alignment @xcite using (x w , x l , y) triplets. we emphasize that preference judgments are difficult to obtain for long-form responses due to numerous aspects of the text that must be considered with respect to the instructions. therefore, we perform model alignment with correct instruction x w and corrupted instruction x l instead. full details on fine-tuning libraries, hardware configurations, and hyperparameters can be found in appendix c. ## automatic evaluation our automatic assessment demonstrates that both suri-i-orpo and suri-sft increase the length of the generated texts while maintaining a reasonable level of repetition. compared to baseline models, suri-i-orpo is more likely to assign higher log probabilities to tokens in the response given the correct instruction than the corrupted instruction. ## human evaluation while our automatic assessments provide insights into the lexical information of the text, they do not capture its semantic content. therefore, we conduct a human evaluation to determine if and how the constraints are satisfied by the outputs of suri-sft and suri-i-orpo. human evaluation on 30 test set generations reveals that while both finetuned models satisfy constraints, suri-i-orpo is preferred by humans for its ability to seamlessly incorporate the constraints into the final outputs. ## conclusion in this work, we investigate the challenge of complex instruction following for generating long-form text. we introduce suri, a dataset of long humanwritten responses accompanied by backtranslated and corrupted instructions. we demonstrate the effectiveness of suri in improving the constraintfollowing capabilities of llms for long-form gen-eration through supervised fine-tuning and i-orpo. human and automated evaluations show that our models generate high-quality, long-form responses while effectively satisfying constraints.