Dolomites: Domain-Specific Long-Form Methodical Tasks (2025)

Abstract: Experts in various fields routinely perform methodical writing tasks to plan, organize, and report their work. From a clinician writing a differential diagnosis for a patient, to a teacher writing a lesson plan for students, these tasks are pervasive, requiring to methodically generate structured long-form output for a given input. We develop a typology of methodical tasks structured in the form of a task objective, procedure, input, and output, and introduce DoLoMiTes, a novel benchmark with specifications for 519 such tasks elicited from hundreds of experts from across 25 fields. Our benchmark further contains specific instantiations of methodical tasks with concrete input and output examples (1,857 in total) which we obtain by collecting expert revisions of up to 10 model-generated examples of each task. We use these examples to evaluate contemporary language models, highlighting that automating methodical tasks is a challenging long-form generation problem, as it requires performing complex inferences, while drawing upon the given context as well as domain knowledge. Our dataset is available at https://dolomites-benchmark.github.io/.

## introduction experts in various fields regularly use writing as a means for planning, organizing, and sharing their work. for instance, a teacher might draft a lesson plan for what they would like to teach in their next class, and a lawyer might draft a patent application for an invention. experts generally follow a consistent and methodical approach to conduct these writing tasks. in the lesson plan example, a teacher would know the lesson objectives, format, and profile of the class, and would produce a plan with the topics to be covered and activities to improve learning. importantly, the teacher follows a systematic procedure to write this lesson plan, using their expertise and what they know about the current context (e.g., the class profile). across fields, from law to visual arts and engineering, experts accomplish on a regular basis such methodical tasks, i.e., writing tasks which loosely follow a standard template for what is usually given as input and what is required from the output. these tasks often follow a structured and consistent procedure as they are performed regularly and tend to be fairly time-consuming, taking from a few hours to several days (see figure 2 ). as large language models (lms) become more capable and widely accessible to a more sophisticated set of users @xcite @xcite @xcite @xcite , they hold great potential for assisting experts with methodical writing tasks, increasing their efficiency and allowing them to focus on complex problem-solving activities @xcite . given their potential for assisting experts, it would be beneficial to evaluate language models on a realistic set of methodical writing tasks. however, we currently do not have benchmarks that contain a typology of such tasks. the most natural source for such data would be query logs @xcite or chat histories @xcite . however, these data sources do not specifically reflect domain-specific use cases and do not allow us to study specific use cases in a controlled manner. in this work, we bridge this gap by eliciting 519 methodical task descriptions (see a few examples in figure 1 ) from 266 experts across 25 different fields (section 3.1). these writing tasks figure 1 : a sample of methodical tasks from law, biology and medicine in dolomites. each task in dolomites follows a standard template, containing a task objective, task procedure, additional notes about the task, and finally, input sections that are usually expected for the task, and output sections that need to be produced as part of the task. these tasks are instantiated with examples that represent plausible inputs and outputs for the task (section 3.4). figure 2 : dolomites contains descriptions of 519 methodical tasks elicited from domain experts across various fields. we instantiate these tasks with examples that contain plausible inputs and outputs, formulating a challenging long-form generation problem that requires domain expertise and structured problem-solving. to evaluate the ability of existing models to assist experts with these tasks, we collect examples (see figure 2 ), where we instantiate each task with plausible inputs and outputs (section 3.4). examples are created semi-automatically: we first retrieve web documents that could potentially serve as samples of the task, and then generate an example using a language model based on the retrieved web documents. these examples are then significantly post-edited by the same experts (who contributed the task) for improving adherence to the task description, factual correctness, and level of detail. we use our benchmark, called dolomites (short for domain-specific long-form methodical tasks), to evaluate current models in their ability to generate accurate and detailed outputs (section 4). we formulate the modeling problem as long-form generation, where models are provided the task description and the example input and asked to generate the example output. our experiments reveal that there is significant headroom in improving performance on methodical tasks (section 5) which are inherently difficult (requiring reasoning skills and domain-knowledge), and in terms of improving automatic evaluation of long-form text. in addition to well-known shortcomings @xcite , conventional metrics are not designed to capture expert knowledge. we hope that dolomites can serve as a reference for domain-specific use cases of language models and provide a means for evaluating future models. we release our dataset and code at https:// dolomites-benchmark.github.io . ## problem formulation in this section, we first describe the types of writing tasks considered in this work. we refer to these tasks as methodical writing tasks due to two properties that are common to their execution. firstly, each task requires structured problem-solving, where the task follows a specific order where each step logically flows from the previous ones. for instance, in the medicine task in figure 1 , the task requires producing an assessment of the patient, then a plan of care and finally a full evaluation of the patient. secondly, every task usually follows a consistent execution across inputs, where there is a standard specification of the input, the output and the procedure for the task. in the same medicine task, given a patient's subjective and objective data, the task structure and procedure would mostly stay consistent across patients. to elicit descriptions of tasks from experts, we operationalize our definition of a methodical task into a standard template (see figure 1 for examples). we require that every task contains a brief task objective, a task procedure walking a beginner through how this task is conducted, input and output sections, which include information that is typically given, and information that needs to be generated. both input and output sections are formatted in the form of section titles and section descriptions. finally, we collect additional notes about the task, which can include best practices or common mistakes, and missing context that is important when conducting the task. we further expect our tasks to meet the following criteria: (1) they are purely textual and do not involve other modalities in the input or output; aside from task descriptions, our dataset contains specific instantiations of methodical tasks (see figure 2 ). we create examples by populating descriptions like those shown in figure 1 with plausible input and output sections (see section 3.4). ## results and discussion human evaluation. to evaluate the reliability of automatic metrics, we measure how well the above automatic evaluation measures correlate with human judgments. specifically, we sample 200 pairs of model outputs, where each pair comes from two randomly chosen models. we (the authors) then label which model output is better (or if they are tied) according to their task adherence, factual correctness, and depth. 3 on these 200 pairs, we also get automatic preference judgments from all the evaluation measures discussed in section 4.2 (we convert float scores for two outputs into binary judgments). the percentage agreements between human labels (with and without pairs with ties) and all evaluation measures a single reference for some of these examples, especially when the task is more subjective. pairwise lm evaluation results. we show the win rates according to different lm evaluators in fine-grained lm evaluation results. finally, we show ratings of models according to finer-grained aspects of response quality in ## discussion as part of dolomites, we present various data artifacts that can be used to study the abilities of language models in assisting domain experts with writing tasks. we outline these artifacts and their intended use below. task collection. we present a collection of 519 writing tasks spanning 25 fields that is representative of work undertaken by domain experts on a regular basis. we believe this is the first collection of tasks built with input from domain experts about scenarios in which language models can be useful to them. these tasks can be used to identify applications of lms in various fields and to cater model development to ecologically valid tasks. ## conclusion we introduce dolomites, a benchmark that is closely tied to realistic use cases of domain experts. the generalization of these use cases as methodical tasks provides a way to study capabilities of language models across tasks and domains. we consider a scenario where ai systems can act as tools for experts to amplify their problem-solving capabilities @xcite and perform their tasks more efficiently. we verify that our tasks are representative across fields and that human oversight is necessary if language models propose initial outputs for these tasks. evaluation of a broad range of contemporary language models suggests that there is a large room for models to improve on generating outputs for our tasks. future directions are many and varied. the tasks in dolomites constitute a mere sample from 25 fields in english language. we hope to further expand the set of tasks to cover a wider range of scenarios and languages. we could also consider tasks that involve modalities other than text in input or output, and multi-turn settings, where models continually improve their outputs through feedback and revision. on the modeling front, we will consider sophisticated generation techniques such as the one proposed by @xcite , that first generate a plan of the output and then fill in different sections, potentially with attributions to sources @xcite . our experimental results revealed that automatic evaluation of generated text is particularly challenging. our data contains a single reference output for an example input and does not model diverse perspectives of experts and the innate subjectivity of tasks @xcite . while conventional metrics do not account for this subjectivity, it is unclear if lm-based evaluators innately capture this subjectivity. more research is needed to ensure language model responses are given credit for alternative, but valid responses. for computational linguistics (volume 1: long papers), pages 9440-9450, bangkok, thailand. association for computational linguistics. https://doi.org/10.18653/v1 /2024.acl-long.511 congying xia, chen xing, jiangshu du, xinyi yang, yihao feng, ran xu, wenpeng yin, and caiming xiong. 2024. fofo: a benchmark to evaluate llms' format-following capability. in proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers), pages 680-699, bangkok, thailand. association for computational linguistics. https://doi.org/10 .18653/v1/2024.acl-long.40 wenting zhao, xiang ren, jack hessel, claire cardie, yejin choi, and yuntian deng. 2023. (inthe) wildchat: 570k chatgpt interaction logs in the wild. in the twelfth international conference on learning representations. lianmin zheng, wei-lin chiang, ying sheng, siyuan zhuang, zhanghao wu, yonghao zhuang, zi lin, zhuohan li, dacheng li, eric xing, hao zhang, joseph e. gonzalez, and ion stoica. 2023. judging llm-as-a-judge with mt-bench and chatbot arena. in thirty-seventh conference on neural information processing systems datasets and benchmarks track.