Beyond Text: Characterizing Domain Expert Needs in Document Research (2025)

Abstract: Working with documents is a key part of almost any knowledge work, from contextualizing research in a literature review to reviewing legal precedent. Recently, as their capabilities have expanded, primarily text-based NLP systems have often been billed as able to assist or even automate this kind of work. But to what extent are these systems able to model these tasks as experts conceptualize and perform them now? In this study, we interview sixteen domain experts across two domains to understand their processes of document research, and compare it to the current state of NLP systems. We find that our participants processes are idiosyncratic, iterative, and rely extensively on the social context of a document in addition its content, and that approaches in NLP and adjacent fields that explicitly center the document as an object, rather than as merely a container for text, tend to better reflect our participants’ priorities. We call on the NLP community to more carefully consider the role of the document in building useful tools that are accessible, personalizable, iterative, and socially aware.

## introduction from contextualizing scientific research in literature reviews, to understanding the functioning of complex organizations, experts conduct a wide variety of tasks that depend on document research. document research, i.e. reading, understanding, and otherwise working with collections of documents is a process that underlies almost all knowledge work. as such, there is a rich body of literature that aims to understand how experts in various fields read documents, characterizing goals, processes, and how their experiences and knowledge inform what and how they read @xcite @xcite . more recently, primarily text-based nlp tools such as llms have been proposed as "solutions" to document research. general purpose commercial models are billed as being able to "understand" both single documents and even whole corpora, context length limits willing, and there are a growing number of purpose built tools targeted at particular professions, from legal document tasks @xcite @xcite to aiding in the process of scientific discovery (ai4science and quantum, 2023), with some claims going as far as to argue that some parts of the scientific process could soon be wholly automated @xcite . but to what degree are llms able to model document research and understanding as currently carried out by experts? in this study, we interview 16 domain experts working in materials science, law, and policy, to understand their processes of document research: their goals, the uses they have for these documents, and how they evaluate the documents' content for relevance and quality. in our analysis, we derive tasks common to the experts we interview, and assess the degree to which modern tools from nlp and adjacent fields address those tasks. we find that our experts' processes are highly personal and varied, involve iteratively constructing mental model, and are consistently informed not only by the content of the documents, but by the material and social context of their production. tools that design around this context, such as citation-aware tools for scientific support @xcite , better reflect experts' processes, though they tend to be limited to domains that explicitly center publication structure. by contrast, modern, general-purpose systems from nlp tend to reflect a common, informationcentric view: documents are merely containers for information, and that information within documents can therefore be segmented, decontextualized and displayed without regard for the source document. this is reflected in the affordances of the systems themselves, which often operate either on individual sentences, or on segments whose lengths are determined by an underlying model's context length or a chunking algorithm @xcite , which may or may not align with semantic boundaries @xcite . the experts we interviewed view documents in ways that cohere much more strongly with theory found in the science and technology studies (sts) literature: that documents are not merely conduits for information, but are traces of social processes whose details are crucial to the documents' interpretation and use. for our experts, these details can inform evaluating a document's provenance, assessing a document against background knowledge of a field, or considering the global context in which a document exists. in other words, a document "serves not simply to communicate, but also to coordinate social practices" @xcite . we therefore call for the nlp community to develop accessible systems and methodologies that are more personalizable, iterative and socially aware ( §6) in order to more accurately reflect the views and priorities of their users, and the rich social context in which documents are produced and consumed. ## methods we recruited 16 participants, beginning with a convenience sampling method @xcite , in which the authors began by interviewing existing non-computer science collaborators across the projects they worked on, and then by snowball sampling @xcite , in which participants were asked to recommend other interview candi-dates. our participants, six women and ten men, were drawn from collaborations in the materials science, law, and policy communities, were all based in the u.s. and had a wide age distribution, with five participants between ages 25 and 34, six between 35 and 44, three between 45 and 54, and two 55-64. ten of our participants were associated with materials science, though many identified themselves as belonging to other disciplines, such as chemical or mechanical engineering. all of our participants in this group were either professors or postdoctoral researchers whose primary focus was the synthesis, characterization, or modeling of materials. as such, the document research that they described to us was primarily the process of literature review: keeping up to date with the subfields they already worked in, or learning about new subfields that became interesting to their work. the remaining six participants were academics and professionals whose jobs involve reading, researching, interpreting, or otherwise engaging with law, public policy, or governmental records. this population's document research operated on many more kinds of documents, but the informational goals were largely similar, e.g. staying abreast of relevant policy, legal precedent, or public reactions to policy. these populations are neither very similar nor dissimilar; we use them as a way of understanding what themes in how experts work with documents might generalize across groups with nominally different tasks, and what themes might be specialized to a single domain. in essence, we aim to establish a loose lower bound of the variety of tasks that professionals across different domains carry out. we conducted semi-structured interviews (weiss, 1995) with our participants that lasted between 27 and 73 minutes, with the median interview lasting 53 minutes. interviews were conducted with a dedicated notetaker, and we recorded the interview audio with participant consent. we followed an interview guide (included in appendix a), that developed four broad themes: the participant's current work and positionality, their current process for document research including how it fit into their work, how they evaluate documents for relevance and/or quality, and finally what existing tools they use to perform document research. we developed this set of questions in collaboration with materials scientists on one author's project, and was evaluated with a test interview before wider interviewing, with only minor changes to the wording of some questions. the guide did not change between the two subpopulations we interviewed. we conducted the interviews between february and july 2024. following the procedures of grounded theory @xcite , at the conclusion of each interview, authors produced an analytical memo detailing the themes from that interview @xcite . after a sufficient number of interviews for recurring, coherent themes to emerge, authors began a process of independently open coding the data, developing a thematic taxonomy that generalized across interviews. after this, the authors met regularly to discuss and refine the open codes into a preliminary closed coding frame @xcite . authors then annotated each interview with the themes from this closed coding frame. in analysis meetings, the authors further refined this closed coding frame by adding, merging, and removing codes, then iteratively re-coding the data. the analysis of this paper emerged from closed-coded versions of the data, and was validated against the original transcripts for appropriate context. ## summary of interviews document research implied a wide variety of activities to our participants. despite the variety of tasks, however, several common threads emerged that bridged the disciplinary gap. we conceptualize the tasks carried out by our participants across domains to broadly fall into three categories: local context tasks, global context tasks, and corpus construction. in the following sections, we first characterize the logistics of the documents that our participants work with, describe some of the examples of each of these three types of task, as well as some broader themes across participants.