The Grid: A semi-automated tool to support expert-driven modeling (2024)

Abstract: When building models of human behavior, we often struggle to find data that capture important factors at the right level of granularity. In these cases, we must rely on expert knowledge to build models. To help partially automate the organization of expert knowledge for modeling, we combine natural language processing (NLP) and machine learning (ML) methods in a tool called the Grid. The Grid helps users organize textual knowledge into clickable cells aLong two dimensions using iterative, collaborative clustering. We conduct a user study to explore participants’ reactions to the Grid, as well as to investigate whether its clustering feature helps participants organize a corpus of expert knowledge. We find that participants using the Grid’s clustering feature appeared to work more efficiently than those without it, but written feedback about the clustering was critical. We conclude that the general design of the Grid was positively received and that some of the user challenges can likely be mitigated through the use of LLMs.

## introduction the increasing availability of text data has transformed our ability to model human behavior in social and economic systems. we can now monitor and model phenomena entirely through preexisting text sources like social media, news articles and journal papers. however, these data sometimes fail to capture the causal information we need to build models. for example, news articles may describe what has happened in a region (e.g., "farmers harvest early") but not why (e.g., "granivorous birds nearby"). in these cases, one of the best ways to interpret and supplement existing data is to ask local experts for causal explanations of how people think and behave. despite the value of expert knowledge, the process of converting it into models remains largely manual and expensive. fortunately, nlp and ml capabilities have drastically improved since the heyday of expert systems @xcite @xcite . if we can partially automate the work required to process expert knowledge, then we can drive more accurate and nuanced modeling of human behavior. while existing nlp and ml methods are powerful, processing expert knowledge presents different challenges than processing large pre-existing text corpora. with this in mind, we combine nlp, ml and visualization methods in a tool designed to satisfy the following criteria based on our experience building models from expert knowledge. first, nlp tools for processing expert knowledge must allow users to explore text quickly at multiple levels of abstraction. existing approaches often force a trade-off between digestible summaries and thorough analysis. for example, knowledge graphs can quickly orient users to important topics and relationships, but as the size of the knowledge base grows, topics must be aggregated for the graphs to remain interpretable by humans. similarly, while large language models (llms) are becoming ever more adept at answering questions and providing summaries, they alone do not support multiple levels of abstraction; rather, they require prompts that may be difficult to write during the early stages of analysis when the characteristics and objectives of the user's model are not yet defined. second, nlp tools for processing expert knowledge should assign work based on the different capabilities of humans and machines. many popular topic modeling methods are fully automated, but users are likely to have domain expertise, some familiarity with their corpora, and objectives for analysis and model-building. this expertise should be used to guide the machine. machines should relieve users of repetitive work and discover patterns that users might not detect, without overriding user decisions. tools should also support a range of processes and strategies from human users. in this paper we introduce the grid (figure 1 ), an expert knowledge tool designed to satisfy these two criteria. we first describe the mechanics of the grid, and then we report results from a user study. based on our results, we conclude that the grid supports the efficient organization of expert knowledge and report on challenges and potential solutions for future work on expert-driven modeling tools. ## the grid the grid is a tool for visualizing and curating expert knowledge. grids organize textual knowledge into clickable cells along two dimensions. the rows of the grid represent structural characteristics of the corpus that do not change across topics, and the columns represent topics from the corpus that the user and the grid work together to discover. the difference between rows and columns is illustrated in figure 1 . figure 1 .a shows a grid that was created to organize knowledge about the work of an artist, so the rows represent calendar years while the columns represent art media, locations, exhibitions and so on. figure 1 .b shows a grid that was created to organize interviews with an expert on rice production in senegal, so the rows represent interviewee and interview date and the columns represent agronomic topics. the color of each cell in the grid indicates how much text it contains. clicking on a grid cell reveals the sentences it contains and clicking on a sentence reveals the surrounding context (figure 1b ). the user can move and copy sentences between columns, rename columns, and generate columns anchored by keywords. since the rows in the grid represent immutable characteristics of the corpus, (e.g., dates or other properties properties of the data points), the user cannot manipulate rows in the same way. the next sections describe how the user and the grid work together to curate columns through iterative clustering. ## study methodology we conducted a user study to explore users' reactions to the grid and to investigate whether iterative, human-machine clustering helps users organize text more efficiently. we asked study participants to curate an 80-sentence corpus in the agricultural domain (see the section titled study corpus) using the grid and then take a timed test about concepts in the corpus. participants were assigned to three conditions with differing levels of automation. in the following section, we discuss the details of the study design. ## discussion and conclusions the grid combines nlp, ml and visualization methods to assist users in the organization of expert knowledge corpora. we have presented results from a user study meant to evaluate this combination of methods. here we draw conclusions about whether the grid successfully satisfies the criteria laid out in the introduction. first, we conclude that the grid allowed users to process the knowledge corpus quickly at multiple levels of abstraction. the organizational power and visualization of the grid was well-received by participants with diverse expertise and skill sets. participants in all conditions appreciated the speed with which they could organize information and even participants in the somewhat frustrating placebo condition were able to answer test questions using their grids. the high test results in all conditions may in part be a ceiling effect; however, we do not discount the role of the grid in allowing participants to rapidly familiarize themselves with a corpus they had not seen before. moreover, participants were afforded a large amount of flexibility in how they used the grid. participants were able to use different combinations of column creation and sentence dragging to organize information, and they reported preferences for different strategies in the written feedback. participants often shifted between large organizational moves like column creation and more precise moves like sentence dragging, indicating that the grid allowed them to work at different levels of abstraction. the number of columns for each participant was similar, but the column names were different, indicating that the grid allowed participants to organize information in the way that made the best sense to them. whether the grid successfully assigned work based on the different capabilities of humans and machines is less clear. the participants using the grid's clustering algorithm appeared to work more efficiently than those without it, but written feedback about the clustering was critical. participants in the placebo and treatment conditions reported that behavior of the grid's clustering algorithm was confusing and sometimes counterproductive, while participants in the control condition praised the much simpler automated column populating. this indicates that efficiency is not sufficient for a satisfying user experience and that future work on collaborative algorithms should focus on transparency. for example, the inclusion of llms in the collaborative process could allow for explanations of why sentences are grouped together in grid columns. the grid tool provides support for organizing expert knowledge in an expert-driven modeling pipeline. while our user study revealed some challenges in the design of such tools, we find the results encouraging and suspect that many of the lessons learned, such as the frustration with the clustering algorithm, may be mitigated in future versions by enlisting llms to provide explanations for the user. the grid can be expanded to include other parts of the knowledge engineering process, such as a semi-automated model generation step after knowledge has been organized. we conclude that semi-automated tools like the grid can play valuable roles in multiple research communities and have the potential to support more nuanced and local models of human behavior. the code for the grid tool is available at https: //github.com/allegra-cohen/grid .