Process-Level Representation of Scientific Protocols with Interactive Annotation (2021)

Abstract: We develop Process Execution Graphs (PEG), a document-level representation of real-world wet lab biochemistry protocols, addressing challenges such as cross-sentence relations, long-range coreference, grounding, and implicit arguments. We manually annotate PEGs in a corpus of complex lab protocols with a novel interactive textual simulator that keeps track of entity traits and semantic constraints during annotation. We use this data to develop graph-prediction models, finding them to be good at entity identification and local relation extraction, while our corpus facilitates further exploration of challenging long-range relations.

## introduction there is a drive in recent years towards automating wet lab environments, where menial benchwork procedures such as pipetting, centrifuging, or incubation are software-controlled, and either executed by fully automatic lab equipment @xcite , or with a human-in-the-loop @xcite . these environments allow reliable and precise experiment reproducbility while relieving researchers from tedious and laborious work which is prone to human error @xcite . to achieve this, several programmatic formalisms are developed to describe an experiment as an executable program. for example, autoprotocol (lee and @xcite defines a mix predicate taking three arguments: mode, speed, and duration. in this work, we focus on a crucial first step towards such conversion -the extraction and representation of the relations conveyed by the protocol in a formal graph structure, termed process execution graphs (peg), exemplified in figure 1 . pegs capture both concrete, exact quantities ("30 minutes"), as well as vague instructions ("swirl gently"). a researcher can then port the peg (either manually or automatically) to their specific lab equipment, e.g., specifying what constitutes a gentle swirl setting and adding missing arguments, such as the temperature of the figure 2 : example interaction with our simulator, showing predicate grounding ("chill" is a temp_type operation) input assignment ("vial" is an argument of "chill"), validation (warning for a missing argument) and auto-complete driven by state-tracking, where only legal instructions in a given state are presented. formally, pegs are directed, acyclic labeled graphs, capturing how objects in the lab (e.g., cells, tubes) are manipulated by lab operations (e.g., mixing, incubating), and in what order. importantly, pegs capture relations which may span across multiple sentences and implicit arguments. for example, the peg in figure 1 explicitly captures the relation between culture tubes, mentioned in the first sentence, and swirl and incubate which appear in later sentences. to annotate long and complex lab protocols, we develop a text-based game annotation interface simulating objects and actions in a lab environment (see example in figure 2 ). our annotators are given wet-lab protocols written in natural language taken from biochemistry publications, and are asked to repeat their steps by issuing textual commands to the simulator. the commands are deterministically converted to our peg representation. this interface takes much of the burden off annotators by keeping track of object traits and commonsense constraints. for example, when the annotator issues a transfer command for a container, the simulator moves all its contents as well. we find that in-house annotators were able to effectively use this interface on complex protocols, achieving good agreement. finally, we use this data to explore several models, building upon recent advances in graph prediction algorithms @xcite . we thoroughly analyze model performance and find that our data introduces interesting new challenges, such as complex coreference resolution and long-range, cross-sentence relation identification. in conclusion, we make the following contributions: ## background and motivation several formalisms for programmatic lab controller interfaces were developed in recent years @xcite . for instance, autoprotocol defines 35 lab commands, including spin, incubate, and mix. foot_0 while these define wet-lab experiments in a precise and unambiguous manner, they do not readily replace their natural language description in scientific publications, much like a model implementation in python does not replace its high-level description in ml papers. similarly to ml model descriptions, lab protocols are often not specified enough to support direct conversion to low-level programs. for example, the protocol in figure 1 does not specify the swirling (mixing) speed or its duration. our process execution graph (peg) captures the predicate-argument structure of the protocol, allowing it to be more lenient than a programming language (for example, capturing that gently modifies swirl). better suited to represent underspecified natural language, pegs can serve as a convenient scaffold to support downstream tasks such as text-to-code assistants @xcite . for example, by asking researchers to fill in missing required arguments for swirl. to annotate pegs, we leverage the sentencelevel annotations of @xcite (wlp henceforth). wlp, exemplified at the top of figure 1 , collected sentence-level structures using the brat annotation tool @xcite . for example, capturing that cells, culture tubes are arguments for add. however, wlp does not capture cross-sentence implicit relations such that culture tubes are an argument for incubate. these are abundant in lab protocols, require tracking entities across many sentences, and are not easy to annotate using brat (see discussion in §4). we vastly extend upon wlp annotations, aiming to capture the full set of expressed protocol relations, using a novel text-based games annotation interface which lends itself to procedural text annotation. nodes peg nodes are triggered by explicit text spans in the protocol, e.g., "swirl", or "ice". nodes consist of two types: (1) predicates, marked in orange: denoting lab operations, such as add or incubate; and (2) arguments, marked in blue: representing physical lab objects (e.g., culture tubes, cells), exact quantities (30 minutes), or abstract instructions (e.g., gently). ## data collection: the x-wlp corpus in this section, we describe in detail the creation of our annotated corpus: x-wlp. the protocols in x-wlp are a subset (44.8%) of those annotated in the wlp corpus. these were chosen because they are covered well by autoprotocol's ontology (for details on ontology coverage, see §a.1). in total, we collected 3,708 sentences (54.1k x-wlp (ours) msptc csp propara # words 54k 56k 45k 29k # words / sent. 14.6 26 25.8 9 # sentences 3,708 2,113 1,764 3,300 # sentences / docs. 13.29 9 n/a 6.8 # docs. 279 230 n/a 488 ## models we present two approaches for peg prediction. first, in §5.1 we design models for separate graph sub-component prediction, which are chained to form a pipeline peg prediction model. second, in §5.2 we present a model which directly predicts the entire peg using a span-graph prediction approach. ## experiments in §5, we presented a pipelined approach to peg prediction based on scibert and a messagepassing neural network that jointly learns span and relation representations. next, we describe the details of our experiments and present empirical results demonstrating that x-wlp supports training models that can predict pegs from natural language instructions. data. x-wlp is our main dataset including 279 fully annotated protocols. statistics of x-wlp are presented in model setup. the pipeline framework employs a separate model for each task, by default using the propagated predictions from previous tasks as input. in addition, we evaluate the model for each task with gold input denoted as pipeline(gold). finally, the multi-task framework learns all tasks together and we decompose its performance into the component subtasks. ## related work natural language processing (nlp) for scientific procedural text is a rapidly growing field. todate, most approaches have focused on textmining applications @xcite and typically annotate only shallow, sentence-level semantic structures (e.g., fig. 1 , top ). examples include wlp @xcite and materials science procedures @xcite . recent interest in automation of lab procedures has also led to sentence-level annotation of procedural texts with action sequences designed to facilitate execution @xcite . however, as noted in recent concurrent work @xcite , neither sentence-level semantic structures nor action sequences are sufficient for the goal of converting text to a machine-executable synthesis procedure; for this purpose, a more structured, process-level semantic representation is required. in particular, executable representations require a structured declaration of the locations and states of the different materials throughout a process, details not represented by sentence-level annotations. our simulator can naturally represent such information by maintaining a stateful model of the process. simulation fidelity can be controlled by implementing the execution semantics of operations to the level of detail required. mehr et al. ( 2020 ) have similarly proposed a process-level executable representation, but use an nlp pipeline consisting primarily of rules and simple pattern matching, relying on a human-inthe-loop for corrections; linking our approach with their framework is a promising future direction. structurally, pegs are similar to abstract meaning representation (amr; banarescu et al. 2013), allowing us to use agreement and performance metrics developed for amr. in contrast with the sentence-level amr, a major challenge in this work is annotating and predicting procedure-level representations. 8 another line of research focuses on procedural text understanding for more general domains: simple scientific processes @xcite , open domain procedural texts @xcite , and cooking recipes @xcite . these works represent process-level information and entity state changes, but typically feature shorter processes, simpler language and an open ontology, compared with our domain-specific terminology and grounded ontology. our framework also provides a link to text-based game approaches to procedural text understanding. tamari et al. (2019 ) modelled scientific procedures with text-based games but used only synthetic data. our simulator enables leveraging recent advances on text-based games agents (e.g., @xcite ) towards natural language understanding. ## conclusion we developed a novel meaning representation and simulation-based annotation interface, enabling the collection of process-level annotations of experimental procedures, as well as two parsers (pipeline and joint modelling) trained on this data. our dataset and experiments present several directions for future work, including the modelling of challenging long range dependencies, application of text-based games for procedural text understanding, and extending simulation-based annotation to new domains.