Text-to-Text Extraction and Verbalization of Biomedical Event Graphs (2022)

Abstract: Biomedical events represent complex, graphical, and semantically rich interactions expressed in the scientific literature. Almost all contributions in the event realm orbit around semantic parsing, usually employing discriminative architectures and cumbersome multi-step pipelines limited to a small number of target interaction types. We present the first lightweight framework to solve both event extraction and event verbalization with a unified text-to-text approach, allowing us to fuse all the resources so far designed for different tasks. To this end, we present a new event graph linearization technique and release highly comprehensive event-text paired datasets, covering more than 150 event types from multiple biology subareas (English language). By streamlining parsing and generation to translations, we propose baseline transformer model results according to multiple biomedical text mining benchmarks and NLG metrics. Our extractive models achieve greater state-of-the-art performance than single-task competitors and show promising capabilities for the controlled generation of coherent natural language utterances from structured data.

## introduction in recent years, events have become an influential formalism for modeling complex relations mentioned within the text as semantic graphs @xcite . in bioinformatics, an event generally refers to an interaction between one or more biomedical entities (e.g., proteins, genes, diseases, drugs), each contributing with a specific role (e.g., theme, cause, site). for instance, biomedical events include molecular reactions, organismlevel outcomes, and adverse drug reactions. their expressive power and flexibility have supported * equal contribution. 1 the data and the code to reproduce our baseline results are available at https://github.com/disi-unibonlp/bio-ee-egv many practical applications like literature-based knowledge discovery @xcite , biological network construction @xcite , diagnosis prediction @xcite , document summarization @xcite , and question answering @xcite ). text-to-event (or event extraction, ee) and eventto-text (or event graph verbalization, egv) systems effectively bridge natural language and symbolic representations. they provide a step towards decoupling concept units (what to say) from language competencies (how to say it) @xcite . strongly linked to natural language understanding, ee is a fundamental task to automatically identify, monitor, and aggregate the relational knowledge disseminated within life science papers, speeding up medical progress and promoting discoveries. yet, although much attention has been paid to ee, no research efforts have been directed to its inverse task, namely egv. even if under-explored, egv targets the generation of informative text constrained on semantic graphs, holding a lot of potential in applications like conversational agents and summarization systems @xcite @xcite . most state-of-the-art (sota) approaches handle structured prediction by employing task-specific architectures and discriminative models. ordinarily, they need to be adapted to the target events and their schema, or they are not flexible enough to work with different domains or analysis granularities (sentence-vs document-level). the extraction process is typically divided into subtasks, executed in a pipeline or joint manner, where the output of several classifiers needs to be integrated. additionally, each task is often associated with its own output space, limiting knowledge sharing and multi-task learning (mtl). classes (i.e., event, argument role, and entity types) are specified implicitly through numerical indices, and models contain no prior information about their meaning. furthermore, modern deep learning solutions require a non-trivial amount of examples to train, but event annotations are expensive to produce-a discrepancy that results in multiple, stand-alone, and closed-domain datasets with few records and potentially overlapping labels @xcite . on a parallel track, transfer learning has been the pinnacle of the latest breakthroughs in natural language processing (nlp). large pre-trained language models (plms) are powerful backbones that can be fine-tuned for different tasks to achieve impressive performance in wide-ranging applications @xcite . plms capture contextual information and latent linguistic/relational knowledge @xcite , incorporating syntax and semantics. in that sense, textual representations and conditional generative modeling can be seen as natural ways of encoding different events in a shared predictive space. in this paper, we design a framework to solve both ee and egv as text-to-text problems, thus leveraging sota plms and disposing of the need for complex and hardly adaptable architectures. concretely, we propose a way to decompose events into text sequences, neatly preserving structure and labels. above it, we present the biomedical textto-event (biot2e) and event-to-text (bioe2t) datasets, two corpora of textualized biomedical event graphs paired with their mention. precisely, we aggregate and preprocess gold annotations coming from 10 popular ee benchmarks, intending to systematize the community work matured with public evaluation programs and solving the low coverage issue. among the exciting multimodal opportunities enabled by these datasets, we show that out-of-the-box transformer models can effectively learn @xmath0. we achieve this symmetry by using the same architecture for parsing and generation, as well as for all event instances, originally belonging to separate ee tasks with independent output spaces. to the best of our knowledge, this is the first study to handle such a variety of event schema without distinct models or additional task-specific modules. our key contributions are the following: graph-text paired data. many graph-text paired datasets have sprung up. nevertheless, annotating text or semantic graphs is expensive, especially for specific fields like biology. most of the resources are domain-general and focus on knowledge graphs (kgs). although there are datasets assembled by crowdsourced human annotators-such as webnlg @xcite , one common thread is using nlp tools and automatic alignment heuristics to forge silver pairs massively, e.g., mapping wikipedia sentences to wikidata triples @xcite or wikipedia paragraphs to freebase subgraphs @xcite . predicate linkers and plms are already used to inherently construct kgs from the biomedical literature @xcite , but the relations extracted for each document are generally not openly released. in contrast, we present the first datasets directly pairing scientific sentences to biomedical event graphs, usable as evaluation gold standards thanks to expert user provenance. ## event linearization seq2seq models require that both the input and target be presented as a linear sequence of tokens. in this section, we describe our format design concept to reformulate event graphs as strings. events have an n-ary and potentially nested structure, with optional modifiers (e.g., "polarity", "certainty level") reshaping the described interaction. like many other relational data, events can be conveniently and naturally modeled as rooted directed acyclic graphs @xcite . with this formalization, triggers and entities are nodes, while argument roles define edges. each trigger, entity, and trigger-trigger/trigger-entity association is assigned to its type according to a predefined ontology foot_1 . we revisit the formulation by @xcite and put forward a formal event language designed to be easy and deterministically reversible to event graphs. while being more complex to learn, our linearization comes with the advantage of enclosing entire events in single expressions, reducing the overhead of generating different output sequences for trigger and argument annotations. each node is surrounded by the special tokens @xcite , which represent semantic structure indicators. inside, a sequence of |-separated tags reports the text span, the type (described in natural words), and a list of @xmath1, where x is the argument role and y is the target trigger. note that the same entity can be coupled to different events (triggers) with distinct roles (i.e., double tagging). the root trigger is a source vertex and has not incoming edges. trigger nodes also specify event-level modifiers as additional @xmath2, in the form @xmath3.g., "po-@xmath4. so, the information on event components (i.e., nodes and their embedded interconnections) are all within [. . . | . . .] patterns, which can be nested in case of sub-events. the final string minimizes the number of tokens to be submitted or generated so as to make encoding/decoding more efficient. event constituents are sorted by their order of appearance in the .a2 for consistency. we define a context-free grammar ( §a.1) and test it with jflap @xcite . figure 2 depicts a practical example of textualized nested bio-event. ## datasets based on §3, we build new corpora suitable for textor graph-conditioned sequence modeling. here, we present the construction process of biot2e and bioe2t, together with their main properties. ## experimental setup in this section, we provide the formal definition of text-to-event parsing and event-to-text generation. then, describe the setup of the experiments we conducted to evaluate our framework in both tasks. ## conclusion this paper presented the first sequence-to-sequence framework for both biomedical event extraction and verbalization. concretely, we proposed biot2e and bioe2t, two highly comprehensive datasets with parallel text-event gold annotations, constructed through a novel linearization technique. by training autoregressive language models on them, we achieved an average f1-score of 0.81 on ten benchmarks, making considerable improvements over previously published work. in stark contrast with discriminative solutions, we employed the same architecture to perform previously distinct tasks, exploiting pre-trained knowledge and label semantics. experimental results also proved the usefulness of (i) knowledge sharing between different biomedical spheres in event-based tasks, (ii) events in improving model understanding for nlp tasks in general, like document summarization. we hope that our contributions will lead to further progress in natural language understanding and generation as transfer learning becomes even more vital for graph-to-text and text-to-graph translations. future directions at the edge of our knowledge, this is the first work that proposes single deep neural models capable of effectively extracting (and back-translating to text) such a variety of biomedical events and their components. this high ontological coverage opens the door to numerous applications and research blueprints. future work should tackle: (i) document-level granularities; (ii) prompting-based purely generative models @xcite ; (iii) text←→graph boosting approaches echoing autoencoders and cycle-gt @xcite ; (iv) few-shot learning; (v) event aggregation towards automatic corpus-level knowledge graph learning @xcite @xcite ; (vi) conversion of events to logic and constrained decoding algorithms @xcite ; (vii) infusion of events in pre-trained language models for tasks like biomedical multi-document summarization @xcite and information retrieval @xcite .