Detecting Anatomical and Functional Connectivity Relations in Biomedical Literature via Language Representation Models (2021)

Abstract: Understanding of nerve-organ interactions is crucial to facilitate the development of effective bioelectronic treatments. Towards the end of developing a systematized and computable wiring diagram of the autonomic nervous system (ANS), we introduce a curated ANS connectivity corpus together with several neural language representation model based connectivity relation extraction systems. We also show that active learning guided curation for labeled corpus expansion significantly outperforms randomly selecting connectivity relation candidates minimizing curation effort. Our final relation extraction system achieves F_1 = 72.8% on anatomical connectivity and F_1 = 74.6% on functional connectivity relation extraction.

## introduction the nih common fund's stimulating peripheral activity to relieve conditions (sparc) program aims to transform our understanding of nerve-organ interactions to help spur the development of effective bioelectronic treatments. bioelectronic medicine represents the convergence of molecular medicine, neuroscience, engineering and computing to develop devices to diagnose and treat diseases @xcite . one of the projects within this large consortium is to create a systematized and computable wiring diagram of the autonomic nervous system, a part of the "wiring system" that travels throughout the body transmitting messages between the peripheral organs and the brain or spinal cord. while diagrams of nerves are currently available in medical texts @xcite , the sparc program seeks to map these connections at higher levels of detail and with greater accuracy. additionally, the diagrams in these medical texts are not generally queryable, nor are they sufficiently detailed to include the granular paths that these nerves travel. such information would be needed, for example, to understand where reliable access points to a particular nerve might be so that stimulation only affects the most relevant nerve or to understand the mechanisms behind stimulation applied at particular locations. many scientific studies contain information about individual nerves and at times the paths they traverse, but to our knowledge, no systematic approach has been attempted to bring these large quantities of information together into a computationally accessible format. the sparc project is building a cross-species connectivity knowledge base that contains detailed information about individual nerves, their pathways, cells of origin and synaptic targets. to date, this knowledge base has been populated through the development of detailed models of circuitry by experts funded through the sparc project using the apinatomy platform @xcite . apinatomy provides a modeling language for representing the complexity of functional and anatomical circuitry in a standardized form. the circuitry contained in these models represent expert knowledge derived from the synthesis of the expert's own work and the synthesis of, in some cases , hundreds of scientific publications. however, to ensure that information in the sparc knowledge base is comprehensive and up to date, i.e., it represents the current state of knowledge about autonomic nervous system (ans) connectivity, we sought to augment the expert-based model approach with experimental information derived from the primary scientific literature. as there are thousands of papers and additional sources like text books, we utilized natural language processing to identify sentences that contained information on neuronal connectivity in the ans. the task was approached by first gathering the relevant scientific literature by matching bodily structures at a variety of anatomical levels (i.e. gasserian ganglion, vagus nerve, brainstem, etc.) from a constructed set of vocabulary at sentence level. then, annotators classified each structure to structure relationship using only the information provided within the sentence based on the connectivity types defined in our annotation guideline. this structured data were then used to train our connectivity relations models. data from two curators was used to assess the inter-curator agreement to determine if the annotation guidelines are sufficient to "teach" the task to humans. we assessed connectivity statements into several types including, anatomical connectivity, functional connectivity, structural connectivity, topological connectivity and general connectivity as well as no connectivity. the general connectivity and no connectivity categories can be thought of as statements that are too vague to be of much direct use for our use case. the most important statements are anatomical connectivity, elucidating which parts are connected physically and functional connectivity, elucidating which parts are connected functionally. a definition and an example for each connectivity type used for annotation is shown in in relation extraction, long-range relations are usually handled using dependency parse tree information. in traditional feature-based models, paths in the dependency parse tree between entities are used used as features @xcite which suffered from the sparsity of the feature patterns. more recently, neural models are increasingly employed for relation extraction instead of feature engineering using vectorized word embeddings. the dependency information is represented as computation graphs along the parse tree @xcite . sequence models, on the other hand, work at the surface level and represent long distance relationships via either convolutional or recurrent neural networks and an attention mechanism @xcite . in biomedical domain, relation extraction work is traditionally focused on protein-protein, genedisease or protein-chemical interactions. several labeled datasets, such as gad @xcite (a gene-disease relation dataset) and chemprot @xcite (a proteinchemical multi-relation dataset) are publicly available. neural sequence models have also been applied to protein-chemical relation extraction task @xcite . recently, sentence level transformer based language representation models such as bert @xcite have shown superior downstream performance on many nlp tasks. a biomedical domain adapted version of bert called biobert @xcite has been shown state of the art performance on several biomedical relation extraction tasks. while most of the transformer based language representation models are pretrained on sentences where a predefined percentage of the tokens are masked and the model learns to predict the masked tokens, a recently introduced language representation model, electra @xcite learns to discriminate if a token in the original input is replaced by a language generator model or not. the generator model is a bert like generative model that is co-trained with the discriminative model. while there are efforts to extract brain connectivity information from neuroscience literature @xcite , their focus is in the cognitive parts of the brain instead of ans. in this paper, we introduce a labeled ans connectivity corpus, together with four biomedical domain adapted electra models, that we have used to develop an anatomical and functional connectivity relation extraction system that outperforms biobert. ## experiments we conducted our experiments in two phases. in the first phase, all the binary connectivity relation candidates in the 805 sentences extracted from the open access subset of pubmed central is annotated by a curator. the curated base set is then randomly split into 80/20% train/test set. afterwards, ten randomly initialized models are trained. the reported results are average of 10 runs together with model params architecture steps train time/hardware mid 50m hidden:512, layers:12 1.2m 6.5d on 8 tpuv3s base 110m hidden:768, layers:12 1.2m 12.5d on 8 tpuv3s mid-tall 88m hidden:512, layers:24, batch:128 1m 5.5d on 8 tpuv3s mid combined 50m hidden:512, layers:12 1.2m 6.5d on 8 tpuv3s as our baseline model, we have used a graph convolution over dependency parse tree neural model @xcite where the dependency graph structure is represented by an adjacency matrix over which convolution operations are performed. the model uses word embedding vectors for input encoding and stacked layers of graph convolution network (gcn) layers to encode relations. the input encodings can be further contexualized via a bi-directional long-short-term memory (lstm) layer, which we have used in our experiments. for word embeddings we have used 300 dimensional common crawl (840b tokens) trained glove @xcite vectors. the dependency parse trees for the input sentences were generated via stanford corenlp @xcite package. all the other models are fine-tuned from pretrained transformers based language representation models. we have downloaded bio-electra++ from zenodo 1 . besides our four biomedical corpus pretrained electra models, we have used biobert @xcite version 1.1 and elec-tra base models. the binary anatomical structure entities are masked in candidate sentences as in @xcite . besides that, no further preprocessing is done. all the models are trained for three epochs, using the the default learning rate and maximum allowed batch size for our 8gb nvidia rtx 2070 gpu. the test performance of models tested are summarized in ## discussion connectivity relations constituted only about 12% of the connectivity relationship candidates in our corpus. taking this into account, the anatomical and functional connectivity detection performance of our final classifier is good enough to be used for ans connectivity knowledge base construction with drastically reduced domain expert curation. when looking at our model's performance, we considered errors at the level of individual connectivity relations labels, meaning we could (and did) have some sentences with multiple errors. we defined errors as cases where relation labels tagged by the model and the annotator did not agree. we performed our error analysis in two phases. in phase 1, our analysis was performed using only binary connectivity data (i.e. did annotators mark a relationship using any type of connectivity or as no relation) from 40 connectivity errors: 19 false positives and 21 false negatives. false positives were defined as instances when the model predicted connectivity when there was actually no relationship (as defined by the annotator). inversely, false negatives were cases when the model predicted structures to have no relationship when there was actually some type of connectivity. in phase 2, our the first identified error pattern was mislabeled data due to human error. we noticed 4 instances where the data was mislabeled. the second type of error occurred because a solid line of demarcation between connectivity types was difficult to establish due to ambiguities in our curation guidelines and the overall difficulty of the task. when we began annotating, connectivity between structures could only be defined as anatomical, functional or having no relation. after discussing the differences in our annotations though, it became apparent we needed to add additional connectivity types to clarify the lines of demarcation between each. as a result, we added structural, topological, and general connectivities, and while we did see improved classifier performance after adding these, it appears we weren't entirely successful in our attempts to explicate our connectivity types. in the example, "the central amygdaloid nucleus (cea) and the bed nucleus of the stria terminalis (bnst), which is considered to be a component of the "extended amygdala", establish important connections with the hypothalamus and other brain areas controlling visceral and sensory information." bnst and the extended amygdala were incorrectly identified as having anatomical connectivity. this "part of the whole" pattern was seen multiple times in our errors (binary, anatomical and functional), primarily as a false positive. in future works, it might benefit curation efforts to add additional connectivity type(s), e.g. fractional connectivity for this "part of the whole" pattern, in order to further elucidate lines of demarcation. additionally, because connections between structures are not always obvious, even to human curators (i.e. if a is connected to b and b is connected to c, is a always connected to c?), the lines of demarcation separating connectivity types may always remain somewhat hazy. we were able to identify a few additional patterns by comparing the syntax and vocabulary of sentences with errors to that of sentences without errors. in general though, sentences with errors tended to have far more complex sentence structure than sentences without. more specifically, error prone sentences generally contained far more prepositional phrases and compound subject and verb phrases. for example, we saw multiple errors within the following sentence: "chemoreceptors in the carotid body or aortic body in the walls of the internal carotid artery or the aorta sense the level of oxygen or carbon dioxide in the blood and convey these signals via the glossopharyngeal and vagus nerves to the nucleus of the tractus solitarius." just from a cursory glance, it becomes obvious that this sentence is complicated; it contains multiple subject and verb phrases clouded by prepositional phrases. unfortunately, the convoluted nature of the sentence hurts readability for both humans and machines. because humans also tend to have issues understanding these highly complex sentences, we feel the best solution is for authors to limit the complexity of their sentences to reasonable levels when possible. if a sentence is too complex for a human to understand, it will most likely be too complex for a computer. additionally, we noticed persistent issues when subjects were not explicit. unresolved pronouns (e.g. pronouns whose antecedents are unknown) and ambiguous body structures (i.e. fibers) tended to cause errors wherein the model would correctly identify that the sentence contained connectivity but would incorrectly identify which structures are connected. with regards to verb usage, our model seemed to perform better when the connectivity between structures was described in active voice rather than passive. one potential explanation is that sentences using active voice tend to be more clear and simple than sentences using passive voice. lastly, our model seemed to perform worse the further apart the two connecting structures were within the sentence. ## conclusions in this paper, we introduced a labeled corpus for ans connectivity relations which is further expanded via active learning. the labeled ans connectivity relation corpus is used to develop relation extraction systems mostly based on language representation neural models. we have introduced four biomedical domain pretrained electra @xcite based discriminative language representation models, two of which have outperformed biobert @xcite on the ans connectivity relation extraction task. using active learning guided curation, the labeled corpus is expanded minimizing the curation effort while significantly improving ans connectivity relation extraction performance. based on the observed benefits of the active learning, we are planning to use our bio-electra based relation extraction system in a web based tool for ans connectivity knowledge base construction with active learning based continuous learning ability.