Leveraging LLM s and Web-based Visualizations for Profiling Bacterial Host Organisms and Genetic Toolboxes (2024)

Abstract: Building genetic tools to engineer microorganisms is at the core of understanding and redesigning natural biological systems for useful purposes. Every project to build such a genetic toolbox for an organism starts with a survey of available tools. Despite a decade-long investment and advancement in the field, it is still challenging to mine information about a genetic tool published in the literature and connect that information to microbial genomics and other microbial databases. This information gap not only limits our ability to identify and adopt available tools to a new chassis but also conceals available opportunities to engineer a new microbial host. Recent advances in natural language processing (NLP), particularly large language models (LLMs), offer solutions by enabling efficient extraction of genetic terms and biological entities from a vast array of publications. This work present a method to automate this process, using text-mining to refine models with data from bioRxiv and other databases. We evaluated various LLMs to investigate their ability to recognize bacterial host organisms and genetic toolboxes for engineering. We demonstrate our methodology with a web application that integrates a conversational LLM and visualization tool, connecting user inquiries to genetic resources and literature findings, thereby saving researchers time, money and effort in their laboratory work.

## introduction our planet currently faces significant challenges concerning biological resources, including limited renewable energy sources, lack of innovative treatments for endemic infectious diseases, water pollution, insufficient arable land resulting in food crises, and the degradation of ecosystems (wef, 2020; @xcite , among other urgent issues. we postulate -as others have -that the capacity to domesticate and genetically engineer non-model microorganisms from relevant environments could facilitate the development of potential solutions to many of these urgent global problems @xcite . although recent technological advances have been made at a rapid pace to address several of these challenges, the information needed for each potential new model organism is dispersed across the literature and is not readily accessible to many practitioners. this situation complicates every new synthetic biology, bioenergy, and biomanufacturing project involving a non-model organism @xcite . the disorganized nature of the information not only impedes machine-readable approaches but also hinders the assessment of the scope of work and identification of knowledge gaps, subsequently offering limited guidance for investment to overcome technological barriers. for instance, despite decades of progress in the field of synthetic biology, it remains challenging to pinpoint suitable microbial targets for specific applications and conditions, as well as the genetic tools required for cultivating and engineering non-model microorganisms @xcite @xcite . a comprehensive literature mining tool that monitors emerging technologies and genetic tools critical for biotechnology professionals would be highly beneficial. this envisioned tool would allow us to identify information gaps and detect opportunities concealed within extensive literature. for instance, the tool should efficiently ascertain whether a chosen organism is suitable for laboratory domestication and which genetic tools are available for that organism, streamlining the search process and conserving time, effort, and funding for numerous lab-oriented projects. recent advancements in natural language processing (nlp) and large language model (llm) have enabled the analysis of textual data at an unprecedented scale (e.g., millions of documents), allowing for the extraction of significant contextual information in ways that were previously unachievable. these innovative techniques offer considerable potential to bridge the knowledge gap discussed earlier. in this study, we propose employing nlp on biological literature to identify organism traits (as depicted in figure 1) and systematically compiling this extracted knowledge. this approach facilitates the development of automated, curated centralized systems crucial for the cultivation and engineering of microorganisms. this work specifically focuses on the extraction of information about bacterial organism hosts and genetic toolbox for them from literature. as detailed in this paper, our contribution is threefold: our findings demonstrate the efficacy of finetuning llms on an annotated dataset, which enhances the models' performance in those entity recognition. ## related work llms demonstrated significant improvements in addressing a multitude of nlp tasks that are critical to the fields of biology and biomedicine @xcite . instructed on a broad spectrum of text corpora, encompassing web crawls, medical records, and rigorously selected datasets, llms are equipped with the proficiency to integrate information from diverse sources. these sources range from scientific publications and databases to various other forms of informational repositories. this integrative ability enables llms to identify complex interconnections, nuanced contextual aspects, and insights that may remain obscure to traditional methods. the biomistral model @xcite , built upon the mistral foundational model and subsequently pre-trained on pubmed central, underwent evaluation on a benchmark encompassing 10 medical question-answering (qa) tasks. this assessment revealed superior performance compared to the original model and existing open-source medical counterparts. mcinnes et al. (2022) presents the development of a synthetic biology knowledge system, wherein a text processing pipeline utilizes nlp techniques to extract and correlate information from the literature aimed at synthetic biology researchers. the pipeline integrates named entity recognition, relation extraction, concept grounding, and topic modeling methodologies to extract pertinent information from published literature. subsequently, this extracted information is utilized to establish connections between articles and elements within the knowledge system. the findings demonstrate the effectiveness of each component when applied to synthetic biology literature and propose avenues for further enhancing the pipeline's capabilities. gong et al. (2023) explored the potential of various llms such as gpt-4, gpt-3.5, palm2, claude2, and sensenova in addressing conceptual biology questions, including those related to synthetic biology, such as principles of genetic circuit design and crispr-based genome editing techniques. while the findings revealed the adeptness of llms in logical reasoning and their potential to support biology research by facilitating tasks such as data analysis, hypothesis formulation, and knowledge synthesis, the authors underscored the necessity for further refinement and validation before fully harnessing the potential of llms to expedite biological discovery. in this study, we assessed the effectiveness of llms in recognizing and extracting pertinent information regarding host organisms and their associated genetic engineering tools. of particular emphasis was the evaluation of open-source llms, chosen for their heightened adaptability and transparency in contrast to proprietary counterparts, thereby enabling users to exercise greater customization and oversight over model operations. the main objective was to gauge the efficacy of these models in discerning insights from a collec-tion of biological literature and resources, thereby augmenting our comprehension of llms' applicability in biological inquiry and ability to inform prospective applications within this domain. ## host organisms and genetic toolbox curation to the best of our knowledge, there are no publicly available datasets specifically tailored for the recognition task involving bacterial host organisms and their associated genetic toolbox by machine learning models. in order to enhance the proficiency of machine learning models in identifying such entities through training on labeled datasets, we undertook an annotation endeavor aimed at labeling both organism hosts and genetic tool types as depicted in biological literature. to facilitate this annotation task, a comprehensive list of terms describing the bacterial genetic toolbox was curated, which is provided in appendix a. when conducting a search using the genetic toolbox related terms, numerous papers unrelated to our specific focus emerge. these papers span topics ranging from human genetics to plant research. while we may consider including them at a later stage, our current emphasis lies on bacteria. therefore, we incorporate the compound word "bacteria" along with relevant keywords to refine our search results. for the annotation process, a corpus comprising 434 pdf papers was assembled, meticulously selected by two domain experts. additionally, 376 xml articles containing any of the terms from the curated list described in appendix a were obtained by querying the biorxiv database within subject areas encompassing biochemistry, bioengineering, bioinformatics, microbiology, molecular biology, synthetic biology, and systems biology. from this corpus, a total of 795 text snippets were extracted from abstracts and main body texts, each comprising a target sentence containing one of the bacterial genetic toolbox terms, accompanied by two preceding and two succeeding sentences, all of which are part of the same paragraph. to ensure nonredundant annotation, we eliminated duplicate snippets that contain multiple genetic toolbox terms. to facilitate the annotation process, a doccano web server @xcite was employed, thereby streamlining the task of annotating textual data. an annotation sample is depicted in figure 2. in order to label entities within the text, a framework comprising 14 distinct entity labels was defined. subsequently, a total of 1,777 annotated entities were obtained across the entire corpus following the completion of the annotation process. ## evaluation of llms for recognizing our study aimed to assess the potential of llms for the task of entity type recognition, utilizing annotated datasets. to this end, we employed a selection of llms, namely falcon @xcite , mpt (mosaicml-nlp-team, 2023), llama 2 @xcite , solar @xcite , mistral (jiang et al., 2023), mixtral (jiang et al., 2024), and llama 3 (meta-ai, 2024). model evaluation was conducted utilizing a question answering formatted prompt paired with text snippets acquired from our annotation task. an illustrative sample of such a prompt for the entity type recognition task is provided below. question: given the options: "plasmid", "organism host", "promoter", "genome engineering", "cloning method", "reporter", "regulator", "antibiotic marker", "genetic screen", "rbs", "counter selection", "terminator", "dna transfer", "operator", which one is the entity type of j23108 in this text? text: plasmids were cloned using gibson assembly or inverse pcr, propagated in e. coli tg1 competent cells in lb media, and isolated through miniprep (qiagen.) reporter plasmids had a p15a origin of replication, chloramphenicol resistance, and the terminator trrnb downstream of the sfgfp coding sequence. plasmids for overexpressing ribosomal proteins in vivo had a cole1 origin of replication, ampicillin resistance, the synthetic constitutive e. coli promoter j23108 from the registry of standard biological parts, and the terminator trnnb after the protein expression gene. ## preliminary evaluation of a llm for extracting host organisms and genetic tools from raw articles our primary objective is to develop an automated tool capable of extracting host organisms and genetic tools for engineering purposes from unprocessed textual data. such a tool would offer practitioners and domain experts a more streamlined and convenient means of accessing this information. as a preliminary phase, we initially evaluated the performance of a plain llm in identifying bio-logical entities and genetic tools within scientific literature. employing a question-answering framework, we devised a structured prompt using the full text of each article. the following exemplifies a sample prompt for this specific task, incorporating the entire text of an article. question: which biological entities and genetic tools are mentioned in this paper, and what are their types? choose from the following options: "plasmid", "organism host", "promoter", "genome engineering", "cloning method", "reporter", "regulator", "antibiotic marker", "genetic screen", "rbs", "counter selection", "terminator", "dna transfer", "operator" paper: title: a biobricks ® toolbox for multiplexed metabolic engineering of central carbon metabolism in the tetracenomycin pathway. full-text: the tetracenomycins are aromatic anticancer polyketides that inhibit peptide translation via binding to the large ribosomal subunit. here, we expressed the elloramycin biosynthetic gene cluster ... answer: the biological entities and genetic tools mentioned in this paper are: 1. organism host: "streptomyces coelicolor m1146, streptomyces glaucescens gla.0, streptomyces olivaceus ttü2353, amycolatopsis sp. a23, e. coli jm109, e. coli et12567/puz8002 2. promoter: erme*p promoter, sf14p promoter, laci promoter 3. genome engineering: cos16f4ie cluster, actinophage integrases (e.g. fc31, fbt1, sv1, tg1, sam2, vwb) 4. cloning method: restriction digestion, ligation, 3a cloning, isocaudomer cloning 5. reporter: red fluorescent protein (rfp) coding device 6. regulator: laci repressor 7. antibiotic marker: ampicillin, kanamycin, apramycin, viomycin, hygromycin, nalidixic acid 8. genetic screen: blue-white colony screening 9. rbs: b0034 strong ribosome binding site 10. counter selection: nalidixic acid 11. terminator: b0015 transcriptional terminator .... as a preliminary evaluation, we employed 233 articles used in our annotation task. due to the length of a full text, only the mixtral 8x7b model was adopted for this evaluation, which can have up to 32k input length. the post-processing of the model output generated a json formatted data. the following illustrates an example of postprocessed model prediction. 3 . the result shows 0.8816 (micro f1) and 0.8734 (macro f1) for 6,962 entities. the model displays inherent uncertainty when encountering ambiguous terminology. for example, the term "genetic screen" has been utilized across diverse contexts, resulting in confusion within the model. this assertion is supported by the individual accuracy measurements presented in ## chatbot for genetic tool engineering complementing the development of a llm to assist research in synthetic biology and biomanufacturing, kbase @xcite provides a web application for users to interact with this model through a chatbot interface. starting from a simple question, users can ask for information about bacteria and genetic tools of interest and receive responses from the trained model. conversations are logged, allowing users to provide feedback on the efficacy of the chatbot's responses, which serves as valuable input for refining and enhancing the system in future iterations (see figure 3 ). using the "outlines" python package (willard and louf, 2023) for structured output generation, we identify any biological entities and their associated tools in the model's response. the genus of each entity is collected, and a pruned ncbi taxonomy tree @xcite is rendered that highlights these organisms in the context of their genus-level neighbors. as a result of the continuous evolution of taxonomic nomenclature, the information output by the model may not reflect the current information in ncbi databases. therefore, this tool performs additional checks against previous names and synonyms for organisms identified in the model output. the web application downloads the most current version of the ncbi taxonomy database at container startup. users may select any organisms on the species tree, and the selection will be summarized on the right side of the visualizer. the top right provides summary information from the bacdive database @xcite to rapidly identify culture conditions amenable to isolation and growth. the bottom right section summarizes genetic tool database information from the phage-host daily @xcite , virus-host @xcite , and plasmid @xcite databases. additional information identified by the model is also described here. this combination of prepackaged curated databases in conjunction with information extracted from the model's large text corpora provides a comprehensive summary of the available strains, tools, and publications describing the organism in question (see figure 4 ). to facilitate the identification of organisms that do not have isolation or genetic tool information, users may elect to visualize other relative organisms without entries in the accompanying databases. similarly, while the tree visualization is species-focused, users may also select to view all strains for a given species in order to highlight strain-level differences in isolation and genetic tool usage. the incorporation of this tool into the kbase infrastructure serves as an additional avenue for researchers to access pertinent information, establishing connections not only with biologically relevant organisms for laboratory investigations but also with the broader ecosystem of kbase, facilitating subsequent analyses and dissemination of findings. integration with the kbase platform is undertaken in adherence to established standards pertaining to containerization, user-oriented tool development, and deployment protocols. the tool is being developed with the intent of serving as a reusable proof-of-concept that caters to a diverse audience. ## conclusion a significant bottleneck within the domains of synthetic biology and biomanufacturing pertains to the identification of suitable microbial targets tailored to specific applications and environmental conditions, alongside the selection of genetic tools conducive to the cultivation and engineering of non-model microorganisms. this bottleneck poses a direct hindrance to the progress of investigations in synthetic biology and obstructs redesign efforts. while peer-reviewed publications serve as the primary repository for biological experimental data, manual curation proves insufficient for managing the extensive volume of available literature. consequently, there arises a need for the automated extraction of data related to environmental conditions and genetic tools from literature sources. recent advancements in nlp and llms have presented promising avenues for addressing this challenge. this study aims to evaluate the potential applicability of llms in alleviating this issue by demonstrating the assessment of various llms in recognizing bacterial host organisms and genetic toolbox, and evaluating the efficacy of annotated datasets for these entities derived from scientific literature in enhancing the models' predictive capabilities. additionally, we introduce a web-based interface through which users can interact with the llm and access answers augmented with external biological resources. we anticipate that users will utilize these tools to extract pertinent information from literature sources concerning biological entities and genetic tools and components, encompassing organism names and various tools such as promoters, plasmids, and phages, which are essential for the engineering of microorganisms. a a list of genetic toolbox keywords transformation method, bacteria transposons, bacterial conjugative, bacterial genetic parts, bacterial genetic tools, bacterial genome editing, bacterial inducer, bacterial integrative vector, bacterial molecular toolbox, bacterial plasmid collection, bacterial promoter library, bacterial replicative vector, bacterial reporter, bacterial transcription terminator, bacterial vectors, bglbrick, biobrick, counter-selection marker, cre-lox, genetic toolkit, lambda red system, standard biological parts, synbio reporter system, synbio toolkit