Importance of investing time and money in integrating large language model-based agents into outbreak analytics pipelines
Authors: Albert Jan van Hoek, Sebastian Funk, Stefan Flasche, Billy J Quilty, Esther van Kleef, Anton Camacho, Adam J Kucharski
Journal: The Lancet Microbe (2024)

Outbreaks such as mpox and COVID-19 underscore the need for swift, evidence-based responses to limit their spread and save lives. Such evidence should be generated in real time. Generating these insights requires synthesis of many different sources of information and a combination of multiple distinct analytical techniques, from basic descriptive analysis to complex dynamic models.1Polonsky JA Baidjoe A Kamvar ZN et al.Outbreak analytics: a developing data science for informing the response to emerging pathogens.Philos Trans R Soc Lond B Biol Sci. 2019; 37420180276Crossref PubMed Scopus (99) Google Scholar Timely generation of policy-relevant evidence relies on skilled staff and prebuilt outbreak analytics pipelines, alongside rapid literature reviews and ad-hoc data curation processes. Although these workflows are constantly improving, they continue to involve several time-intensive manual tasks such as abstract selection and data extraction, cleaning, and merging. Performing subsequent research and writing up results are also time intensive. At some instances, specific skills such as coding are required. In particular, multiple governments or organisations require regular situational updates against a background of changing data and epidemic dynamics, creating major bottlenecks in the ability to deliver analysis at scale. To improve and enhance the efficiency of real-time outbreak analytics and to reduce the number of potential bottlenecks, considerable opportunity to make use of newly emerging large language models (LLMs) exists. Recently, several advanced LLMs have become available, including GPT-4 from OpenAI and Gemini from Google. These LLMs are equipped with capabilities such as generating and interpreting text, writing and executing code, performing analytical tasks, and processing images. Integration of LLMs within outbreak analytic workflows can be achieved both directly and indirectly. The direct approach would be for analysts to interact with LLMs directly to assist in writing codes for analysis or to provide tailored feedback on written text, similar to the common tools currently available. The indirect approach would involve having the analyst interact with specialised LLM-based agents. LLM-based agents are computer programs that can deploy multiple LLMs with specific roles and specialisations to perform dedicated tasks, similar to how human teams divide projects on the basis of the experience and knowledge of the team members.2Xi Z Chen W Guo X et al.The rise and potential of large language model based agents: a survey.arXiv. 2023; (preprint)https://arxiv.org/abs/2309.07864Google Scholar Such agents can perform tasks to achieve specific goals, either on their own or collaboratively with other agents, with the outputs of one agent's task becoming an input for the next agent or in dialogue with the analyst. We observe several key advantages in utilising specialised LLM-based agents. A primary advantage of using LLM-based agents, compared with using LLMs directly through prompting, lies in their ability to formalise LLM team interactions, enabling optimisation of workflows and quality control to enhance consistency and reproducibility. Furthermore, using multiple agents allows for more review steps before presenting the answer to the analyst, which can yield a higher-quality output than that obtained using direct prompting. Moreover, simultaneous deployment of LLM-based agents allows multiple tasks to be performed simultaneously and at scale. Through collaboration, multiple agents could complete more complex tasks. Finally, LLM agents, equipped with memory capabilities, enhance their performance over time by learning from their previous experience to accomplish tasks more effectively. Given these characteristics and the anticipated increasing capabilities of LLMs, developing several specifically defined LLM-based agents as part of the outbreak analytics pipeline has considerable potential for improving its efficiency. Once developed and tested, these workflows can be scaled and replicated globally, providing round-the-clock operation and support and creating a more equitable distribution of analytical knowledge and expertise. However, as with all new technologies, the integration of these new LLMs should be approached with caution. Pilot studies, real-world testing, and validation against established benchmarks are necessary to understand the limitations of the models; learn about their stability and costs; acquire experience with optimal use; learn all ethical implications; and develop consistency, trust, and acceptance. This learning will also allow for implementation of necessary governance structures and oversight to prevent misuse or unintended consequences, as misinterpretation of data or failure to capture nuances could lead to erroneous conclusions or recommendations, having serious consequences. Furthermore, integrating the use of LLMs will require training of staff and adaptation of hardware and software infrastructures to support these pipelines. Although LLMs in outbreak responses can lead to important efficiency gains in time and quality, this article is not a call for all tasks to be replaced by LLM agents. A sufficiently nuanced interpretation of outbreak data is difficult, with profound consequences, for which human skill, responsibility, and input are key. Therefore, creating an environment in which skilled staff can focus more of their time on these problems is crucial. Cautious use of this approach might also empower less technically trained staff to solve these problems. We declare no competing interests. During the preparation of this manuscript, the author used OpenAI GPT-4 using GPT "Academic Assistant Pro" to generate an initial sketch draft based on an extended outline with relevant topics in bullet points produced by the main author. After using this tool/service, the authors substantially reworked and edited the content and then performed an AI check for grammar and spelling of a near final version. The authors take full responsibility for the content of the publication.