MIMIR : A Customizable Agent Tuning Platform for Enhanced Scientific Applications (2024)

Abstract: Recently, large language models (LLMs) have evolved into interactive agents, proficient in planning, tool use, and task execution across various tasks. However, without agent-tuning, open-source models like LLaMA2 currently struggle to match the efficiency of larger models such as GPT-4 in scientific applications due to a lack of agent tuning datasets. In response, we introduce MIMIR, a streamlined platform that leverages large LLMs to generate agent-tuning data for fine-tuning smaller, specialized models. By employing a role-playing methodology, MIMIR enables larger models to simulate various roles and create interaction data, which can then be used to fine-tune open-source models like LLaMA2. This approach ensures that even smaller models can effectively serve as agents in scientific tasks. Integrating these features into an end-to-end platform, MIMIR facilitates everything from the uploading of scientific data to one-click agent fine-tuning. MIMIR is publicly released and actively maintained at https://github. com/gersteinlab/MIMIR, along with a demo video for quick-start, calling for broader development.

## introduction recently, large language models (llms) have undergone a significant evolution, transitioning into interactive agents that have demonstrated considerable progress in many scientific scenarios @xcite @xcite . the commendable performance of these models across various downstream tasks has incited researchers to propose methods for utilizing llms to generate instruction datasets @xcite @xcite . the quality and diversity of such data are instrumental in aligning, 1 https://www.youtube.com/watch?@xmath0. * means contributed equally. while there is an abundance of datasets available for instruction tuning @xcite @xcite @xcite , datasets specifically focused on agent tuning @xcite are in short supply. this imbalance has inculcated reliance on proprietary llms such as chatgpt or gpt-4 as mainstay tools for reasoning and planning in scientific applications. this dependency raises significant privacy concerns, especially when integrating sensitive domain knowledge, such as ehr data, into model training, as highlighted by @xcite and @xcite . furthermore, concentrating solely on finetuning llms with tool-learning datasets might inadvertently compromise their ability to master rare domain-specific knowledge and perform complex reasoning, a concern raised by @xcite . for example, an llm trained on a dataset focused on diagnosing cardiovascular diseases might overlook a critical anomaly that falls outside its training scope, such as an unusual symptom of a rare cancer. additionally, if we finetune gpt-3.5, incorporating confidential patient data into a public model can heighten the risk of privacy breaches. in this paper, we introduce mimir foot_0 , a novel streamlined platform, as illustrated in figure [doctor] these symptoms could be indicative of a variety of conditions, including a viral or bacterial infection, allergies, or even a more serious illness. ## human evaluation of the generated data experiments in the biomedical domain in our study, we selected a diverse set of data samples to form our investigation set for source input in mimir. specifically, we chose 25 random samples from each of the following biomedical domain datasets we described before: medqa, medmcqa, and mmlu medical topics. although these datasets traditionally consist of medical multiple-choice questions, for our evaluation, we removed the multiple-choice options and tasked the model with generating long-form answers. these answers required a detailed reasoning path to address the complex medical questions. this approach ensures a comprehensive dataset, facilitating an in-depth biomedical data analysis. in addition, to demonstrate the effectiveness of our agent tuning, we also conducted standard benchmark tests (multiple-choice) on these datasets. ## conclusion and future work mimir is a streamlined platform for agent tuning, focusing on scientific expertise and advanced applications. it integrates domainspecific datasets and user-uploaded topics, utilizing various contemporary reasoning frameworks and tools. our platform is particularly useful in accelerating scientific discovery in biology and medicine by incorporating diverse tools and automating tool selection.