SCITUNE : Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions (2024)

Abstract: Instruction finetuning is a popular paradigm to align large language models (LLM) with human intent. Despite its popularity, this idea is less explored in improving LLMs to align existing foundation models with scientific disciplines, concepts and goals. In this work, we present SciTune as a tuning framework to improve the ability of LLMs to follow multimodal instructions generated from scientific publications. To test our methodology, we train a large multimodal model LLaMA-SciTune that connects a vision encoder and LLM for science-focused visual and language understanding. LLaMA-SciTune significantly outperforms the state-of-the-art models in the generated figure types and captions in SciCap and VisText benchmarks. In comparison to the models that are finetuned with synthetic data only, LLaMA-SciTune surpasses human performance on average and in many sub-categories on the ScienceQA benchmark. Our results demonstrate that human-generated scientific multimodal instructions remain highly valuable in tuning LLMs to perform well on science tasks, despite their lower volume and relative scarcity compared to synthetic data.

## introduction instruction finetuning has gained significant traction in the nlp community as a means of enhancing the capabilities of large language models (llms), allowing them to accurately balance desired outcomes, context, and human preferences, leading to more relevant and coherent responses. more recently, ai assistants have been trained to comprehend and execute multimodal visionand-language instructions, aligned with human intent, to accomplish diverse real-world tasks in complex multimodal environments. in one of the latest developments, @xcite , llava @xcite and llama-adapter @xcite have focused on expanding language-only instruction models to incorporate multimodal capabilities, thereby granting llms the ability to perform visual grounded reasoning tasks. recent research suggests that high-quality multimodal pretraining data and instructions, such as high-resolution images and diverse multimodal data are important for effective model performance @xcite . however, one of the primary bottlenecks is the scarcity of highquality data for multimodal pretraining and instruction tuning. to mitigate the challenges associated with data scarcity, many recent multimodal models rely on synthetically generated instructions (training data distilled from responses from other models) for fine-tuning instead of human annotations @xcite @xcite . however, using synthetic data to align ai models can create confusion and uncertainty, since synthetic data, being artificially generated, often fails to capture the complexities of human values @xcite . this can cause ai models to learn from biased @xcite , ungrounded @xcite or inaccurate data @xcite . additionally, training models on recursively generated data can lead to a loss of true data distribution, resulting in less varied and misaligned outputs @xcite . furthermore, models tuned with synthetic data often fail to meet the standards required by certain scientific subdomains, such as medicine @xcite . for example, several open-source medical vision-language models such as llava-med @xcite , med-flamingo @xcite , medvint @xcite , and radfm @xcite failed to satisfy the trustfulness, fairness, safety, privacy, and robustness conditions in the recently introduced cares @xcite benchmark. as a result, ai systems relying on synthetic data might behave unpredictably and could potentially cause unintended or harmful outcomes @xcite . we believe this is mainly due to the absence of alignment methods designed to synchronize existing foundation models with scientific disciplines, concepts, and goals and to ensure that the generated content meets the standards and expectations of the scientific community. our hypothesis is that scientifically aligned multimodal models can learn from unique patterns and structures present in scientific language generated by human scientists, thus would be able to follow precise instructions about complex procedures, protocols, and guidelines in the scientific environments. this leads to the question: to what extent, can llms align solely with human-curated scientific multimodal instructions? to this end, we build on top of the llava @xcite model architecture to perform scientific multimodal instruction tuning (scitune) on top of a decoder-based pretrained llm and vision encoder. the scitune training method includes two stages for scientific multimodal instruction tuning, i) scientific concept alignment to learn across various scientific visual signals (e.g., plots, charts, equation, diagram, etc.), and textual signals (e.g., captions, optical character recognition (ocr) and paragraph mentions), ii) scientific instruction tuning to fine-tune on a multimodal scientific reasoning task. to validate our approach, we train our models on top of llama @xcite and the clip @xcite vision encoder model. we show that our model surpasses human performance on the scienceqa multimodal reasoning benchmark and performs significantly better than state-of-the-art vision-language models in a variety of scientific image understanding tasks. our results demonstrate that human-curated scientific multimodal instructions remain highly valuable in tuning llms to perform well on science tasks, despite their lower volume and relative scarcity compared to synthetic data. ## methodology in this section, we describe the scitune framework in two stages of scientific multimodal concept alignment and multimodal task-specific instruction tuning and the design choices and multimodal architecture used for the experiments. ## experiments in this section, we report the performance of llama-scitune models across a variety of sciencefocused downstream tasks. our goal is to assess the performance of the models in visual grounded language understanding and multimodal reasoning tasks. for example, we want to show how much difference the training stages contribute to the model performance, or whether adding various scientific modalities in the instruction template improves the overall performance. note that our objective is not to introduce a model that tops the leaderboard across various downstream tasks. instead, we aim to demonstrate the extent to how human-curated scientific multimodal instructions can be useful in aligning vision-language models. to this end, we trained three llama-scitune-scicap models and finetuned them with scientific multimodal reasoning dataset (e.g., scienceqa) for the corresponding llama-scitune-scienceqa models. three llama-scitune-scicap models differ on the text input types (e.g., caption, figure first, we report the performance of llama-scitune-scicap in two science-focused visual grounded tasks to assess the scientific concept alignment training stage (section 3.1). finally, we use the scienceqa benchmark to test the multimodal reasoning abilities of llama-scitune-scienceqa across three scientific subject areas (section 3.2). ## conclusion in this work, we present scientific multimodal instruction tuning to align llms with scientific concepts and goals. to this end, we use humangenerated multimodal instructions curated from visual signals (e.g., plots, charts, equations), and textual signals (e.g., captions, optical character recognition (ocr) and paragraph mentions) found within scientific publications. we train several models built on top of llama language decoder model and clip vision encoder model and test the models on science-focused multimodal downstream tasks. in evaluation, we show that the resulting llama-scitune-scicap models can perform better on classifying scientific visuals and generating figure captions compared with sota visionto-language models. furthermore, the llama-scitune-scienceqa model surpasses the human performance in scienceqa, the standard multimodal science-focused reasoning qa benchmark. our results suggest human-curated scientific multimodal data remains highly valuable despite the advancements in synthetic data generation techniques. while it is fast and easy to generate large volume of synthetic training data with closedsource models such as gpt-4, they may contain inaccuracies or biases due to lack of expert review. in contrast, although existing human-generated scientific multimodal datasets are comparatively smaller, they provide reliable ground truth for tuning llms, which leads to better generalization and performance in downstream science applications. xuhui zhou, zhe su, tiwalayo eisape, hyunwoo kim, and maarten sap. 2024. is this the real life? is this just fantasy? the misleading success of simulating social interactions with llms. arxiv preprint arxiv:2403.05020. deyao zhu, jun chen, xiaoqian shen, xiang li, and mohamed elhoseiny. 2023. minigpt-4: enhancing vision-language understanding with advanced large language models. arxiv preprint arxiv:2304.10592. andy zou, zifan wang, j zico kolter, and matt fredrikson. 2023. universal and transferable adversarial attacks on aligned language models. arxiv preprint arxiv:2307.15043.