Infrared- LL a VA : Enhancing Understanding of Infrared Images in Multi-Modal Large Language Models (2024)

Abstract: Expanding the understanding capabilities of multi-modal large language models (MLLMs) for infrared modality is a challenge due to the single-modality nature and limited amount of training data. Existing methods typically construct a uniform embedding space for cross-modal alignment and leverage abundant visual image data to indirectly understand infrared images. However, they ignore the supervisory signals of infrared-modality-specific attributes, which may lead to biased understanding of infrared images. To address this issue, we propose a debating multi-agent generation system which transfers knowledge from visible images to generate infrared image-text pairs and infrared instruction data. Moreover, we construct an infrared question-answering benchmark based on common infrared tasks. Experimental results from incremental fine-tuning on existing models and our Infrared-LLaVA-7B trained from scratch on infrared data demonstrate the effectiveness of the generated data and the feasibility of the generation approach.

## introduction with the rapid development of large language models (llms) @xcite @xcite , existing multi-modal large language models (mllms) have shown impressive capabilities on generic visual modality tasks such as image-text retrieval and phrase grounding @xcite @xcite @xcite . but less attent ion has been paid to infrared images one of the scarce modalities @xcite , due to single-modality nature and low resourc of training data. the infrared images have more stable information compared to visual images which have large information loss in low light or other extremely * corresponding author harsh environments @xcite . thus it is worth exploring and researching how to utilize the limited data sources of the infrared images to align the infrared images' feature space with the llm's text embedding space, and to further perform infrared instruction supervised fine-tuning. this would enable a mllm to correctly accept infrared image inputs and perform infrared image question-answering tasks. previous work @xcite ) has achieved some infrared image comprehension using only visible image-text pairs and visible image instructions for alignment and instruction finetuning, based on the unified embedding space implemented by imagebind @xcite . however, the method does not involve the infrared image encoder during alignment and instruction fine-tuning. the lack of supervisory signals for infrared images may lead to biased understanding of infrared images. hence it is necessary to construct infrared image-text pairs and infrared instructions to enhance the model's understanding ability for infrared images. meanwhile, given to the lack of benchmarks for evaluating infrared-related capabilities of mllms, it is necessary to establish an infrared multi-modal question-answering benchmark to assess mllm's understanding ability for infrared images. to address these issues, we propose a debating multi-agent system. the system generates infrared image-text pairs and infrared instructions by transferring knowledge from visual image-text pairs and corresponding bounding box annotations, as illustrated in figure 1 . the system utilizes multiple independent gpt-3.5 models to construct multiple agents. each agent respectively performs infrared image-text pairs generation, instruction generation, and instruction evaluation. furthermore, the system ensures the quality of generated data by self-updating through the debates between different agents. most importantly, the entire process is auto-mated and does not require training, saving a significant amount of time and economic costs compared to annotating the data from scratch. additionally, we build an infrared template question-answering benchmark based on annotated real infrared images to evaluate the mllm's understanding ability for infrared images, as shown in figure 2 . this benchmark is constructed based on common infrared task datasets from multiple infrared scenarios, with high consistency and accuracy. we use this multi-agent to generate 118k synthetic infrared images, 500k synthetic infrared image-text pairs, and 12k synthetic infrared instruction data based on coco @xcite , called infrared instruction dataset. we demonstrated the effectiveness of the synthetic instruction data through incremental fine-tuning on other models, as show in figure 5 . specifically, with the aid of this dataset, we developed an infrared multi-modal large language model called infrared-llava-7b, as show in figure 3 . infrared-llava-7b outperformed pandagpt-7b, pandagpt-13b and imagebind-llm trained in the infrared template benchmark, showing its competitive performance. our contributions are as follows: ## infrared instruction dataset in this section, we present our debating multi-agent system to build the infrared instruction dataset. subsection 3.1, subsection 3.2, and subsection 3.3 explain the composition and workflow of each agent. subsection 3.4 describes the process of interaction in the multi-agent system. as a result, we generate 118k synthetic infrared images based on coco @xcite , 500k paired captions, and 12k infrared image instructions. ## infrared template benchmark existing research lacks a benchmark for questionanswering tasks specifically for infrared images. to further evaluate the mllm's ability to understand and respond to infrared images and commands, we build an infrared benchmark based on existing infrared dataset, as shown in figure 2 . ## infrared-llava given the lack of a mllm trained on infrared data, we construct an infrared-specific mllm, infrared-llava-7b, that utilizes infrared data for a twostage training process involving alignment and supervised fine-tuning. ## conclusion in this paper, we propose an debating multi-agent data generation system, which includes a caption agent that generates infrared images and captions, a task agent that generates instructions, and a debate agent that evaluates the instructions. using this system, we transfer the visible image dataset coco to get 118k infrared images and 500k infrared image-text pairs, as well as 12k infrared instruction. we validate the effectiveness of this dataset through incremental fine-tuning on other mllms. we further construct the infrared-llava-7b, which is fully aligned and fine-tuned based on infrared data, demonstrating the importance of infrared instruction data. we explore the impact of alignment methods, sft structures, and alignment layer types on the model's infrared questionanswering performance using infrared-llava-7b.