FLIQA - AD : a Fusion Model with Large Language Model for Better Diagnose and MMSE Prediction of A lzheimer’s Disease (2025)

Abstract: Tracking a patient’s cognitive status early in the onset of the disease provides an opportunity to diagnose and intervene in Alzheimer’s disease (AD). However, relying solely on magnetic resonance imaging (MRI) images with traditional classification and regression models may not fully extract finer-grained information. This study proposes a multi-task Fusion Language Image Question Answering model (FLIQA-AD) to perform AD identification and Mini Mental State Examination (MMSE) prediction. Specifically, a 3D Adapter is introduced in Vision Transformer (ViT) model for image feature extraction. The patient electronic health records (EHR) information and questions related to the disease work as text prompts to be encoded. Then, an ADFormer model, which combines self-attention and cross-attention mechanisms, is used to capture the correlation between EHR information and structure features. After that, the extracted brain structural information and textual content are combined as input sequences for the large language model (LLM) to identify AD and predict the corresponding MMSE score. Experimental results demonstrate the strong discrimination and MMSE prediction performance of the model, as well as question-answer capabilities.

## introduction alzheimer's disease (ad) is one of the most common forms of dementia. it takes several years from the onset of normal cognition (nc) to ad, so it provides an opportunity for early diagnosis and intervention. the mini-mental state examination (mmse) is a widely used cognitive assessment tool for evaluating the progression of cognitive and behavioral states. alternatively, magnetic resonance images (mri) can obtain more detailed structural 1 the code is following: https://github.com/ junhao667/fliqa-ad.git changes, such as the presence of senile plaques (sp) and atrophy of the cerebral cortex @xcite . ad identification and mmse score are interrelated, which underscores the necessity of combining mri and other non-imaging data for dementia analysis @xcite . therefore, some researchers have introduced multi-task learning to predict mmse and detect ad jointly. for instance, in @xcite an interaction module is designed to connect the shared features to the tasks. to include the demographic text information, a deep multi-task multi-channel learning (dm 2 l) framework is proposed for classification and regression @xcite . to solve the task relevance issue, feature relevance is exploited by adding three multi-task interaction layers between two task backbones @xcite . however, such work tends to perform better on ad identification or mmse score prediction tasks exclusively, and a decline in performance is observed on multi-target tasks. using an additional interaction module for interacting still requires extracting features for different tasks. simply designing multiple interaction layers without incorporating any electronic health records (ehr) prompts information will not assess early-stage ad effectively due to ignoring demographic characteristics. in recent years, the vision language pre-trained (vlp) model has provided a better reference for solving the above challenges. for example, clip @xcite learns representations from natural language supervision and performs well for zero-shot transfer to various downstream tasks. blip-2 @xcite uses an efficient pretraining strategy that freezes the visual encoder and large language model. the modal gap is bridged by training the q-former. llava @xcite trains a projection layer to connect the frozen visual encoder and large language model (llm), with better zero-shot capabilities. inspired by these works, in this study, we propose an ad mri diagnostor, fliqa-ad, for better diagnosis and prediction of mmse. specifically, the diagnostor is constructed by the vision encoder module, adformer fusion module, and llm module as shown in fig. 1 . in the vision encoder module, a 3d adapter is used to convert 3d images into processable tokens, preserving the spatial structure information of the images. then, we utilize the bio-clinicalbert model, which has been pre-trained on specialized diagnostic question-answering texts @xcite , as the text encoder. the patient's ehr information and questions related to the disease will be used as text prompts. to extract the most diagnostically beneficial visual features from different types of patients, adformer is proposed to fuse the ehr information and vision features through a cross-attention manner. finally, llm is used as a decoder that outputs ad detection and mmse scores from the text and visual features input. ## limitation this paper proposes a fliqa-ad model based on ehr information and mri images to diagnose ad. however, in the medical domain, especially alzheimer's disease, text and image information is extremely scarce due to privacy protection and other issues, and the amount compiled in this paper is limited, which greatly limits the open questionanswering ability of this model. secondly, the model is pre-trained on the adni dataset. when it is transferred to the oasis dataset, although we have performed a series of preprocessing to keep the basic features of the image consistent, the performance on oasis has declined due to differences in information such as image resolution. in the experiments in this paper, we found that increasing the trainable dataset can improve the model's ability on image datasets that are significantly different from the training set. that may be the thing worth trying in the future.