B rain L oc: Brain Signal-Based Object Detection with Multi-modal Alignment (2025)

Abstract: Object detection is a core challenge in computer vision. Traditional methods primarily rely on intermediate modalities such as text, speech, or visual cues to interpret user intent, leading to inefficient and potentially distorted expressions of intent. Brain signals, particularly fMRI signals, emerge as a novel modality that can directly reflect user intent, eliminating ambiguities introduced during modality conversion. However, brain signal-based object detection still faces challenges in accuracy and robustness. To address these challenges, we present BrainLoc, a lightweight object detection model guided by fMRI signals. First, we employ a multi-modal alignment strategy that enhances fMRI signal feature extraction by incorporating various modalities including images and text. Second, we propose a cross-domain fusion module that promotes interaction between fMRI features and category features, improving the representation of category information in fMRI signals. Extensive experiments demonstrate that BrainLoc achieves state-of-the-art performance in brain signal-based object detection tasks, showing significant advantages in both accuracy and convenience.

## introduction current intelligent systems typically rely on intermediate modalities such as speech @xcite @xcite ), text @xcite , and images @xcite to understand human intent, but these modalities are merely indirect channels and abstract expressions of consciousness. this indirectness is particularly evident in object detection tasks within computer vision. while traditional text-based detection systems (such as grounding dino @xcite ) have achieved high accuracy, they face significant limitations in practical applications: users need to describe texture and spatial information through text, a process that requires careful thinking and manual input, resulting in high cognitive costs. moreover, the modality conversion process may lead to distortion and ambiguity of user intent. to address these challenges, functional magnetic resonance imaging (fmri) @xcite signals demonstrate unique advantages. fmri is a non-invasive brain imaging technique that records neural activity patterns under visual stimulation by measuring blood oxygen level-dependent signal changes. this type of signal can directly reflect user intent, offering a promising pathway toward more natural and intuitive human-computer interaction. compared to electroencephalogram (eeg) @xcite , fmri possesses a higher information entropy, enabling it to capture richer semantic and visual details. this advantage makes fmri particularly well-suited for supporting tasks such as object detection. imagine a future where people wear smart glasses that can capture current scenes in real-time and, based on the user's brain responses, quickly and accurately locate target objects, highlighting them on the display. this technology has significant applications not only in daily life scenarios but also in military target detection, industrial automation, and medical assistance. to this end, we combine the advantages of traditional object detection systems and brain signals to propose brainloc. as shown in fig. 1 , even in complex scenes containing multiple similar targets (such as multiple apples), our system can accurately identify and locate the specific target (such as a red apple) that the user has in mind. the key idea is to build a lightweight feature extractor that enhances fmri signal comprehension through multi-modal alignment strategies, enabling it to capture finegrained visual information from user intent. additionally, we design a cross-domain fusion module that organically combines visual information (such as color) from fmri features with semantic information through attention mechanisms, thereby improving localization accuracy. experimental results demonstrate that brainloc not only achieves state-of-the-art performance in localization accuracy but also shows unique advantages in handling complex scenes and fine-grained recognition tasks. our main contributions are as follows: ## method as shown in fig. 2 , we introduce the brainloc, which can detect objects in people's minds with only images and fmri signals as input. first, we employ a multi-modal alignment strategy to train a lightweight feature extractor that converts fmri signals into features (see sec. 3.1). second, we design a cross-domain fusion module that deeply integrates fmri features with category features and predicts the category of the detected object (see sec. ## conclusion in this paper, we presented brainloc, a novel lightweight brain-based object detection model that leverages fmri signals to identify and locate target objects in complex scenes. experimental results demonstrate that brainloc achieves sota performance in brain-based localization tasks, combining the precision of traditional systems with the convenience of brain-based approaches. this significant advancement highlights the potential of brain-computer interface (bci) technologies @xcite in various applications, including assistive technologies, military target detection, and industrial automation. given that fmri data typically contain less signal noise compared to eeg, our research primarily focused on fmri. in future work, we plan to shift our efforts toward eeg to enhance real-time applications. overall, brainloc represents a significant step forward in brain-based object detection, offering high accuracy and portability, and is expected to inspire further research and development in the field of bci.