Decoding Semantic Representations in the Brain Under Language Stimuli with Large Language Models (2025)

Abstract: Brain decoding technology is paving the way for breakthroughs in the interpretation of neural activity to recreate thoughts, emotions, and movements. Tang et al. (2023) introduced a novel approach that uses language models as generative models for brain decoding based on functional magnetic resonance imaging (fMRI) data. Building on their work, this study explored the use of three additional language models along with the GPT model used in previous research to improve decoding accuracy. Furthermore, we added an evaluation metric using an embedding model, providing higher-level semantic similarity than the BERTScore. By comparing the decoding performance and identifying the factors contributing to good performance, we found that high decoding accuracy does not solely depend on the ability to accurately predict brain activity. Instead, the type of text (e.g., web text, blogs, news articles, and books) that the model tends to generate plays a more significant role in achieving more precise sentence reconstruction.

## introduction brain decoding technology has recently gained considerable attention for its potential. this technology, which analyzes brain activity in real time to decode thoughts, emotions, and movements, is expected to bring major breakthroughs in areas such as medicine, rehabilitation, communication support, scientific research, and beyond. many brain-machine interfaces (bmis) designed for practical use rely on invasive methods like electrocorticography (ecog), which require brain surgery @xcite . although these methods provide clearer data, allowing for accurate analysis of brain activity even in complex tasks, they come with surgical risks and practical limitations, making them unsuitable for large-scale deployment. in contrast, non-invasive bmis using functional magnetic resonance imaging (fmri) or electroencephalography (eeg) are safer and more costeffective alternatives. however, these methods face challenges, including noisy data and lower temporal or spatial resolution, which restrict their applications to simpler tasks such as recognizing a limited set of words or basic motion commands @xcite . non-invasive bmi technologies remain far from being practically deployed, with several challenges yet to be addressed. @xcite took a novel approach by not directly decoding stimuli from non-invasive data, but instead utilizing neural data to support the reconstruction process. their method involved using a language model to generate several possible next words, then selecting the one that most closely aligns with the brain's current state. although this method is based on off-line brain decoding using data acquired through fmri, its innovative approach has sparked widespread interest from researchers. in this study, we extend the work of @xcite by using three additional language models, along with the fine-tuned gpt model @xcite they employed for language generation, in order to reconstruct sentences with higher similarity scores to the actual stimulus sentences, and compare the accuracy of the decoders. we investigate whether higher accuracy of the encoding model that predicts brain state leads to more precise decoding, as well as the factors that contribute to decoding accuracy. encoding models generally estimate brain states from vectors that represent stimuli, typically extracted from deep learning models. since the introduction of word2vec @xcite , which represents the meaning of words in natural language as vectors, it has become possible to extract features from language stimuli presented to the human brain. more recently, intermediate representations from language models such as bert @xcite , gpt-2 @xcite , and llama @xcite have been increasingly used as vectors that capture sentence features for brain state estimation @xcite @xcite . the performance of encoding models depends on the language model used. @xcite reported that there is a scaling law between the number of parameters in the language model used for feature extraction and the accuracy of the resulting encoding model. as the number of parameters in the model increases, the accuracy of the encoding model improves in a logarithmic-linear fashion. in this study, we introduce the pre-trained gpt, the original model before fine-tuning in the research of @xcite , and investigate how fine-tuning affects decoder accuracy. additionally, while @xcite and other studies using encoding models have commonly employed gpt or gpt-2 for feature extraction, we use the powerful language models llama3 and opt to build a more accurate encoding model. furthermore, we introduce a new evaluation approach that provides further insights into their performance to evaluate the effectiveness of tang et al.'s decoding methods. ## conclusion in this study, we examined and expanded upon tang et al.'s research, which proposed the use of language models for brain decoding. specifically, in addition to the fine-tuned gpt model used in previous study, we constructed decoders using three additional language models, clarified the accuracy of the encoding models and the token rate models used in the decoders, and compared their decoding performance. regardless of the language model used, we confirmed that the decoders could significantly reconstruct sentences similar to the actual stimuli presented to participants. although larger models like llama3-8b and opt-6.7b demonstrated superior performance in predicting brain activity, we found that the gpt (120m) models achieved higher decoding scores. we hypothesize that this result is attributable, at least, to the training dataset of the gpt models being more similar to the actual stimulus sentences. moreover, this study added a similarity evaluation metric using an embedding model by computing higher-level semantic similarities between sentences, demonstrating that all language models successfully reconstructed sentences with significantly high scores at most time points. while this study focused solely on evaluating the similarity between the actual stimulus sentences and the decoded sentences, such similarity does not necessarily guarantee an accurate reflection of brain status. unlike this study, when the stimulus dataset in the fmri data is not explicitly known, using language models trained on more diverse datasets could potentially result in a better reconstruction of brain states.