U ni C o RN : Unified Cognitive Signal R econstructio N bridging cognitive signals and human language (2023)

Abstract: Decoding text stimuli from cognitive signals (e.g. fMRI) enhances our understanding of the human language system, paving the way for building versatile Brain-Computer Interface. However, existing studies largely focus on decoding individual word-level fMRI volumes from a restricted vocabulary, which is far too idealized for real-world application. In this paper, we propose fMRI2text, the first open-vocabulary task aiming to bridge fMRI time series and human language. Furthermore, to explore the potential of this new task, we present a baseline solution, UniCoRN: the Unified Cognitive Signal ReconstructioN for Brain Decoding. By reconstructing both individual time points and time series, UniCoRN establishes a robust encoder for cognitive signals (fMRI & EEG). Leveraging a pre-trained language model as decoder, UniCoRN proves its efficacy in decoding coherent text from fMRI series across various split settings. Our model achieves a 34.77% BLEU score on fMRI2text, and a 37.04% BLEU when generalized to EEG-to-text decoding, thereby surpassing the former baseline. Experimental results indicate the feasibility of decoding consecutive fMRI volumes, and the effectiveness of decoding different cognitive signals using a unified structure.

## introduction language serves as a window into the cognitive processes unfolding within our minds, communicating a vast amount of information through its syntax and semantics @xcite . advances in cognitive neuroscience have enabled us to directly observe the cognitive processes that underlie language use through the analysis of non-invasive cognitive signals, such as functional magnetic resonance imaging (fmri) and electroencephalogram (eeg). however, this also poses a challenge in understanding the relationship between these signals and the external stimuli that give rise to them * corresponding author within the mind. deciphering cognitive signals into human language not only enhances our grasp of the linguistic system, but also facilitates the development of practical brain-computer interfaces (bcis) by leveraging our comprehension of decoded signals @xcite . although brain decoding has gained great success from word-level to sentence-level decoding on eeg @xcite , relatively little research has been dedicated to directly generating text, particularly complete sentences, from fmri volumes. this is largely attributed to the challenges posed by the relatively low temporal resolution of fmri, which makes it challenging to acquire word-level fmri frames within a sentence. in this study, we propose fmri2text, the first open-vocabulary task that decodes fmri time series into the corresponding texts under naturalistic settings. despite the early efforts in fmri decoding @xcite @xcite , these methods are limited in the ways that they: (1) primarily rely on predefined regions of interest (rois) for feature extraction, underutilizing the rich spatial data inherent in full fmri volumes. this may oversimplify the complex, distributed nature of cognitive processes @xcite . (2) do not effectively leverage the sequential information embedded in fmri time series, missing valuable insights into the dynamics of cognitive processes @xcite . as a standard encoder-decoder framework, uni-corn leverages the robust decoding abilities of pre-trained language models. crucially, it constructs an effective encoder through both snapshot and series reconstructions, harnessing the power of seq2seq models. this allows unicorn to analyze individual signal "snapshots" (such as a single fmri volume or an eeg time point) and capture the "series" or temporal dependencies among these snapshots, thus maximizing the information extracted from the cognitive signals. in summary, our contributions are as follows: ## related work cognitive signals cognitive signals represent the dynamic neural activity associated with information processing and cognitive functions, and are crucial in building bci systems @xcite . these signals are captured at individual time points or as part of a time series, with each data point providing a snapshot of brain activity at a specific point in time. while ecog is often used in high-performance bci systems @xcite @xcite , its semi-invasive nature limits its potential for widespread application in healthy individuals. in non-invasive bci systems, eeg is most commonly used due to its high temporal resolution and cost-effectiveness, while other techniques such as fmri have also been employed in recent years @xcite @xcite . in spite of its relatively lower temporal resolution, fmri allows for the mapping of brain-wide responses to linguistic stimuli at a highly detailed spatial resolution of millimeters @xcite @xcite . this makes fmri particularly ideal for bci systems that translate brain signals into text, a process that involves the participation of multiple brain regions @xcite . brain decoding recent research has been directed towards resolving the issue of decoding cognitive signals into human language through the introduction of new multi-modal tasks and models. most recently-proposed tasks in this field focus on aligning cognitive signals with a limited vocabulary up to a thousand for word-level decoding @xcite @xcite or incorporating them into sentence embeddings for sentence-level decoding using pairwise classification @xcite . wang and ji introduce a novel brain decoding task called eeg-to-text decoding (eeg2text for short), which achieves sentence-level decoding by converting each word-level eeg signal into corresponding text stimuli using pre-trained language models, thereby extending the problem from a closed vocabulary to an open vocabulary. ## task definition as shown in figure 1 , the subject is instructed to read or listen to the text stimuli, while an fmri volume is acquired every fixed repetition time (tr). given an fmri time series of length t , f := {f 1 , f 2 , ..., f t }, the task is to decode the corresponding text tokens w := {w 1 , w 2 , ..., w n } of the stimuli used during the acquisition of the fmri volumes from an open vocabulary v. as mentioned in section 2, many studies have aimed to link cognitive signals with human language. we summarize three related tasks and compare them to fmri2text in the three representative tasks share a common characteristic of relying on cognitive signals that operate at the word level. however, this approach may not be practical for real-world fmri applications, as the poor temporal resolution of fmri necessitates tightly controlled experimental manipulations under these settings @xcite . in contrast, fmri2text leverages cognitive signals from more naturalistic settings, using text and speech as stimuli in a manner closer to real-world language use @xcite . here, each fmri frame corresponds to a specific timeframe, and is aligned with an undetermined number of tokens rather than a fixed one, better reflecting the variable and dynamic nature of natural language processing. another distinct feature that differentiates fmri2text from prior fmri-related tasks is its incorporation of multiple sequential frames as input. the inherent low signal-to-noise ratio of fmri has directed prior studies towards a focus on individual frames. however, this approach overlooks the valuable temporal information embedded within the interrelations of successive frames, which is particularly crucial when dealing with continuous data streams such as cognitive signals. ## method in this section, we introduce the unicorn structure and use the fmri2text task as an explicit demonstration. as shown in figure 2 , unicorn consists of two stages: (1) the cognitive signal reconstruction to train the encoder specifically for cognitive signals, and (2) the cog2text decoding to convert the embeddings of the cognitive signals from the first stage to human language. ## ablation study to further validate the effectiveness of unicorn, we conduct ablation studies on both fmri2text and eeg2text, to assess how the two phases of signal reconstruction affect the model's performance. as shown in conversely, ## conclusion in this paper, we introduce a novel open-vocabulary brain decoding task fmri2text, aiming to decode linguistic stimuli from multiple fmri frames collected under naturalistic conditions. building upon this, we present unicorn, a two-stage framework that integrates both temporal and spatial aspects of cognitive signals through snapshot and series reconstruction. the efficacy of unicorn is validated under various split settings, illuminating the opportunities that this task provides. furthermore, we adapt the framework to eeg2text, demonstrating its capacity to generate semantically and syntactically more accurate results, thereby introducing a fresh perspective to brain decoding tasks.