Cross-Modal Cloze Task: A New Task to Brain-to-Word Decoding (2022)

Abstract: Decoding language from non-invasive brain activity has attracted increasing attention from both researchers in neuroscience and natural language processing. Due to the noisy nature of brain recordings, existing work has simplified brain-to-word decoding as a binary classification task which is to discriminate a brain signal between its corresponding word and a wrong one. This pairwise classification task, however, cannot promote the development of practical neural decoders for two reasons. First, it has to enumerate all pairwise combinations in the test set, so it is inefficient to predict a word in a large vocabulary. Second, a perfect pairwise decoder cannot guarantee the performance on direct classification. To overcome these and go a step further to a realistic neural decoder, we propose a novel Cross-Modal Cloze (CMC) task which is to predict the target word encoded in the neural image with a context as prompt. Furthermore, to address this task, we propose a general approach that leverages the pre-trained language model to predict the target word. To validate our method, we perform experiments on more than 20 participants from two brain imaging datasets. Our method achieves 28.91% top-1 accuracy and 54.19% top-5 accuracy on average across all participants, significantly outperforming several baselines. This result indicates that our model can serve as a state-of-the-art baseline for the CMC task. More importantly, it demonstrates that it is feasible to decode a certain word within a large vocabulary from its neural brain activity.

## introduction neural decoding, i.e., using brain activity to make predictions of stimuli or mental states, is a challenging cross-discipline research area. it is crucial for developing brain-computer interfaces (bcis) that allow people to communicate using brain signals instead of verbal or body language @xcite . with the development of brain imaging technology and computational models, two lines of work emerge. one is invasive decoding, which depends on invasive brain recording methods such as electrocorticography (ecog). in recent years, several breakthroughs have been made in this field and demonstrated the feasibility to decode speech @xcite @xcite or handwriting @xcite from neural activity at high accuracy and speed. despite the impressive performance, invasive decoding is unlikely to be used except in rare medical situations since it needs invasive surgery on the brain. in contrast, non-invasive decoding uses atraumatic neuroimaging technologies such as functional magnetic resonance imaging (fmri) to collect brain signals, having wider applicable groups and applications prospects. however, progress in this field is relatively slow after the pioneering work of @xcite that shows the feasibility to discriminate an fmri image between two words. for a decade, this pairwise classification task (shown in recently, @xcite argues that we should move on to a more difficult but direct classification task rather than staying on the simple pairwise classification. in their work, they experi- ## analysis ablation study the feature extraction step is a key step in our method. in this step, we use retrieval-based method. hence the direct classification accuracy can also be used to evaluate the quality of feature vectors. we perform ablation experiments to understand the relative importance of each facet of our method. as shown in ## conclusions and future work in this paper, intending to build practical neural language decoders, we investigate the feasibility of large-vocabulary zero-shot brain-to-word decoding. large-vocabulary classification is much harder than pairwise classification. by introducing context as a prompt, we formalize it as a cross-modal cloze task, which alleviates the decoding difficulty while keeping the essence of neural decoding. furthermore, if we assume the past and the future content of brain activity has been decoded, our cmc task can be viewed as a simplified version of brain-totext decoding. based on this task, we find that decoding brain activity into words from a large vocabulary is possible to a certain extent, which lays the foundation for decoding text word by word from the brain. to move towards brain-to-text decoding, we can use a generative pre-trained language model to replace bert. the biggest challenge lies in how to align fmri signals to individual words when the stimuli are presented as a continuous time series of words. in the future, we are going to address this problem since it is fundamental for building powerful neural language decoders that translate brain activity into text.