Towards Brain-to-Text Generation: Neural Decoding with Pre-trained Encoder-Decoder Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Decoding language from non-invasive brain signals is crucial in building widelyapplicable brain-computer interfaces (BCIs).
However, most of the existing studieshave focused on discriminating which one in two stimuli corresponds to the givenbrain image, which is far from directly generating text from neural activities.
Tomove towards this, we ﬁrst propose two neural decoding tasks with incrementaldifﬁculty.
The ﬁrst and simpler task is to predict a word given a brain imageand a context, which is the ﬁrst step towards text generation.
And the secondand more difﬁcult one is to directly generate text from a given brain image anda preﬁx.
Furthermore, to address the two tasks, we propose a general approachthat leverages the powerful pre-trained encoder-decoder model to predict a word orgenerate a text fragment.
Our model achieves 18.20% and 7.95% top-1 accuracy ina vocabulary of more than 2,000 words on average across all participants on thetwo tasks respectively, signiﬁcantly outperforming their strong baselines.
Theseresults demonstrate the feasibility to directly generate text from neural activities ina non-invasive way.
Hopefully, our work can promote practical non-invasive neurallanguage decoders a step further.
Introduction
Decoding language from human brain activities is crucial in building BCIs that translate brain signalsinto a coherent text.
This technology is considered transformative for helping those who are unableto communicate due to some severe neuromuscular disorders [1].
Meanwhile, it also offers a tool forneuroscientists to study brain mechanisms.
Two lines of neural decoding research have dominatedthis ﬁeld: invasive decoding, based on invasive brain recording methods such as electrocorticography(ECoG); and non-invasive decoding, depending on atraumatic neuroimaging technologies such asfunctional magnetic resonance imaging (fMRI).
In recent years, several breakthroughs have beenmade in invasive decoding and demonstrated the feasibility to decode speech [2–4] or handwriting[5] from neural activities at high accuracy and speed.
Nevertheless, invasive decoding is unlikely tobe used except in rare medical situations since it needs invasive surgery on the brain.
In contrast, non-invasive decoding is applicable to normal people without doing any harm.
Therehave been some successful attempts in decoding words [6–8] and sentences [9, 10] from fMRI datain the form of pairwise classiﬁcation.
The pairwise classiﬁcation is a binary classiﬁcation task thatdiscriminates which one in two stimuli corresponds to the given fMRI image.
The major limitationsof this setting are: 1) to predict a word or a sentence, it has to enumerate all pairwise combinations inthe test set and thus is inefﬁcient; and 2) for sentence decoding, it can only select a ﬁxed sentencefrom two options and thus can not produce ﬂexible sentences.
Until now, little work has been done to directly generate text from fMRI images.
Early work hasattempted to generate natural language descriptions for brain activities evoked by motion pictures [11]. [Figure] [Figure]
They formulate the neural decoding problem as an image-captioning task and employ a pre-trainedimage-captioning system to perform this task.
Despite the novel method, they adopt perplexity as theevaluation metric, which can not reﬂect the relevance of the generated text to the given fMRI image.
Recently, Affolter et al.
[12] have tried to generate text conditioned on fMRI images using GPT-2[13].
They ﬁrst train a classiﬁer that takes an fMRI image as input and outputs a probability vectorover a vocabulary of 180 words.
Then to generate the next word for the history, they directly adjustthe output probability vector produced by GPT-2 by using the top-5 words predicted by the classiﬁer.
However, in their experiment, they use ground-truth information in prediction by limiting the top-5words to always contain the target word.
To understand to what extent can we currently decode text from fMRI data, we take advantage of theuseful and easily accessible context and design two neural decoding tasks with incremental difﬁculty.
The ﬁrst task, as shown in Figure 1(a), is to predict a word given an fMRI image and a maskedsentence related to that word.
The second task, as shown in Figure 1(b), is to generate a text fragmentcontaining the target word given an fMRI image and a preﬁx.
In general, the ﬁrst task is an easier onewhile the second task is more closer to practical neural decoders.
To address the two tasks, we propose a general approach that leverages the powerful pre-trainedtransformer-based encoder-decoder model BART [14] by formulating them into a mask-ﬁlling taskand a text generation task respectively.
In summary, our contributions are: 1) we propose two neuraldecoding tasks to explore the feasibility of decoding text from fMRI data, paving the road to buildpractical neural decoders that translate neural activities into a coherent text; 2) we propose a generalapproach that leverages the powerful pre-trained encoder-decoder model to address the two neuraldecoding tasks; 3) we validate the effectiveness of our method and demonstrate the feasibility togenerate text from fMRI data.
Neural decoding method
To address the two neural decoding tasks – the mask-ﬁlling task and the text generation task, wepropose a general approach as illustrated in Figure 1(c)-(d).
Our method contains two steps: 1)extracting semantic features from the fMRI image, and 2) fusing the extracted features into BART topredict a word or generate a text fragment.
We choose BART as the backbone of our model for tworeasons.
One is that BART is a conditional language model.
And the other is that BART is pre-trainedusing an in-ﬁlling scheme, where spans of text are replaced with a single mask token.
Hence, BARTis applicable and effective to both mask-ﬁlling and text generation tasks.
Feature Extraction Assume we have N fMRI images corresponding to N word stimuli, we ﬁrstrepresent the word stimuli using word vectors derived from the last hidden layer of BART’s encoder 1 .
The word embedding matrix is denoted as E ∈ R N ∗ d , where d refers to the dimension of the wordvectors.
Following the conventions in the pairwise classiﬁcation [6–10], we train ridge regressionmodels to map fMRI samples to their corresponding word vectors under cross-validation setting.
Foreach fMRI sample in the hold-out test data, its predicted word vector ˆ e is used as a query to retrieve knearest neighbor word vectors in the ground-truth semantic space E using the cross-domain similaritylocal scaling (CSLS) method [15].
The feature vector for each fMRI sample is obtained using thefollowing equation:
[Equation]
where i = 1 , ..., N and e i 1 , ..., e i k are the k retrieved word vectors.
Intuitively, the feature vectorcan be viewed as a new representation of the fMRI sample in the semantic space where we performfeature fusion.
Feature Fusion As shown in Figure 1(c)-(d), feature fusion is performed at the last hidden layer of BART’s encoder. The information from the neural activity and the information from the context arefused together in that layer and then propagated forward to the decoder.
Operating at this particularlayer allows the decoder to decide to what extent is the added information used in generating thenext word since they interact by cross-attention.
To be speciﬁc, assume we have an fMRI imagecorresponding to a word and a masked sentence / text fragment, the text is fed into BART as before.
And then the hidden states of the mask token h mask in the last hidden layer of the encoder is updatedusing the following equation:
[Equation]
where f i is the feature vector corresponding to the fMRI image.
Then the hidden states from theencoder are propagated forward to the decoder as it originally does.
In general, our decoding method is quite straightforward.
For simplicity, we do not ﬁne-tune BARTin our experiment but use it to derive word vectors as well as to predict a word / words.
The featureextraction step and the feature fusion step are decoupled and can be optimized independently.
Experiments
Datasets The brain imaging data is from [9], which contains 180 fMRI images corresponding to180 content words collected from 15 human participants 2 .
We select the most informative 5,000voxels following the voxel selection method proposed by [9] to reduce the dimension of fMRI data.
To prepare the context for our two decoding tasks, we use the sentences in the presentation scriptsin the fMRI experiment and mask the target word in each sentence as illustrated in the upper partof Figure 2.
Two samples of the dataset for each task are shown in the lower part of Figure 2.
As awhole, we create a neural decoding dataset of 1,080 samples per task per participant.
[Figure]
For details about the dataset and the experiment paradigms, please refer to https://osf.io/crwz7/.
Models We adopt BART without any extra information fusion and ﬁne-tuning as our baseline 3 .
Forneural decoding, we build our model on the same BART without ﬁne-tuning.
The hyperparameter kis tuned to 5.
In the text generation task, we use greedy decoding, which is to select the most probableword on each step of generation.
All the experiments are done under an 18-fold cross-validationsetting, repeatedly using 16 folds for training, 1 fold for validation, and 1 fold for testing.
Evaluation metrics For the mask-ﬁlling task, we adopt top-1 accuracy as an evaluation metricfollowing [12].
In the text generation task, if the generated text contains the target word, then it isdeemed correct, so the evaluation metric is also top-1 accuracy.
Results of the mask-ﬁlling task For the mask-ﬁlling task, as shown in Table 1, the strong baselineachieves 17.13% accuracy, demonstrating the effectiveness of BART in predicting words.
Based on
BART, our method achieves 18.20% accuracy on average across 15 participants, outperforming thebaseline by 1.07% absolute improvement.
The minimum and maximum accuracy are 17.87% and18.52% respectively, both surpassing the baseline.
Furthermore, the improvements on 14 participantsare statistically signiﬁcant under paired t-test with p-value < 0 .
05 while the p-value for the worstsubject is 0.051.
These results demonstrate the feasibility to directly predict a word in a largevocabulary from an fMRI image, which is a departure from the traditional pairwise classiﬁcation task.
Results of the text generation task As shown in Table 2, the strong baseline for the text generationtask obtains 6.48% accuracy, which is much lower than the performance in the mask-ﬁlling task.
This result shows that the text generation task is much harder than the mask-ﬁlling one.
Despitethe difﬁculty of the task, our model is able to achieve 7.95% accuracy on average across all participants, signiﬁcantly outperforming the baseline by 1.47% absolute improvement.
The minimum andmaximum accuracy are 7.41% and 8.24% respectively, both surpassing the baseline.
Moreover, theperformances of our model on 14 participants are signiﬁcantly better than the baseline.
These results,for the ﬁrst time, demonstrate that it is feasible to generate text related to the stimuli from fMRIimages without using the ground-truth information.
The experiment results show the effectiveness touse the pre-trained encoder-decoder model in neural decoding.
[Table] [Table]
Conclusions and future work
In this paper, we have proposed two neural decoding tasks to understand to what extent can wecurrently decode text from brain images.
From the perspective of engineering, these two tasks aredepartures from the traditional pairwise classiﬁcation task. They can help to promote the developmentof practical non-invasive neural decoders that translate brain activities into a coherent text.
From theperspective of neuroscience, a good neural decoder can serve as a useful adjunct to basic research incognitive neuroscience.
Future work will be move on to explore how voxels selected from differentbrain regions affect the accuracy of neural decoders.
In this way, we can infer where in the brainsemantics are represented.
In the ﬁeld of natural language processing (NLP), pre-training has become a dominant paradigm.
Language models pre-trained on text from a wide variety of sources have achieved remarkable resultson language understanding and language generation tasks.
In this paper, we have experimentedwith a pre-trained transformer-based conditional language model.
In the future, we are going toexperiment on more pre-trained models to investigate how different model architectures and pretraining objectives affect the performance of neural decoders.
Intuitively, models that better correlatewith brain activation data may better explain brain mechanisms.
Through contrastive analysis ofdifferent pre-trained models in neural decoding, hopefully, we can provide some insights into howhumans learn a language.