Mind Reader: Reconstructing complex images from brain activities
Anonymous Author(s)
Abstract
Understanding how the brain encodes external stimuli and how these stimuli canbe decoded from the measured brain activities are long-standing and challengingquestions in neuroscience.
In this paper, we focus on reconstructing the compleximage stimuli from fMRI (functional magnetic resonance imaging) signals.
Unlikeprevious works that reconstruct images with single objects or simple shapes, ourwork aims to reconstruct image stimuli that are rich in semantics, closer to everydayscenes, and can reveal more perspectives.
However, data scarcity of fMRI datasetsis the main obstacle to applying state-of-the-art deep learning models to thisproblem.
We find that incorporating an additional text modality is beneficial for thereconstruction problem compared to directly translating brain signals to images.
Therefore, the modalities involved in our method are: (i) voxel-level fMRI signals,(ii) observed images that trigger the brain signals, and (iii) textual description of theimages.
To further address data scarcity, we leverage an aligned vision-languagelatent space pre-trained on massive datasets.
Instead of training models fromscratch to find a latent space shared by the three modalities, we encode fMRIsignals into this pre-aligned latent space.
Then, conditioned on embeddings in thisspace, we reconstruct images with a generative model.
The reconstructed imagesfrom our pipeline balance both naturalness and fidelity: they are photo-realisticand capture the ground truth image contents well.
Introduction
In an effort to understand visual encoding and decoding processes, researchers in recent years havecurated multiple datasets recording fMRI signals while the subjects are viewing natural images[3, 7, 32, 39].
In particular, the Natural Scene Dataset (NSD [3]) was built to meet the needs ofdata-hungry deep learning models, sampling at an unprecedented scale compared to all prior workswhile having the highest resolution and signal-to-noise ratio (SNR).
In addition, all the images usedin NSD are sampled from MS-COCO [20], which has far richer contextual information and moredetailed annotations compared to datasets that are commonly used in other fMRI studies (e.g., Celeb
A face dataset [21], ImageNet [9], self-curated symbols, grayscale datasets).
This dataset, therefore,offers the opportunity to explore the decoding of complex images that are closer to real-life scenes.
Human visual decoding can be categorized into stimuli category classification [1], stimuli identification [36], and reconstruction.
We focus on stimuli reconstruction in this study.
Different fromprevious efforts in reconstructing images from fMRI [5, 10, 11, 22, 30, 32, 33], we approach theproblem with one more modality, that of text.
The benefits of adding the text modality are threefold:first, the brain is naturally multimodal.
Research [6, 12, 23] indicates that the brain is not only capableof learning multisensory representations, but a larger portion of the cortex is engaged in multisensoryprocessing: for example, both visual and tactile recognition of objects activate the same part of the
[Figure]
object-responsive cortex [24].
Visual-linguistic pathways along the border of the occipital lobe [25]also bring a more intertwined view of the brain’s representation of these two modalities.
Second,multimodal deep models tend to explain the brain better (having higher representation correlations)than the visual-only models, even when compared with activities in the visual cortex [8].
Lastly, ourgoal is to reconstruct complex images that have multiple objects in different categories with intricateinteractions: it is natural to incorporate contextual information as an additional modality.
Instead of training a model to map all three modalities (fMRI, image, text) to a unified latent space, wepropose to map fMRI to a well-aligned space shared by image and text, and use conditional generativemodels to reconstruct seen images from representations in that space.
This design addresses the datascarcity issue of brain datasets by separating fMRI from the other two modalities.
In this way, a largeamount of data is readily available to learn the shared visual-language representation and to train agenerative model conditioned on this representation.
Furthermore, pre-trained models can be utilizedto make the whole reconstruction pipeline more efficient and flexible.
Our contributions are as follows: (1) to the best of our knowledge, this is the first work on reconstructing complex images from human brain signals.
It provides an opportunity to study the brain’svisual decoding in a more natural setting than using object-centered images.
Compared to previousworks, it also decodes signals from more voxels and regions, including those outside the visualcortex, that are responsive to the experiment.
This inclusion allows us to study the behavior andfunctionality of more brain areas.
(2) We address the data scarcity issue by incorporating additionaltext modality and leveraging pre-trained latent space and models.
For the reconstruction, we focus onsemantic representations of the images while taking low-level visual features into account.
(3) Ourresults show we can decode complex images from fMRI signals relatively faithfully.
We also performmicrostimulation on different brain regions to study their properties and showcase the potential usagesof the pipeline.
Method
Recent developments in contrastive models allow more accurate embeddings of images and theirsemantic meanings in the same latent space.
This performance is realized using massive datasets:models such as CLIP [26] and ALIGN [16] utilize thousands of millions of image-text pairs forrepresentation alignment.
In comparison, brain imaging datasets that record pairs of images and fMRIrange from 1.2k to 73k samples, making it difficult to learn brain encoding and decoding modelsfrom scratch.
However, we can utilize aligned embeddings obtained from pre-trained contrastivemodels as the intermediary and generate images conditioned on these embeddings.
In our pipeline,as shown in fig. 1, we first map fMRI signals to CLIP embeddings of the observed image and itscaptions, then pass these embeddings to a conditional generative model for image reconstruction.
2.1 Caption Screening
Each image x img in the COCO dataset has five captions { x cap 1 , · · · , x cap 5 } collected through Amazon’s Mechanical Turk (AMT), and in nature, these captions vary in their descriptive ability.
Fig.
2shows a sample image with its five captions, and we can tell captions (2) and (3) are more objectiveand informative than caption (4) when it comes to describing the content of that image, thus aremore helpful to serve as the image generation condition.
We utilize pre-trained CLIP encoders toscreen the high-quality captions since representations in the CLIP space are trained to be image-textaligned.
A caption with an embedding more aligned to the image embedding is more descriptive thana less aligned one; it is also less general and more specific to this particular image because of thecontrastive loss in CLIP.
For the screening, we pass each image together with its five captions to the
CLIP model, which outputs corresponding probabilities that the captions and image are proper pairs.
We keep captions with probabilities larger than half of the highest probability.
After screening outless informative captions, we have one to three high-quality captions per image.
2.2 Mapping fMRI signals to CLIP space
Each fMRI signal that reflects a specific image is a 3D data volume, and the value on position ( i, j, k )is the relative brain activation on this voxel triggered by the image.
We apply an ROI (region ofinterest) mask on this 3D volume to extract signals of cortical voxels that are task-related and havegood SNRs.
The signal is then flattened into a 1D vector and voxel-wise standardized within eachscan session.
The end results x fmri are used by our image reconstruction pipeline.
We choose to usethe ROI with the widest region coverage, and the length N of x fmri ranges from 12682 to 17907 fordifferent brains in the NSD dataset.
Our goal is to train two mapping models, f mi and f mc in fig.
1 (collectively denoted as f m ), thatencodes x fmri ∈ R N to h img
=
C img ( x img ) ∈ R 512 and h cap =
C txt ( x cap ) ∈ R 512 respectively.
Here 93 C img , C txt are CLIP image and text encoders, and x cap is one of the image captions chosen randomly 94 from the vetted caption pool.
We construct both f m as a CNN with one Conv1D layer followed byfour residual blocks and three linear layers.
The training objective is a combination of MSE loss, 96 cosine similarity loss, and contrastive loss on cosine similarity.
We use the infoNCE definition [37]of contrastive loss, for the i th sample in a batch of size B :
[Equation]
For the mapping model f mi that encodes fMRI to image embeddings, we have h ( i ) img ′
=
f mi ( x ( i ) fmri ) .
The training objective is:
[Equation]
where τ, α 1 , α 2 , α 3 are non-negative hyperparameters selected through sweeps.
The loss L mcfor caption embedding mapping model f mc is defined similarly.
Although CLIP embeddings aretrained to be aligned, there are still systematic differences between image and text embeddings, withembeddings under each modality showing outlier values at a few fixed positions.
In addition, wealso notice the generated images emphasize either image content (object proximity, shape, etc.) orsemantic features depending on which condition we use.
Therefore, including both embeddings asthe conditions for a generator can cover both ends, and that is why we train two mapping modelsfor the two modalities.
Since the outlier indices are fixed for each modality across images, clippingthe value should not affect image-specific information. Therefore, before normalizing the ground truth embeddings into unit vectors, we set h = clamp( h , − 1 . 5 , 1 . 5) .
This can greatly improve themapping performance during training.
2.3 Image reconstruction with CLIP embedding conditioning
The mapping models output fMRI-mapped CLIP embeddings h ′ img and h ′ cap that serve as conditionsfor the generative model.
We aim to generate images that have both naturalness (being photorealistic) and high fidelity (can faithfully reflect objects and relationships in the observed image).
Ourgeneration model is built upon Lafite [40], a text-to-image generation model: it adapts unconditional
StyleGAN2 [19, 18] to conditional image generation contexted on CLIP text embeddings.
In our generator G , both conditions h ′ img and h ′ cap are injected into the StyleSpace: each of them goesthrough two fully connected (FC) layers and is transformed into condition codes c img and c cap .
Thesecondition codes are max-pooled and then concatenated with the intermediate latent code w ∈ W ,which is obtained from passing the noise vector z ∈ Z through a mapping network (see fig. 1).
Usinga mapping network to transform z into an intermediate latent space W is the key of StyleGAN as Wis shown to be much less entangled than Z [34].
The conditioned style s is then passed to differentlayers of G as in StyleGAN2, generating image x img ′ :
[Equation]
We align the semantics of generated x img ′ and condition vectors by passing x img ′ through pre-trained
CLIP encoders and apply contrastive loss (eq. (5) L c 2 ) between them.
For further alignment ofthe lower-level visual features, such as prominent edges, corners and shapes, we also pass theimage through resnet50 and align the position-wise averaged representation obtained from Layer2(eq.
(5) L c 3 ).
The discriminator D has three heads that share a common backbone: the first head D d classifiesimages to be real/fake, the second and the third semantic projection heads D si , D sc map x img ′ to h ′
imgand
h ′ cap .
The latter two ensure the generated images are faithful to the conditions.
It is also shownthat contrastive discriminators are useful for preventing discriminator overfitting and improvingthe final model performance [17, 15].
Applying contrastive loss (eq. (5) L c 1 ) between the outputsfrom discriminator semantic projection heads and the condition vectors fed to G can therefore helpstabilize the training.
To summarize the objective function, the standard GAN loss is used to ensurethe naturalness of generated x img ′ :
[Equation]
where σ denotes the Sigmoid function.
Meanwhile, contrastive losses are used to align the semanticsof generated images and the fMRI-mapped condition vectors that supposedly residing in the CLIPspace:
[Equation]
The overall training objectives are: L G =
L
[Equation]
, L D =
L
[Equation]
+
[Equation] [Equation]
where λ 1 , λ 2 , λ 3 , are non-negative hyperparameters.
The whole generation pipeline, consisting of mapping models and GAN, is trained in two stages.
First, mapping models f mi and f mc are trained on fMRI-CLIP embedding pairs.
Next, startingfrom the trained mapping model weights and Lafite language-free model weights, we modify thelosses and model structure and finetune the conditional generator.
For the additional condition vectorprojection layers in G and semantic head in D , we duplicate the weights in the existing parallellayers to make the model converge faster.
Note that Lafite is pre-trained on the Google Conceptual
Captions 3M dataset [31] then finetuned on the MS-COCO dataset, both of which are much largerthan NSD.
Finetuning from it allows us to exploit the natural relationships between semantics andimages with sparse fMRI data.
We can still utilize a two-stage training to compensate for data scarcityeven if no pre-trained conditional GAN like Lafite is available, for example, when using a differentgenerator architecture.
Only this time, we should firstly train the conditional GAN on a large imagedataset with noise perturbed h img and h cap as the pseudo input condition vectors.
Results
3.1 Data and experimental setup
The NSD data is collected from eight subjects.
We focus on reconstructing observed scenes from asingle subject’s brain signals.
The reasons are twofold: first, it is more accurate to utilize individualbrain coordinates instead of mapping voxels into a shared space, which can result in information lossduring the process.
More importantly, brain encoding and perception are different among individuals.
This project aims to get the best reconstruction for a single individual, thus training models on onesubject’s data.
Nevertheless, the commonality of this encoding process among the population is anexciting topic for future explorations.
We use subject one from NSD: the available data contains 27750 fMRI-image sample pairs on 9841images.
Each image repeats up to three times during the same or different scan sessions.
Note thatbrain responses to the same image can differ drastically during the repeats (fig. 6).
The dataset issplit image-wise: 23715 samples corresponding to 8364 images are used as the train set, and 4035samples corresponding to the remaining 1477 images are used as the validation set.
Therefore, ourpipeline never sees the image it will be tested on during the training.
We use 1pt8mm-resolution scansand only consider fMRI signals from voxels in the nsdgeneral ROI provided by the dataset.
This ROI covers voxels responsive to the NSD experiment (voxels with high SNR) in the posterior aspectof the cortex, and contains 15724 voxels for subject one ( x fmri ∈ R 15724 ).
Images are all scaled to256 × 256 .
Additional experiment settings , including hyperparameters of two training phases, areprovided in appendices A.1 and A.2.
Our experiments are conducted on one Tesla V100 GPU andone Tesla T4 GPU.
All codes will be publicly available.
3.2 Mapping models from fMRI to CLIP embeddings
Evaluation Criteria
In the first training stage, mapping models f mi and f mc are trained to encodefMRI signals to CLIP embeddings.
We use two criteria to evaluate the mapper performance to decidewhich one to use in the next stage.
The first criterion is FID (Fréchet Inception Distance) [13] betweengenerated image and ground truth using the trained mapper and a pre-trained generator.
Given a
Lafite model pre-trained on MS-COCO (language-free setting), we can replace its conditional vectorwith the outputs of our mapping models to generate images conditionally.
These FIDs can indicatethe starting points of the finetuning processes: the lower the FID, the better the candidate model.
Secondly, we use the success rate of image "retrieval" in a batch of size 300.
For the i th samplein the batch, if the cosine similarity between h ( i ) ′
and h ( i ) is larger compared to between h ( i ) ′
andh ( j ) , j ̸
= i , then it counts as one successful forward retrieval.
For backward retrieval, we count thenumber of correct matches of h ( i ) to all h ( j ) ′ .
Configuration comparisons We tested different configurations on the mapping models, including:(1)
Whether to place the threshold at ± 1 . 5 as mentioned in section 2.2; (2) When training f mi ,whether to perform image augmentations before passing images through the CLIP encoder; (3) Whentraining f mc , whether to use the CLIP text embedding of a fixed caption, a random valid caption, oruse the average embedding of all valid captions; (4) Which loss function to use: MSE only, cos (cosinesimilarity) only, Contra only, MSE + cos , MSE + cos + Contra ; (5) Whether auxiliary networks help.
We tested adding an auxiliary discriminator with GAN loss, as well as adding auxiliary expandernetworks with VICReg loss [4].
We found: (1) Clamping ground truth embeddings significantly increase performance; (2) Usingimage augmentations increase f mi performance.
This further indicates CLIP embeddings are moresemantic related; (3) For f mc , selecting a random caption from the valid caption pool each time isbetter than using a fixed one or using the average embedding of all valid captions; (4) Using MSE + cosas the loss gives the best base models, but then finetune these base models with MSE + cos + Contracan further lower the starting FID for pipeline finetuning, making the training in the next stageconverge faster; (5) Adding auxiliary networks and objectives will not improve the performance.
Ingeneral, although h cap and h img are already relatively well aligned, f mc can still map x fmri closer toh cap than h img , whereas f mi maps x fmri to an embedding that is equally close to both, while beingable to capture a few extreme values in h img (see appendix A.3 for numerical details and mappedembedding visualizations).
We think this difference reflects that it is easier to map fMRI signals to amore semantic representation (from the text space) than to a visual one.
[Table]
[Table]
[Table]
To verify fMRI-mapped embeddings h ′ are semantically well aligned with ground truth CLIPembeddings, we examined the mismatches during the image retrieval.
For four incorrect retrievals,fig.
3 shows which images’ h ( j ) ′ are closer to the ground truth images’ h ( i ) than h ( i ) ′ .
Notably, thesemismatches are semantically close to the ground truth images.
This indicates that the mapping modelscan successfully map fMRI signals into a semantically disentangled space.
Embeddings in this spaceare suitable for providing contexts to a conditional generative model.
We also tested another mappingmodel f mr that maps fMRI signals to representations obtained from resnet50 Layer2 .
Unlike the
CLIP embedding space, the resnet vector encodes more lower-level visual features.
We see a jumpin the image retrieval rate when we combine the representations obtained from f mi , f mc with f mr(table 3).
However, the generative model is difficult to train when taking in two conditions fromdistinct embedding spaces.
Therefore, we add the low-level vision constraint into the contrastive loss
L c 3 instead.
3.3 Conditional image generation
Quantitative results In the second training stage, we finetune the conditional StyleGAN2.
1 Thereis no standard metric to measure image reconstruction quality from fMRI signals for complex images.
Since previous works focused on reconstructing simpler images, the metrics typically involve pixelwise MSE or correlation measures.
However, when it comes to complex images, it seems morereasonable to use a perceptual metric, such as FID, which is based on Inception V3 [35] activationsand is widely used in GAN.
We also detail another metric, n-way identification accuracy, that reflectsmore of the fidelity and uniqueness of the generated images, in appendix A.4.
We perform theablation studies on the pipeline to answer the following questions: (1) Which mapping model trainedin stage one leads to the best final performance?
f mc or f mi or using both?
(2) Which pre-trained
GAN leads to the best final performance?
For this, we compare using Lafite pre-trained on either thelangue-free (LF) setting or the fully supervised setting.
Whether including the contrastive loss
L c 3 between lower-level visual features can further improve the performance of a semantic-basedgenerative model?
Finally, we tested (4) whether finetune the whole pipeline end-to-end or freezingthe mapping models is better?
The new mapping model losses are set to L ′ m =
L m + λ 4 L GAN G iftrained end-to-end.
Results are reported in table 1.
We observed the following: (1) in terms of FID, using h ′ img obtainedfrom f mi as the generator condition is better than using the h ′ cap from f mc or using two conditionalheads.
On the other hand, f mc and the two-head setting achieve as good or even better performanceas f mi does in terms of n-way identification accuracy.
In addition, if training time or resource is the concern, using two heads and pre-trained LF-Lafite with only condition feeding interface changesand cloned weights in the new branches can already give reasonably good results. (2) Training thepipeline on LF-Lafite is much better than on the fully supervised Lafite.
This result is expected for thegenerator conditioned on h ′ img since the supervised version is conditioned on CLIP text embeddings.
However, the same discrepancy exists for the generator conditioned on h ′ cap .
This may reflect theflexibility of pre-trained generators to adapt to the slight changes in the embedding space.
It alsoshows the crucial impact of a pre-trained model on final performance when training data is limited.(3)
The addition of low-level visual feature constraint L c 3 is beneficial for the model performance,especially faithfulness.
It also seems to have more effects on single-head models than the two-headone.
For the end-to-end pipeline training, we test performance with λ 4 =
[0 . 1 , 1 , 10] , all of whichgive worse performance than keeping the mapping model weights frozen (reported values are fromλ 4 = 1 ).
In particular, we found that h ′ tends to collapse to having nonzero values at only a fewpositions if the mappers are finetuned together with GAN.
Qualitative results We show several generated images in fig.
Although the generator takes inboth the noise vector z and fMRI-mapped embeddings, the results vary much more with the lattercondition, while z only contributes to variations on some minor details.
In general, the generatedimages capture both semantics and visual features relatively well, even on complex images containinginteractions of multiple objects.
Since each stimulus is repeated up to three times to the subject, wehave multiple fMRI scans corresponding to the same image.
The semantic differences in the generatedimages conditioned on these multiple scans could potentially reveal brain processing discrepanciesof the same stimulus.
For example, the three generations for the second image in fig.
4b emphasizerespectively: (1) the overall scene and the fence, (2) people with green suits, and (3) overhead flagsand the fence; these might reflect the variations in the subject’s attention or interpretations of thatimage.
Eyetracking data can be further examined to study attention’s effect on generated images.
It is challenging to perform one-to-one comparisons with previous deep image reconstruction workssince the images in the MS-COCO dataset have much higher complexities than artificial shapes, faces,or images containing a single centered object (like in ImageNet).
We show results from a few bestmodels for reconstructing images from fMRI in fig.
5a.
There is also a recent survey [27] coveringmore models and results if readers are interested.
As our dataset is different, we search for similarimages in the NSD validation set and show our generations in fig.
5b.
Compared to other methods, ourpipeline can generate more photo-realistic images that reflect objects’ shapes and backgrounds well.
It also utilizes more voxel activities than previous works (15724 voxels versus a few hundred).
Moreimportantly, it is able to reconstruct the relationships of different components when the images aremore complex.
As natural scenes around us are rarely isolated objects and always information-laden,we think reconstructing images through semantic alignment and conditioning is more beneficial andrealistic than focusing on lower-level visual features. Microstimulation
In neuroscience, microstimulation refers to the electrical current-driven excitation of neurons and is used to identify the functional significance of a population of neurons.
Here,
[Figure]
DNN
DNN+DGN
GAN
EncDec EncDec
SSGAN
D-VAE/GAN [32] [32] [33] [5] [11] [10] [30]
[Figure]
we "microstimulate" the input fMRI signals of voxels in different brain ROIs, aiming to identify theroles of individual regions.
In the NSD dataset, there are four floc (functional localizer) experimentstargeting regions responsible for faces, bodies, places, and words.
A typical standardized fMRI signalhas a value range around [ − 4 , 4] .
For the experiment, we locate the corresponding task-specificvoxels based on ROI masks and increase the voxel activities to 10 while keeping the activities inunrelated voxels unchanged (see appendix A.5 for visual results).
We observe the emergence ofbodies or words when we increase the voxel activities in "bodies" or "words" ROIs.
For voxels in"places" ROIs, elevating the signals will result in mesh-like patterns in the background, and this istrue across different images.
For "faces" ROIs, the generated images under elevated facial area signalsseem to contain many small repeated patterns/perturbations.
Interestingly, this appears to result from FFA (fusiform face area) signal changes since increasing only OFA (occipital face area) regions’activity does not result in similar patterns.
Overall, increasing a specific task ROI’s signal acrossfMRI samples results in CLIP embedding changes in similar positions.
This means the disentangledspace of CLIP embedding aligns well with how the human brain processes visual cues.
Apart from task-specific ROIs, we also changed brain region activities based on their roles in the visualprocessing hierarchy.
We use the streams mask in the dataset to identify early visual cortex ROIs,intermediate ROIs, and higher-level ROIs.
We then zero out voxels at each level.
Our observationsare: (1) when silencing the early visual cortex, objects and the whole scene are prone to be in dullcolors, and objects tend to have sharp shapes.
Meanwhile, the mapped embedding in the CLIP spacewill constantly have a lower value at almost all positions compared to mapped from unchanged signals.
(2) Silencing the higher-level ROIs has the opposite effect: more colors, more shapes, and crowdedscenes.
This is reasonable since the lower-level visual regions will bring up all the details when theylack high-level control.
This time, the embeddings in the CLIP space have values consistently higherthan normal.
Finally, (3) silencing the intermediate ROIs seems to have the least visual impact or CLIP embedding changes among the three.
We performed the above microstimulation experimentson our pipeline with existing ROIs; however, it is potentially helpful for testing new ROI definitionsand hypotheses.
Further discussions
Prior to our current pipeline design, we experimented with a DALL-E -like structure [29] sincewe can view the image reconstruction problem as signal-to-signal translation.
In particular, weapplied VQVAE [38] on both fMRI and image to represent them as discrete latent codes and train a
Transformer model to autoregressively generate text and image tokens from fMRI tokens.
However,it was challenging to train the Transformer-based model to converge with limited fMRI-image data.
Incorporating the caption as text tokens to serve as the bottleneck between fMRI and image tokenswhile utilizing pre-trained models on the text and image modality did not help either.
We think thissuggests the need to introduce a semantic medium to avoid direct translations between fMRI andimage, as well as a solution to data scarcity.
We address both issues with the semantic space of CLIP embeddings.
First, CLIP space is semanticallyinformative and visually descriptive: for example, we can use image-text CLIP embedding alignmentprobabilities to screen captions.
Mapping fMRI signals to representations in this latent space willretain rich information about the image that needs to be reconstructed.
Second, the pre-training ofthe generative model can be separated entirely from fMRI data, meaning it can utilize much largerdatasets than the one we use.
However, there is a trade-off between generating a semantically similarscene and faithfully reconstructing each pixel.
Although trained with additional contrastive losstargeting low-level visual features, the generated images by our pipeline are still leaning towards theformer.
We consider this a reasonable choice since brains are more likely to perceive the image as awhole rather than identifying each pixel, especially with multiple objects in the scene.
Nevertheless,this results in worse reconstructions for images with fine details but less semantic, such as singlefaces.
The reconstruction of complex images with better aligned low-level visual features is worthfurther studies.
There are many more areas to explore.
First, our study focuses on reconstructing a single subject’sbrain signals.
Applying the model to different subjects and observing the differences when generatingthe same image would be interesting.
Since the data contains behavioral measures like valence andarousal towards each image, one can test if the generated images reflect personalized attention andperceptions.
Second, other latent spaces can be examined.
Although CLIP is one of the best-alignedcomputation models for the brain, other multimodal models like TSM [2] seem to have a betteralignment [8] with the visual cortex.
In addition, other conditional generative models, such asdiffusion models, can be explored.
In particular, DALL-E 2 [28] generates images conditioned on
CLIP embeddings through diffusion, and it also provides an alternative solution to the differencesexhibited in the image the text CLIP embeddings by learning a Prior model.
Third, given theadditional text modality, our pipeline opens up new opportunities to study visual imagery evenwithout ground truth images.
For example, one can either use mapping models trained on givenfMRI-image pairs and pre-trained generators to reconstruct imagined scenes, or study the mappingbetween brain signals and the text embeddings of the mental images’ descriptions.
Lastly, we focuson the decoding (brain-to-image) process, but the encoding (image-to-brain) process of compleximages is equally important and exciting.
With current brain signal recording devices, the negative social impact of this work is minimal:portable devices like EEG have poor spatial resolutions, making them unlikely to provide enoughimage-related details; On the other hand, fMRI scanners are used under highly controlled settingswith designed procedures, therefore unlikely to have subject-unapproved privacy violations.
However,when new devices that can address these issues become readily available, regulations would beneeded on collecting and inspecting user data, since they potentially reveal sensitive information thatusers are unwilling to share through neural decoding.
With pre-trained components, the pipelinemay also misinterpret brain signals or be hacked to generate from manipulated inputs (no matterhow unlikely it is) and produce over-confident false reconstructions because of the training datadistributions.
Several tricks may alleviate this issue, for example, training an input discriminatorand placing it before the entire pipeline to filter out suspicious inputs. Or, using a parallel pipelinetargeting pixel-level reconstruction as a check: if the two systems agree with each other abovea certain threshold, the reconstruction results are accepted, otherwise discarded. Future pipelineimprovements should also focus on exploring high-performing models pre-trained on large (thusmore generalizable) and unbiased datasets (more in appendix A.8).  Conclusion
The paper proposes a pipeline to reconstruct complex images observed by subjects from their brainsignals.
With more objects and relationships presented in the image, we bring in an additional textmodality to better capture the semantics.
To achieve high performance with limited data
, we utilizepre-trained semantic space that aligns visual and text modalities.
We first encode fMRI signals to thisvisual-language latent space and use a generative model conditioned on the mapped embeddings toreconstruct the images.
We also introduce additional contrastive loss to incorporate low-level visualfeatures into this semantic-based pipeline.
As a result, the reconstructed images by our method areboth photo-realistic and, most of the time, can faithfully reflect the image content.
This brain signalto image decoding pipeline opens new opportunities to study human brain functions through strategicinput alterations and can even potentially be helpful for human-brain interfaces.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Seesection 4(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] In thesupplemental material.(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See appendix A.2.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See tables 4 and 5 in appendix A.4.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See section 3.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See section 3.3.
(b) Did you mention the license of the assets?
[No] The code and the data are proprietary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Additional codes are provided in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] The dataset paper containing the details is cited.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] The dataset paper containing the details is cited.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Appendix
[Figure]
A.1 Data
[Figure]
fMRI data fMRI activities differ when the same individual sees the same image at different times(fig. 6).
Although we use activities and signals interchangeably throughout the paper, what we meanare fMRI betas in the NSD dataset.
Betas are not direct measurements of BOLD (blood-oxygenationlevel dependent) changes, but the inferred activities from BOLD signals through GLM (generallinear models).
The reason for using betas instead of direct measurements is that image stimuli areshown consecutively to the subjects without prolonged delay, and activities triggered by the previousimage can interfere with the next one if there is no proper separation.
Authors of NSD proved theeffectiveness of their GLM approach with much improved SNR in the betas over raw measurements[3].
[Figure]
Image augmentation during training Based on conclusions from StyleGAN2-ADA [18], weperform the following image augmentations before passing images into the CLIP encoder whentraining the fMRI-CLIP mapping model:
• perform random sized crop with a scale between 0.8 to 1.
• perform horizontal flip with probability p = 0 .
• perform ColorJitter(0 . 4 , 0 . 4 , 0 . 2 , 0 . 1) with p = 0 .
4 .• perform grayscale with p = 0 .
2 .• perform Gaussian blur with p = 0 .
5 and kernel size 23.• perform random masking with 0.3 masking ratio.
We test mapping models trained with and without the above augmentations, and found augmentationscan improve fMRI to CLIP image embedding mapping performance (details are in table 2).
CLIP embeddings and thresholding See fig.
7 for the visualizations of CLIP embeddings thatshow image and text embedding differences, effects of thresholding, image augmentation, and randomcaption selection.
A Experiment hyperparameters
The following hyperparameters are used in our experiments:
• τ = 0 .
5 in eq.
(1) for all the contrastive losses.
• for fMRI-CLIP mappers f mi , f mc (losses are in eq.
(2)), the models are first trained with
α 1 = 0 .
4 , α 2 = 0 .
6 , α 3 = 0 , then finetuned with α 1 = 0 .
2 , α 2 = 0 .
3 , α 3 = 0 .
• mappers are trained with batch size 32 (on a single GPU) when not including contrastive
loss, and batch size 128 when including contrastive loss or using VICReg loss.
Learning rate is 0.0004.
• λ 1 = 5 , λ 2 = 10 , λ 3 = 10 for the losses of conditional StyleGAN2.
• conditional StyleGAN2 is trained with batch size 16 × number of GPUs (in our case B =
since we used two GPUs).
Learning rate is 0.0025.
[Figure]
A.3 Results for the fMRI-CLIP mapping models f m
Mapping models f mi and f mc are trained under different settings detailed in section 3.2, here welist the numerical results of the summarized findings in table 2.
Simply put, forward retrieval checksthe correct match of " which ground truth CLIP embedding is the closest to the fMRI-mapped one? "
while the backward retrieval checks " which fMRI-mapped embedding is the closest to the groundtruth CLIP one? ".
When multiple losses are involved, we use hyperparameter settings as in A.2.
Fig.
8 visualizes the mapping results of the best setting (models trained with threshold, imageaugmentation, use a random valid caption each time, pre-trained with MSE+cos loss then finetunedwith MSE+cos+Contra loss).
Combining the mapped embeddings from multiple mappers boosts the retrieval performance, especially the backward one (as shown in table 3).
To use multiple mapping models, we first calculate a
B × B batch similarity matrix between the mapped embeddings for each model.
Then we combinethe similarity matrices with a weighted sum (weights are obtained through grid search) and performimage retrievals based on this combined similarity matrix.
The mapping model f mr that encodesfMRI to ResNet embeddings has a correct forward retrieval 6 and backward retrieval 50.
But whenits similarity matrix is combined with mapped-CLIP embedding similarity matrices, the performanceis far above that of both ResNet and CLIP embeddings.
[Figure] [Table] [Table]
A.4 Additional quantitative results (generator)
In addition to using FID as a metric
[Table]
we also perform 2-way identification for images reconstructed bymodels under different settings, and n-way identification of generated images with n = 2 , 5 , 10 , 50under the best setting (finetuned from LF, with L c 3 , with mapping models f m frozen).
For n-wayidentification
[Table]
we reconstruct an image from the fMRI signal for each sample in the validation set.
Foreach generated image, we compare it with a set of n randomly selected images, including the groundtruth one.
Then based on the cosine similarity of their Inception V3 embeddings (before FC layers,the length-2048 vector)
[Table]
we identify which image the generated one corresponds to.
This processis repeated ten times because of the randomness of the n-sample selection.
Results are reported intables 4 and 5.
The n-way identification accuracy of the two-head setting ( f mi & f mc ) is slightlybetter most of the time (table 5)
[Table]
but this advantage is minor when taking the standard deviations intoaccount.
Note that when performing n-way identification [Table] previous image reconstruction works are
[Table] [Table]
typically tested on a validation set that contains 50 images of 50 different categories [14].
However,there are multiple objects involved in each image in the complex images we aim to reconstruct; itis not straightforward to separate them into different categories and pick one from each.
Therefore,we leave the validation set as is (1477 image-fMRI pairs in total) [Figure] and there will be overlappingcategories in it; for example, several images contain scenes of animals in a natural environment.
A.5 Visual results from microstimulation experiments
[Figure]
Fig 9 shows generated images under different microstimulation experiments.
Fig 10 shows theresults regarding changes of mapped fMRI embeddings in the CLIP space when perturbing voxels indifferent visual cortex levels.
A.6 Input interpolations and the potential extension to movie reconstruction
In addition to reconstructing observed images [Table] we found utilizing the CLIP space can also result in asmooth transition when decoding from interpolations of two fMRI scans (fig. 11). Combined withthe ability to capture complex semantics
[Table]
this pipeline can be helpful for movie reconstruction frombrain signals.
Temporal constraints can also be added, which could, in turn, benefit the reconstructionof each frame.
A.7 CLIP space as the intermediary
[Figure]
[Table]
In this section
[Table]
we show that multimodal embedding space, particularly the CLIP space, is beneficialfor brain signal decoding.
To this end, we trained a set of multi-label category classifiers to classifyif a certain object category exists in the image based on the following inputs: (1) image-triggeredfMRI x fmri ∈ R 15724 ; (2) image CLIP embeddings h img ∈ R 512 ; (3) CLIP embeddings mappedfrom image-triggered fMRI h ′ img ∈ R 512 ; (4) image ResNet embeddings ResNet( x img ) ∈ R 2048(obtained from Layer4 , the final block before fully connected layers).
All classifier models consist of3 linear layers with ReLU activations in between, and finished with a Sigmoid activation.
For fMRIsignals, we use (2048, 512) as the hidden dimension; for CLIP embeddings (setting (2) and (3)), weuse (384, 256) as hidden dimensions; and for the ResNet embedding, we use (512, 256) as hiddendimensions.
The final output covers 171 classes, including 80 things categories (bounded objects, like“person”, “car”), and 91 stuff categories (mostly unbounded objects, like “tree”, “snow”).
2 Binarycross-entropy loss is used for each class to predict its existence in the input image.
Fig 12 shows the category-wise AUC-ROC.
The result demonstrates that CLIP embeddings containthe most object-level information about the image out of all the input sources.
Following it, fMRIsignals are also surprisingly very predictive, considering they carry a lot of noise.
The performancediscrepancy between settings (2) and (3) is minimal, meaning mapping fMRI signals into the CLIPspace retains most of the fMRI signals’ information: this provides strong support for the validity ofour design.
Lastly, ResNet embeddings perform poorly compared with other input sources.
Therefore,even with a perfect mapping model, projecting fMRI signals into this space will lose informationabout the image since the expressiveness of the embedding is bounded by the lower performer.
Inaddition, we note that both CLIP embeddings and fMRI have poorer performance on stuff categoriesthan on things categories, whereas ResNet embeddings do not.
This can indicate brain signals alignbetter with the multimodal CLIP space than with single-modality ResNet space.
Previous brainsignal decoding work utilizing pre-trained generators all relied on image-only embedding spaces(ResNet-50 [22], VGG19 [33]), and we believe moving to a multimodal latent space is a crucial steptowards better brain signal decodings.
[Figure]
Our pipeline relies on two pre-trained components.
The first and the most crucial one is the CLIPencoder that provides the latent space where we project fMRI signals.
The second is a conditional
GAN (Lafite) that generates images, which could be swapped for other generators.
In what follows,we will discuss these two components separately.
CLIP One exciting aspect of CLIP is the size of its training dataset, which consists of 30 million
Flicker images that should cover most of the natural image statistics.
This coverage is also proved bysubsequent works that generate images guided by CLIP embeddings through their abilities to performgenerations in various styles.
In addition, as we observed in A.7, CLIP embeddings can retain around98% of object-level information in fMRI with a very well-aligned performance across categories.
Albeit its incredible expressive power, CLIP does have a much lower dimensionality than the originalsignal: no matter how faithful, it is a compression.
By the nature of compression, CLIP only retainsthe most crucial information and removes most of the redundancies in the original signal.
Indeed, ifwe mask fMRI at different ratios, from 0 to 1, and perform the multi-label classification (the sametask as in A.7) using (1) masked fMRI or (2) CLIP mapped from masked-fMRI, we will notice avery drastic difference in the performance drop rate.
As shown in fig.
14, prediction performancefrom fMRI only drops drastically after the masking ratio becomes larger than 0.9, indicating brainredundancies to represent the objects.
In contrast, if we map the masked fMRI into the CLIP spaceand use these embeddings for prediction, the performance drop is almost at a constant rate.
Thisdiscrepancy makes the CLIP space more vulnerable to adversarial attacks than the fMRI space sincea small change would cause the generated images to derail from the ground truth.
In addition, CLIPembeddings also carry more biases than fMRI, as its mean AUC-ROC is much larger even withall-masked inputs.
One should consider these traits of CLIP embeddings when applying this systemand design defense mechanisms accordingly.
Lafite As for the generator, we utilize a conditional GAN pre-trained on the MS-COCO dataset(containing 328K images), from which NSD drew its experiment images.
This naturally provides analignment in the data distribution.
Although MS-COCO images are about everyday objects, humans,and scenes, the data statistics could vary when we move to other settings.
Therefore, future studies areneeded to extend current generators to one trained on broader sources (e.g., DALL-E 2, mentioned insection 4, used 650M images sampled from CLIP and DALL-E training data).
This should minimizethe dataset biases, although one should not interpret results without considering the training/testingdiscrepancies.
To show that our concept works across different generators, but dataset biases indeed play an importantrole, we test our pipeline with a Lafite pre-trained on the Google Conceptual Captions 3M dataset(CC3M, consisting of 3.3 million images) as the generator without any extra finetuning .
We used ourtrained f mc as the mapping function.
The results are shown in fig.
All generated images have thewatermark where CC3M sources its images.
In addition, when trying to generate out-of-distributionimages, the quality decreases in terms of photo-realism.
Nonetheless, semantic alignments are stillshown in these reconstructions.
We also want to note that pre-trained models provide excellent basesfor finetuning.
For example, Lafite finetuned its COCO model on the CC3M model within threehours, compared to four days to reach the same performance if training from scratch.
Therefore, ifthe pipeline is known to be used on certain types of images, a small-scale dataset and some lighttraining should greatly help the model to fit into the desired data distribution.
A.9 More examples
As mentioned in section 2.2, we found that the generated results conditioned on embeddings of different modalities tend to emphasize different aspects: more visual (colors, shape, etc.)
if conditionedonly on h ′ img , and more semantic if conditioned only on h ′ txt .
This could reflect the slight differencebetween the latent space of the two modalities.
We show the examples conditioned on either one ofthese two conditions, or both, in fig.
The pipeline tends to fail under the following conditions: (1) the image only contains close-updetails without too much semantic information; (2) the presented scene is novel (in the trainingdataset).
The model also tends to: (3) generate based on data biases: adding windows to indoorscenes, adding people to food scenes, generating colored images when the inputs are black-and-white,etc.; (4) change or ignore the background; (5) Mix-up colors (assigning colors in the scene to awrong object); (6) generate the wrong number of objects/people.
We showcase these failures togetherwith more other generated images in fig.
Given that the model is confident (in terms of GAN’sdiscriminator output staying at the same level) when generating results based on training data biases,future extensions should focus on exploring generators pre-trained on a much larger dataset, asdiscussed in A.8.
[Figure]