This paper tackles the problem of learning brain-visual representations for understanding and neural processes behind human visual perception , with a view towards replicating these processes into machines. The core idea is to learn plausible representations through the combined use of human neural activity evoked by natural imagesas a supervision mechanism for deep learning models. To accomplish this, we propose a multimodal approach that uses two different deep encoders, one for images and one for EEGs, trained in a siamese configuration  for learning a joint manifold that maximizes a compatibility measure between visual features and brain representation. The learned manifold is then used to perform image classification and saliency detection as well as to shed light on the possible representations generated by the human brain when perceiving the visual world. Performance analysis shows that neural signals can be used to effectively supervise the training of deep learning models, as demonstrated by the achieved performance in both image classification and saliency detection . Furthermore, the learned brain-visual manifold is consistent with cognitive neuroscience literature about visual perception and , most importantly, highlights new associations between brain areas, image patches and computational kernels. In particular, we are able to approximate brain responses to visual stimuli by training an artificial model with image features correlated to neural activity .
This work presents a novel method of exploring human brain-visual representations , with a view towards replicating these processes in machines. The core idea is to learn plausible computational and biological representations by correlating human neural activity and natural images. Thus, we first propose a model, EEG-ChannelNet, to learn a brain manifold for EEG classification. After verifying that visual information can be extracted from EEG data, we introduce a multimodal approach that uses deep image and EEG encoders, trained in a siamese configuration , for learning a joint manifold that maximizes a compatibility measure between visual features and brain representations. We then carry out image classification and saliency detection on the learned manifold. Performance analyses show that our approach satisfactorily decodes visual information from neural signals. This, in turn, can be used to effectively supervise the training of deep learning models, as demonstrated by the high performance of image classification and saliency detection on out-of-training classes. The obtained results show that the learned brain-visual features lead to improved performance and simultaneously bring deep models more in line with cognitive neuroscience work related to visual perception and attention .
R
paper tackles the problem of learning
work presents a novel method of exploring human
D
for understanding and neural processes behind human visual perception
R
into
in
R
representations through the combined use of
computational and biological representations by correlating
R
evoked by natural imagesas a supervision mechanism for deep learning models. To accomplish this, we propose
and natural images. Thus, we first propose a model, EEG-ChannelNet, to learn a brain manifold for EEG classification. After verifying that visual information can be extracted from EEG data, we introduce
R
two different deep encoders, one for images and one for EEGs,
deep image and EEG encoders,
A
,
R
representation. The learned manifold is then used to perform
representations. We then carry out
R
as well as to shed light on the possible representations generated by the human brain when perceiving the visual world. Performance analysis shows that neural signals
on the learned manifold. Performance analyses show that our approach satisfactorily decodes visual information from neural signals. This, in turn,
R
achieved performance in both
high performance of
R
. Furthermore,
on out-of-training classes. The obtained results show that
R
manifold is consistent
features lead to improved performance and simultaneously bring deep models more in line
R
literature about
work related to
R
, most importantly, highlights new associations between brain areas, image patches and computational kernels. In particular, we are able to approximate brain responses to visual stimuli by training an artificial model with image features correlated to neural activity
attention