Original: F OREGROUND - ATTENTION IN NEURAL DECODING : G UIDING L OOP -E NC -D EC TO RECONSTRUCT VISUAL STIMULUS IMAGES FROM F MRI
Revised: F OREGROUND - ATTENTION IN NEURAL DECODING : G UIDING L OOP -E NC -D EC TO RECONSTRUCT VISUAL STIMULUS IMAGES FROM F MRI

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The reconstruction of visual stimulus images from functional Magnetic Resonance
Revised: The reconstruction of visual stimulus images from functional Magnetic Resonance

Original: Imaging (fMRI) has received extensive attention in recent years, which provides a possibility to interpret the human brain.
Revised: Imaging (fMRI) has received extensive attention in recent years, which provides a possibility to interpret the human brain.

Original: Due to the high-dimensional and highnoise characteristics of fMRI data, how to extract stable, reliable and useful information from fMRI data for image reconstruction has become a challenging problem.
Revised: Due to the high-dimensional and highnoise characteristics of fMRI data, how to extract stable, reliable and useful information from fMRI data for image reconstruction has become a challenging problem.

Original: Inspired by the mechanism of human visual attention, in this paper, we propose a novel method of reconstructing visual stimulus images, which ﬁrst decodes the distribution of visual attention from fMRI, and then reconstructs the visual images guided by visual attention.
Revised: Inspired by the mechanism of human visual attention, in this paper, we propose a novel method of reconstructing visual stimulus images, which ﬁrst decodes the distribution of visual attention from fMRI, and then reconstructs the visual images guided by visual attention.

Original: We deﬁne visual attention as foreground attention (F-attention).
Revised: We deﬁne visual attention as foreground attention (F-attention).

Original: Because the human brain is strongly wound into sulci and gyri, some spatially adjacent voxels are not connected in practice.
Revised: Because the human brain is strongly wound into sulci and gyri, some spatially adjacent voxels are not connected in practice.

Original: Therefore, it is necessary to consider the global information when decoding fMRI, so we introduce the self-attention module for capturing global information into the process of decoding F-attention.
Revised: Therefore, it is necessary to consider the global information when decoding fMRI, so we introduce the self-attention module for capturing global information into the process of decoding F-attention.

Original: In addition, in order to obtain more loss constraints in the training process of encoder-decoder, we also propose a new training strategy called Loop-Enc-Dec.
Revised: In addition, in order to obtain more loss constraints in the training process of encoder-decoder, we also propose a new training strategy called Loop-Enc-Dec.

Original: The experimental results show that the F-attention decoder decodes the visual attention from fMRI successfully, and the Loop-Enc-Dec guided by F-attention can also well reconstruct the visual stimulus images.
Revised: The experimental results show that the F-attention decoder decodes the visual attention from fMRI successfully, and the Loop-Enc-Dec guided by F-attention can also well reconstruct the visual stimulus images.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, reconstructing visual stimulus images from fMRI has gradually gained attention, which provides the possibility of ”mind reading” in the future (Fig. 1).
Revised: In recent years, reconstructing visual stimulus images from fMRI has gradually gained attention, which provides the possibility of ”mind reading” in the future (Fig. 1).

Original: Existing work has shown that there is a certain mapping relationship between visual stimuli and brain activity (Poldrack & Farah, 2015), which provides us with the possibility and basis for reconstructing visual stimuli from fMRI data.
Revised: Existing work has shown that there is a certain mapping relationship between visual stimuli and brain activity (Poldrack & Farah, 2015), which provides us with the possibility and basis for reconstructing visual stimuli from fMRI data.

Original: The fMRI data which is collected from the human brain records the variations in blood oxygen level dependent (BOLD) and reﬂects the activity of nerves in human brain.
Revised: The fMRI data which is collected from the human brain records the variations in blood oxygen level dependent (BOLD) and reﬂects the activity of nerves in human brain.

Original: Through the analysis of fMRI data, the correlation between brain activity and visual tasks can be explored, helping us to understand human visual mechanism better.
Revised: Through the analysis of fMRI data, the correlation between brain activity and visual tasks can be explored, helping us to understand human visual mechanism better.

Original: Prior studies.
Revised: Prior studies.

Original: In recent years, there has been a lot of research in this ﬁeld.
Revised: In recent years, there has been a lot of research in this ﬁeld.

Original: The main methods can be roughly divided into the following two categories: Linear regression models , which encode fMRI data into image pixel values, and ﬁnally achieve image reconstruction.
Revised: The main methods can be roughly divided into the following two categories: Linear regression models , which encode fMRI data into image pixel values, and ﬁnally achieve image reconstruction.

Original: Deep learing models , such as DCNN and GAN model frameworks.
Revised: Deep learning models , such as DCNN and GAN model frameworks.

Original: For the ﬁrst type of methods, Yoichi Miyawaki et al. applied a multi-scale linear weighting model to predict the pixel value of each image and obtain the reconstruction results of the black and white simple images (Miyawaki et al., 2008).
Revised: For the ﬁrst type of methods, Yoichi Miyawaki et al. applied a multi-scale linear weighting model to predict the pixel value of each image and obtain the reconstruction results of the black and white simple images (Miyawaki et al., 2008).

Original: Fujiwara et al.
Revised: Fujiwara et al.

Original: proposed to use Bayesian canonical correlation analysis (BCCA) to build a reconstruction model, which extracts image information from measured data and reconstruct images (Fujiwara et al., 2013).
Revised: proposed to use Bayesian canonical correlation analysis (BCCA) to build a reconstruction model, which extracts image information from measured data and reconstruct images (Fujiwara et al., 2013).

Original: Marcel AJ van Gerve et al. adopted a hierarchical generative model composed of Boltzmann machines with restricted conditions, which explores the reconstruction of feature hierarchies based on learning (Van Gerven et al., 2010).
Revised: Marcel AJ van Gerve et al. adopted a hierarchical generative model composed of Boltzmann machines with restricted conditions, which explores the reconstruction of feature hierarchies based on learning (Van Gerven et al., 2010).

Original: Yu et al.
Revised: Yu et al.

Original: advanced a correlation network framework that could be ﬂexibly combined with diverse pattern representation models.
Revised: advanced a correlation network framework that could be ﬂexibly combined with diverse pattern representation models.

Original: They revealed the effective connectivity of human brain and reconstructed the visual stimulus images (Yu & Zheng, 2017; Yu et al., 2018).
Revised: They revealed the effective connectivity of human brain and reconstructed the visual stimulus images (Yu & Zheng, 2017; Yu et al., 2018).

Original: For the second type of methods, Guohua Shen et al. proposed an end-to-end deep convolution model to reconstruct visual stimulus images directly by using fMRI data, and at the same time they used
Revised: For the second type of methods, Guohua Shen et al. proposed an end-to-end deep convolution model to reconstruct visual stimulus images directly by using fMRI data, and at the same time they used

Original: GAN framework to optimize the training of the model (Shen et al., 2019a).
Revised: GAN framework to optimize the training of the model (Shen et al., 2019a).

Original: In response to the current problem of the lack of data sets used to reconstruct stimulus images from fMRI data, Roman Beliy et al.
Revised: In response to the current problem of the lack of data sets used to reconstruct stimulus images from fMRI data, Roman Beliy et al.

Original: suggest an approach for self-supervised training on unlabeled fMRI data (Beliy et al., 2019).
Revised: suggest an approach for self-supervised training on unlabeled fMRI data (Beliy et al., 2019).

Original: Yunfeng Lin et al.
Revised: Yunfeng Lin et al.

Original: proposed a model called DCNN-GAN by combining a reconstruction network and a GAN module.
Revised: proposed a model called DCNN-GAN by combining a reconstruction network and a GAN module.

Original: They utilized the CNN for hierarchical feature extraction and the DCNN-GAN to reconstruct more realistic images (Lin et al., 2019).
Revised: They utilized the CNN for hierarchical feature extraction and the DCNN-GAN to reconstruct more realistic images (Lin et al., 2019).

Original: Tao Fang et al.
Revised: Tao Fang et al.

Original: deﬁned a Shape-Semantic
Revised: deﬁned a Shape-Semantic

Original: GAN, considering the functional differences of different visual cortex areas, and reconstruct visual stimulus images under the guidance of shape and semantic information (Fang et al., 2020).
Revised: GAN, considering the functional differences of different visual cortex areas, and reconstruct visual stimulus images under the guidance of shape and semantic information (Fang et al., 2020).

Original: The ideas of DCNN and GAN have also been applied to various other model frameworks to achieve the reconstruction of visual stimulus images (Seeliger et al., 2018; VanRullen & Reddy, 2019; Zhang et al., 2020; Qiao et al.,  2020).
Revised: The ideas of DCNN and GAN have also been applied to various other model frameworks to achieve the reconstruction of visual stimulus images (Seeliger et al., 2018; VanRullen & Reddy, 2019; Zhang et al., 2020; Qiao et al., 2020; Mozafari et al., 2020).

Original: Our contributions.
Revised: Our contributions.

Original: Because of the complex neural information in the visual cortex, how to extract valuable information from high-dimensional fMRI data has been a challenging problem.
Revised: Because of the complex neural information in the visual cortex, how to extract valuable information from high-dimensional fMRI data has been a challenging problem.

Original: Inspired by human visual attention mechanism, we introduce visual attention to this work for the ﬁrst time.
Revised: Inspired by human visual attention mechanism, we introduce visual attention to this work for the ﬁrst time.

Original: We ﬁrst decode the human visual attention from a small amount of fMRI data successfully, and then use visual attention to guide the work of reconstructing visual stimulus images from fMRI.
Revised: We ﬁrst decode the human visual attention from a small amount of fMRI data successfully, and then use visual attention to guide the work of reconstructing visual stimulus images from fMRI.

Original: Existing studies have shown that the human visual system is more inclined to focus on prominent objects, and the neural representation of these prominent objects in the brain is more obvious (Ungerleider & G, 2000; Braun et al., 2001).
Revised: Existing studies have shown that the human visual system is more inclined to focus on prominent objects, and the neural representation of these prominent objects in the brain is more obvious (Ungerleider & G, 2000; Braun et al., 2001).

Original: The distribution of attention leads to an information bottleneck, only the most prominent objects are allowed to appear in the inferior temporal cortex, especially the ventral visual stream that encodes the identity of the object.
Revised: The distribution of attention leads to an information bottleneck, only the most prominent objects are allowed to appear in the inferior temporal cortex, especially the ventral visual stream that encodes the identity of the object.

Original: The visual attention mechanism is crucial for simulating the neural response of the higher visual system (Poggio & Anselmi, 2016; Khosla et al., 2020).
Revised: The visual attention mechanism is crucial for simulating the neural response of the higher visual system (Poggio & Anselmi, 2016; Khosla et al., 2020).

Original: Based on the above conclusions, we propose Foreground-attention (F-attention).
Revised: Based on the above conclusions, we propose Foreground-attention (F-attention).

Original: We abstract the distribution of human visual attention as the most prominent foreground object distribution area in the images.
Revised: We abstract the distribution of human visual attention as the most prominent foreground object distribution area in the images.

Original: Through the constructed F-attention decoder, the fMRI corresponding to the natural stimulus images can be decoded into visual attention distribution (Fig. 2a).
Revised: Through the constructed F-attention decoder, the fMRI corresponding to the natural stimulus images can be decoded into visual attention distribution (Fig. 2a).

Original: Because human brain is strongly wound into sulci and gyri, some spatially adjacent voxels are not directly connected in the human brain (Tallinen et al., 2014), the correlation between voxels is not only determined by location.
Revised: Because human brain is strongly wound into sulci and gyri, some spatially adjacent voxels are not directly connected in the human brain (Tallinen et al., 2014), the correlation between voxels is not only determined by location.

Original: Therefore, we need to consider global information when decoding fMRI, so we ﬁrst introduce the self-attention module (Vaswani et al., 2017; Wang et al., 2018;
Revised: Therefore, we need to consider global information when decoding fMRI, so we ﬁrst introduce the self-attention module (Vaswani et al., 2017; Wang et al., 2018;

Original: Zhang et al., 2019) into the fMRI decoding process, the self-attention module can capture global information of the input data and expand the receptive ﬁeld (Fig. 3c).
Revised: Zhang et al., 2019) into the fMRI decoding process, the self-attention module can capture global information of the input data and expand the receptive ﬁeld (Fig. 3c).

Original: At the same time, we propose a new encoding and decoding framework called Loop-Enc-Dec (Fig.
Revised: At the same time, we propose a new encoding and decoding framework called Loop-Enc-Dec (Fig.

Original: 2b,c).
Revised: 2b,c).

Original: The ﬁrst step we pre-train the fMRI encoder on the data set to realize the encoding process from images to fMRI.
Revised: The ﬁrst step we pre-train the fMRI encoder on the data set to realize the encoding process from images to fMRI.

Original: The second step we train the end-to-end encoder-decoder model under the guidance of F-attention, and input the images decoded by the decoder into the fMRI encoder for encoding to add the re-encoding constraint in the loss function, that is called “Loop-Enc-Dec”.
Revised: The second step we train the end-to-end encoder-decoder model under the guidance of F-attention, and input the images decoded by the decoder into the fMRI encoder for encoding to add the re-encoding constraint in the loss function, that is called “Loop-Enc-Dec”.

Original: In the training process, natural pictures without fMRI data are also added for end-to-end self-enc-dec training, which increases the stability and generalization of the model.
Revised: In the training process, natural pictures without fMRI data are also added for end-to-end self-enc-dec training, which increases the stability and generalization of the model.

Original: After experimental evaluation, the performance of Loop-Enc-Dec under the guidance of F-attention is better than that of mainstream methods.
Revised: After experimental evaluation, the performance of Loop-Enc-Dec under the guidance of F-attention is better than that of mainstream methods.

Original: The main contributions of this article are as follows:
Revised: The main contributions of this article are as follows:

Original: • As far as we know, we ﬁrst introduced the visual attention mechanism to the work of reconstructing visual stimulus images from fMRI.
Revised: • As far as we know, we ﬁrst introduced the visual attention mechanism to the work of reconstructing visual stimulus images from fMRI.

Original: According to neuroscience research, we abstract visual attention as the attention distribution of foreground objects (F-attention), and build a visual attention decoder to decode the human visual attention distribution from fMRI successfully.
Revised: According to neuroscience research, we abstract visual attention as the attention distribution of foreground objects (F-attention), and build a visual attention decoder to decode the human visual attention distribution from fMRI successfully.

Original: • To our best knowledge, we ﬁrst introduced the self-attention module into the work of fMRI decoding to capture global information of fMRI data.
Revised: • To our best knowledge, we ﬁrst introduced the self-attention module into the work of fMRI decoding to capture global information of fMRI data.

Original: • We propose a new enc-dec framework called Loop-Enc-Dec, guided by F-attention, which adds more loss constraints to the training of the image reconstruction model.
Revised: • We propose a new enc-dec framework called Loop-Enc-Dec, guided by F-attention, which adds more loss constraints to the training of the image reconstruction model.

Original: Evaluating the quality of reconstructed images, our method outperforms previous works.
Revised: Evaluating the quality of reconstructed images, our method outperforms previous works.

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section, we ﬁrst introduce the data set used in the experiments,and thengive a clear deﬁnition of the problem to be solved in this paper. Next we introduce the details of F-attention and Loop-EncDec framework proposed in this article, as well as the step-by-step training strategy for the model.
Revised: In this section, we ﬁrst introduce the data set used in the experiments, then we introduce the details of F-attention and Loop-Enc-Dec framework proposed in this article, as well as the step-by-step training strategy for the model.

Original: Finally, our evaluation method is given.
Revised: Finally, our evaluation method is given.

Original: 2. F MRI DATA SET
Revised: 2. F MRI DATA SET

Original: We use a publicly available benchmark data sets (Horikawa & Kamitani, 2017), which is widely used in the work of visual stimulus image reconstruction.
Revised: We use a publicly available benchmark data sets (Horikawa & Kamitani, 2017), which is widely used in the work of visual stimulus image reconstruction.

Original: In the image presentation experiment, fMRI signals were measured while subjects viewed a sequence of object images from ImageNet (Deng et al., 2009).
Revised: In the image presentation experiment, fMRI signals were measured while subjects viewed a sequence of object images from ImageNet (Deng et al., 2009).

Original: The image presentation experiment consisted of two sessions: the training image session and the testing image session.
Revised: The image presentation experiment consisted of two sessions: the training image session and the testing image session.

Original: In the training image session, 1200 images fromobject categories (8 images from each category) were each presented once.
Revised: In the training image session, 1200 images fromobject categories (8 images from each category) were each presented once.

Original: In the testing image session, 50 images from 50 object categories (one image from each category) were each presented 35 times.
Revised: In the testing image session, 50 images from 50 object categories (one image from each category) were each presented 35 times.

Original: The data collectors performed their analysis for each combination of feature types/layers and brain regions of interest (ROIs; V1–V4, the lateral occipital complex (LOC), fusiform face area (FFA), parahippocampal place area (PPA), lower visual cortex (LVC; V1–V3), higher visual cortex (HVC; covering regions around LOC, FFA and PPA) and the entire visual cortex (VC; covering all of the visual subareas listed above), the voxel size is 3×3×3 mm 3 )
Revised: The data collectors performed their analysis for each combination of feature types/layers and brain regions of interest (ROIs; V1–V4, the lateral occipital complex (LOC), fusiform face area (FFA), parahippocampal place area (PPA), lower visual cortex (LVC; V1–V3), higher visual cortex (HVC; covering regions around LOC, FFA and PPA) and the entire visual cortex (VC; covering all of the visual subareas listed above), the voxel size is 3×3×3 mm 3 )

Original: (Horikawa & Kamitani, 2017).
Revised: (Horikawa & Kamitani, 2017)

Original: 2. P ROBLEM DEFINITION

Original: The problem we need to solve in this paper is to reconstruct the visual stimulus images from the neural representation fMRI of the human brain (Fig. 1).

Original: While the subjects are seeing the visual stimulus images, the fMRI data corresponding to the human brain is collected.

Original: After preprocessing, the fMRI is used as input to construct a model to reconstruct the visual stimulus images seen by the subjects.

Original: The visual stimulus image is deﬁned as y , and the corresponding fMRI is deﬁned as x , then the problem is simpliﬁed to construct a reconstruction model f ,

Original: [Equation]

Original: M stands for visual stimulus image data set, F stands for fMRI data set.

Original: 2. P ROPOSED MODEL
Revised: 2. P ROPOSED MODEL

Original: Summarizing the proposed model.
Revised: Summarizing the proposed model.

Original: We ﬁrst trian the F-attention decoder on fMRI data(Fig.
Revised: We ﬁrst trian the F-attention decoder on fMRI data(Fig.

Original: 2a), then pre-train the image encoder(Fig. 2b), next we ﬁx the weight of the encoder, train the decoder under the guidance of F-attention(Fig. 2c).
Revised: 2a), then pre-train the image encoder(Fig. 2b), next we ﬁx the weight of the encoder, train the decoder under the guidance of F-attention(Fig. 2c).

Original: When testing, we ﬁrst use fMRI as the input of
Revised: When testing, we ﬁrst use fMRI as the input of

Original: F-attention decoder to decode the distribution of F-attention, then input the fMRI into the image decoder to reconstruct the visual stimulus images guided by corresponding F-attention.
Revised: F-attention decoder to decode the distribution of F-attention, then input the fMRI into the image decoder to reconstruct the visual stimulus images guided by corresponding F-attention.

Original: 2.3. F- ATTENTION
Revised: 2.2. F- ATTENTION

Original: In order to get more useful information from fMRI data to guide the work of visual stimulus image reconstruction, we introduced the human visual attention mechanism into the work.
Revised: In order to get more useful information from fMRI data to guide the work of visual stimulus image reconstruction, we introduced the human visual attention mechanism into the work.

Original: We abstracted the human visual attention as foreground attention (F-attention) based on neuroscience research (Poggio & Anselmi, 2016;
Revised: We abstracted the human visual attention as foreground attention (F-attention) based on neuroscience research (Poggio & Anselmi, 2016;

Original: Khosla et al., 2020).
Revised: Khosla et al., 2020).

Original: By constructing a F-attention decoder, we decode human visual attention distribution from fMRI data.
Revised: By constructing a F-attention decoder, we decode human visual attention distribution from fMRI data.

Original: The detailed structure of F-attention decoder

Original: Visual attention labeling.
Revised: Visual attention labeling.

Original: For the purpose of obtaining the human visual attention distribution of the training set, we ﬁrst label the train data (Fig. 3b).
Revised: For the purpose of obtaining the human visual attention distribution of the training set, we ﬁrst label the train data (Fig. 3b).

Original: According to the deﬁnition of F-attention, we ﬁrst extract the prominent objects in the natural images, and then perform the binary value on the images.
Revised: According to the deﬁnition of F-attention, we ﬁrst extract the prominent objects in the natural images, and then perform the binary value on the images.

Original: Through the transformation operation, the main distribution areas of F-attention are obtained.
Revised: Through the transformation operation, the main distribution areas of F-attention are obtained.

Original: Then we assigned attention weights to different regions in the images (the bluer the color, the higher the attention weights, in the train set, the weights of the foreground object areas are 1, the background areas are 0), the ﬁnal F-attention maps are obtained as the human visual attention labels of the training set.
Revised: Then we assigned attention weights to different regions in the images (the bluer the color, the higher the attention weights, in the train set, the weights of the foreground object areas are 1, the background areas are 0), the ﬁnal F-attention maps are obtained as the human visual attention labels of the training set.

Original: F-attention decoder.
Revised: F-attention decoder.

Original: Then we construct F-attention-decoder to realize the decoding process from fMRI data to human visual attention(Fig. 3c).
Revised: Then we construct F-attention-decoder to realize the decoding process from fMRI data to human visual attention(Fig. 3c).

Original: First, the preprocessed fMRI data is integrated into a vector, then the vector is sequentially connected to a fully connection module, convolution and upsampling modules, a self-attention module, convolution and upsampling modules, and we ﬁnally get the F-attention map.
Revised: First, the preprocessed fMRI data is integrated into a vector, then the vector is sequentially connected to a fully connection module, convolution and upsampling modules, a self-attention module, convolution and upsampling modules, and we ﬁnally get the F-attention map.

Original: The self-attention module calculation formula is as follows:
Revised: The self-attention module calculation formula is as follows, d k is 1:

Original: [Equation]
Revised: [Equation]

Original: In the training process, we use the mean square error loss (MSE) to constrain.
Revised: In the training process, we use the mean square error loss (MSE) to constrain.

Original: After 400 rounds of iterations, the error decreases and stabilizes.
Revised: After 400 rounds of iterations, the error decreases and stabilizes.

Original: The loss function is as follows:
Revised: The loss function is as follows:

Original: [Equation]
Revised: [Equation]

Original: Where FA i represents the visual attention distribution corresponding to the images in the training set, and FA ∗ i stands for the visual attention distribution decoded by the F-attention decoder.
Revised: Where FA i represents the visual attention distribution corresponding to the images in the training set, and FA ∗ i stands for the visual attention distribution decoded by the F-attention decoder.

Original: 2.3. L OOP -E NC -D EC
Revised: 2.2. L OOP -E NC -D EC

Original: The Loop-Enc-Dec reconstruction model we proposed is divided into two steps for training, pretraining encoder (Fig. 2b) and training Loop-Enc-Decoder guided by F-attention (Fig. 2c).
Revised: The Loop-Enc-Dec reconstruction model we proposed is divided into two steps for training, pretraining encoder (Fig. 2b) and training Loop-Enc-Decoder guided by F-attention (Fig. 2c).

Original: Pre-train fMRI encoder.
Revised: Pre-train fMRI encoder.

Original: The fMRI encoder realizes the encoding process from natural images to fMRI.
Revised: The fMRI encoder realizes the encoding process from natural images to fMRI.

Original: The structure of the encoder is mainly composed of residual network, convolution and full connection.
Revised: The structure of the encoder is mainly composed of residual network, convolution and full connection.

Original: The front end of the encoder is used as ResNet50 network (He et al., 2016) pre-trained on the ImageNet dataset, this part can extract image features (Fig. 4).
Revised: The front end of the encoder is used as ResNet50 network (He et al., 2016) pre-trained on the ImageNet dataset, this part can extract image features (Fig. 4).

Original: The loss function is composed of the weighted sum of the mean square error (MSE) and the cosine similarity, as shown below: 
Revised: The loss function is composed of the weighted sum of the mean square error (MSE) and the cosine similarity, as shown below: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Where α and β are hyperparameters, x i is the original fMRI label, and x ∗ i is the encoding result.
Revised: Where α and β are hyperparameters, x i is the original fMRI label, and x ∗ i is the encoding result.

Original: In this paper, α is assigned a value of 0.85 and β is -0.15.
Revised: In this paper, α is assigned a value of 0.85 and β is -0.15.

Original: This loss function is also used in the training process of Loop-Enc-Dec.
Revised: This loss function is also used in the training process of Loop-Enc-Dec.

Original: Train Loop-Enc-Decoder guided by F-attention.
Revised: Train Loop-Enc-Decoder guided by F-attention.

Original: We ﬁx the weight of pre-trained fMRI encoder, then connect an image decoder to form an end-to-end enc-dec model.
Revised: We ﬁx the weight of pre-trained fMRI encoder, then connect an image decoder to form an end-to-end enc-dec model.

Original: The so-called ‘Loop’ idea is to re-input the reconstructed images into the encoder for cyclic encoding in the process of training
Revised: The so-called ‘Loop’ idea is to re-input the reconstructed images into the encoder for cyclic encoding in the process of training

Original: Enc-Dec, increasing the fMRI encoding loss constraint, so that the original end-to-end model is transformed into a loop structure.
Revised: Enc-Dec, increasing the fMRI encoding loss constraint, so that the original end-to-end model is transformed into a loop structure.

Original: The image decoder is composed of full connection, convolution and upsampling.
Revised: The image decoder is composed of full connection, convolution and upsampling.

Original: At the same time, F-attention is used as a weight to multiply the convolution feature to guide the training process of the model (Fig. 4).
Revised: At the same time, F-attention is used as a weight to multiply the convolution feature to guide the training process of the model (Fig. 4).

Original: The F-attention ( FA ) decoded by pre-training F-attention decoder and the convolution features ( ConvF ) are used as a Hadamard product, the output is weighted λ and added to the original input convolution feature weighted µ .
Revised: The F-attention ( FA ) decoded by pre-training F-attention decoder and the convolution features ( ConvF ) are used as a Hadamard product, the output is weighted λ and added to the original input convolution feature weighted µ .

Original: Then we complete the guidance of the attention weight, as shown in the following formula:
Revised: Then we complete the guidance of the attention weight, as shown in the following formula:

Original: [Equation]
Revised: [Equation]

Original: In the training process, λ is assigned a value of 0.7 and µ is 0.3.
Revised: In the training process, λ is assigned a value of 0.7 and µ is 0.3.

Original: We also add unlabeled ImageNet images for end-to-end unsupervised self-enc-dec to enhance the generalization ability of the model.
Revised: We also add unlabeled ImageNet images for end-to-end unsupervised self-enc-dec to enhance the generalization ability of the model.

Original: The loss function in the training process consists of three parts, as follows:
Revised: The loss function in the training process consists of three parts, as follows:

Original: [Equation]
Revised: [Equation]

Revised: Next, we will introduce the three components of Loss LoopEncDec in detail.

Revised: Loss

Revised: image .

Revised: During training,

Revised: as

Revised: shown in Fig.

Revised: 4, the image decoder has two

Revised: inputs

Revised: one is

Revised: the original

Revised: fMRI

Original: Details of image decoder training strategy and the loss function.
Revised: data, we input it into image decoder to generate image img 1 , and the other is

Original: During training, the image decoder has two inputs, one is the original fMRI data, and the other is the output of pre-trained fMRI encoder.
Revised: the output of pre-trained fMRI encoder(encoding the visual stimulation image), we input it into image decoder to decode image img 2 . Then as shown in Fig.

Original: These two inputs are decoded by the image decoder to generate two images, which we named img fMRI , img feature , then we loop img fMRI , img feature back into the fMRI encoder to encode two different results, which we names fea imgfMRI , fea imgfea .
Revised: 4, we use V GG 19 (Simonyan & Zisserman, 2014) as a feature extractor to extract the convolution features of visual stimulation images and reconstructed images ( img 1 , img 2 ), then calculate the loss between convolution features as shown in Eq. 6.

Original: And we named the image obtained by self-enc-dec of ImageNet data img imageNet .
Revised: [Equation] Where V represents the feature extractor composed of pre-trained V GG 19 .

Original: The three parts of the loss function are deﬁned as follows:
Revised: 2) Loss Loop . Then we loop img 1 , img 2 back into the fMRI encoder to encode two different features, which we names fea 1 , fea 2 . We use Loss fMRI to calculate the loss between encoded features ( fea 1 , fea 2 ) and raw fRMI data, as shown in Eq.

Original: [Equation]
Revised: [Equation]

Original: Where V represents the feature extractor composed of pre-trained V GG 19 (Simonyan & Zisserman, 2014), img is  the original stimulus image, and imgNet is theimageNet image  without fMRI tags.
Revised: Where x represents the raw fRMI. Loss fMRI is deﬁned in Eq. 3) Loss imageNet . In order to enhance the generalization and robustness of the model, we introduce the image data without fMRI label from imageNet in the training process.

Revised: Firstly, the image is input into the fMRI encoder for encoding, and then the encoding result is input into the image decoder for decoding to obtain the reconstructed image img 3 .

Revised: We also calculate the loss between V GGconvolution features.

Revised: [Equation]

Revised: Where Loss fMRI is Eq. 3, and imgNet is the imageNet image without fMRI labels.

Original: 2. E VALUATION METHOD
Revised: 2. E VALUATION METHOD

Original: This paper uses a quantitative method to evaluate the reconstructed visual stimulus images.
Revised: This paper uses a quantitative method to evaluate the reconstructed visual stimulus images.

Original: The evaluation method is called pairwise similarity comparison (Shen et al., 2019b).
Revised: The evaluation method is called pairwise similarity comparison (Shen et al., 2019b; Fang et al., 2020), we use 2-way method used in Fang et al.

Original: There are a total of 50 natural stimulus images in the test set.
Revised: There are a total of 50 natural stimulus images in the test set.

Original: For each reconstructed image, it is calculated SSIM (Wang et al., 2004) value separately from the 50 natural stimulus images.
Revised: For each reconstructed image, it is calculated SSIM (Wang et al., 2004) value separately from the 50 natural stimulus images.

Original: We deﬁne the SSIM between the reconstructed image and the corresponding original image as SSIM pair , and the SSIM with the remaining 49 as SSIM random , then SSIM pair and SSIM random compare the SSIM values in pairs, a total of 50×49=2450 comparisons, we count the correct of times ( SSIM pair is greater than SSIM random ), and calculate the correct rate(Shen et al., 2019b). 
Revised: We deﬁne the SSIM between the reconstructed image and the corresponding original image as SSIM pair , and the SSIM with the remaining 49 as SSIM random , then SSIM pair and SSIM random compare the SSIM values in pairs, a total of 50×49=2450 comparisons, we count the correct of times ( SSIM pair is greater than SSIM random ), and calculate the correct rate(Shen et al., 2019b). We will supplement the details of this part in the appendix.

Original:  R ESULTS
Revised:  R ESULTS

Original: In the following, we will present the F-attention decoding results, visual stimulus image reconstruction results, some ablation experiments involving the model, anddescribe some details of the training process, ﬁnally verify some conclusions in neuroscience.
Revised: In the following, we will present the F-attention decoding results, visual stimulus image reconstruction results, some ablation experiments involving the model, and verify some conclusions in neuroscience.

Original: All our experiments are performed on the data set Horikawa

Revised: All our experiments are performed on the data set Horikawa

Original: (Horikawa & Kamitani, 2017).
Revised: (Horikawa & Kamitani, 2017).

Original: There are a total of 1200 images in the training set and a total of 50 images in the test set.
Revised: There are a total of 1200 images in the training set and a total of 50 images in the test set.

Original: There is no overlap in image categories between the training set and the test set.
Revised: There is no overlap in image categories between the training set and the test set.

Original: 3. F- ATTENTION DECODEING RESULTS
Revised: 3. F- ATTENTION DECODEING RESULTS

Original: We use entire visual cortex (VC) of fMRI data labeled with visual attention distribution for training.
Revised: We use entire visual cortex (VC) of fMRI data labeled with visual attention distribution for training.

Original: A total of 280 rounds of experiments are performed.
Revised: A total of 280 rounds of experiments are performed.

Original: The learning rate is set by using the SGDR method (Loshchilov & Hutter, 2016).
Revised: The learning rate is set by using the SGDR method (Loshchilov & Hutter, 2016).

Original: According to SGDR, we divide the 280 rounds of training into four small rounds.
Revised: According to SGDR, we divide the 280 rounds of training into four small rounds.

Original: The learning rate of each small round rises to the maximum value of 0.
Revised: The learning rate of each small round rises to the maximum value of 0.

Original: through a linear function, and then decays to the minimum value of 4e-5 through cosine annealing.
Revised: through a linear function, and then decays to the minimum value of 4e-5 through cosine annealing.

Original: Each small round repeats this operation to realize the periodic restart of the learning rate and improve the convergence speed and generalization of the model, the details are shown in the appendix.
Revised: Each small round repeats this operation to realize the periodic restart of the learning rate and improve the convergence speed and generalization of the model, the details are shown in the appendix.

Original: The results of decoding the F-attention distribution from fMRI are shown in Fig.
Revised: The results of decoding the F-attention distribution from fMRI are shown in Fig.

Original: 5, the more blue the area, the higher the weight of visual attention.
Revised: 5, the more blue the area, the higher the weight of visual attention.

Original: From the results, it can be seen that the F-attention decoder we design can decode the distribution of visual attention successfully, whether the object is an animal, a man-made object, or a human, such as the fourth image in the second row, the duck in the grass can be accurately found, the seventh image in the fourth row, visual attention distribution area of the person lying on the hammock can also be accurately found.
Revised: From the results, it can be seen that the F-attention decoder we design can decode the distribution of visual attention successfully, whether the object is an animal, a man-made object, or a human, such as the fourth image in the second row, the duck in the grass can be accurately found, the seventh image in the fourth row, visual attention distribution area of the person lying on the hammock can also be accurately found.

Original: 3. V ISUAL STIMULUS IMAGES RECONSTRUCTION RESULTS
Revised: 3. V ISUAL STIMULUS IMAGES RECONSTRUCTION RESULTS

Original: In this part, we present the reconstruction results of our method and compare the results with other existing works (Shen et al., 2019a; Beliy et al., 2019; Fang et al.,  2020), the comparison results are shown in Fig.
Revised: In this part, we present the reconstruction results of our method and compare the results with other existing works (Shen et al., 2019a; Beliy et al., 2019; Fang et al., 2020; Mozafari et al., 2020), the comparison results are shown in Fig.

Original: Through the evaluation method of pairwise similarity comparison mentioned above, we quantitatively evaluate the results of the experiment.
Revised: Through the evaluation method of pairwise similarity comparison mentioned above, we quantitatively evaluate the results of the experiment.

Original: It can be seen from the experimental results that our method can effectively reconstruct the images seen by humans from fMRI, and better decode the shape and direction of the prominent object.
Revised: It can be seen from the experimental results that our method can effectively reconstruct the images seen by humans from fMRI, and better decode the shape and direction of the prominent object.

Original: Compared with the previous work (reconstruction results of the visual stimulus images shown in Fang et al.
Revised: Compared with the previous work (reconstruction results of the visual stimulus images shown in Fang et al.

Original: , some improvements have been obtained.
Revised: , some improvements have been obtained.

Original: From a subjective perspective, our results can better pay attention to the distribution of foreground objects in the images.
Revised: From a subjective perspective, our results can better pay attention to the distribution of foreground objects in the images.

Original: For example, for the reconstruction of the ﬁrst goldﬁsh image, the position and color of the goldﬁsh in the image are more accurate.
Revised: For example, for the reconstruction of the ﬁrst goldﬁsh image, the position and color of the goldﬁsh in the image are more accurate.

Original: From a quantitative comparison point of view, our method experiment results achieve 68.9 % by pairwise SSIM comparison, slightly better than 68.4 % (Fang et al., 2020), and better than 66.8 %
Revised: From a quantitative comparison point of view, our method experiment results achieve 68.9 % by pairwise SSIM comparison, slightly better than 68.4 % (Beliy et al., 2019), and better than 66.8 %

Original: (Fang et al., 2020) and 63.8 % (Shen et al., 2019a).
Revised: (Fang et al., 2020), 63.8 % (Shen et al., 2019a) and 54.3 % (Mozafari et al., 2020), indicating the effectiveness of our method and foreground-attention can guide loop-Enc-Dec to complete the reconstruction of visual stimulation images successfully.

Original: 3. M ODEL ABLATION EXPERIMENTS
Revised: 3. M ODEL ABLATION EXPERIMENTS

Original: Here, we conduct ablation experiments to discuss the role of F-attention and Loop-Enc-Dec in reconstruction of visual stimulus images, the results are shown in Fig.
Revised: Here, we conduct ablation experiments to discuss the role of F-attention and Loop-Enc-Dec in reconstruction of visual stimulus images, the results are shown in Fig.

Original: First, we give the results obtained using the complete method of this paper (the second row of Fig. 7), the pairwise SSIM value is 68.9 % .
Revised: First, we give the results obtained using the complete method of this paper (the second row of Fig. 7), the pairwise SSIM value is 68.9 % .

Original: Then on the basis of the complete method, the F-attention guidance is removed from the training of the decoder, and only the Loop-Enc-Dec structure is used for step-by-step training.
Revised: Then on the basis of the complete method, the F-attention guidance is removed from the training of the decoder, and only the Loop-Enc-Dec structure is used for step-by-step training.

Original: First, the fMRI encoder is pre-trained, then the Loop-Enc-Dec is trained.
Revised: First, the fMRI encoder is pre-trained, then the Loop-Enc-Dec is trained.

Original: We ﬁnally use the image decoder to test and get the reconstruction results (the third row of Fig. 7), the pairwise SSIM value is 66.7 % .
Revised: We ﬁnally use the image decoder to test and get the reconstruction results (the third row of Fig. 7), the pairwise SSIM value is 66.7 % .

Original: Then we remove the Loss Loop on the basis of the complete method, just use the image loss of end-to-end encoding, and at the same time train the image decoder under the guidance of F-attention and obtain the reconstruction results (the fourth row of Fig. 7), The value of pairwise
Revised: Then we remove the Loss Loop on the basis of the complete method, just use the image loss of end-to-end encoding, and at the same time train the image decoder under the guidance of F-attention and obtain the reconstruction results (the fourth row of Fig. 7), The value of pairwise

Original: SSIM is 65.0 % .
Revised: SSIM is 65.0 % .

Original: Finally, on the basis of the complete method, we remove the F-attention guidance and the Loss Loop , and only use the end-to-end enc-dec for step-by-step training.
Revised: Finally, on the basis of the complete method, we remove the F-attention guidance and the Loss Loop , and only use the end-to-end enc-dec for step-by-step training.

Original: The image reconstruction results are shown in the ﬁfth row of Fig.
Revised: The image reconstruction results are shown in the ﬁfth row of Fig.

Original: 7, the pairwise SSIM value is 62.5 % .
Revised: 7, the pairwise SSIM value is 62.5 % .

Original: Comparing the ablation experimental results, it is not difﬁcult to see that our proposed F-attention and Loss Loop have played a positive role in the image reconstruction work.
Revised: Comparing the ablation experimental results, it is not difﬁcult to see that our proposed F-attention and Loss Loop have played a positive role in the image reconstruction work.

Original: Comparing the results in the second row with the results in the ﬁfth row, it can be seen that under the combined effect of
Revised: Comparing the results in the second row with the results in the ﬁfth row, it can be seen that under the combined effect of

Original: F-attention and Loss Loop , the image reconstruction quality has been greatly improved, including the shapes and colors of the prominent objects in the images, 68.9 % is better than 62.5 % in pairwise
Revised: F-attention and Loss Loop , the image reconstruction quality has been greatly improved, including the shapes and colors of the prominent objects in the images, 68.9 % is better than 62.5 % in pairwise

Original: SSIM quantiﬁcation.
Revised: SSIM quantiﬁcation.

Original: Comparing the ablation experimental results separately.
Revised: Comparing the ablation experimental results separately.

Original: Comparing the results of the second row and the third row, it can be seen that F-attention plays a certain guiding role in the model, making the reconstruction of foreground objects in the images more prominent, such as the goldﬁsh in the water in the ﬁrst column, and the people lying in a hammock in the second column, pairwise SSIM quantitative comparison shows that 68.9 % is better than 66.7 % .
Revised: Comparing the results of the second row and the third row, it can be seen that F-attention plays a certain guiding role in the model, making the reconstruction of foreground objects in the images more prominent, such as the goldﬁsh in the water in the ﬁrst column, and the people lying in a hammock in the second column, pairwise SSIM quantitative comparison shows that 68.9 % is better than 66.7 % .

Original: The guiding effect of F-attention can also be obtained by comparing the results of fourth rows and ﬁfth rows.
Revised: The guiding effect of F-attention can also be obtained by comparing the results of fourth rows and ﬁfth rows.

Original: The reconstruction results of foreground objects of fourth row are more prominent, and pairwise SSIM quantitative comparison shows that 65.0 % is better than 62.5 % .
Revised: The reconstruction results of foreground objects of fourth row are more prominent, and pairwise SSIM quantitative comparison shows that 65.0 % is better than 62.5 % .

Original: Comparing the results of the second row and the fourth row, it can be seen that the Loss Loop has made the details of the reconstruction images more clear and accurate.
Revised: Comparing the results of the second row and the fourth row, it can be seen that the Loss Loop has made the details of the reconstruction images more clear and accurate.

Original: For example, the face of the leopard in the ﬁfth column is clearer, and the shape and details of stone pan are more accurate in the seventh column.
Revised: For example, the face of the leopard in the ﬁfth column is clearer, and the shape and details of stone pan are more accurate in the seventh column.

Original: The pairwise SSIM quantitative comparison shows that 68.9 % is better than 65.0 % .
Revised: The pairwise SSIM quantitative comparison shows that 68.9 % is better than 65.0 % .

Original: The constraint effect of Loss Loop can also be obtained by comparing the results of the third row and the ﬁfth row.
Revised: The constraint effect of Loss Loop can also be obtained by comparing the results of the third row and the ﬁfth row.

Original: The details and quality of the image reconstruction in the third row are better.
Revised: The details and quality of the image reconstruction in the third row are better.

Original: The pairwise SSIM quantitative comparison shows that 66.7 % is better than 62.5 % .
Revised: The pairwise SSIM quantitative comparison shows that 66.7 % is better than 62.5 % .

Original: According to the above ablation experiments, we proved the guiding role of F-attention in image reconstruction and the constraint role of Loss Loop in the training process.
Revised: According to the above ablation experiments, we proved the guiding role of F-attention in image reconstruction and the constraint role of Loss Loop in the training process.

Original: 3. V ALIDATION OF NEUROSCIENCE RESEARCH
Revised: 3. V ALIDATION OF NEUROSCIENCE RESEARCH

Original: We verify some results of neuroscience research through experiments in the following.
Revised: We verify some results of neuroscience research through experiments in the following.

Original: It is generally believed that in the human visual cortex, the low-level visual (LVC) cortex often pays attention to the shape and contour information of objects, and has signiﬁcant directional sensitivity, while the high-level visual cortex (HVC) may correspond to the category information, movement information and other higher-level information of objects (Carandini et al., 2005; DiCarlo et al., 2012; Bracci et al., 2017; Zeman et al., 2020).
Revised: It is generally believed that in the human visual cortex, the low-level visual (LVC) cortex often pays attention to the shape and contour information of objects, and has signiﬁcant directional sensitivity, while the high-level visual cortex (HVC) may correspond to the category information, movement information and other higher-level information of objects (Carandini et al., 2005; DiCarlo et al., 2012; Bracci et al., 2017; Zeman et al., 2020).

Original: We reconstruct the visual stimulus images with the fMRI data of the low-level visual cortex and the high-level visual cortex respectively.
Revised: We reconstruct the visual stimulus images with the fMRI data of the low-level visual cortex and the high-level visual cortex respectively.

Original: The results are shown in the Fig.
Revised: The results are shown in the Fig.

Original: 8, the pairwise SSIM value of the images reconstructed from LVC is 64.7 % , and HVC is 55.7 % .
Revised: 8, the pairwise SSIM value of the images reconstructed from LVC is 64.7 % , and HVC is 55.7 % .

Original: By comparing the experimental results, we can ﬁnd that the LVC reconstruction results largely restore the shape and direction information of the foreground objects in the images, while the HVC reconstruction results lose the shape and direction information of the foreground objects.
Revised: By comparing the experimental results, we can ﬁnd that the LVC reconstruction results largely restore the shape and direction information of the foreground objects in the images, while the HVC reconstruction results lose the shape and direction information of the foreground objects.

Original: Quantitatively, the pairwise SSIM value of LVC is also better than HVC.
Revised: Quantitatively, the pairwise SSIM value of LVC is also better than HVC.

Original: This results verify that low-level visual cortex (LVC) pays more attention to the shape and contour information of objects than high-level visual cortex (HVC).
Revised: This results verify that low-level visual cortex (LVC) pays more attention to the shape and contour information of objects than high-level visual cortex (HVC).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, inspired by human visual attention, we deﬁne the Foreground-attention distribution of people when they view natural images, and successfully decode the visual attention distribution (Foreground-attention) from the fMRI data of human brain.
Revised: In this paper, inspired by human visual attention, we deﬁne the Foreground-attention distribution of people when they view natural images, and successfully decode the visual attention distribution (Foreground-attention) from the fMRI data of human brain.

Original: We achieve the decoding and interpretation of the visual attention of the human brain.
Revised: We achieve the decoding and interpretation of the visual attention of the human brain.

Original: The results show that areas with higher attention are concentrated on prominent objects in the images, such as animals and animal faces in the images, while lower attention is given to messy background information.
Revised: The results show that areas with higher attention are concentrated on prominent objects in the images, such as animals and animal faces in the images, while lower attention is given to messy background information.

Original: This result is in line with the research conclusions on human brain visual attention (Ungerleider & G, 2000; Braun et al., 2001;
Revised: This result is in line with the research conclusions on human brain visual attention (Ungerleider & G, 2000; Braun et al., 2001;

Original: Poggio & Anselmi, 2016; Khosla et al., 2020).
Revised: Poggio & Anselmi, 2016; Khosla et al., 2020).

Original: At the same time, we propose a new enc-dec training strategy called Loop-Enc-Dec, and apply the decoded F-attention to guide it, then reconstruct the visual stimulus images from the fMRI data successfully.
Revised: At the same time, we propose a new enc-dec training strategy called Loop-Enc-Dec, and apply the decoded F-attention to guide it, then reconstruct the visual stimulus images from the fMRI data successfully.

Original: Compared with the previous work, the reconstructed visual images in this paper are more realistic and get a higher score on pairwise
Revised: Compared with the previous work, the reconstructed visual images in this paper are more realistic and get a higher score on pairwise

Original: SSIM.
Revised: SSIM.

Original: From the perspective of neuroscience, we verify the following conclusions through experiments: The low-level visual cortex pays more attention to the shape of objects in the scene and has signiﬁcant directionality, while the high-level visual cortex is insensitive to shape and direction information.
Revised: From the perspective of neuroscience, we verify the following conclusions through experiments: The low-level visual cortex pays more attention to the shape of objects in the scene and has signiﬁcant directionality, while the high-level visual cortex is insensitive to shape and direction information.

Original: A  PPENDIX A. D ETAILS OF MODELS AND TRAINING
Revised: A A PPENDIX A. D ETAILS OF MODELS AND TRAINING

Original: In this paper, the details of the F-attention decoder we designed are as follows, the input fMRI data is a one-dimensional vector, and the fully connected layer is connected, the output size is 512*10*10, then reshape it into the feature of size (10, 10, 512).
Revised: In this paper, the details of the F-attention decoder we designed are as follows, the input fMRI data is a one-dimensional vector, and the fully connected layer is connected, the output size is 512*10*10, then reshape it into the feature of size (10, 10, 512).

Original: The convolutions has step size of 2 and a kernel size of 4.
Revised: The convolutions has step size of 2 and a kernel size of 4.

Original: The number of self-attention channels is 64.
Revised: The dimension of Query and Key is 68*68*8, and the dimension of Value is 68*68*64.

Original: In Loop-Enc-Dec, the front-end ResNet feature extractor in the fMRI decoder selects ‘conv3-block4-out’ as the feature output, the output feature size is 14*14*512, followed by two layers of residual networks, one layer of max-pooling and convolution (the size of the convolution kernel is 3, the step size is 1, padding is ’same’), and ﬁnally outputs the encoded results through a layer of fully connection.
Revised: In Loop-Enc-Dec, the front-end ResNet50 feature extractor in the fMRI decoder selects ‘conv3block4-out’ as the feature output, the output feature size is 14*14*512, followed by two layers of residual networks, one layer of max-pooling and convolution (the size of the convolution kernel is 3, the step size is 1, padding is ’same’), and ﬁnally outputs the encoded results through a layer of fully connection.

Original: In order to eliminate the checkerboard effect that appears in the reconstructed image, in image decoder, we set the size of the convolution kernel to be twice the convolution step length, and use linear interpolation for each upsampling to expand the feature to four times the original.
Revised: In order to eliminate the checkerboard effect that appears in the reconstructed image, in image decoder, we set the size of the convolution kernel to be twice the convolution step length, and use linear interpolation for each upsampling to expand the feature to four times the original.

Original: A. C HANGES IN LEARNING RATE USING SGDR
Revised: A. C HANGES IN LEARNING RATE USING SGDR

Original: The linear function and cosine annealing function used in each round are as follows:
Revised: The linear function and cosine annealing function used in each round are as follows:

Original: [Equation]
Revised: [Equation]

Original: E cur counts how many epochs have been performed since the last restart, E w represents the ‘warm’ epochs of each small round , E per represents the total epochs of each small round, l max represents the maximum learning rate, and l min represents the termination learning rate.
Revised: E cur counts how many epochs have been performed since the last restart, E w represents the ‘warm’ epochs of each small round , E per represents the total epochs of each small round, l max represents the maximum learning rate, and l min represents the termination learning rate.

Original: A total of 280 rounds of experiments were performed.
Revised: A total of 280 rounds of experiments were performed.

Original: The learning rate is set using the SGDR method (Loshchilov & Hutter, 2016).
Revised: The learning rate is set using the SGDR method (Loshchilov & Hutter, 2016).

Original: According to SGDR, we divide the 280 rounds of training into four small rounds.
Revised: According to SGDR, we divide the 280 rounds of training into four small rounds.

Original: The learning rate of each small round rises to the maximum value of 0.
Revised: The learning rate of each small round rises to the maximum value of 0.

Original: through a linear function, and then decays to the minimum value of 4e-5 through cosine annealing.
Revised: through a linear function, and then decays to the minimum value of 4e-5 through cosine annealing.

Original: Each small round repeats this operation to realize the periodic restart of the learning rate and improve the convergence speed and generalization of the model. 
Revised: Each small round repeats this operation to realize the periodic restart of the learning rate and improve the convergence speed and generalization of the model. [Table]

Revised: A. T HE DETAILS OF SSIM

Revised: SSIM is an evaluation index to measure the similarity of two images, the calculation formula is as follows:

Revised: [Equation]

Revised: Where x and y are two images, µ x is the mean of x , µ y is the mean of y , σ 2 x is the variance of x , σ 2 y is the variance of y , σ xy is the covariance of x and y , c

Revised: 1 = ( k 1 L ) 2 , c 2 = ( k 2 L ) 2 , is a constant used to maintain stability.

Revised: L is the dynamic range of pixel values.

Revised: k 1 =0.01, k 2 =0.03.

Revised: A. H OW TO DETERMINE THE VALUE OF λ AND µ IN E Q .

Revised: We tested the sequential combination parameters of µ = 0.1, 0.3, 0.5, 0.7, 0.9 and λ = 0.9, 0.7, 0.5, 0.3 and 0.1 respectively, and then selected the best combination, λ = 0.7 and µ = 0.3, as shown in

Revised: Table 1.

Revised: It may not be the best among all the values of λ and µ , but it is appropriate.

Revised: A. C HANGE OF LOSS FUNCTION DURING TRAINING

Revised: In the process of training, the loss function Eq.

Revised: 5 of each part can converge well.

Revised: As shown in Fig.

Revised: 10, ﬁgure a) shows the downward trend of total loss Loss Loop − Enc − Dec in the training process, ﬁgure b) shows the downward trend of Loss Loop , ﬁgure c) and d) show the change trend of Loss image , and ﬁgure e) shows the change trend of Loss imageNet .

Revised: The overall trend of error decreases and can be stabilized to a certain level.

Revised: The cyclical rise and fall is due to the using of SGDR learning rate change strategy.