Original:  U NSUPERVISED E XPECTATION L EARNING FOR M ULTISENSORY B INDING
Revised: C ONTINUOUS AND U NSUPERVISED E XPECTATION L EARNING FOR M ULTISENSORY B INDING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Expectation learning is a continuous  learning process which usesknown multisensory bindings to modulate unisensory perception.
Revised: Expectation learning is a continuous unsupervised learning process which uses multisensory bindings to modulate unisensory perception.

Original: When perceiving an event, we have an expectation on what we should see or hear which affects our unisensory perception. Expectation learning is known to enhance the unisensory perception of previously known multisensory events.
Revised: As humans, we learn to associate a barking sound with the visual appearance of a dog, and we continuously ﬁne-tune this association over time, as we learn, e.g., to associate highpitched barking with small dogs.

Original: In this work, we  present a novel hybrid deep recurrent model based on audio/visual autoencoders, for unimodal stimulus representation and reconstruction, and a recurrent self-organizing network for  multisensory binding of the representations. The model adapts concepts of expectation learning to enhance the unisensory representation based on the learned bindings.
Revised: In this work, we address the problem of building a computational model that captures two important properties of expectation learning, namely continuity and the lack of any external supervision other than temporal co-occurrence. To this end, we present a novel hybrid neural model based on audio/visual autoencoders and a recurrent self-organizing network for stimulus reconstruction and multisensory binding.

Original: We demonstrate that the proposed model is capable of reconstructing signals from one modality by processing input of another modality for 43, Youtube videos in the animal subset of the AudioSet corpus.
Revised: We demonstrate that the proposed model is capable of learning concept bindings, i e. dog barking with dogs, by evaluating it on unisensory classiﬁcation tasks for audi-visual stimuli using the 43,500 Youtube videos in the animal subset of the AudioSet corpus.

Original: Our experiments also show that when using expectation learning, the proposed model presents state-ofthe-art performance in representing and classifying unisensory stimuli.
Revised: In addition, our analysis and discussion explain how the expectation learning mechanism enforces the generation of high-level bindings and how they contribute to audiovisual recognition.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multisensory binding is one of the most important processes that humans use to understand their environment.
Revised: Multisensory binding is one of the most important processes that humans use to understand their environment.

Original: By using different sensory mechanisms, we are able to collect and process distinct information streams from the same experience which leads to a highly complex association learning.
Revised: By using different sensory mechanisms, we are able to collect and process distinct information streams from the same experience which leads to a highly complex association learning.

Original: This mechanism allows us to improve the perception of individual stimuli (Frassinetti et al.
Revised: This mechanism allows us to improve the perception of individual stimuli (Frassinetti et al.

Original: , solve contextual, spatial and temporal conﬂicts (Diaconescu et al.
Revised: , solve contextual, spatial and temporal conﬂicts (Diaconescu et al.

Original: (2011)), and progressively acquire and integrate novel information (Dorst & Cross (2001)).
Revised: (2011)), and progressively acquire and integrate novel information (Dorst & Cross (2001)).

Original: An important process of multisensory binding is known as the expectation effect (Yanagisawa  (2016)).
Revised: An important process of multisensory binding is known as the expectation effect (Yanagisawa (2016) (2016)).

Original: When perceiving an event, we compare it to other events we have experienced before, and make certain assumptions based on our experience.
Revised: When perceiving an event, we compare it to other events we have experienced before, and make certain assumptions based on our experience.

Original: For instance, when seeing a cat, we expect it to meow and not to bark.
Revised: For instance, when seeing a cat, we expect it to meow and not to bark.

Original: This effect modulates our multisensory association in terms of top-down expectation.
Revised: This effect modulates our multisensory association in terms of top-down expectation.

Original: When a cat barks to us, we assume that our perception is inconsistent. This means that either unisensory perception failed, the spatial or temporal attention was misleading, or we are learning a new species of a barking cat.
Revised: In consequence, when a cat barks to us, we assume that our perception is inconsistent, and assume that either unisensory perception failed, the spatial or temporal attention was misleading, or we create a new species of a barking cat.

Original: For each of these scenarios, our brain adapts to the situation and we update our multisensory knowledge.
Revised: For each of these scenarios, our brain adapts to the situation and we update our multisensory knowledge.

Original: This learning process referred to learning by expectation (Ashby & Vucovich (2016)), strongly suggests the role of unsupervised learning for multisensory binding, and leads to an adaptive mechanism for learning of novel concepts (Chen et al.
Revised: This learning process referred to learning by expectation (Ashby & Vucovich (2016)), strongly suggests the role of unsupervised learning for multisensory binding, and leads to an adaptive mechanism for learning of novel concepts (Ellingsen et al.

Revised: In this work, we propose a novel method that mimics the human ability to perform expectation learning and ask the following questions:

Revised: Q.1 How can we build a computational model that allows for continuous unsupervised learning of multisensory bindings?

Revised: Q.2 Can we adapt the expectation learning from humans to this model and use it to generate expected unisensory visual stimuli from auditory stimuli and vice versa?

Revised: Q.3 Can we exploit the generated expected stimuli to improve unisensory classiﬁcation?

Revised: In Section 2 we investigate several other multisensory integration approaches and identify a lack of methods that are both unsupervised and continuous.

Original: Most existing computational models for multisensory learning apply explicit weighted connections while integrating the sensor information which are learned using early (Wei et al.
Revised: To address Q.1, we employ autoencoders to learn auditory and visual representations, which allows for unsupervised learning. As a novelty and core mechanism to address continuity, we propose to link the autoencoders with a recurrent Grow-When-Required (GWR) neural network that changes its size as demanded, thus allowing for the continuous learning of multisensory binding (see Section 3). We address Q.2 by hypothesizing that the recurrent GWR network learns prototypes of multisensory bindings, which allows us to reconstruct auditory information from visual stimuli and vice-versa.

Revised: For example, when perceiving the sound of a cat, we expect the model to reconstruct the image of a cat, while when a dog enters a scene, the sound of the dog will be reconstructed.

Revised: This expectation learning mechanism is described in Section 4.

Revised: By extending the GWR association mechanism, we expect the model to be able to create concept-level bindings.

Revised: Speciﬁcally, we hypothesize that by activating the neural units that represent prototypical concepts such as cats, dogs, and horses, the model will reconstruct prototypical auditory and visual stimuli in the absence of any sensory input.

Revised: Our novel method is inspired by the multisensory imagery effect (Spence &

Revised: Deroy (2013)), i.e., the ability of humans to simplify absent stimuli into concepts, and to use the abstract concepts to reconstruct unisensory information to enhance the overall perception.

Revised: with Q. we ask whether this effect can be used to improve the classiﬁaction performance and hypothesize that our approach improves unisensory classiﬁcation by reconstructing unisensory stimuli based on multisensory bindings.

Revised: To the best of our knowledge, there exists no standard benchmark to evaluate audio-visual bindings.

Revised: Therefore, we propose a series of experiments to measure how far the expectation learning mechanism improves unisensory classiﬁcation.

Revised: To this end, we employ the Youtube AudioSet corpus (Gemmeke et al.

Revised: (2017)), which contains human-labeled samples of Youtube videos based on the audio information.

Revised: We select the animal subset of the corpus consisting of 44k samples to train the multisensory bindings in an unsupervised manner.

Revised: We then exploit the multisensory bindings by using them to train a classiﬁer for 24 different animal classes.

Revised: We then employ the classiﬁer to recognize absent stimuli, i.e., recognizing auditory stimuli when visual stimuli is present and viceversa.

Revised: In Section Section 5 we summarize our efforts to perform a fair and thorough evaluation of our model.

Revised: We perform the experiments in three steps and assess the contribution of each step of the expectation and binding mechanism to compare it with state-of-the-art solutions for unisensory classiﬁcation.

Revised: In Section Section 6 we summarize the results of our experiments.

Revised: We show that the expectation learning improved the recognition of unisensory stimuli and it presents competitive performance with state-of-the-art solutions.

Revised: To conﬁrm our hypotheses, we discuss the results in Section 7, providing evidence that correlates our network behavior with the multisensory imagery effect.

Revised: Also, we discuss the capabilities and limitations of our model.

Revised: We conclude in Section 8 that the expectation learning mechanism improves the quality of the multisensory association by providing a better unisensory classiﬁcation.

Revised:  R ELATED W ORK

Revised: Most existing computational models for multisensory learning apply explicit weighted connections while integrating the sensor information which are learned using early (Wei et al.

Original: or late (Liu et al.
Revised: or late (Liu et al.

Original: (2016); de Boer et al.
Revised: (2016); de Boer et al.

Original: (2016)) fusion techniques.
Revised: (2016)) fusion techniques.

Original: These weighted connections are usually tuned in a data-driven way whereby the data distribution  affects the multisensory binding directly.
Revised: These weighted connections are usually tuned in a data-driven manner, whereby the data distribution directly affects the multisensory binding.

Original: This makes these models suitable to be used for real-world and complex problems but they arevery sensitive to the training data used when performing the multisensory integration.
Revised: Such existing methods have the drawback that they require supervision and that they are sensitive to the training data distribution when performing the multisensory integration.

Original: More accurate models apply the neurophysiology ﬁndings on unisensory biasing for multisensory computational models (Pouget et al.
Revised: More accurate models apply the neurophysiological ﬁndings on unisensory biasing for multisensory computational models (Pouget et al.

Original: ; Rowland et al.
Revised: ; Rowland et al.

Original: ; Kayser & Shams (2015)).
Revised: ; Kayser & Shams (2015)).

Original: Such models, although similar to the brain’s neural behavior, are usually not feasible to be used on real-world data, as they are mostly applied to simple stimuli scenarios, and do not scale well.
Revised: Such models, although similar to the brain’s neural behavior, are usually not feasible to be used on real-world data, as they are mostly applied to simple stimuli scenarios, and do not scale well.

Original: There exist other complex models that implement attention mechanisms based on multisensory information, but the most recent focus in this area is on data-driven fusion models (Hori et al.
Revised: There exist other complex models that implement attention mechanisms based on multisensory information, but the most recent focus in this area is on data-driven fusion models (Hori et al.

Original: ; Barros et al.
Revised: ; Barros et al.

Original:  Mortimer & Elliott (2017)).
Revised: ; Mortimer & Elliott (2017)).

Original: The introduction of expectation learning would give these models the ability to adapt better to novel situations and learn with its own errors in a online and continuous way.
Revised: The introduction of expectation learning would give these models the ability to adapt better to novel situations and learn with its own errors in an online and continuous way.

Original: Recent contributions make use of data-driven learning for multisensory representations Arandjelovic & Zisserman (2017); Senocak et al.
Revised: Recent contributions make use of data-driven learning for multisensory representations Arandjelovic & Zisserman (2017); Senocak et al.

Original: ; Owens & Efros (2018);
Revised: ; Owens & Efros (2018);

Original: Kim et al.
Revised: Kim et al.

Original: Such solutions make use of different transfer learning and attention mechanisms to improve unisensory recognition and localization.
Revised: Such solutions make use of different transfer learning and attention mechanisms to improve unisensory recognition and localization.

Original: Although they present impressive result in this speciﬁc tasks, they still rely on strongly labelled data points or are not suitable for online learning given that they have a extensive training proceeding.
Revised: Although they present impressive result in this speciﬁc tasks, they still rely on strongly labelled data points or are not suitable for online learning given that they have a extensive training proceeding.

Original: In particular, the work by Arandjelovic and Zisserman (Arandjelovi´c & Zisserman (2017)) introduces a data-driven model for multisensory binding with bottom-up modulation for spatial attention.
Revised: In particular, the work by Arandjelovi ´ c & Zisserman (2017) introduces a data-driven model for multisensory binding with bottom-up modulation for spatial attention.

Original: Their model uses the network’s activity to spatially identify which part of an image a certain sound is related to.
Revised: Their model uses the network’s activity to spatially identify which part of an image a certain sound is related to.

Original: Although the model is data-driven, the authors claim that it learns real-world biasing on a multisensory description for unisensory retrieval by using a large amount of real-world training data.
Revised: Although the model is data-driven, the authors claim that it learns real-world biasing on a multisensory description for unisensory retrieval by using a large amount of real-world training data.

Original: Their results show that the model can also use a number of unisensory channels to compensate absent ones and identify congruent and incongruent stimuli.
Revised: Their results show that the model can also use a number of unisensory channels to compensate absent ones and identify congruent and incongruent stimuli.

Original: Overall, this approach has produced impressive results on spatial location and cross-sensory retrieval.
Revised: Overall, this approach has produced impressive results on spatial location and cross-sensory retrieval.

Original: A similar approach was presented by Zhou et al.
Revised: A similar approach was presented by Zhou et al.

Original: (Zhou et al.
Revised: (2017), but

Original: , but focused on audio generation.
Revised: focused on audio generation.

Original: Their model relies on a sequence-to-sequence generator to associate audio events to visual information.
Revised: Their model relies on a sequence-to-sequence generator to associate audio events to visual information.

Original: The same generator is use to generate audio for newly presented video scenes.
Revised: The same generator is use to generate audio for newly presented video scenes.

Original: This requires an external teacher to identify congruent and incongruent stimuli which makes it impossible to be used in online learning scenarios.
Revised: This requires an external teacher to identify congruent and incongruent stimuli which makes it impossible to be used in online learning scenarios.

Original: Also, all of them are deeply dependent on an end-to-end deep learning strategy.
Revised: Also, all of them are deeply dependent on an end-to-end deep learning strategy.

Original: Furthermore, it cannot learn novel information without the need of extensive retraining the entire model again.
Revised: Furthermore, it cannot learn novel information without the need of extensive retraining the entire model again.

Original: In this work, we propose a computational model based on expectation learning for multisensory binding.

Original: Our model implements a multichannel autoencoder to learn audio-visual representations.

Original: It also introduces the use of temporal binding to learn expectation by means of a recurrent GrowWhen-Required (GWR) neural network.

Original: The capability to grow and shrink when necessary allows us to model a continuous learning scenario of multisensory binding.

Original: As the recurrent GWR network learns prototypes of multisensory binding, we use such prototypes to reconstruct unisensory stimuli.

Original: This allows us to reconstruct auditory information from visual stimuli and vice-versa.

Original: Our hybrid approach allowed us to have a strong and robust encoding/decoding mechanism for realworld data based on our autoencoders and a temporal multisensory binding solution.

Original: The recurrent

Original: GWR performed particularly well on creating prototypical concepts which are important for our expectation learning mechanism, as we do not want to learn multisensory bindings of instances but of higher abstraction concepts.

Original: Our model is trained fully unsupervised, different from most of the current state-of-the-art solutions for multisensory binding.

Original: This allows us to learn concepts in an online manner, without the constraint of having a strong supervised training process.

Original: Our model develops the temporal aspects of multisensory binding to continuously reconstruct the perceived stimuli.

Original: When perceiving the sound of a cat, the model can reconstruct the image of a cat, while when a dog enters a scene, the sound of the dog will be reconstructed.

Original: Also, by clustering the multisensory bindings on the GWR, we can create expected stimuli based on the learned concepts.

Original: This means that our model is able to reconstruct auditory and visual stimuli from prototypical concepts such as cats, dogs, and horses in the absence of sensory input.

Original: We hypothesize that our model can use the reconstruction of multimodal stimuli based on multisensory binding to boost the representation of unisensory stimuli.

Original: To the best of our knowledge, there is no standard audio-visual binding association benchmark, and thus we evaluate our model using the animal sub-sets of the AudioSet corpus (Gemmeke et al.

Original: This corpus contains human-labeled samples of Youtube videos based on the audio information.

Original: We use a total of 24 different animal classes and more than 44k samples to train the model.

Original: To proper measure the models capability to learn different abstraction levels of multisensory binding, we make use of overlapping animal classes.

Original: We measure our models behavior in two steps: ﬁrst, we evaluate the performance of the model to recognize multimodal stimuli when one of the modalities is absent.

Original: Second, we analyze the learning behavior of the model and correlate it to multisensory imagery (Spence & Deroy (2013)) effect, where humans simplify absent stimuli into concepts to enhance perception.

Original:  MULTISENSORY T EMPORAL B INDING
Revised:  M ULTISENSORY T EMPORAL B INDING

Original: To reconstruct auditory and visual stimuli, we developed neural networks based on autoencoders for each of the unisensory channels.
Revised: To reconstruct auditory and visual stimuli, we developed neural networks based on autoencoders for each of the unisensory channels.

Original: These networks encode high-dimensional data into a latent representation and reconstruct real-world audio-visual information.
Revised: These networks encode high-dimensional data into a latent representation and reconstruct real-world audio-visual information.

Original: The binding between auditory and visual information is realized by means of a recurrent GWR network.
Revised: The binding between auditory and visual information is realized by means of a recurrent GWR network.

Original: The GWR is a self-organizing network that learns to create conceptual prototypes of a data distributions in an unsupervised, incremental learning manner.
Revised: The GWR is a self-organizing network that learns to create conceptual prototypes of a data distributions in an unsupervised, incremental learning manner.

Original: To address the temporal aspects of coincident binding, we extend the
Revised: To address the temporal aspects of coincident binding, we extend the

Original: Gamma-GWR (Parisi & Wermter (2017)) which endows prototype neurons with a number of temporal contexts to learn the spatiotemporal structure of the data distribution.
Revised: Gamma-GWR (Parisi & Wermter (2017)) which endows prototype neurons with a number of temporal contexts to learn the spatiotemporal structure of the data distribution.

Original: An overview of our model is illustrated in Figure
Revised: An overview of our model is illustrated in Figure

Original: 2. V ISUAL CHANNEL
Revised: 3. V ISUAL CHANNEL

Original: To process high-level information in the visual channel, we developed a variational autoencoder (Chen et al. Which, instead of using a standard bottleneck layer, performs sampling over
Revised: To process high-level information in the visual channel, we realized a variational autoencoder (VAE) (Kingma & Welling (2013)) which enforces the encoded latent variables to follow a Gaussian distribution.

Original: Gaussian unit reconstructions of individual nodes, which makes it robust to overﬁtting and which allows it to learn more general representations than a standard autoencoder.
Revised: The VAE produced better results when recognizing the animals from the AudioSet dataset when compared to normal deep autoencoders. We hypothesize that this improvement is due to the great variance on the image quality, perspective and resolution of the visual information of our dataset. Most likely the VAE learns to represent the important characteristics of the animals through the latent vector sampling, instead of learning to reconstruct the entire encoded image.

Original: To train the autoencoder, we implemented a composite loss function based on the  reconstruction error and the Kullback-Leibler (KL) divergence between the encoded representation and asample from the Gaussian distribution.
Revised: To train the VAE, we implemented a composite loss function based on the image reconstruction error and the Kullback-Leibler (KL) divergence between the encoded representation and a the Gaussian unit.

Original: This composite loss function is important to learn representations of general concepts, represented by Gaussian distributions, instead of reconstructing input images from memorized parameters.
Revised: This composite loss function is important to enforce that the encoded representations learn general concepts of the animals, instead of reconstructing input images from memorized parameters.

Original: Our model receives as input a 128x128 color image.
Revised: Our model receives as input a 128x128 color image.

Original: The input data is processed by our encoding architecture which is composed of a series of four convolution layers, with a stride of 2x2, and kernel sizes of dimension 3x3.
Revised: The input data is processed by our encoding architecture which is composed of a series of four convolution layers, with a stride of 2x2, and kernel sizes of dimension 3x3.

Original: The ﬁrst convolution layer has three channels and the subsequent three layers have 64 ﬁlters.
Revised: The ﬁrst convolution layer has three channels and the subsequent three layers have 64 ﬁlters.

Original: The latent representation starts with a fully connected layer withunits.
Revised: The latent representation starts with a fully connected layer withunits.

Original: We compute the standard deviation and mean of this layer’s output, generate a Gaussian distribution from it and sample an input for another fully connected hidden layer with 64 units, which is our ﬁnal latent representation.
Revised: We compute the standard deviation and mean of this layer’s output, generate a Gaussian distribution from it and sample an input for another fully connected hidden layer with 64 units, which is our ﬁnal latent representation.

Original: The decoding layer has the same structure as our encoding layer but in the opposite direction and applying transpose convolutions.
Revised: The decoding layer has the same structure as our encoding layer but in the opposite direction and applying transpose convolutions.

Original: 2. A UDITORY CHANNEL
Revised: 3. A UDITORY CHANNEL

Original: For the auditory channel, we implement a recurrent autoencoder based on Gated Recurrent Units (GRU) (Cho et al.
Revised: For the auditory channel, we implement a recurrent autoencoder based on Gated Recurrent Units (GRU) (Cho et al.

Original: Recurrent units allow us to reconstruct audio with better quality than using non-recurrent layers since auditory signals are sequential, and each audio frame depends highly on previous contextual information.
Revised: Recurrent units allow us to reconstruct audio with better quality than using non-recurrent layers since auditory signals are sequential, and each audio frame depends highly on previous contextual information.

Original: As input and output of the auditory autoencoder, we compute a Mel spectrum which we generate from the raw waveform.
Revised: As input and output of the auditory autoencoder, we compute a Mel spectrum which we generate from the raw waveform.

Original: To reconstruct the audio from the output Mel spectrum, we develop a Convolutional Bottleneck CBHG-layer (Lee et al.
Revised: To reconstruct the audio from the output Mel spectrum, we employ a convolutional bottleneck CBHG network model (Lee et al.

Original: (2017)) which consists of a 1-D convolutional bank, a highway network and a bi-directional GRU layer.
Revised: (2017)) which consists of a 1-D convolutional bank, a highway network and a bi-directional GRU layer.

Original: This network receives as input the Mel spectrum, and outputs a linear frequency spectrum which is then transformed into waveform using the Grifﬁn Lim algorithm.
Revised: This network receives as input the Mel spectrum, and outputs a linear frequency spectrum which is then transformed into waveform using the Grifﬁn Lim algorithm.

Original: This approach of transforming Mel coefﬁcients to  waveform achieved better audio synthesis quality than performing Grifﬁn Lim on the Mel spectrum directly (Wang et al.
Revised: This approach of transforming Mel coefﬁcients to a linear spectrum and then to waveform achieved better audio synthesis quality than performing Grifﬁn Lim on the Mel spectrum directly (Wang et al.

Original: (2017)), which is important for our expectation learning.
Revised: , and it improves the audio data of our expectation learning approach.

Original: We performed hyperparameter optimization for the autoencoder and found that an audio spectrum window length of 50ms, a window shift of 12.5 ms with 80 Mel coefﬁcients and 1000 linear frequencies yield best results.
Revised: We performed hyperparameter optimization for the autoencoder and found that an audio spectrum window length of 50ms, a window shift of 12.5 ms with 80 Mel coefﬁcients and 1000 linear frequencies yield best results.

Original: We also found that 80 units for the dense bottleneck layer and two GRU layers with 128 units each for both the encoder and decoder network are sufﬁcient for achieving a high audio quality.
Revised: We also found that 80 units for the dense bottleneck layer and two GRU layers with 128 units each for both the encoder and decoder network are sufﬁcient for achieving a high audio quality.

Original: An additional number of Mel coefﬁcients, GRU layers, and neural units did not signiﬁcantly improve the reconstruction quality.
Revised: An additional number of Mel coefﬁcients, GRU layers, and neural units did not signiﬁcantly improve the reconstruction quality.

Original: The number of bottleneck units is important for the multisensory binding as it determines the number of connections between the binding layer and the audio encoder and decoder.
Revised: The number of bottleneck units is important for the multisensory binding as it determines the number of connections between the binding layer and the audio encoder and decoder.

Original: 2. S ELF -O RGANIZING T EMPORAL B INDING
Revised: 3. S ELF -O RGANIZING T EMPORAL B INDING

Original: To learn coincident bindings between audio and visual stimuli, wemake useof an unsupervised binding layer.
Revised: To learn coincident bindings between audio and visual stimuli, we use an unsupervised binding layer.

Original: This layer is implemented as a recurrent GWR network which is an unsupervised model for learning spatiotemporal prototype representations.
Revised: This layer is implemented as a recurrent GWR network, which is an unsupervised model for learning spatiotemporal prototype representations.

Original: The layer receives as input the latent representation of our visual and auditory channels which occur coincidentally.
Revised: The layer receives as input the latent representation of our visual and auditory channels which occur coincidentally.

Original: To synchronize the two data streams, we make sure that the videos have a temporal resolution of 20 frames per second. As a result, eachframe of our video  is associated with 12.5 ms of auditory information.
Revised: To synchronize the two data streams, we resample video and audio streams to a temporal resolution of 20 frames per second, i.e., each video frame is associated with 12.5 ms of auditory information.

Original: In contrast to traditional self-organizing models with winner-take-all dynamics for the processing of spatial patterns, the Gamma-GWR (Parisi & Wermter (2017)) computes the winner neuron taking into account the activity of the network for the current input and a temporal context.
Revised: In contrast to traditional selforganizing models with winner-take-all dynamics for the processing of spatial patterns, the GammaGWR (Parisi & Wermter (2017)) computes the winner neuron taking into account the activity of the network for the current input and a temporal context.

Original: Each neuron of the map consists of a weight vector w j and a number K of context descriptors c kj (with w j , c kj ∈ R n ).
Revised: Each neuron of the map consists of a weight vector w j and a number K of context descriptors c kj (with w j , c kj ∈ R n ).

Original: As a result, recurrent neurons in the map will encode prototype sequence-selective snapshots of the input.
Revised: As a result, recurrent neurons in the map will encode prototype sequence-selective snapshots of the input.

Original: Given a set of N neurons, the best-matching unit (BMU),w b , with respect to the input x ( t ) ∈ R n is computed as:
Revised: Given a set of N neurons, the best-matching unit (BMU), b , with respect to the input x ( t ) ∈ R n is computed as:

Original: [Equation]
Revised: [Equation]

Original: where α i and β ∈ (0; 1) are constant values that modulate the inﬂuence of the current input with respect to previous neural activity, w I  − 1 is the weight of the winner neuron at t − 1 , and C k ∈ R n is the global context of the network ( C k ( t 0 ) = 0 ).
Revised: where α i and β ∈ (0; 1) are constant values that modulate the inﬂuence of the current input with respect to previous neural activity, w I ( t − 1) is the weight of the winner neuron at t − 1 , and C k ∈ R n is the global context of the network ( C k ( t 0 ) = 0 ).

Original: New connections are created between the BMU and the second-BMU of an input.
Revised: New connections are created between the BMU and the second-BMU of an input.

Original: When a BMU is computed, all the neurons the BMU is connected to are referred to as its topological neighbors.
Revised: When a BMU is computed, all the neurons the BMU is connected to are referred to as its topological neighbors.

Original: Given the current input x ( t ) , the activity of the network a ( t ) is deﬁned by the activity of the BMU as a ( t ) =

Original: exp( d b ) where d b is deﬁned by Eq. 6.

Original: Each neuron is equipped with a habituation counter h i ∈ [0 , 1] expressing how frequently it has ﬁred based on a simpliﬁed model of how the efﬁcacy of a habituating synapse reduces over time.
Revised: Each neuron is equipped with a habituation counter h i ∈ [0 , 1] expressing how frequently it has ﬁred based on a simpliﬁed model of how the efﬁcacy of a habituating synapse reduces over time.

Original: In the Gamma-GWR, the habituation rule is given by ∆ h i =
Revised: In the Gamma-GWR, the habituation rule is given by ∆ h i =

Original: τ i · κ · (1 − h i ) − τ i , where κ and τ i are constants that control the decreasing behavior of the habituation counter Marsland et al. (2002)
Revised: τ i · κ · (1 − h i ) − τ i , where κ and τ i are constants that control the decreasing behavior of the habituation counter (Marsland et al. (2002)).

Original: To establish whether a neuron is habituated, its habituation counter h i must be smaller than a given habituation threshold h T .
Revised: To establish whether a neuron is habituated, its habituation counter h i must be smaller than a given habituation threshold h T .

Original: The network is initialized with two neurons and, at each learning iteration, it inserts a new neuron whenever the activity of the network a ( t ) of a habituated neuron is smaller than a given threshold a T , i.e., a new neuron r is created if a ( t ) < a T and h b < h T .
Revised: The network is initialized with two neurons and, at each learning iteration, it inserts a new neuron whenever the activity of the network a ( t ) of a habituated neuron is smaller than a given threshold a T , i.e., a new neuron r is created if a ( t ) < a T and h b < h T .

Original: The training of the neurons is carried out by adapting the BMU b and its topological neurons n according to:
Revised: The training of the neurons is carried out by adapting the BMU b and its topological neurons n according to:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) i is a constant learning rate.
Revised: where (cid:15) i is a constant learning rate.

Original: The learning process of the Gamma-GWR is unsupervised and driven by bottom-up sensory observations, thereby either allocating new neurons or adapting existing ones in response to novel input.
Revised: The learning process of the Gamma-GWR is unsupervised and driven by bottom-up sensory observations, thereby either allocating new neurons or adapting existing ones in response to novel input.

Original: In this way, ﬁne-grained multisensory representations can be acquired and ﬁne-tuned through experience.
Revised: In this way, ﬁne-grained multisensory representations can be acquired and ﬁne-tuned through experience.

Original: As an extension of the Gamma-GWR, we implement temporal synapses for the purpose of predicting future frames from an onset frame.
Revised: As an extension of the Gamma-GWR, we implement temporal synapses for the purpose of predicting future frames from an onset frame.

Original: We implement temporal connections as sequence-selective synaptic links that are incremented between those two neurons that are consecutively activated.
Revised: We implement temporal connections as sequence-selective synaptic links that are incremented between those two neurons that are consecutively activated.

Original: When the two neurons i and j are activated at time t − 1 and t respectively, their synaptic link P ( i,j ) is strengthened.
Revised: When the two neurons i and j are activated at time t − 1 and t respectively, their synaptic link P ( i,j ) is strengthened.

Original: Thus, at each learning iteration, we set ∆ P ( I − 1 ,b )
Revised: Thus, at each learning iteration, we set ∆ P ( I − 1 ,b )

Original: = 1 , where I − 1 and b are respectively the indexes of the BMUs at time t − 1 and t .
Revised: = 1 , where I − 1 and b are respectively the indexes of the BMUs at time t − 1 and t .

Original: As a result, for each neuron i ∈ N , we can retrieve the next neuron v of a prototype sequence by selecting
Revised: As a result, for each neuron i ∈ N , we can retrieve the next neuron v of a prototype sequence by selecting

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This approach results in the learning of trajectories of neural activations that can be reconstructed in the absence of sensory input.
Revised: This approach results in the learning of trajectories of neural activations that can be reconstructed in the absence of sensory input.

Original:  E XPECTATION L EARNING
Revised:  E XPECTATION L EARNING

Original: As the self-organizing layer is updated in an unsupervised Hebbian manner, it learns to associate audio-visual stimuli online.
Revised: As the self-organizing layer is updated in an unsupervised Hebbian manner, it learns to associate audio-visual stimuli online.

Original: That means that the binding process is entirely data-driven, without the necessity of supervision.
Revised: That means that the binding process is entirely data-driven, without the necessity of supervision.

Original: More speciﬁcally, after ﬁnding the BMU related to a unimodal perceived stimulus, the associated absent stimuli will be reconstructed based on the prototypical concept that this neuron learned.
Revised: More speciﬁcally, after ﬁnding the BMU related to a unimodal perceived stimulus, the associated absent stimuli will be reconstructed based on the prototypical concept that this neuron learned.

Original: This is possible because each neuron in the self-organizing layer processes the union of the auditory and visual encodings at training time, where both signals are provided.
Revised: This is possible because each neuron in the self-organizing layer processes the union of the auditory and visual encodings at training time, where both signals are provided.

Original: This reconstruction and expectation learning capability is the basis for our following proposal of a re-training mechanism for the self-organizing layer.
Revised: This reconstruction and expectation learning capability is the basis for our following proposal of a re-training mechanism for the self-organizing layer.

Original: We ﬁrst pre-train our self-organizing binding to generate prototype neurons with strong audio-visual encodings.
Revised: We ﬁrst pre-train our self-organizing binding to generate prototype neurons with strong audio-visual encodings.

Original: This allows the model to learn a prior association between auditory and visual concepts.
Revised: This allows the model to learn a prior association between auditory and visual concepts.

Original: After the network has learned these associations, we use unknown data to ﬁne-tune the bindings with the expectation learning.
Revised: After the network has learned these associations, we use unknown data to ﬁne-tune the bindings with the expectation learning.

Original: We ﬁrst encode a visual or auditory stimulus ( s ), and compute the BMU ( b av ) using only the associated auditory or visual weights:
Revised: We ﬁrst encode a visual or auditory stimulus ( s ), and compute the BMU ( b av ) using only the associated auditory or visual weights:

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) w j,s represents the audio or visual representation encoded on the neuron’s weights.
Revised: where (cid:101) w sj represents the audio or visual representation encoded on the neuron’s weights.

Original: In this case, the global context of the network at any time step ( (cid:101) C s,k t )  is represented by the stimulus encoding, the same happens with the BMU context ( (cid:101) c
Revised: In this case, the global context of the network at any time step ( (cid:101) C s k ( t ) ) is represented by the stimulus encoding, the same happens with the BMU context ( (cid:101) c

Original: j,k ).
Revised: j,k ).

Original: We then use the auditory and vision parts of the multisensory representation stored on b av to reconstruct the auditory ( a (cid:48) ) and visual ( v (cid:48) ) information using the speciﬁc channel decoding D v for vision and D a for audio:
Revised: We then use the auditory and vision parts of the multisensory representation stored on b av to reconstruct the auditory ( a (cid:48) ) and visual ( v (cid:48) ) information using the speciﬁc channel decoding D v for vision and D a for audio:

Original: [Equation]
Revised: [Equation]

Original: By doing this for both perceived auditory and visual signals, we create two extra pairs of multisensory stimuli by combining the perceived auditory and visual ones with the reconstructed auditory and visual.
Revised: By doing this for both perceived auditory and visual signals, we create two extra pairs of multisensory stimuli by combining the perceived auditory and visual ones with the reconstructed auditory and visual.

Original: We bind the encoded information of the reconstructed audio-visual information with the original perceived stimuli and re-train the self-organizing layer with the new pairs.
Revised: We bind the encoded information of the reconstructed audio-visual information with the original perceived stimuli and re-train the self-organizing layer with the new pairs.

Original: By pairing the perceived and the reconstructed stimuli representations, we enforce the self-organizing layer to learn concepts and not instances of the animals.
Revised: By pairing the perceived and the reconstructed stimuli representations, we enforce the self-organizing layer to learn concepts and not instances of the animals.

Original: That means that animals which sound similar will be paired together. This mechanism helps us enforce the connections of coincident stimuli without needing a large amount of  data.
Revised: In consequence, animals which sound similar will be paired together, and connections of coincident stimuli will be learned with relatively small amounts of training data.

Original: In the other hand, incongruence will make the model  pair different audio-visual stimuli which will create prototype neurons which will be forgotten quickly by the self-organizing layer as they do not happen often.
Revised: Incongruences will cause the model to pair different audio-visual stimuli, thus creating new prototype neurons, but these will be forgotten quickly by the self-organizing layer as they occur less frequently.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Our goal is to evaluate the performance of the model to reconstruct audio/visual stimuli based on unimodal perception.
Revised: Our goal is to evaluate the performance of the model to reconstruct audio/visual stimuli based on unimodal perception.

Original: Also, we intend to evaluate the conceptual relations learned by the network.
Revised: Also, we intend to evaluate the conceptual relations learned by the network.

Original: Thus, although there exists several datasets with multimodal information, the animal subset of the AudioSet corpus (Gemmeke et al.
Revised: Although there exists several datasets with multimodal information, the animal subset of the AudioSet corpus (Gemmeke et al.

Original: (2017)) presents a unique advantage for our evaluation. It contains natural scenarios with different levels of conceptual binding - from images of cats and meowing sounds association to high pitch barking  with small dogs.
Revised: (2017)) presents a unique advantage for our evaluation: It contains natural scenarios with different levels of conceptual binding, including broader prototype associations like images of cats linked to meowing, but also more ﬁne-grained associations like high-pitched barking liked to with small dogs.

Original: Each video  has a duration of 10s and it is possible that, e.g., there is both a cat and a dog present in the video.
Revised: Each video in the dataset has a duration of 10s and it is possible that, e.g., there is both a cat and a dog present in the video.

Original: As there are no standard published results on this speciﬁc task for the AudioSet corpus, we run a series of baseline recognition experiments that serves as main comparison to measure our model’s performance.
Revised: As there are no standard published results on this speciﬁc task for the AudioSet corpus, we run a series of baseline recognition experiments that serves as main comparison to measure our model’s performance.

Original: To obtain a precise measure of the contribution of the expectation learning, we decided to cluster some overlapping classes and use 16 single labels, one per video:Cats (”Cat” + ”Meow”+ ”Purr”), Dogs (”Bark”+”Dog”+”Howl”), Pigs (”Oink” + ”Pig”), Cows (”Moo”+”Cattle, bovinae”),
Revised: To obtain a precise measure of the contribution of the expectation learning, we decided to cluster some overlapping classes and use 16 single labels, one per video:

Original:  Owls (”Owl” + ”Coo”), Birds, Goats, Bee (”Bee, was, etc..”), Chickens (”Chicken, rooster”), Ducks (”Duck”), Pidgeons (”Pidgeon, dove”), Crows (”Crow”), Horses (”Horse”), Frogs (”Frogs”), Flies (”Fly, houseﬂy”), Lions (”Roaring cats (lions, tigers)”).
Revised: Cats (”Cat” + ”Meow”+ ”Purr”), Dogs (”Bark”+”Dog”+”Howl”), Pigs (”Oink” + ”Pig”), Cows (”Moo”+”Cattle, bovinae”), Owls (”Owl” + ”Coo”), Birds, Goats, Bee (”Bee, was, etc..”), Chickens (”Chicken, rooster”), Ducks (”Duck”), Pidgeons (”Pidgeon, dove”), Crows (”Crow”), Horses (”Horse”), Frogs (”Frogs”), Flies (”Fly, houseﬂy”), Lions (”Roaring cats (lions, tigers)”).

Original: We used the unbalanced training subset consisting of approximately 43.500 videos to train our model and evaluated it with the test subset consisting of approximately 20.000 videos.
Revised: We used the unbalanced training subset consisting of approximately 43.500 videos to train our model and evaluated it with the test subset consisting of approximately 20.000 videos.

Original: The labels of this dataset were crowdsourced based on the video descriptions.
Revised: The labels of this dataset were crowdsourced based on the video descriptions.

Original: In our ﬁrst baseline experiment, we train classic deep learning architectures for individual stimuli recognition. We make use of  the Inception V3 network (Ioffe & Szegedy (2015)) for the visual stimuli, and the SoundNet (Aytar et al.
Revised: We perform two sets of experiment: one to evaluate the contribution of the expectation learning to the multisensory binding and one to compare the performance of our model with currently successful deep learning models for unisensory recognition. The ﬁrst set of experiments is divided into three steps. In Exp 1 we train the multisensory bindings of the GWR using half of the training subset in order to guarantee that the model learned strong audiovisual prior bindings.

Original:  for the auditory stimuli.
Revised: In Exp 2 we continue the training of the Exp 1 network using the other half of the training subset. This experiment serves as a baseline for learning bindings without expectation [Table] and as a main comparison point for the contribution of the expectation learning mechanism.

Original: Our second baseline experiment makes use of the learned representations of the individual autoencoers.
Revised: And ﬁnally, Exp 3 , where we repeat the continuation of the training of the Exp 1 network with the other half of the training subset but now using the expectation learning mechanism when creating the GWR associations. To evaluate the performance contribution of each of our experimental steps on the association learning, we implemented a supervised classiﬁer for each of the channels channel (auditory and visual). It receives as input the encoded representation from the GWR of a perceived audio or visual stimuli.

Original: We implement a supervised classiﬁer  for each channel.
Revised: To evaluate the capability of the model to learn meaningful associations, we classify always an absent stimuli, i.e. when perceiving an auditory stimuli, the network uses the associated visual stimuli as input to the classiﬁer and vice-versa. That means that when perceiving 50ms of audio, we have an associated representation of 4 frames and vice-versa. As the videos from the AudioSet dataset have a lenght of 10s, we use a simple voting scheme to obtain the ﬁnal label. For every 50ms for audio and every 4 frames for video we produce one label and after having all the labels for a 10s video, we select the one which appears more often.

Original: The classiﬁer is composed of a dense layer with 128 units and an output softmax layer.
Revised: Each classiﬁer is composed of a dense layer with 128 units and an output softmax layer.

Original: We optimized our autoencoders and classiﬁer to maximize the recognition accuracy on the training subset using a tree-structured Parzen estimator (TPE) (Bergstra et al.
Revised: We optimized our autoencoders and classiﬁer to maximize the recognition accuracy on the training subset using a tree-structured Parzen estimator (TPE) (Bergstra et al.

Original: and use the optimal parameters through all of our experiments.
Revised: and use the optimal parameters through all of our experiments.

Original: This is an important step, as we need the encoding/decoding processes of our model towork robustly with real-world data. To evaluate  our expectation model in a proper manner, we ﬁrst trained our network using half of the training subset in order to guarantee that the model learned prior meaningful audio-visual bindings.
Revised: Our second set of experiments was designed to evaluate how our proposed model compares to deep learning networks for auditory and visual stimuli recognition. We compare our model with the

Original: Then we use the other half of the training subset to update the networks bindings using the expectation learning.
Revised: Inception V3 network (Ioffe & Szegedy (2015)) for the visual stimuli, and the SoundNet (Aytar et al. for the auditory stimuli.

Original: We measure the performance of the model to generate meaningful information.
Revised: These models present competitive results on different audiovisual recognition tasks Kiros et al.

Original: For that, we present one audio or visual stimulus of a video and generate the absent stimulus. We then e feed the autoencoder with the generated stimulus and calculate the accuracy of the generated stimulus regarding the true label of the video.
Revised: ; Jiang et al.

Original: To measure the contribution of the expectation learning we also computed the recognition accuracy for each class of the evaluation subset when training only with half of the training subset. This is important to explicit the impact of the expectation learning.
Revised: ; Kumar et al. ; Jansen et al. Herein, our goal is not to propose the best benchmark model for audio-visual recognition, but to assess the contributions of the expectation learning on the reconstruction of absent stimuli.

Original: For all of our experiments We trained the models 10 times and report the mean accuracy and standard deviation for each modality.
Revised: For all experiments, we trained the models 10 times and report the mean accuracy and standard deviation for each modality.

Original: We used the same 10% of the training subset as a validation set for each experiment. We used  the accuracy convergence on the validation subset to stop the training of each model.
Revised: We used the same 10% of the training subset as a validation set for each experiment, and used an early stopping mechanism based on the accuracy of the validation subset to prevent overﬁtting.

Original: This guarantees that each model is trained for an optimized number of epochs to maximize the performance.

Original:  R ESULTS
Revised:  R ESULTS

Original: Our ﬁnal results are shown on Table 1.
Revised: Our ﬁnal results are depicted in Table 1.

Original: The contribution of the expectation is highlighted by a comparison with the results of training the autoencoder with half the training data.

Original: Training the model with half of the data, to createthe strong binding associations, was enough to obtain a baseline 
Revised: Our ﬁrst experiment, Exp 1 , demonstrates that training the model with half of the data, to create strong binding associations, was enough to obtain a baseline performance.

Original: [Table] performance. Training the model with all the data points improved 8% of recognition accuracy for audio and more than 17% for audio.
Revised: Continuing to train the model using standard GWR associations ( Exp 2 ) gives as an, expected, improvement of 8% of recognition accuracy for audio and more than 17% for vision when compared to Exp 1 experiment.

Original: While training with the expectation learning, we obtained an improvement of more than 12% for audiorecognition and more than 20% for vision.
Revised: The results of Exp 3 shows that the expectation mechanism improved the recognition of unisensory stimuli, when compared to Exp 2 . We obtained an improvement of more than 4% on audio and 3% on vision.

Original: The performance of the network follows the general behavior of other models to recognize vision stimuli better than auditory stimuli.
Revised: The performance of the network follows the general behavior of other models to recognize vision stimuli better than auditory stimuli.

Original: This effect is demonstrated by the results of the SoundNet and Inception-V3 models.
Revised: This effect is demonstrated by the results of the SoundNet and Inception-V3 models.

Original: This is probably due the dataset presenting challenging audio stimuli with much background noise.
Revised: This is probably due the dataset presenting challenging audio stimuli with much background noise.

Original: When compared to SoundNet and Inception-V3, our expectation model  presents better audiotry recognition, and competitive vision recognition performance.
Revised: When compared to SoundNet and Inception-V3, our expectation model (represented by Exp 3 experiment) presents better auditory recognition, and comparable vision recognition performance.

Original: When compared to training the autoencoder with all the datapoints the expectation learning improved the recognition of auditory stimuli by more than 4% while improved the visual stimuli in 3%.

Original: This behavior is a side effect of the expectation model which enforces the multimodal binding.

Original: The auditory stimuli is more affected, as it presents much more noisy information.
Revised: The auditory stimuli is more affected, as it presents much more noisy information.

Original: The network then relies more on the visual stimuli and creates neurons with strong visual encoding.
Revised: The network then relies more on the visual stimuli and creates neurons with strong visual encoding.

Original: This effect is represented by creating neurons with similar visual encoding associated to different auditory encoding.
Revised: This effect is represented by creating neurons with similar visual encoding associated to different auditory encoding.

Original: When training with expectation learning, the network created an average of 5400 neurons, while training without the expectation it created 4000 neurons.
Revised: When training with expectation learning, the network created an average of 5400 neurons, while training without the expectation it created 4000 neurons.

Original: The latent representation from the auditory and visual channels encode different characteristics of the stimulus, which are then bound by our self-organizing layer.
Revised: The latent representation from the auditory and visual channels encode different characteristics of the stimulus, which are then bound by our self-organizing layer.

Original: The expectation learning enforced the generation of robust bindings, especially for distinct animals.
Revised: The expectation learning enforced the generation of robust bindings, especially for distinct animals.

Original: The network ended up creating speciﬁc neurons for cats and dogs, and shared neurons for chickens and ducks, for example.
Revised: The network ended up creating speciﬁc neurons for cats and dogs, and shared neurons for chickens and ducks, for example.

Original: This explains the improvement on the recognition of the reconstructed stimuli of easily separable animals, as illustrated in Figure 2.
Revised: This explains the improvement on the recognition of the reconstructed stimuli of easily separable animals, as illustrated in Figure 2.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: As the self-organizing layer is updated in an unsupervised manner, it learns to associate audiovisual stimuli online.
Revised: As the self-organizing layer is updated in an unsupervised manner, it learns to associate audiovisual stimuli online.

Original: That means that our model learning mechanism is entirely data-driven without a strong teaching supervisor.

Original: More speciﬁcally, after ﬁnding the BMU related to a single perceived stimulus, the associated absent stimuli can be reconstructed based on the concept that this neuron learned, and not on the originalhigh-abstraction data.
Revised: Moreover, by activating the BMU related to a speciﬁc perceived stimulus, the associated absent stimuli can be reconstructed based on the concept that this neuron learned. However, the reconstructed data is, of course, not identical to the original data.

Original: When sending an image of a dog to the network, it will reconstruct the sound of barking, but not  theexact sound that this speciﬁc dog would make.
Revised: For example, when processing an image of a dog, the network will reconstruct an appropriate barking sound, but not exactly the sound that this speciﬁc dog would make.

Original: This effect is related to theconcept of multisensory imagery  (Spence & Deroy (2013)) where humans tend to simplify and clusterthe absent stimuli when asked to reconstruct it.
Revised: This mimics precisely the multisensory imagery effect (Spence & Deroy (2013)) of humans, who tend to simplify and cluster absent stimuli when asked to reconstruct them.

Original: That means that every time you see adifferent small yellow bird, you will expect it to sound very similar to the ones you have seen before.
Revised: For example, every time one sees a small yellow bird, the person will expect it to sound very similar to the ones she/he has seen before.

Original: This is an important effect that helps our model to reconstruct concepts instead of speciﬁc instances.
Revised: This is an important effect that helps our model to reconstruct concepts instead of speciﬁc instances.

Original: As there is no quantitative metric to measure this effect, thus we proceed with a simple overlapping analysis which gives us an indication of how well the model is binding and clustering audio-visual information.
Revised: To provide an indication of this effect, and as an additional indicator for multisensory concept formation, we perform an overlapping analysis to estimate how well the model is binding and clustering audio-visual information.

Original: We ﬁrst train the model with  expectation learning  and then we classify every single neuron  using both audio and visual classiﬁers which generate two labels for each neuron: one for auditory and one for visual information.
Revised: To this end, we ﬁrst train the model with the expectation learning mechanism and then we classify every single neuron of the GWR using both audio and visual classiﬁers which generate two labels for each neuron: one for auditory and one for visual information.

Original: The total overlap between visual and auditory labels for each prototype neuron in our self-organizing layer is 93%, suggesting that our prototype neurons are very concise when storing audio-visual information.
Revised: The total overlap between visual and auditory labels for each prototype neuron in our self-organizing layer is 93%, suggesting that our prototype neurons are very concise when storing audio-visual information.

Original: The training of the binding layers causes the prototype neurons to be associated to high-abstraction concepts and to reconstruct absent modality-speciﬁc information which is thebasis for our expectation learning mechanism.
Revised: Performing the same experiment on the network training without the expectation mechanism gave us an overlap of 85% of the neurons. Another effect that we investigate is multisensory correspondence (Spence & Driver (2000)).

Original: For example, when humans see a barking dog, they are continuously reinforcing the association between dogs and barking sounds. We also create associations between small dogs and high-pitch barking, an example of an effect called multisensory correspondence
Revised: The effect causes humans not only to associate dogs with barking, but also, more speciﬁcally, small dogs with high-pitched barking. The associations between the stimuli are continuously reinforced as perceptive stimuli are experienced.

Original: Spence & Driver (2000).

Original: This effect was also observed  in some examples where the variety of animals was higher, such as the dogs.
Revised: We observed this effect in some examples where the variety of animals was higher, such as the dogs.

Original: We illustrate one of these examples in Figure 3.
Revised: We illustrate one of these examples in Figure 3.

Original: It is possible to see that when reconstructing the  visual information based on an auditory stimulus of different dogs barking, a high-pitchdog barkingsound generated images related to a small dog.
Revised: The ﬁgure depicts the reconstruction of visual information based on an auditory stimulus of different dogs barking. A high-pitch barking generated images related to a small dog.

Original: It is also possible to see that, when the sound of more than one dog appears barking, the network generated an image of more than one dog.
Revised: Furthermore, when the simultaneous barking of more than one dog is processed, the network generates an image of several dogs.

Revised: We expect this effect to become more visible with larger datasets that contain more diverse samples.

Revised: One important limitation of our approach is that both multisensory imagery and multisensory correspondence only occurs when both auditory and visual stimuli can be understood and represented as a simpliﬁed concept.

Revised: A human cannot reconstruct precisely the characteristics of how the voice of a person will sound when reading a text, for example.

Revised: Our experiments demonstrate that our model learns to associate high-level animal concepts, and even multisensory correspondences, but would not be applied to reconstruct information that demands a much higher precision, i.e.person identiﬁcation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Multisensory binding is a crucial aspect of how humans understand the world.
Revised: Multisensory binding is a crucial aspect of how humans understand the world.

Original: Consequently, the development of computational systems able to adapt this aspect into information processing is important to many research ﬁelds.
Revised: Consequently, the development of computational systems able to adapt this aspect into information processing is important to many research ﬁelds.

Original: An extensive number of models has been proposed that incorporate different aspects of multisensory binding.
Revised: An extensive number of models has been proposed that incorporate different aspects of multisensory binding.

Original: However, none of them made use of expectation learning which is a mechanism that helps humans learn from their mistakes in a continuous manner. In this paper, we introduce a model that adapts different concepts of expectation learning.
Revised: However, our approach combines several novelties.

Original: Our model makes use of autoencoders to be able to reconstruct audio-visual stimuli and use a recurrent self-organizing network  to learn multisensory bindings.
Revised: It combines a Grow-When-Required (GWR) network with autoencoders to realize continuous unsupervised expectation learning.

Original: We show that our expectation learning mechanism provides a robust solution for reconstructingabsent, modality-speciﬁc stimuli and discussed the effects of such mechanism on  the learning behavior of our model. Our experiments demonstrate that the pro- posed model and expectation learning mechanisms can improve unisensory recognition.
Revised: In addition, we propose to exploit expectation learning by reconstructing stimuli that can be used as additional training data to generate a signiﬁcant positive effect on perceptive tasks like classiﬁcation. We are, therefore, the ﬁrst to introduce a quantitative metrics to measure the quality of multisensory bindings, while at the same time providing a novel proof-ofconcept for a data augmentation mechanism to improve the accuracy and performance of unimodal classiﬁcation methods.

Original: We would like to extend our model to deal with spatial expectation, which would give us a complete view on learning based on contextual, temporal, and spatial incongruence.
Revised: An interesting future research direction is to also address spatial expectation, because this would provide a complementary component to integrate contextual, temporal, and spatial correspondence.

Original: We also observed the development of continuous and incremental learning in our self-organizing binding layer. We would like to explore these characteristics with further experiments on life-long and transfer learning.
Revised: Realizing the transfer of learned multisensory bindings is another unexplored research area that we plan to investigate as a follow-up to this work.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Given the double-blind review nature of ICLR we willonly update the Acknowledgmentssession after the reviewing process.
Revised: Given the double-blind review nature of ICLR, we will update the Acknowledgments after the reviewing process.

Original:  A PPENDIX 8. T RAINING PARAMETERS 8.1. V ISION CHANNEL
Revised:  A PPENDIX 9. T RAINING PARAMETERS 9.1. V ISION CHANNEL

Original: Table 2 exhibits all the important parameters used to train our visual channel.
Revised: Table 2 exhibits all the important parameters used to train our visual channel.

Original: We used ADAM optimizer with an adaptive learning rate.
Revised: We used ADAM optimizer with an adaptive learning rate.

Original: [Table]
Revised: [Table]

Original:  A UDITORY CHANNEL
Revised: 9.1. A UDITORY CHANNEL

Original: Table 3 exhibits all the important parameters used to train our auditory channel.
Revised: Table 3 exhibits all the important parameters used to train our auditory channel.

Original: We follow the same training procedure as the vision channel [Table] and also use ADAM optimizer with an adaptive learning rate.
Revised: We follow the same training procedure as the vision channel, and also use ADAM optimizer with an adaptive learning rate.

Original: [Table]
Revised: [Table]

Original: 8.1. S ELF - ORGANIZING BINDING LAYER
Revised: 9.1. S ELF - ORGANIZING BINDING LAYER

Original: Table 4 exhibits all the important parameters used to train our gamma Growing-When-Required (GWR) network.
Revised: Table 4 exhibits all the important parameters used to train our gamma Growing-When-Required (GWR) network.

Original: We use a small insertion threshold, which helps the network to maintain a limited number of neurons, reinforcing the generation of high-abstract clusters. [Table]
Revised: We use a small insertion threshold, which helps the network to maintain a limited number of neurons, reinforcing the generation of high-abstract clusters

Original:  D ETAILED EXPERIMENTAL RESULTS 8.2. B ASELINE E XPERIMENTS
Revised: . 9. D ETAILED EXPERIMENTAL RESULTS 9.2. B ASELINE E XPERIMENTS

Original: Table 6 details the mean accuracy of our baseline experiments (SoundNet and Inception V3) for each of the classes of the evaluation subset. 
Revised: Table 5 details the mean accuracy of our baseline experiments (SoundNet and Inception V3) for each of the classes of the evaluation subset. [Table]

Original: [Table]
Revised: [Table]

Original: E XPECTATION LEARNING E XPERIMENTS
Revised: E XPECTATION LEARNING E XPERIMENTS

Original: Table 5 exhibits the detailed accuracy per class when evaluating our expectation learning model with the evaluation subset.
Revised: Table 6 exhibits the detailed accuracy per class when evaluating our expectation learning model with the evaluation subset.

Original: Here we detail the results based on reconstructed audio, when the audio is absent on the perceived stimuli [Table] and on reconstructed vision, when the vision is absent.
Revised: Here we detail the results based on reconstructed audio, when the audio is absent on the perceived stimuli, and on reconstructed vision, when the vision is absent.

Original: We detail the experiments with and without the expectation.
Revised: We detail the experiments with and without the expectation.

Revised: 9. E XAMPLES

Revised: Figure 4 illustrates some selected examples of impact of the expectation learning.

Revised: The image exempliﬁes cases where the network trained with expectation learning improved the recognition.

Revised: Each row shows an input audio example (and the class associated to it), and the reconstruction of a network trained with and without the expectation.

Original: [Table]
Revised: [Table]

Revised: Frogs 51,8 (3.7) 59,4 (2.7) 80,6 (2.7) 82,1 (3.4)

Revised: Flies 57,8 (3.0) 58,3 (2.5) 84,9 (1.6) 86,7 (2.6)

Revised: Lions 60,3 (2.9) 68,5 (2.6) 90,4 (2.4) 93,2 (3.8)