Original: On the Beneﬁts of Early Fusion in Multimodal Representation Learning
Revised: On the Beneﬁts of Early Fusion in Multimodal Representation Learning

Original: Abstract
Revised: Abstract

Original: Intelligently reasoning about the world often requires integrating data from multiple modalities, as any individual modality may contain unreliable or incompleteinformation.
Revised: Intelligently reasoning about the world often requires integrating data from multiple modalities, as any individual modality may contain unreliable or incomplete information.

Original: Prior work in multimodal learning fuses input modalities only after signiﬁcant independent processing.
Revised: Prior work in multimodal learning fuses input modalities only after signiﬁcant independent processing.

Original: On the other hand, the brain performsmultimodal processing almost immediately.
Revised: On the other hand, the brain performs multimodal processing almost immediately.

Original: This divide between conventionalmultimodal learning and neuroscience suggests that a detailed study of early multimodal fusion could improve artiﬁcial multimodal representations.
Revised: This divide between conventional multimodal learning and neuroscience suggests that a detailed study of early multimodal fusion could improve artiﬁcial multimodal representations.

Original: To facilitatethe study of early multimodal fusion, we create a convolutional LSTM networkarchitecture that simultaneously processes both audio and visual inputs, and allows us to select the layer at which audio and visual information combines.
Revised: To facilitate the study of early multimodal fusion, we create a convolutional LSTM network architecture that simultaneously processes both audio and visual inputs, and allows us to select the layer at which audio and visual information combines.

Original: Ourresults demonstrate that immediate fusion of audio and visual inputs in the initial
Revised: Our results demonstrate that immediate fusion of audio and visual inputs in the initial

Original: C-LSTM layer results in higher performing networks that are more robust to theaddition of white noise in both audio and visual inputs.
Revised: C-LSTM layer results in higher performing networks that are more robust to the addition of white noise in both audio and visual inputs.

Revised: Find the full paper at: https://arxiv.org/pdf/2011.07191.pdf

Original:  Introduction
Revised:  Introduction

Original: Multimodal learning is important for many tasks, including audio visual speech recognition (Yuet al., 2020;
Revised: Multimodal learning is important for many tasks, including audio visual speech recognition (Yu et al., 2020;

Original: Zhou et al., 2019;
Revised: Zhou et al., 2019;

Original: Su et al., 2017), emotion recognition (Park et al., 2020; Cao et al.,2014), multimedia event detection (Song et al., 2019), depth-based object detection (Wang et al. ,2015b,a), urban dynamics modeling (Zhang et al., 2017), image-sentence matching (Liu et al. and biometric recognition (Song et al., 2019).
Revised: Su et al., 2017), emotion recognition (Park et al., 2020; Cao et al., 2014), multimedia event detection (Song et al., 2019), depth-based object detection (Wang et al., 2015b,a), urban dynamics modeling (Zhang et al., 2017), image-sentence matching (Liu et al., 2019) and biometric recognition (Song et al., 2019).

Original: In many cases, an individual modality doesnot contain sufﬁcient information to classify the scene.
Revised: In many cases, an individual modality does not contain sufﬁcient information to classify the scene.

Original: Therefore, utilizing multiple modalities iscrucial, particularly in complex tasks or domains prone to noisy data.
Revised: Therefore, utilizing multiple modalities is crucial, particularly in complex tasks or domains prone to noisy data.

Original: One important design decision in multimodal learning is how to best combine, or fuse, the differentinput modalities (Baltruˇsaitis et al., 2018; Li et al., 2018).
Revised: One important design decision in multimodal learning is how to best combine, or fuse, the different input modalities (Baltruˇsaitis et al., 2018; Li et al., 2018).

Original: Prior work on multimodal learning haslargely relied on extensive unimodal featurization and other preprocessing before fusing the differentmodalities (Katsaggelos et al., 2015; Atrey et al., 2010).
Revised: Prior work on multimodal learning has largely relied on extensive unimodal featurization and other preprocessing before fusing the different modalities (Katsaggelos et al., 2015; Atrey et al., 2010).

Original: On the other hand, it is known that biological neural networks engage in multimodal fusion in the very early layers of sensory processingpathways (Schroeder & Foxe, 2005; Budinger et al., 2006).
Revised: On the other hand, it is known that biological neural networks engage in multimodal fusion in the very early layers of sensory processing pathways (Schroeder & Foxe, 2005; Budinger et al., 2006).

Original: This divide between conventional multimodal learning and neuroscience suggests that a detailed study of early multimodal fusion couldyield insights for improving multimodal representation learning.
Revised: This divide between conventional multimodal learning and neuroscience suggests that a detailed study of early multimodal fusion could yield insights for improving multimodal representation learning.

Original: In this paper, we study the beneﬁts of early fusion in multimodal learning.
Revised: In this paper, we study the beneﬁts of early fusion in multimodal learning.

Original: To facilitate this study,we design a convolutional LSTM (C-LSTM) architecture that enables audio and visual input fusionat various layers in the architecture.
Revised: To facilitate this study, we design a convolutional LSTM (C-LSTM) architecture that enables audio and visual input fusion at various layers in the architecture.

Original: We ﬁnd that early fusion outperforms late fusion, and that earlyfusion enables robust performance over a range of signal to noise ratios.
Revised: We ﬁnd that early fusion outperforms late fusion, and that early fusion enables robust performance over a range of signal to noise ratios.

Original: These results shed new lighton the power of immediate fusion as a means to improve model performance in the presence of noise.
Revised: These results shed new light on the power of immediate fusion as a means to improve model performance in the presence of noise.

Original: Integrating multimodal inputs as soon as possible can be generalized to other multimodal domains, such as audio-visual speech recognition and emotion recognition to increase their performance andrepresentational power.
Revised: Integrating multimodal inputs as soon as possible can be generalized to other multimodal domains, such as audio-visual speech recognition and emotion recognition to increase their performance and representational power.

Original:  Related Work
Revised:  Related Work

Original: Multimodal Representation Learning.
Revised: Multimodal Representation Learning.

Original: Baltruˇsaitis et al.
Revised: Baltruˇsaitis et al.

Original: (2018) thoughtfully broke down the mainproblems within multimodal machine learning into ﬁve categories: representation, translation, alignment, fusion, and co-learning.
Revised: (2018) thoughtfully broke down the main problems within multimodal machine learning into ﬁve categories: representation, translation, alignment, fusion, and co-learning.

Original: Turk (2014) also posits that multimodal integration, also referred toas the fusion engine, is the key technical challenge for multimodal systems.
Revised: Turk (2014) also posits that multimodal integration, also referred to as the fusion engine, is the key technical challenge for multimodal systems.

Original: When dealing with fusion, there are currently two common paradigms: early fusion (Atrey et al. and immediate fusion (Katsaggelos et al., 2015).
Revised: When dealing with fusion, there are currently two common paradigms: early fusion (Atrey et al., 2010) and immediate fusion (Katsaggelos et al., 2015).

Original: In early fusion, audio and visual modalitiesare ﬁrst featurized before being passed to two independent modeling process units that do not differentiate between features from different modalities (Katsaggelos et al., 2015).
Revised: In early fusion, audio and visual modalities are ﬁrst featurized before being passed to two independent modeling process units that do not differentiate between features from different modalities (Katsaggelos et al., 2015).

Original: On the other hand,immediate fusion is when the audio and visual modalities are ﬁrst featurized and then sent to a joinmodeling process unit (Katsaggelos et al., 2015).
Revised: On the other hand, immediate fusion is when the audio and visual modalities are ﬁrst featurized and then sent to a joint modeling process unit (Katsaggelos et al., 2015).

Original: This unfortunate terminology does not take intoaccount the possibility of fusing the inputs before any substantial featurization, which does occur inbiological neural networks.
Revised: This unfortunate terminology does not take into account the possibility of fusing the inputs before any substantial featurization, which does occur in biological neural networks.

Revised: Therefore, we will propose and utilize our own terminology below.

Original: Connections to Neuroscience.
Revised: Connections to Neuroscience.

Original: In the brain, multisensory integration was traditionally believedto occur only after single modality inputs underwent extensive processing in unisensory regions(Schroeder & Foxe, 2005).
Revised: In the brain, multisensory integration was traditionally believed to occur only after single modality inputs underwent extensive processing in unisensory regions (Schroeder & Foxe, 2005).

Original: However, we now know that in many species, including humans, thatmultisensory convergence occurs much earlier in low level coritical structures (Schroeder & Foxe,2005).
Revised: However, we now know that in many species, including humans, multisensory convergence occurs much earlier in low level coritical structures (Schroeder & Foxe, 2005).

Original: In fact, primary sensory cortices may not be unimodal at all (Budinger et al., 2006).
Revised: In fact, primary sensory cortices may not be unimodal at all (Budinger et al., 2006).

Original: This mayin part be because of individual neuron’s abilities to be modulated by multiple modalities (Meredith & Allman, 2009).
Revised: This may in part be because of individual neuron’s abilities to be modulated by multiple modalities (Meredith & Allman, 2009).

Original: In a striking discovery, Allman & Meredith (2007) found that 16% of visualneurons in the posterolateral lateral suprasylvian that were previously believed to be only visuallyresponsive were signiﬁcantly facilitated by auditory stimuli.
Revised: In a striking discovery, Allman & Meredith (2007) found that 16% of visual neurons in the posterolateral lateral suprasylvian that were previously believed to be only visually responsive were signiﬁcantly facilitated by auditory stimuli.

Original: This philosophical departure from individual modality processing towards early multimodal convergence in neuroscience lays a promisinggroundwork for high-impact explorations in multimodal machine learning.
Revised: This philosophical departure from individual modality processing towards early multimodal convergence in neuroscience lays a promising groundwork for high-impact explorations in multimodal machine learning.

Original:  Models
Revised:  Models

Original: In order to maintain the advantages of modality speciﬁc inductive biases, such as convolution orrecurrence, while also allowing for the immediate fusion of audio and visual inputs, we created amultimodal convolutional long short term memory network that generates fused audio-visual representations with appropriate inductive biases.
Revised: In order to maintain the advantages of modality speciﬁc inductive biases, such as convolution or recurrence, while also allowing for the immediate fusion of audio and visual inputs, we created a multimodal convolutional long short term memory network that generates fused audio-visual representations with appropriate inductive biases.

Original: Our convolutional long short term memory, or CLSTM, architecture performs the standard LSTM operations at every point of a convolution, usingshared weights and separate cell and hidden state values.
Revised: Our convolutional long short term memory, or CLSTM, architecture performs the standard LSTM operations at every point of a convolution, using shared weights and separate cell and hidden state values.

Original: Further architectural details can be foundin A.1 and [-A.
-]
Revised: Further architectural details can be found in A.1 and [-A.
-]A.

Original: Using our C-LSTM architecture, we constructed multiple different models in order to study thebeneﬁts of multimodal fusion.
Revised: Using our C-LSTM architecture, we constructed multiple different models in order to study the beneﬁts of multimodal fusion.

Original: • The full C-LSTM model that allows for fusing in the early layers (akin to how biological networks fuse sensory processing in early layers).
Revised: • The full C-LSTM model that allows for fusion in the early layers (akin to how biological networks fuse sensory processing in early layers).

Original: • Restricting fusion to the intermediate convolutional layers.•
Revised: • Restricting fusion to the intermediate convolutional layers.

Revised: Which in this case is fusion in the second layer.

Original:  Restricting fusion to the fully connected layers (akin to prior work that performed unimodalfeaturization prior to fusion).• Only processing visual or audio input.
Revised: • Restricting fusion to the fully connected layers (akin to prior work that performed unimodal featurization prior to fusion). • Only processing visual or audio input.

Original: We conﬁrmed the ability of our model to effectively operate on both audio and visual data alone byoccluding each of the modalities in A.3.
Revised: We conﬁrmed the ability of our model to effectively operate on both audio and visual data alone by occluding each of the modalities in 5.3. [Figure]

Original:  Dataset
Revised:  Dataset

Original: We constructed a multimodal dataset based on the well-known MNIST dataset (LeCun et al., 2010)and the Free Spoken Digit dataset (Jackson et al., 2018).
Revised: We constructed a multimodal dataset based on the well-known MNIST dataset (LeCun et al., 2010) and the Free Spoken Digit dataset (Jackson et al., 2018).

Original: We selected these datasets because of their[Figure] tractability.
Revised: We selected these datasets because of their tractability.

Original: This allows us to combine them to create a multimodal task which we know will besolvable.
Revised: This allows us to combine them to create a multimodal task which we know will be solvable.

Original: This allows us to artiﬁcially manipulate the difﬁculty of the task via the addition of noise.
Revised: This allows us to artiﬁcially manipulate the difﬁculty of the task via the addition of noise.

Original: We prefer a regime in which we can break the system to understand its principles of fusion andrepresentation.
Revised: We prefer a regime in which we can break the system to understand its principles of fusion and representation.

Original: For more details see A.4.
Revised: For more details see A.3.

Original:  Experiments5.1 Fusion Comparison
Revised:  Experiments 5.1 Fusion Comparison

Original: To examine the value of early compared to late multimodal fusion, we created models which replaceearly multimodal C-LSTM layers with separate streams for each modality, as described in A.2.
Revised: To examine the value of early compared to late multimodal fusion, we created models which replace early multimodal C-LSTM layers with separate streams for each modality, as described in A.2.

Original: Theimmediate fusion model, the second layer fusion model, and the fully connected layer fusion modelwere trained with the same signal to noise ratios of training data.
Revised: The initial layer fusion model, the second layer fusion model, and the fully connected layer fusion model were trained with the same signal to noise ratios of training data.

Original: We then tested the accuracy ofeach of these models for a range of values of the signal to noise ratios of both the audio and visualinputs.
Revised: We then tested the accuracy of each of these models for a range of values of the signal to noise ratios of both the audio and visual inputs.

Original: Then we compare the accuracy of the immediate fusion model to the accuracy of each of thelate fusion models.
Revised: Then we compare the accuracy of the initial layer fusion model to the accuracy of each of the late fusion models.

Original: As seen in Figure 1, the immediate fusion model is more accurate not only for the signal to noiseratio that the models were trained at, but also for the majority of the other signal to noise ratios.
Revised: As seen in Figure 1, the initial layer fusion model is more accurate not only for the signal to noise ratio that the models were trained at, but also for the majority of the other signal to noise ratios.

Original: In particular, the initial layer fusion model always outperforms or equally performs to the fullyconnected fusion model.
Revised: In particular, the initial layer fusion model always outperforms or equally performs to the fully connected fusion model.

Original: Furthermore, the initial layer fusion model outperforms the second layerfusion model, except for when the audio input is degraded well beyond the audio training signal tonoise ratio.
Revised: Furthermore, the initial layer fusion model outperforms the second layer fusion model, except for when the audio input is degraded well beyond the audio training signal to noise ratio.

Original: Additionally, initial layer fusion appears to allow the model to be much more robust to increasesin the signal to noise ratio beyond the training values, especially in the case of increased audio SNR.
Revised: Additionally, initial layer fusion appears to allow the model to be much more robust to increases in the signal to noise ratio beyond the training values, especially in the case of increased audio SNR.

Original: The main characteristic of the SNR values in which immediate fusion does not outperformlate fusion is low audio SNR and relatively high visual SNR, and this only occurs in the case wherefusion occurs in the second layer.
Revised: The main characteristic of the SNR values in which initial fusion does not outperform late fusion is low audio SNR and relatively high visual SNR, and this only occurs in the case where fusion occurs in the second layer.

Original: This suggests that immediate multimodal fusion encourages themultimodal model to use both input modalities.
Revised: This suggests that initial layer multimodal fusion encourages the multimodal model to use both input modalities.

Original: While the speciﬁc areas and degree to which early fusion outperforms delayed fusion varies with the
Revised: While the speciﬁc areas and degree to which early fusion outperforms delayed fusion varies with the

Original: SNRs of the training data, the general trends are similar, and these fusion plots are representative ofmodels trained at other audio and visual SNRs; see A.5 for the same plot at other audio and visualtraining levels. [Figure]
Revised: SNRs of the training data, the general trends are similar, and these fusion plots are representative of

Original: 5.2 Robustness to Noise
Revised: 5.2 Robustness to Noise

Original: An advantage of multimodal processing is the network’s resilience to noise in the inputs.
Revised: An advantage of multimodal processing is the network’s resilience to noise in the inputs.

Original: To examineour multimodal model’s resilience to white noise we trained 16 models at distinct audio-visual SNRcombinations and then tested each model in 100 different audio-visual SNR regimes.
Revised: To examine our multimodal model’s resilience to white noise we trained 16 models at distinct audio-visual SNR combinations and then tested each model in 100 different audio-visual SNR regimes.

Original: The results ofthese experiments can be seen in ﬁgure 2.
Revised: The results of these experiments can be seen in ﬁgure 2.

Original: In the bottom right quadrant of ﬁgure 2 we see that themodel is mostly audio dependent.
Revised: In the bottom right quadrant of ﬁgure 2 we see that the model is mostly audio dependent.

Original: However, when the visual SNR is 1 and 0.5, there is strong visualdependence.
Revised: However, when the visual SNR is 1 and 0.5, there is strong visual dependence.

Original: The addition of noise to both the training and test data is designed to provide a setting that allowsthe exploration of the limits of this model while using a simple dataset.
Revised: The addition of noise to both the training and test data is designed to provide a setting that allows the exploration of the limits of this model while using a simple dataset.

Original: These results mirror ourexpectations of how a multimodal model would behave both to various training and testing SNRs.
Revised: These results mirror our expectations of how a multimodal model would behave both to various training and testing SNRs.

Revised: 5.3 Comparison to Unimodal Models

Revised: To verify that joint audio visual representations are a result of both modalities, we tested our multimodal model on unimodal inputs by setting one input to zero.

Revised: This created unimodal visual models and unimodal audio models without changing the underlying architecture.

Revised: For each of these unimodal models, we trained the model at four SNR values, and tested the accuracy across our previously selected set of signal to noise ratios.

Revised: The accuracy of these unimodal models for the SNR values is displayed in ﬁgures 3 and 4.

Original: Conclusion
Revised: As expected, the unimodal models perform relatively well at the SNR that they are trained at or higher, although as the training SNR decreases, the models perform worse overall, and the perfor- mance on higher SNR data decreases. This effect is particularly noticeable in the audio only model, ﬁgure Additionally, these unimodal models demonstrate that the C-LSTM architecture is at minimum capable of using information from either modality to perform the classiﬁcation task. However, the difference in performance between the audio and visual unimodal models shows that the model is not equally sensitive to each modality or to noise in each modality.

Revised: This discrepancy motivated us to choose an audio SNR of 0.5 and a visual SNR of 0.25 for the model investigation discussed above in ﬁgure 1.

Revised: Choosing these audio and visual SNR values allowed us to investigate early fusion in a setting where multimodal fusion would be beneﬁcial, as discussed above in section 3.

Revised:  Conclusion

Original: In this paper, we developed a C-LSTM architecture to investigate the effects of fusion depth on noiserobustness.
Revised: In this paper, we developed a C-LSTM architecture to investigate the effects of fusion depth on noise robustness.

Original: Motivated by the neuroscientiﬁc literature suggesting that sensory inputs are combinedearly in processing, we proposed that truly immediate fusion of modalities would provide beneﬁtsin neural networks as well.
Revised: Motivated by the neuroscientiﬁc literature suggesting that sensory inputs are combined early in processing, we proposed that truly immediate fusion of modalities would provide beneﬁts in neural networks as well.

Original: Our immediate fusion model demonstrates robustness to changes ininput noise and an improvement in accuracy relative to late fusion models with analogous architectures.
Revised: Our initial layer fusion model demonstrates robustness to changes in input noise and an improvement in accuracy relative to late fusion models with analogous architectures.

Original: Future directions include investigating the effects of immediate multimodal fusion in deepernetworks and on problems with more inherent difﬁculty, as well as the extension of immediate multimodal processing to other multimodal domains such as translation, alignment, and co-learning.
Revised: Future directions include investigating the effects of immediate multimodal fusion in deeper networks and on problems with more inherent difﬁculty, as well as the extension of immediate multimodal processing to other multimodal domains such as translation, alignment, and co-learning.

Original: However, we believe that the results demonstrated here and in A.6 show the importance of trulyimmediate fusion and could help with other domain speciﬁc multimodal learning tasks.
Revised: However, we believe that the results demonstrated here and in A.5 show the importance of truly immediate fusion and could help with other domain speciﬁc multimodal learning tasks.

Original: A Appendix 
Revised: A Appendix A.1 Multimodal Convolutional LSTM Model

Original: A.1 Multimodal Convolutional LSTM Model

Original: At each point of convolution, the ﬁrst layer takes:
Revised: At each point of convolution the ﬁrst layer takes:

Original: • The section of the input image to be multiplied by our convolutional kernel, denoted as v .•
Revised: • The section of the input image to be multiplied by our convolutional kernel, denoted as v .

Original:  The section of the hidden state to be multiplied by our convolutional kernel , denoted as h t − It
Revised: • The section of the hidden state to be multiplied by our convolutional kernel, denoted as h t − 1 .

Original: is

Original: initialized

Original: to

Original:  zeros for the ﬁrst time step in 
Revised: It is initialized to zeros for the ﬁrst time step in the ﬁrst layer.

Original: the ﬁrst layer. • The spectrogram value of the audio input, at a given time step, denoted as a t .
Revised: • The spectrogram value of the audio input, at a given time step, denoted as a t .

Original: Our model then computes the LSTM gate values using: v , h t − 1 , and a t via the equations in Figure3, under the Initial LSTM Gate Values section.
Revised: Our model then computes the LSTM gate values using: v , h t − 1 , and a t via the equations in Figure 5, under the Initial LSTM Gate Values section.

Original: The initial C-LSTM layer produces a single multimodal tensor that combines information from the audio and visual inputs.
Revised: The initial C-LSTM layer produces a single multimodal tensor that combines information from the audio and visual inputs.

Original: As in the standard LSTMarchitecture, the hidden state, h t , of the previous layer is used as the input, x t , of the current layer.
Revised: As in the standard LSTM architecture, the hidden state, h t , of the previous layer is used as the input, x t , of the current layer.

Original: Therefore, in our subsequent LSTM layers the gate values at each location are computed from thesection of the combined multimodal input, x t , to be multiplied by our convolutional kernel.
Revised: Therefore, in our subsequent LSTM layers the gate values at each location are computed from the section of the combined multimodal input, x t , to be multiplied by our convolutional kernel.

Original: Equations found in Figure 3, under the Subsequent LSTM Gate Values section.
Revised: Equations found in Figure 5, under the Subsequent LSTM Gate Values section.

Original: By applying the LSTMoperations at each location of a convolution, this architecture allows the LSTM cells to respondto the spatial information from the visual domain as well as the temporal information of the audiodomain.
Revised: By applying the LSTM operations at each location of a convolution, this architecture allows the LSTM cells to respond to the spatial information from the visual domain as well as the temporal information of the audio domain.

Original: This architecture enables us to study the mixing of signals at the initial, second, and fullyconnected layer while maintaining the same inductive biases that are beneﬁcial in processing imageand sequence data.
Revised: This architecture enables us to study the mixing of signals at the initial, second, and fully connected layer while maintaining the same inductive biases that are beneﬁcial in processing image and sequence data.

Original: Varying the Fusion Level.
Revised: Varying the Fusion Level.

Original: The full C-LSTM approach described above performs early fusion, i.e.,mixing the modalities starting in the very ﬁrst layer.
Revised: The full C-LSTM approach described above performs early fusion, i.e., mixing the modalities starting in the very ﬁrst layer.

Original: However, our framework can be easily modiﬁedto only allow for fusion in later layers, or to mask out one modality all together.
Revised: However, our framework can be easily modiﬁed to only allow for fusion in later layers, or to mask out one modality all together.

Original: A.2 Training & Modeling Details
Revised: A.2 Training & Modeling Details

Original: For the full model, we used: an initial merge layer (64 units and 3 × 3 kernels, and a 2 × 2 max poollayer), a second multimodal C-LSTM layer (64 units and 3 × 3 kernels, and a 2 × 2 max pool layer),a dense layer (128 units and ReLU activation), and then a ﬁnal dense output layer (10 units).
Revised: For the full model, we used: an initial merge layer (64 units and 3 × 3 kernels, and a 2 × 2 max pool layer), a second multimodal C-LSTM layer (64 units and 3 × 3 kernels, and a 2 × 2 max pool layer), a dense layer (128 units and ReLU activation), and then a ﬁnal dense output layer (10 units).

Original: The second layer fusion model consists of a separate convolutional layer with 64 units and 3 × 3kernels and an LSTM layer with 64 units.
Revised: The second layer fusion model consists of a separate convolutional layer with 64 units and 3 ×kernels and an LSTM layer with 64 units.

Original: These layers feed into a C-LSTM layer, with 64 units and
Revised: These layers feed into a C-LSTM layer, with 64 units and

Original: Initial LSTM Gate Values: f
Revised: Initial LSTM Gate Values: f

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: o t =σ
Revised: o [Equation] =

Original: [Equation]
Revised: [Equation]

Original: o [ h t−
Revised: o [ h t

Original: [Equation]
Revised: [Equation]

Original: xt
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Cell State Update: c t = f t c t − 1 + i t g t
Revised: Cell State Update: c t = f t c t − 1 + i t g t

Original: Hidden State Update: h t =
Revised: Hidden State Update: h t =

Original: o t tanh( c t )
Revised: o t tanh( c t )

Original: Subsequent LSTM Gate Values:
Revised: Subsequent LSTM Gate Values:

Original: [Equation]
Revised: [Equation]

Original: The fully connected layer fusion model consists of separate processing streams for the visual andaudio data.
Revised: The fully connected layer fusion model consists of separate processing streams for the visual and audio data.

Original: The visual stream consists of two convolutional layers with 64 units and 3 × 3 kernels,while the audio stream consisting of two LSTM layers with 64 units.
Revised: The visual stream consists of two convolutional layers with 64 units and 3 × 3 kernels, while the audio stream consisting of two LSTM layers with 64 units.

Original: The output of the convolutionallayers and the last timestep of the output of the LSTM layers are concatenated and fed into a denselayer with 128 units and ReLU activation, then a ﬁnal dense output layer with 10 units.
Revised: The output of the convolutional layers and the last timestep of the output of the LSTM layers are concatenated and fed into a dense layer with 128 units and ReLU activation, then a ﬁnal dense output layer with 10 units.

Original: We trained all models using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.001.
Revised: We trained all models using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.001.

Original: Each model is trained on 87516 examples randomly selected from our multimodal dataset (which intotal contains 11202076 training data points, 174490 validation data points, 175389 test data points).
Revised: Each model is trained on 87516 examples randomly selected from our multimodal dataset (which in total contains 11202076 training data points, 174490 validation data points, 175389 test data points).

Original: The models can be trained on any combination of audio and visual SNRs.
Revised: The models can be trained on any combination of audio and visual SNRs.

Original: A.3 Comparison to Unimodal Models

Original: To verify that joint audio visual representations are a result of both modalities, we tested our multimodal model on unimodal inputs by setting one input to zero.

Original: This created unimodal visual modelsand unimodal audio models without changing the underlying architecture.

Original: For each of these unimodal models, we trained the model at four SNR values, and tested the accuracy across our previously selected set of signal to noise ratios.

Original: The accuracy of these unimodal models for the SNRvalues is displayed in ﬁgures 4 and 5.

Original: [Figure]

Original: t A u d i o

Original: [Figure]

Original: N

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: t

Original: [Figure]

Original: i

Original: [Figure]

Original: As expected, the unimodal models perform relatively well at the SNR that they are trained at orhigher, although as the training SNR decreases, the models perform worse overall, and the performance on higher SNR data decreases.

Original: This effect is particularly noticeable in the audio only model,ﬁgure 5.

Original: Additionally, these unimodal models prove that the C-LSTM architecture is at minimum capable ofusing information from either modality to perform the classiﬁcation task.

Original: However, the differencein performance between the audio and visual unimodal models shows that the model is not equallysensitive to each modality or to noise in each modality.

Original: This discrepancy motivated us to choosean audio SNR of 0.5 and a visual SNR of 0.25 for further investigation of the model architecture,such that the contribution of each modality will be comparable and will allow us to investigate earlyfusion in a setting where multimodal fusion will be beneﬁcial.

Original: A.4 Input data
Revised: A.3 Input data

Original: For each of these datasets, we ﬁrst combined all the data, testing and training, into one dataset, thensplit each of the two datasets into training, testing, and validation sets with an 8 : 1 : 1 ratio.
Revised: For each of these datasets, we ﬁrst combined all the data, testing and training, into one dataset, then split each of the two datasets into training, testing, and validation sets with an 8 : 1 : 1 ratio.

Original: Next,for each of the training, testing and validation splits, we created a dataset containing all image andaudio pairs with the same label.
Revised: Next, for each of the training, testing and validation splits, we created a dataset containing all image and audio pairs with the same label.

Original: We then augmented the data by adding white noise to the image andaudio data such that the signal to noise ratio could be chosen, as shown in Figure 6.
Revised: We then augmented the data by adding white noise to the image and audio data such that the signal to noise ratio could be chosen, as shown in Figure 6.

Original: This allows us toexplore our model’s response to degradation in either the image or audio input, while utilizing noisetechniques commonly applied to inputs (Borji & Lin, 2020).
Revised: This allows us to explore our model’s response to degradation in either the image or audio input, while utilizing noise techniques commonly applied to inputs (Borji & Lin, 2020).

Original: We then front padded each audio inputwith zeros such that all the audio examples are of equal length.
Revised: We then front padded each audio input with zeros such that all the audio examples are of equal length.

Original: Finally, we took the spectrogram ofeach audio input, with 400 samples, 201 frequency bins, and a stride of 200 samples. I m a g e Original SNR:16.
Revised: Finally, we took the spectrogram of each audio input, with 400 samples, 201 frequency bins

Original: SNR:8. SNR:4. SNR:2. SNR:1.
Revised: [Figure]

Original: SNR:0. SNR:0. SNR:0. SNR:0.
Revised: and a stride of 200 samples.

Original: [Figure]
Revised: [Figure]

Original: A.5 Late 
Revised: A.4 Late Fusion

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A.6 Model Inspection
Revised: A.5 Model Inspection

Original: In order to examine the contribution of audio and visual inputs to the performance of our multimodalclassiﬁer, we considered the state of the network at intermediate timesteps in the recurrent processingof the audio inputs.
Revised: In order to examine the contribution of audio and visual inputs to the performance of our multimodal classiﬁer [Figure] we considered the state of the network at intermediate timesteps in the recurrent processing of the audio inputs.

Original: In ﬁgure 11 we display the activations of the ﬁnal layer of the network acrosstimesteps for a single representative example at various signal to noise ratios.
Revised: In ﬁgure 11 we display the activations of the ﬁnal layer of the network across timesteps for a single representative example at various signal to noise ratios.

Original: These ﬁnal layeractivations correspond to the classiﬁcation of the input into each of the ten digit classes.
Revised: These ﬁnal layer activations correspond to the classiﬁcation of the input into each of the ten digit classes.

Original: We displaythe activations in response to four SNR scenarios: the original input, a scenario where the audio
Revised: We display the activations in response to four SNR scenarios: the original input, a scenario where the audio

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: This visualization demonstrates the value of the multimodal input in this network.
Revised: This visualization demonstrates the value of the multimodal input in this network.

Original: Because theimage information is available to the network for the entire length of the audio, the network initiallyresponds to the image without audio input, because the audio is front zero-padded.
Revised: Because the image information is available to the network for the entire length of the audio, the network initially responds to the image without audio input, because the audio is front zero-padded.

Original: As the networkis evaluated along the time dimension of the audio input, information from the audio input becomesavailable to the network.
Revised: As the network is evaluated along the time dimension of the audio input, information from the audio input becomes available to the network.

Original: Therefore, comparing the activations of the ﬁnal layer can illuminate thecontribution of each modality to the network.
Revised: Therefore, comparing the activations of the ﬁnal layer can illuminate the contribution of each modality to the network.

Original: In this example, the original input demonstrates the capability of the same model to correctly classifythe digit in both the visual and audio regime, because both before and during the audio input, thecorrect class, 3, is assigned the maximum value.
Revised: In this example, the original input demonstrates the capability of the same model to correctly classify the digit in both the visual and audio regime, because both before and during the audio input, the correct class, 3, is assigned the maximum value.

Original: In the example where the audio input has an SNRof 1.0 and the visual input has an SNR of 0.25, as well as the example with audio SNR of 0.5 andvisual SNR of 0.5, the contribution of the audio input becomes more evident, as in both of thesescenarios, before the audio input is available the model shows more confusion, both with all theother classes, and in particular with class 6.
Revised: In the example where the audio input has an SNR of 1.0 and the visual input has an SNR of 0.25, as well as the example with audio SNR of 0.5 and visual SNR of 0.5, the contribution of the audio input becomes more evident, as in both of these scenarios, before the audio input is available the model shows more confusion, both with all the other classes, and in particular with class 6.

Original: In these examples, as the audio information becomesavailable to the network, the correct class comes to dominate the activations.
Revised: In these examples, as the audio information becomes available to the network, the correct class comes to dominate the activations.

Original: Additional examples of ﬁnal layer activations.
Revised: Additional examples of ﬁnal layer activations.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]