Original: L EARNING M ULTIMODAL D ATA A UGMENTATION IN F EATURE S PACE
Revised: L EARNING M ULTIMODAL D ATA A UGMENTATION IN F EATURE S PACE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The ability to jointly learn from multiple modalities, such as text, audio, and visual data, is a defining feature of intelligent systems.
Revised: The ability to jointly learn from multiple modalities, such as text, audio, and visual data, is a defining feature of intelligent systems.

Original: While there have been promising advances in designing neural networks to harness multimodal data, the enormous success of data augmentation currently remains limited to single-modality tasks like image classification.
Revised: While there have been promising advances in designing neural networks to harness multimodal data, the enormous success of data augmentation currently remains limited to single-modality tasks like image classification.

Original: Indeed, it is particularly difficult to augment each modality while preserving the overall semantic structure of the data; for example, a caption may no longer be a good description of an image after standard augmentations have been applied, such as translation.
Revised: Indeed, it is particularly difficult to augment each modality while preserving the overall semantic structure of the data; for example, a caption may no longer be a good description of an image after standard augmentations have been applied, such as translation.

Original: Moreover, it is challenging to specify reasonable transformations that are not tailored to a particular modality.
Revised: Moreover, it is challenging to specify reasonable transformations that are not tailored to a particular modality.

Original: In this paper, we introduce LeMDA, Learning Multimodal Data Augmentation , an easy-to-use method that automatically learns to jointly augment multimodal data in feature space, with no constraints on the identities of the modalities or the relationship between modalities.
Revised: In this paper, we introduce LeMDA, Learning Multimodal Data Augmentation , an easy-to-use method that automatically learns to jointly augment multimodal data in feature space, with no constraints on the identities of the modalities or the relationship between modalities.

Original: We show that LeMDA can (1) profoundly improve the performance of multimodal deep learning architectures, (2) apply to combinations of modalities that have not been previously considered, and (3) achieve state-of-the-art results on a wide range of applications comprised of image, text, and tabular data.
Revised: We show that LeMDA can (1) profoundly improve the performance of multimodal deep learning architectures, (2) apply to combinations of modalities that have not been previously considered, and (3) achieve state-of-the-art results on a wide range of applications comprised of image, text, and tabular data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Imagine watching a film with no sound, or subtitles.
Revised: Imagine watching a film with no sound, or subtitles.

Original: Our ability to learn is greatly enhanced through jointly processing multiple data modalities, such as visual stimuli, language, and audio.
Revised: Our ability to learn is greatly enhanced through jointly processing multiple data modalities, such as visual stimuli, language, and audio.

Original: These information sources are often so entangled that it would be near impossible to learn from only one modality in isolation — a significant constraint on traditional machine learning approaches.
Revised: These information sources are often so entangled that it would be near impossible to learn from only one modality in isolation — a significant constraint on traditional machine learning approaches.

Original: Accordingly, there have been substantial research efforts in recent years on developing multimodal deep learning to jointly process and interpret information from different modalities at once (Baltruˇsaitis et al., 2017).
Revised: Accordingly, there have been substantial research efforts in recent years on developing multimodal deep learning to jointly process and interpret information from different modalities at once (Baltruˇsaitis et al., 2017).

Original: Researchers studied multimodal deep learning from various perspectives such as model architectures (Kim et al., 2021b; P ´ erez-R ´ ua et al., 2019; Nagrani et al., 2021; Choi & Lee, 2019), training techniques (Li et al., 2021;
Revised: Researchers studied multimodal deep learning from various perspectives such as model architectures (Kim et al., 2021b; P ´ erez-R ´ ua et al., 2019; Nagrani et al., 2021; Choi & Lee, 2019), training techniques (Li et al., 2021;

Original: Chen et al., 2019a), and theoretical analysis (Huang et al., 2021; Sun et al., 2020b).
Revised: Chen et al., 2019a), and theoretical analysis (Huang et al., 2021; Sun et al., 2020b).

Original: However, data augmentation for multimodal learning remains relatively unexplored (Kim et al., 2021a), despite its enormous practical impact in single modality settings.
Revised: However, data augmentation for multimodal learning remains relatively unexplored (Kim et al., 2021a), despite its enormous practical impact in single modality settings.

Original: Indeed, data augmentation has particularly proven its value for data efficiency, regularization, and improved performance in computer vision (Ho et al., 2019; Cubuk et al., 2020; M¨uller & Hutter, 2021; Zhang et al., 2017; Yun et al., 2019) and natural language processing (Wei & Zou, 2019;
Revised: Indeed, data augmentation has particularly proven its value for data efficiency, regularization, and improved performance in computer vision (Ho et al., 2019; Cubuk et al., 2020; M¨uller & Hutter, 2021; Zhang et al., 2017; Yun et al., 2019) and natural language processing (Wei & Zou, 2019;

Original: Karimi et al., 2021; Fadaee et al., 2017; Sennrich et al., 2015; Wang & Yang, 2015; Andreas, 2020;
Revised: Karimi et al., 2021; Fadaee et al., 2017; Sennrich et al., 2015; Wang & Yang, 2015; Andreas, 2020;

Original: Kobayashi, 2018).
Revised: Kobayashi, 2018).

Original: These augmentation methods are largely tailored to a particular modality in isolation.
Revised: These augmentation methods are largely tailored to a particular modality in isolation.

Original: For example, for object classification in vision, we know certain transformations such as translations or rotations should leave the class label unchanged.
Revised: For example, for object classification in vision, we know certain transformations such as translations or rotations should leave the class label unchanged.

Original: Similarly, in language, certain sentence manipulations like synonym replacement will leave the meaning unchanged.
Revised: Similarly, in language, certain sentence manipulations like synonym replacement will leave the meaning unchanged.

Original: The most immediate way of leveraging data augmentation in multimodal deep learning is to separately apply well-developed unimodal augmentation strategies to each corresponding modality.
Revised: The most immediate way of leveraging data augmentation in multimodal deep learning is to separately apply well-developed unimodal augmentation strategies to each corresponding modality.

Original: However, this approach can be problematic because transforming one modality in isolation may lead to disharmony with the others.
Revised: However, this approach can be problematic because transforming one modality in isolation may lead to disharmony with the others.

Original: Consider Figure 1, which provides four training examples from
Revised: Consider Figure 1, which provides four training examples from

Original: SNLI-VE (Xie et al., 2019a), a vision-language benchmark dataset.
Revised: SNLI-VE (Xie et al., 2019a), a vision-language benchmark dataset.

Original: Each description is paired with the image on the top left, and the label refers to the relationship between the image and description.
Revised: Each description is paired with the image on the top left, and the label refers to the relationship between the image and description.

Original: A u t o A u g m e n t _ C I
Revised: A u t o A u g m e n t _ C I

Original: F
Revised: F

Original: A
Revised: A

Original: R
Revised: R

Original: [Figure]
Revised: [Figure]

Original: image
Revised: image

Original: [Figure]
Revised: [Figure]

Original: plane is doing tricks while flying down
Revised: plane is doing tricks while flying down

Original: [Figure]
Revised: [Figure]

Original: There is a plane in the air
Revised: There is a plane in the air

Original: [Figure]
Revised: [Figure]

Original: The pilot is aware that the plane is doing a loop.
Revised: The pilot is aware that the plane is doing a loop.

Original: [Figure]
Revised: [Figure]

Original: The plane is falling down.
Revised: The plane is falling down.

Original: [Figure]
Revised: [Figure]

Original: method that randomly chooses one transformation with a random magnitude, the loop is cropped out, which leads to mislabeling for data (a) and (c).
Revised: method that randomly chooses one transformation with a random magnitude, the loop is cropped out, which leads to mislabeling for data (a) and (c).

Original: Mislabeling can be especially problematic for over-parameterized neural networks, which tend to confidently fit mislabeled data, leading to poor performance (Pleiss et al., 2020)
Revised: Mislabeling can be especially problematic for over-parameterized neural networks, which tend to confidently fit mislabeled data, leading to poor performance (Pleiss et al., 2020)

Original: There are two key challenges in designing a general approach to multimodal data augmentation.
Revised: There are two key challenges in designing a general approach to multimodal data augmentation.

Original: First, multimodal deep learning takes input from a diverse set of modalities.
Revised: First, multimodal deep learning takes input from a diverse set of modalities.

Original: Augmentation transformations can be obvious for some modalities such as vision and language, but not others, such as sensory data which are often numeric or categorical.
Revised: Augmentation transformations can be obvious for some modalities such as vision and language, but not others, such as sensory data which are often numeric or categorical.

Original: Second, multimodal deep learning includes a diverse set of tasks with different cross-modal relationships.
Revised: Second, multimodal deep learning includes a diverse set of tasks with different cross-modal relationships.

Original: Some datasets have redundant or totally correlated modalities while others have complementary modalities.
Revised: Some datasets have redundant or totally correlated modalities while others have complementary modalities.

Original: There is no reasonable assumption that would generally preserve labels when augmenting modalities in isolation.
Revised: There is no reasonable assumption that would generally preserve labels when augmenting modalities in isolation.

Original: In this work, we propose LeMDA
Revised: In this work, we propose LeMDA

Original: Le arning Multimodal Data Augmentation ) as a general multimodal data augmentation method.
Revised: Le arning Multimodal Data Augmentation ) as a general multimodal data augmentation method.

Original: LeMDA augments the latent representation and thus can be applied to any modalities.
Revised: LeMDA augments the latent representation and thus can be applied to any modalities.

Original: We design the augmentation transformation as a learnable module such that it is adaptive to various multimodal tasks and cross-modal relationships.
Revised: We design the augmentation transformation as a learnable module such that it is adaptive to various multimodal tasks and cross-modal relationships.

Original: Our augmentation module is learned together with multimodal networks to produce informative data through adversarial training, while preserving semantic structure through consistency regularization.
Revised: Our augmentation module is learned together with multimodal networks to produce informative data through adversarial training, while preserving semantic structure through consistency regularization.

Original: With no constraints over the modalities and tasks, one can simply plug-and-play LeMDA with different multimodal architectures.
Revised: With no constraints over the modalities and tasks, one can simply plug-and-play LeMDA with different multimodal architectures.

Original: We summarize our contributions as follows
Revised: We summarize our contributions as follows

Original: • In Section 3, we introduce LeMDA, a novel approach to multimodal data augmentation.
Revised: • In Section 3, we introduce LeMDA, a novel approach to multimodal data augmentation.

Original: Section 3.1 shows how to use LeMDA with multimodal networks, and Section 3.2 describes how to train the augmentation module to produce informative and label preserving data.
Revised: Section 3.1 shows how to use LeMDA with multimodal networks, and Section 3.2 describes how to train the augmentation module to produce informative and label preserving data.

Original: The method is notable for several reasons: (1) it can be applied to any modality combinations; (2) it is attractively simple and easy-to-use; (3) it is the first
Revised: The method is notable for several reasons: (1) it can be applied to any modality combinations; (2) it is attractively simple and easy-to-use; (3) it is the first

Original: augmentation method to be applied to the joint of text, image, and tabular data, which is essentially uncharted territory.
Revised: augmentation method to be applied to the joint of text, image, and tabular data, which is essentially uncharted territory.

Original: • In Section 4, we show that LeMDA consistently boosts accuracy for multimodal deep learning architectures compared to a variety of baselines, including state-of-the-art input augmentation and feature augmentation methods.
Revised: • In Section 4, we show that LeMDA consistently boosts accuracy for multimodal deep learning architectures compared to a variety of baselines, including state-of-the-art input augmentation and feature augmentation methods.

Original: • In Section 4.4, we provide an ablation study validating the design choices behind LeMDA.
Revised: • In Section 4.4, we provide an ablation study validating the design choices behind LeMDA.

Original: In particular, we study the architecture of the augmentation module, and the effects of consistency regularizer.
Revised: In particular, we study the architecture of the augmentation module, and the effects of consistency regularizer.

Original: We demonstrate that the consistency regularizer clearly outperforms L 2 regularization (Tang et al., 2020b).
Revised: We demonstrate that the consistency regularizer clearly outperforms L 2 regularization (Tang et al., 2020b).

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: Multimodal network architectures.
Revised: Multimodal network architectures.

Original: Multimodal deep learning architectures are categorized as performing early or late fusion, depending on the stage of combining information from each modality.
Revised: Multimodal deep learning architectures are categorized as performing early or late fusion, depending on the stage of combining information from each modality.

Original: In early fusion , the network combines the raw input or token embedding from all the modalities.
Revised: In early fusion , the network combines the raw input or token embedding from all the modalities.

Original: Early fusion architectures can be designed to exploit the interaction between low-level features, making it a good choice for multimodal tasks with strong cross-modal correlations.
Revised: Early fusion architectures can be designed to exploit the interaction between low-level features, making it a good choice for multimodal tasks with strong cross-modal correlations (Barnum et al., 2020; Gao et al., 2020).

Revised: For example, there exist low-level correspondence in image captioning task because different words in the caption may relate to different objects in the image.

Original: We note that feature-space augmentation procedures are typically computationally intractable on early-fusion architectures, because early fusion would require combining a large number of latent features, such as a long sequence of token embeddings.
Revised: We note that feature-space augmentation procedures are typically computationally intractable on early-fusion architectures, because early fusion would require combining a large number of latent features, such as a long sequence of token embeddings.

Original: On the other hand, in late fusion , the focus of our work, input from each modality is independently processed by different backbones.
Revised: On the other hand, in late fusion , the focus of our work, input from each modality is independently processed by different backbones.

Original: The representations provided by different backbones are fused together in later layers, often just before the classifier layer.
Revised: The representations provided by different backbones are fused together in later layers, often just before the classifier layer (Shi et al., 2021a; Wang et al., 2017;

Revised: Sch ¨ onfeld et al., 2019; Mahajan et al., 2020).

Original: This design is straightforward to apply to any new modality and any multimodal task.
Revised: This design is straightforward to apply to any new modality and any multimodal task.

Original: Late fusion often uses pre-trained networks as backbones in each modality, making it more computationally tractable.
Revised: Late fusion often uses pre-trained networks as backbones in each modality, making it more computationally tractable.

Original: In both early and late fusion, there are a variety of methods to fuse information.
Revised: In both early and late fusion, there are a variety of methods to fuse information.

Original: Standard approaches include (1) feed all modalities as token embedding into the network, (2) perform crossattention between modalities, (3) concatenate representations from all modalities, and (4) combine the predictions from each modality in an ensemble (Baltru ˇ saitis et al., 2017).
Revised: Standard approaches include (1) feed all modalities as token embedding into the network, (2) perform cross-attention between modalities, (3) concatenate representations from all modalities, and (4) combine the predictions from each modality in an ensemble (Baltruˇsaitis et al., 2017).

Original: Researchers usually design the multimodal network by considering the task objective, the amount of data available, and the computation budget (Shi et al., 2021b; Chen et al., 2019b; Li et al., 2022; Tsai et al., 2019).
Revised: Researchers usually design the multimodal network by considering the task objective, the amount of data available, and the computation budget (Shi et al., 2021b; Chen et al., 2019b; Li et al., 2022; Tsai et al., 2019; Mahajan & Roth, 2020).

Original: Baltruˇsaitis et al.
Revised: Baltru ˇ saitis et al.

Original: (2017) provides further readings.
Revised: (2017) provides further readings.

Original: Data augmentation for single modality tasks.
Revised: Data augmentation for single modality tasks.

Original: Data augmentation is widely adopted in vision and natural language tasks.
Revised: Data augmentation is widely adopted in vision and natural language tasks.

Original: In vision, we can manually intervene on a per-task basis to apply transformations that should leave our label invariant — e.g., translations, rotations, flipping, cropping, and color adjustments.
Revised: In vision, we can manually intervene on a per-task basis to apply transformations that should leave our label invariant — e.g., translations, rotations, flipping, cropping, and color adjustments.

Original: A transformation on one task may not be suitable for another: for example, flipping may be reasonable on CIFAR-10, but would lose semantic information on MNIST, because a flipped ‘6’ becomes a ‘9’.
Revised: A transformation on one task may not be suitable for another: for example, flipping may be reasonable on CIFAR-10, but would lose semantic information on MNIST, because a flipped ‘6’ becomes a ‘9’.

Original: Accordingly, there are a variety of works for automatic augmentations in vision, including neural architecture search (Ho et al., 2019; Cubuk et al., 2020), reinforcement learning (Cubuk et al., 2019), generative modelling (Ratner et al., 2017), mixing aspects of the existing data (Zhang et al., 2017; Yun et al., 2019), and adversarial training for informative examples (Fawzi et al., 2016;
Revised: Accordingly, there are a variety of works for automatic augmentations in vision, including neural architecture search (Ho et al., 2019; Cubuk et al., 2020), reinforcement learning (Cubuk et al., 2019), generative modelling (Ratner et al., 2017), mixing aspects of the existing data (Zhang et al., 2017; Yun et al., 2019), and adversarial training for informative examples (Fawzi et al., 2016;

Original: Goodfellow et al., 2015; Zhang et al., 2019; Suzuki, 2022; Tang et al., 2020b) .
Revised: Goodfellow et al., 2015; Zhang et al., 2019; Suzuki, 2022; Tang et al., 2020b; Tsai et al., 2017; ? ) .

Original: Similarly, in natural language processing there are a variety of standard interventions (replacement, deletion, swapping) (Wei & Zou, 2019; Karimi et al., 2021; Fadaee et al., 2017), and more automatic approaches such as back-translation (Sennrich et al., 2015), context augmentation (Wang & Yang, 2015; Andreas, 2020; Kobayashi, 2018), and linear interpolation of training data (Sun et al., 2020a).
Revised: Similarly, in natural language processing there are a variety of standard interventions (replacement, deletion, swapping) (Wei & Zou, 2019; Karimi et al., 2021; Fadaee et al., 2017), and more automatic approaches such as back-translation (Sennrich et al., 2015), context augmentation (Wang & Yang, 2015; Andreas, 2020; Kobayashi, 2018), and linear interpolation of training data (Sun et al., 2020a).

Original: Data augmentation is less explored for tabular data, but techniques in vision, such as mixup (Zhang et al., 2017) and adversarial training (Goodfellow et al., 2015) have recently been adapted to the tabular setting with promising results (Kadra et al., 2021).
Revised: Data augmentation is less explored for tabular data, but techniques in vision, such as mixup (Zhang et al., 2017) and adversarial training (Goodfellow et al., 2015) have recently been adapted to the tabular setting with promising results (Kadra et al., 2021).

Original: Latent space augmentation is much less explored than input augmentation, as it is less obvious what transformations to apply.
Revised: Latent space augmentation is much less explored than input augmentation, as it is less obvious what transformations to apply.

Original: To augment latent vectors produced by passing data inputs through a neural network ( feature space augmentation), researchers have considered interpolation, extrapolation, noise addition, and generative models (Verma et al., 2019; Liu et al., 2018; Kumar et al., 2019).
Revised: To augment latent vectors produced by passing data inputs through a neural network ( feature space augmentation), researchers have considered interpolation, extrapolation, noise addition, and generative models (Verma et al., 2019; Liu et al., 2018; Kumar et al., 2019).

Original: Multimodal data augmentation.
Revised: Multimodal data augmentation.

Original: There are a small number of works considering multimodal data augmentation, primarily focusing on vision-text tasks.
Revised: There are a small number of works considering multimodal data augmentation, primarily focusing on vision-text tasks.

Original: In visual question answering, Tang et al.
Revised: In visual question answering, Tang et al.

Original: (2020a) proposes to generate semantic similar data by applying back-translation on the text  and adversarial noise on the image.
Revised: (2020a) proposes to generate semantic similar data by applying back-translation on the text [Figure] 2, and the design of our augmentation network in Section 3.3. and adversarial noise on the image.

Original: Wang et al.
Revised: Wang et al.

Original: (2021) generates text based on images using a variational autoencoder.
Revised: (2021) generates text based on images using a variational autoencoder.

Original: In cross-modal retrieval, Gur et al.
Revised: In cross-modal retrieval, Gur et al.

Original: (2021) query similar data from external[Figure] design of our augmentation network in Section 3.3. knowledge sources for cross-modal retrieval tasks.
Revised: (2021) query similar data from external knowledge sources for cross-modal retrieval tasks.

Original: The state-of-the-art augmentation procedure for visual-language representation learning generates new image-text pairs by interpolating between images and concatenating texts in a method called MixGen (Hao et al., 2022).
Revised: The state-of-the-art augmentation procedure for visual-language representation learning generates new image-text pairs by interpolating between images and concatenating texts in a method called MixGen (Hao et al., 2022).

Original: All prior work on multimodal data augmentation relies on tailored modality-specific transformations.
Revised: All prior work on multimodal data augmentation relies on tailored modality-specific transformations.

Original: By contrast, our proposed approach is fully automatic and can be applied to any arbitrary modality.
Revised: By contrast, our proposed approach is fully automatic and can be applied to any arbitrary modality.

Original: Indeed, for the first time, we consider augmentation jointly over the tabular, image, and language modalities.
Revised: Indeed, for the first time, we consider augmentation jointly over the tabular, image, and language modalities.

Original: Moreover, even for image-text specific problems, we show that our approach outperforms MixGen, the state-of-the-art tailored approach.
Revised: Moreover, even for image-text specific problems, we show that our approach outperforms MixGen, the state-of-the-art tailored approach.

Original:  L E MDA: L EARNING M ULTIMODAL D ATA A UGMENTATION
Revised:  L E MDA: L EARNING M ULTIMODAL D ATA A UGMENTATION

Original: We now introduce LeMDA, a simple and automatic approach to multi-modal data augmentation.
Revised: We now introduce LeMDA, a simple and automatic approach to multi-modal data augmentation.

Original: LeMDA learns an augmentation network G , along with the multimodal task network F to generate informative data that preserves semantic structure.
Revised: LeMDA learns an augmentation network G , along with the multimodal task network F to generate informative data that preserves semantic structure.

Original: In Sections 3.1 and 3.2 we describe how we learn the parameters the augmentation and task networks, respectively.
Revised: In Sections 3.1 and 3.2 we describe how we learn the parameters the augmentation and task networks, respectively.

Original: We summarize the training algorithm for LeMDA in Figure 2 and Algorithm 1.
Revised: We summarize the training algorithm for LeMDA in Figure 2 and Algorithm 1.

Original: In Section 3.4 we provide intuition for the consistency loss.
Revised: In Section 3.4 we provide intuition for the consistency loss.

Original: Finally, in Section 3.3 we describe how we design the augmentation network.
Revised: Finally, in Section 3.3 we describe how we design the augmentation network.

Original: Algorithm Input: Task network before fusion F before ; Task network after fusion F after ; Augmentation network G ; Training set X ; Task loss function L ; Consistency loss L consist ; while F not converged do
Revised: Input: Task network before fusion F before ; Task network after fusion F after ; Augmentation network G ; Training set X ; Task loss function L ; Consistency loss L consist ; while F not converged do

Original: Sample a mini-batch from X
Revised: Sample a mini-batch from X

Original: Compute z ← F before ( x )
Revised: Compute z ← F before ( x )

Original: Generate augment feature G ( z ) ˆ y ← F after ( z ) , ˆ y G ← F after ( G ( z ))
Revised: Generate augment feature G ( z ) ˆ y ← F after ( z ) , ˆ y G ← F after ( G ( z ))

Original: Update the augmentation network G by stochastic gradient −∇ L (ˆ y  ) + ∇ L consist (ˆ y, ˆ y G )
Revised: Update the augmentation network G by stochastic gradient −∇ L (ˆ y G ) + ∇ L consist (ˆ y, ˆ y G )

Original: Update the task network F by stochastic gradient ∇ L (ˆ y )
Revised: Update the task network F by stochastic gradient ∇ L (ˆ y )

Original: + ∇ L (ˆ y G ) end while
Revised: + ∇ L (ˆ y G ) end while

Original: 3. T RAINING THE T ASK N ETWORK
Revised: 3. T RAINING THE T ASK N ETWORK

Original: The task network can be divided into two parts at the fusion layer F ( x ) =
Revised: The task network can be divided into two parts at the fusion layer F ( x ) =

Original: F after ( F before ( x )) where
Revised: F after ( F before ( x )) where

Original: F before denotes the layers before fusion, F after denotes the layers after the fusion.
Revised: F before denotes the layers before fusion, F after denotes the layers after the fusion.

Original: Given a training sample x , we pass x until the fusion layer and obtain the latent features for each modality { z i } Ni =1 = F before ( x ) where N is the number of modalities.
Revised: Given a training sample x , we pass x until the fusion layer and obtain the latent features for each modality { z i } Ni =1 = F before ( x ) where N is the number of modalities.

Original: Taken { z i } Ni =1 as inputs, the augmentation network G generates additional latent vectors G ( { z i } Ni =1 ) .
Revised: Taken { z i } Ni =1 as inputs, the augmentation network G generates additional latent vectors G ( { z i } Ni =1 ) .

Original: Both { z i } Ni =1 and G ( { z i } Ni =1 ) are fed through the rest of target network F after as distinct training data.
Revised: Both { z i } Ni =1 and G ( { z i } Ni =1 ) are fed through the rest of target network F after as distinct training data.

Original: Then, the task network is trained in the standard way, taking the task loss function on both original data and augmented data, to find min E x ∼X ( L (ˆ y )
Revised: Then, the task network is trained in the standard way, taking the task loss function on both original data and augmented data, to find min E x ∼X ( L (ˆ y )

Original: + L (ˆ y G ))
Revised: + L (ˆ y G ))

Original: where ˆ y = F after ( F before ( x )) and ˆ y G
Revised: where ˆ y = F after ( F before ( x )) and ˆ y G

Original: = F after ( G ( F before ( x ))) .
Revised: = F after ( G ( F before ( x ))) .

Original: 3. T RAINING THE A UGMENTATION N ETWORK
Revised: 3. T RAINING THE A UGMENTATION N ETWORK

Original: Inspired by adversarial data augmentation, we optimize parameters for the augmentation network to maximize the task loss such that the task network’s representation is encouraged to be updated by the augmented data.
Revised: Inspired by adversarial data augmentation, we optimize parameters for the augmentation network to maximize the task loss such that the task network’s representation is encouraged to be updated by the augmented data.

Original: At the same time, we introduce a consistency regularizer that encourages a similar output distribution given the original data and the augmented data to preserve the semantic structure.
Revised: At the same time, we introduce a consistency regularizer that encourages a similar output distribution given the original data and the augmented data to preserve the semantic structure.

Original: Formally, we find maxmin E x ∼X ( L (ˆ y )
Revised: Formally, we find max E x ∼X ( L (ˆ y G )) + min E x ∼X

Original: + L consist (ˆ y, ˆ y G )), where L consist (ˆ y, ˆ y G ) denotes a divergence metric between the logit outputs on original data ˆ y and on augmented data ˆ y G such as the Kullback-Leibler divergence.
Revised: ( L consist (ˆ y, ˆ y G )) where L consist (ˆ y, ˆ y G ) denotes a divergence metric between the logit outputs on original data ˆ y and on augmented data ˆ y G such as the Kullback-Leibler divergence.

Original: Confidence masking.
Revised: Confidence masking.

Original: For classification problems, we apply the consistency term only to the samples whose highest probability is greater than a threshold α .
Revised: For classification problems, we apply the consistency term only to the samples whose highest probability is greater than a threshold α .

Original: If the task network can’t make a confident prediction, it is unlikely the prediction provides a good reference to the ground truth label.
Revised: If the task network can’t make a confident prediction, it is unlikely the prediction provides a good reference to the ground truth label.

Original: Design decisions.
Revised: Design decisions.

Original: The simplicity and generality of this approach, combined with its strong empirical performance in Section 4, are LeMDA’s most appealing features.
Revised: The simplicity and generality of this approach, combined with its strong empirical performance in Section 4, are LeMDA’s most appealing features.

Original: The few design decisions for training involve how the consistency regularizer should be defined and to what extent it should be applied.
Revised: The few design decisions for training involve how the consistency regularizer should be defined and to what extent it should be applied.

Original: For example, as an alternative to a KL-based consistency regularizer, we could minimize the
Revised: For example, as an alternative to a KL-based consistency regularizer, we could minimize the

Original: L 2 distance of the augmented feature vector to the original feature vector as a proxy for preserving the label of the augmentation.
Revised: L 2 distance of the augmented feature vector to the original feature vector as a proxy for preserving the label of the augmentation.

Original: We provide ablations of these factors in Section 4.4.
Revised: We provide ablations of these factors in Section 4.4.

Original: 3. T HE D ESIGN OF A UGMENTATION N ETWORK
Revised: 3. T HE D ESIGN OF A UGMENTATION N ETWORK

Original: The augmentation network can take various forms depending on the multimodal learning tasks and the fusion strategies.
Revised: The augmentation network can take various forms depending on the multimodal learning tasks and the fusion strategies.

Original: In our experiments, we use a variational autoencoder (VAE) as the augmentation network, since VAEs have generally been found effective for augmentation purposes (Tang et al., 2020b).
Revised: In our experiments, we use a variational autoencoder (VAE) as the augmentation network, since VAEs have generally been found effective for augmentation purposes (Tang et al., 2020b).

Original: We consider two architectural choices:
Revised: We consider two architectural choices:

Original: MLP-VAE: The encoder and decoder of VAE are MLPs.
Revised: MLP-VAE: The encoder and decoder of VAE are MLPs.

Original: { z i } Ni =1 are concatenated as the input.
Revised: { z i } Ni =1 are concatenated as the input.

Original: Attention-VAE: The encoder and decoder are made of self-attention and feedforward networks.
Revised: Attention-VAE: The encoder and decoder are made of self-attention and feedforward networks.

Original: { z i } Ni =1 are treated as N tokens where each token has an embedding z i 
Revised: { z i } Ni =1 are treated as N tokens where each token has an embedding z i .

Revised: There are two loss terms in the standard VAE, the reconstruction loss, and the KL divergence regularizer.

Revised: We only adopt the KL regularizer on the encoder distribution.

Revised: The updating step for augmentation networks is −∇ L (ˆ y G )

Revised: + ∇ L consist (ˆ y, ˆ y G )

Revised: + + ∇ L VAE , where L VAE refers to the KL divergence regularizer on the latent encoding distribution.

Original: The major deciding factor between MLP-VAE and Attention-VAE is the multimodal task network architectures.
Revised: The major deciding factor between MLP-VAE and Attention-VAE is the multimodal task network architectures.

Original: With late fusion architectures, which is the primary focus of this paper, z i refers to the representation from a single modality backbone (e.g. CLS embedding from a BERT model), and N is the number of modalities or the number of backbone models.
Revised: With late fusion architectures, which is the primary focus of this paper, z i refers to the representation from a single modality backbone (e.g. CLS embedding from a BERT model), and N is the number of modalities or the number of backbone models.

Original: We can concatenate { z i } N i = as one vector input to MLP-VAE, or we can treat { z i } Ni =1 as a sequence of N tokens to AttentionVAE.
Revised: We can concatenate { z i } Ni = as one vector input to MLP-VAE, or we can treat { z i } Ni =1 as a sequence of N tokens to AttentionVAE.

Original: Attention-VAE may be less intuitive here because N is usually a small number in late fusion architectures( 2 or 3 in our experiment).
Revised: Attention-VAE may be less intuitive here because N is usually a small number in late fusion architectures( 2 or 3 in our experiment).

Original: We provide a performance comparison between these two architectures in Section 4.4.
Revised: We provide a performance comparison between these two architectures in Section 4.4.

Original: On the other hand, for early fusion architectures, z i could be a sequence of token embedding for a text or a sequence of patch embedding for an image.
Revised: On the other hand, for early fusion architectures, z i could be a sequence of token embedding for a text or a sequence of patch embedding for an image.

Original: Concatenation will result in a really high dimension input, which makes MLP-VAE less favorable.
Revised: Concatenation will result in a really high dimension input, which makes MLP-VAE less favorable.

Original: 3. I NTUITION ON W HY C ONSISTENCY R EGULARIZER D ISCOURAGES M ISLABELED D ATA
Revised: 3. I NTUITION ON W HY C ONSISTENCY R EGULARIZER D ISCOURAGES M ISLABELED D ATA

Original: D
Revised: D

Original: D
Revised: D

Original: We intuitively prefer D1, because the augmented point should be informative but preserve the same label.
Revised: We intuitively prefer D1, because the augmented point should be informative but preserve the same label.

Original: The consistency loss will prefer D1 over D2, because D2 crosses the model’s decision boundary, even though both points incur the same training loss.
Revised: The consistency loss will prefer D1 over D2, because D2 crosses the model’s decision boundary, even though both points incur the same training loss.

Original: In Figure 3 we provide intuition for the consistency regularizer using a simple illustrative binary classification.
Revised: In Figure 3 we provide intuition for the consistency regularizer using a simple illustrative binary classification.

Original: Darker background corresponds to higher task training loss, the solid green line is the actual decision boundary, and the dashed green line is the model’s decision boundary.
Revised: Darker background corresponds to higher task training loss, the solid green line is the actual decision boundary, and the dashed green line is the model’s decision boundary.

Original: Starting from a point in feature space, moving to D1 and D2 would provide a similar increase in task loss and thus are equally favored by the adversarial loss term.
Revised: Starting from a point in feature space, moving to D1 and D2 would provide a similar increase in task loss and thus are equally favored by the adversarial loss term.

Original: However, D2 crosses the model’s decision boundary, and thus would be heavily penalized by the consistency regularizer — as we would hope, since such a point is likely to have a different class label.
Revised: However, D2 crosses the model’s decision boundary, and thus would be heavily penalized by the consistency regularizer — as we would hope, since such a point is likely to have a different class label.

Original: On the other hand, an L2 regularizer between the original and augmented points in feature space would have no preference between D1 and D2, as they are an equal distance away from the starting point.
Revised: On the other hand, an L2 regularizer between the original and augmented points in feature space would have no preference between D1 and D2, as they are an equal distance away from the starting point.

Original: Empirically, in Section 4, we see the consistency loss confers accuracy improvements over both pure adversarial training and L2 regularizer.
Revised: Empirically, in Section 4, we see the consistency loss confers accuracy improvements over both pure adversarial training and L2 regularizer.

Original: Similar intuition is in Suzuki (2022), which uses the logits distribution from the teacher model (an exponential moving average over the model’s weights) as the soft target such that the augmented data is still recognizable by the teacher, and Xie et al.
Revised: Similar intuition is in Suzuki (2022), which uses the logits distribution from the teacher model (an exponential moving average over the model’s weights) as the soft target such that the augmented data is still recognizable by the teacher, and Xie et al.

Original: (2019b), which designs the unsupervised training objective to encourage similar logits for augmented data.
Revised: (2019b), which designs the unsupervised training objective to encourage similar logits for augmented data.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate LeMDA over a diverse set of real-world multimodal datasets.
Revised: We evaluate LeMDA over a diverse set of real-world multimodal datasets.

Original: We curate a list of public datasets covering image, text, numerical, and categorical inputs.
Revised: We curate a list of public datasets covering image, text, numerical, and categorical inputs.

Original: Table 1 provides a summary of the source, statistic, and modality identity.
Revised: Table 1 provides a summary of the source, statistic, and modality identity.

Original: We introduce baselines in Section 4.1, and describe experimental settings in Section 4.2 We provide the main evaluation result in Section 4.3.
Revised: We introduce baselines in Section 4.1, and describe Dataset # Train #Test Metric Image

Revised: Text

Revised: Tabular

Original: Finally, we investigate the effects of the consistency regularizer and the choices of augmentation model architecture in Section 4.4.
Revised: Hateful Memes Accuracy ✓ ✓ Food Accuracy ✓ ✓

Revised: SNLI-VE

Revised: Accuracy ✓ ✓

Revised: Petfinder

Revised: Quadratic Kappa ✓ ✓ ✓

Revised: Melbourne Airbnb

Original: [Table]
Revised: Accuracy ✓ ✓ News Channel Accuracy ✓ ✓ Wine Reviews

Original: [Table]
Revised: Accuracy ✓ ✓ Kick Starter Funding ROC-AUC ✓ ✓ experimental settings in Section 4.2 We provide the main evaluation result in Section 4.3. Finally, we investigate the effects of the consistency regularizer and the choices of augmentation model architecture in Section 4.4.

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: To the best of our knowledge [Table] there exist no general-purpose multimodal augmentation methods.
Revised: To the best of our knowledge, there exist no general-purpose multimodal augmentation methods.

Original: We compare against a diverse set of state-of-the-art data augmentation methods from vision, language, and vision-text tasks.
Revised: We compare against a diverse set of state-of-the-art data augmentation methods from vision, language, and vision-text tasks.

Original: We additionally consider baselines for feature augmentation, since LeMDA augments in the feature space.
Revised: We additionally consider baselines for feature augmentation, since LeMDA augments in the feature space.

Original: Finally, we compare with state-of-the-art multimodal augmentation methods from the vision-text tasks, although we note, unlike LeMDA, these methods are not general purpose and cannot be directly applied to our datasets that have tabular inputs.
Revised: Finally, we compare with state-of-the-art multimodal augmentation methods from the vision-text tasks, although we note, unlike LeMDA, these methods are not general purpose and cannot be directly applied to our datasets that have tabular inputs.

Original: • Input Augmentation.
Revised: • Input Augmentation.

Original: We apply state-of-the-art input augmentation independently on the data from each modality.
Revised: We apply state-of-the-art input augmentation independently on the data from each modality.

Original: For images, we use TrivialAugment (M¨uller & Hutter, 2021), a simple and effective method for image classification tasks.
Revised: For images, we use TrivialAugment (M¨uller & Hutter, 2021), a simple and effective method for image classification tasks.

Original: For text, we apply EDA (Wei & Zou, 2019) and AEDA (Karimi et al., 2021).
Revised: For text, we apply EDA (Wei & Zou, 2019) and AEDA (Karimi et al., 2021).

Original: We randomly sample one transformation from all transformations proposed in EDA and AEDA with a randomly generated magnitude.
Revised: We randomly sample one transformation from all transformations proposed in EDA and AEDA with a randomly generated magnitude.

Original: • Mixup.
Revised: • Mixup.

Original: Mixup was originally proposed to perform interpolation between two images in the training data for image classification.
Revised: Mixup was originally proposed to perform interpolation between two images in the training data for image classification.

Original: We adopt the original Mixup for images and numerical features and extend it for text and categorical features.
Revised: We adopt the original Mixup for images and numerical features and extend it for text and categorical features.

Original: Specifically, given a pair of data, we construct the mixed data as follows.
Revised: Specifically, given a pair of data, we construct the mixed data as follows.

Original: We generate a random number j uniformly between 0.0 to 1.0.
Revised: We generate a random number j uniformly between 0.0 to 1.0.

Original: If j < α , we use the first data, else we use the second data.
Revised: If j < α , we use the first data, else we use the second data.

Original: • Manifold Mixup.
Revised: • Manifold Mixup.

Original: Manifold Mixup (Verma et al., 2019) performs interpolation between hidden representations and thus can be applied to all modalities.
Revised: Manifold Mixup (Verma et al., 2019) performs interpolation between hidden representations and thus can be applied to all modalities.

Original: We applied Manifold Mixup to the exact feature in the multimodal network as LeMDA.
Revised: We applied Manifold Mixup to the exact feature in the multimodal network as LeMDA.

Original: • MixGen.
Revised: • MixGen.

Original: MixGen (Hao et al., 2022) is a state-of-the-art data augmentation designed specifically for vision-text tasks.
Revised: MixGen (Hao et al., 2022) is a state-of-the-art data augmentation designed specifically for vision-text tasks.

Original: MixGen generates new data by interpolating images and concatenating text.
Revised: MixGen generates new data by interpolating images and concatenating text.

Original: We apply MixGen to datasets only consisting of images and text.
Revised: We apply MixGen to datasets only consisting of images and text

Original: 4. E XPERIMENT S ETUP
Revised: 4. E XPERIMENT S ETUP

Original: We use Multimodal-Net (Shi et al., 2021a) for all the datasets except SNLI-VE.
Revised: We use Multimodal-Net (Shi et al., 2021a) for all the datasets except SNLI-VE.

Original: Multimodal-Net passes input from each modality through separate backbones, concatenates the representation(e.g.
Revised: Multimodal-Net passes input from each modality through separate backbones, concatenates the representation(e.g.

Original: the CLS embedding) from all backbones, and passes them through fusion MLP.
Revised: the CLS embedding) from all backbones, and passes them through fusion MLP.

Original: We use the default hyper-parameters provided by Multimodal-Net and plug LeMDA before the fusion layer.
Revised: We use the default hyper-parameters provided by Multimodal-Net and plug LeMDA before the fusion layer.

Original: We use
Revised: We use

Original: ConvNet as the image backbone and ELECTRA as the text backbone.
Revised: ConvNet as the image backbone and ELECTRA as the text backbone.

Original: To further demonstrate LeMDA’s generalizability, we evaluate LeMDA with early fusion architectures ALBEF (Li et al., 2021) on SNLI-VE.
Revised: To further demonstrate LeMDA’s generalizability, we evaluate LeMDA with early fusion architectures ALBEF (Li et al., 2021) on SNLI-VE.

Original: ALBEF performs cross-attention between image patch embeddings and text token embeddings.
Revised: ALBEF performs cross-attention between image patch embeddings and text token embeddings.

Original: We keep all original configurations except the batch size due to limitations in computation memory.
Revised: We keep all original configurations except the batch size due to limitations in computation memory.

Original: We set the batch size to be half of the default.
Revised: We set the batch size to be half of the default.

Original: We load the 4M pre-trained checkpoints.
Revised: We load the 4M pre-trained checkpoints.

Original: In this setting, we apply LeMDA before the cross-attention layer.
Revised: In this setting, we apply LeMDA before the cross-attention layer.

Original: The augmentation network augments every image patch embedding and every text token embedding.
Revised: The augmentation network augments every image patch embedding and every text token embedding.

Original: For LeMDA, we set the confidence threshold for consistency regularizer α as 0.5, and we study this choice in Section 4.4.
Revised: For LeMDA, we set the confidence threshold for consistency regularizer α as 0.5, and we study this choice in Section 4.4.

Original: For our baselines, we follow the recommended hyperparameters.
Revised: For our baselines, we follow the recommended hyperparameters.

Original: For Mixup and Manifold Mixup, we set α as 0.8, and for MixGen, we set λ as 0.5.
Revised: For Mixup and Manifold Mixup, we set α as 0.8, and for MixGen, we set λ as 0.5.

Original: 4. M AIN R ESULTS
Revised: 4. M AIN R ESULTS

Original: We summarize the performance comparison in Table 2.
Revised: We summarize the performance comparison in Table 2.

Original: Plugging LeMDA in both Multimodal-Net and ALBEF leads to consistent accuracy improvements.
Revised: Plugging LeMDA in both Multimodal-Net and ALBEF leads to consistent accuracy improvements.

Original: There are also some particularly notable improvements, such as a 6% increase in accuracy for both Hateful Memes and Petfinder.
Revised: There are also some particularly notable improvements, such as a 6% increase in accuracy for both Hateful Memes and Petfinder.

Original: Tableillustrates how LeMDA performs comparing to the baselines.
Revised: Tableillustrates how LeMDA performs comparing to the baselines.

Original: We see that single modality input augmentation methods can hurt accuracy, for example, on News Channel, in accordance with the intuition from our introductory example in Figure 1.
Revised: We see that single modality input augmentation methods can hurt accuracy, for example, on News Channel, in accordance with the intuition from our introductory example in Figure 1.

Original: Mixup also can hurt accuracy, for example, on Wine Reviews.
Revised: Mixup also can hurt accuracy, for example, on Wine Reviews.

Original: Similarly, in the latent space, Manifold Mixup fails to improve accuracy across datasets.
Revised: Similarly, in the latent space, Manifold Mixup fails to improve accuracy across datasets.

Original: On Melbourne Airbnb and Wine Reviews, Manifold Mixup results in accuracy drops. 
Revised: On Melbourne Airbnb and Wine Reviews, Manifold Mixup results in accuracy drops. On

Original: On the contrary, LeMDA consistently improves upon original architectures and provides clearly better performance than a wide range of baselines.
Revised: [Table] the contrary, LeMDA consistently improves upon original architectures and provides clearly better performance than a wide range of baselines.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: We now perform three ablation studies to support the design choice of LeMDA.
Revised: We now perform three ablation studies to support the design choice of LeMDA.

Original: [Table]
Revised: [Table]

Original: lead to better performance.

Original: Consistency regularizer consistently outperforms a standard L2 regularizer in feature space.

Original: Moreover, combining the consistency regularizer with an L2 regularizer improves over only using an L2 regularizer.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Augmentation Network Regularizer.
Revised: Augmentation Network Regularizer.

Original: We argue in Section 3.4 that consistency regularizer helps preserve the semantic structure of augmentations.
Revised: We argue in Section 3.4 that consistency regularizer helps preserve the semantic structure of augmentations.

Original: In Table 3, we see that this consistency regularizer significantly improves performance, and also outperforms L2 regularization in feature space.
Revised: In Table 3, we see that this consistency regularizer significantly improves performance, and also outperforms L2 regularization in feature space.

Original: While
Revised: While

Original: L2 regularization attempts to keep augmented features close in distance to the original as a proxy for semantic similarity, the consistency regularization has access to the softmax outputs of the target and augmentation networks, providing direct information about labels.
Revised: L2 regularization attempts to keep augmented features close in distance to the original as a proxy for semantic similarity, the consistency regularization has access to the softmax outputs of the target and augmentation networks, providing direct information about labels.

Original: Architecture Difference.
Revised: Architecture Difference.

Original: We consider the two augmentation architectures introduced in Section 3.3, MLP-VAE and Attention-VAE.
Revised: We consider the two augmentation architectures introduced in Section 3.3, MLP-VAE and Attention-VAE.

Original: In Table 4 we see both architectures increase performance over no augmentation.
Revised: In Table 4 we see both architectures increase performance over no augmentation.

Original: We also see that MLP-VAE generally outperforms Attention-VAE.
Revised: We also see that MLP-VAE generally outperforms Attention-VAE.

Original: We suspect the reason is that Multimodal-Net passes the concatenation of N latent vector into fusion layers,  where N is the number of modalities (2 or 3 in our experiments).
Revised: We suspect the reason is that Multimodal-Net passes the concatenation of N latent vector into fusion layers, [Table] where N is the number of modalities (2 or 3 in our experiments).

Original: For Attention-VAE, this means that the input is only 2 or 3 tokens.
Revised: For Attention-VAE, this means that the input is only 2 or 3 tokens.

Original: However, we note that MLP-VAE is not reasonable for ALBEF, since it would require concatenating thousands of tokens.
Revised: However, we note that MLP-VAE is not reasonable for ALBEF, since it would require concatenating thousands of tokens.

Original: Confidence Masking.
Revised: Confidence Masking.

Original: Here, we investigate the effect of confidence masking, as well as the choice of α in Table 5.
Revised: Here, we investigate the effect of confidence masking, as well as the choice of α in Table 5.

Original: α = 0 means no masking, and all training data are used to calculate consistency loss.
Revised: α = 0 means no masking, and all training data are used to calculate consistency loss.

Original: We see that confidence masking generally leads to higher accuracy, and that the performance is not particularly sensitive to the precise value of α .
Revised: We see that confidence masking generally leads to higher accuracy, and that the performance is not particularly sensitive to the precise value of α .

Original: 4. T HE R ELATIONSHIP BETWEEN M ODALITIES
Revised: 4. T HE R ELATIONSHIP BETWEEN M ODALITIES

Original: We can categorize the relationship between available modalities by looking at P ( y | x ) where y ∼ Y and Y is the target domain.
Revised: We can categorize the relationship between available modalities by looking at P ( y | x ) where y ∼ Y and Y is the target domain.

Original: Let x = { x 1 , x 2 , . . .
Revised: Let x = { x 1 , x 2 , . . .

Original: , x N } consist of N modalities.
Revised: , x N } consist of N modalities.

Original: Perfect Correlation P ( y | x ) =
Revised: Perfect Correlation P ( y | x ) =

Original: P ( y | x n ) .
Revised: P ( y | x n ) .

Original: Essentially, one modality alone provides enough information to make the right prediction.
Revised: Essentially, one modality alone provides enough information to make the right prediction.

Original: Nevertheless, data still comprises multiple modalities for reasons such as easier training (Huang et al., 2021).
Revised: Nevertheless, data still comprises multiple modalities for reasons such as easier training (Huang et al., 2021).

Original: One example could be Food101, where the task is to predict the food from the text of a recipe and the photo of the food.
Revised: One example could be Food101, where the task is to predict the food from the text of a recipe and the photo of the food.

Original: Complementary P ( [Equation] =
Revised: Complementary P ( y | x ) =

Original: [Equation] [Equation] 1 , x 2 , . . .
Revised: P ( y |{ x 1 , x 2 , . . .

Original: , x N } ) .
Revised: , x N } ) .

Original: This category suggests that information aggregated from all modalities is necessary to make the right prediction.
Revised: This category suggests that information aggregated from all modalities is necessary to make the right prediction.

Original: Each modality is complementary to each other, and missing one modality would lead to information loss.
Revised: Each modality is complementary to each other, and missing one modality would lead to information loss.

Original: One example could be Hateful Memes, where only the combined meaning of text and image indicates harmful content.
Revised: One example could be Hateful Memes, where only the combined meaning of text and image indicates harmful content.

Original: The design for LeMDA does not exploit any assumption over the cross-modal relationship.
Revised: The design for LeMDA does not exploit any assumption over the cross-modal relationship.

Original: We observe from Table 2 that LeMDA consistently improves performance regardless of the relationship.
Revised: We observe from Table 2 that LeMDA consistently improves performance regardless of the relationship.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Jointly learning from multiple different modalities will be crucial in our quest to build autonomous intelligent agents.
Revised: Jointly learning from multiple different modalities will be crucial in our quest to build autonomous intelligent agents.

Original: We introduce the first method, LemDA, for jointly learning data augmentation across arbitrary modalities.
Revised: We introduce the first method, LemDA, for jointly learning data augmentation across arbitrary modalities.

Original: LeMDA is simple, automatic, and achieves promising results over a wide range of experiments.
Revised: LeMDA is simple, automatic, and achieves promising results over a wide range of experiments.

Original: Moreover, our results provide several significant conceptual findings about multimodal data augmentation in general: (1) separately augmenting each modality performs much worse than joint augmentation; (2) although feature augmentation is less popular than input augmentation for single-modality tasks because it is less interpretable, feature augmentation is particularly promising for modality-agnostic settings; (3) a learning-based multimodal augmentation policy can outperform even tailored augmentations, and significantly improve accuracy when augmentation transformations are not obvious such as for tabular data.
Revised: Moreover, our results provide several significant conceptual findings about multimodal data augmentation in general: (1) separately augmenting each modality performs much worse than joint augmentation; (2) although feature augmentation is less popular than input augmentation for single-modality tasks because it is less interpretable, feature augmentation is particularly promising for modality-agnostic settings; (3) a learning-based multimodal augmentation policy can outperform even tailored augmentations, and significantly improve accuracy when augmentation transformations are not obvious such as for categorical data.

Original: Our investigation has primarily focused on late-fusion architectures, showing strong results over a wide range of settings.
Revised: Our investigation has primarily focused on late-fusion architectures, showing strong results over a wide range of settings.

Original: In general, applying feature augmentation strategies to early-fusion architectures is an open question.
Revised: In general, applying feature augmentation strategies to early-fusion architectures is an open question.

Original: Early fusion combines a large number of latent features (e.g., a long sequence of token embeddings), resulting in typically intractable computational costs for augmenting every latent feature.
Revised: Early fusion combines a large number of latent features (e.g., a long sequence of token embeddings), resulting in typically intractable computational costs for augmenting every latent feature.

Original: Our experiment with an early-fusion architecture shows however that developing more efficient augmentation networks, or selectively generating only a few important latent vectors, is a promising direction for future work.
Revised: Our experiment with an early-fusion architecture shows however that developing more efficient augmentation networks, or selectively generating only a few important latent vectors, is a promising direction for future work.

Original: ETHICS STATEMENT
Revised: ETHICS STATEMENT

Original: We anticipate no direct harmful social consequences from this paper.
Revised: We anticipate no direct harmful social consequences from this paper.

Original: We believe that data augmentations are beneficial in terms of data efficiency and training efficiency, helping to save costs in data collection.
Revised: We believe that data augmentations are beneficial in terms of data efficiency and training efficiency, helping to save costs in data collection.

Original: We believe that investigating data augmentation for multimodal deep learning settings is helpful to the community in building less data hungry models.
Revised: We believe that investigating data augmentation for multimodal deep learning settings is helpful to the community in building less data hungry models.

Original: REPRODUCIBILITY STATEMENT
Revised: REPRODUCIBILITY STATEMENT

Original: We use AWS for experiments.
Revised: We use AWS for experiments.

Original: We carefully describe out experimental setup, and provide ablations for our design choices.
Revised: We carefully describe out experimental setup, and provide ablations for our design choices.

Original: Moreover, we will make code available upon publication.
Revised: Moreover, we will make code available upon publication.

Revised: A M ORE DETAILS ON THE DESIGN A. D ETAILED ARCHITECTURES OF THE AUGMENTATION NETWORK

Revised: We consider two VAE architectures in LeMDA, depending on the architecture of the task network.

Revised: The latent dimension in VAE is set as 8.

Revised: We adopted the KL divergence regularizer on the encoder distribution.

Revised: Note that we do not use the reconstruction loss between the input and the output.

Revised: In

Revised: MLP-VAE, the encoder and decoder are standard fully connected layers with ReLU as the activation function.

Revised: Dropout is used with p = 0 .

Revised: In Attention-VAE, the encoder is implemented as torch.nn.

Revised: TransformerEncoder.

Revised: We set numlayers as 4 and nhead as 8.

Revised: One fully connected layer is used to mapThe decoder is symmetric as the encoder.

Revised: Features from all modalities are treated as token embedding with no cross-attention.

Revised: We use VAE for its simplicity.

Revised: The main focus of this paper is to demonstrate the effectiveness of a learnable augmentation network for multimodal learning.

Revised: Other generative models, such as diffusion models and GANs, are also valid architectures.

Revised: The main concern may lie in efficiency, and we leave this direction as future work.

Revised: A. I MPLEMENTATION DETAILS OVER THE TRAINING PROCEDURE

Revised: In practice, we iterative train the task and augmentation networks using the same batch of training data.

Revised: Specifically, we perform two separate forward passes using F after for easy implementation with pyTorch Autograd.

Revised: We use two optimizers, one for the task network and one for the augmentation network.

Revised: B E XPERIMENT DETAILS B. A DDITIONAL STUDIES ON THE TRAINING COST

Revised: One limitation of a learning-based approach is the extra training cost.

Revised: LeMDA optimizes the augmentation network along with the task network and does incur extra training costs.

Revised: Here, we investigate the training throughput to provide a more complete understanding of the method.

Revised: We summarize the training throughput(it/second) in Table 6.

Revised: As expected, we observe lower throughput for LeMDA compared to other baselines.

Revised: Multimodal Network

Revised: [Table]

Revised: However, efficiency can be improved.

Revised: The most straightforward direction is to reduce the frequency of updating the augmentation network.

Revised: Currently, the augmentation network is updated every iteration.

Revised: However, the parameters for our task network change slowly, especially in the later stage of training.

Revised: We leave this part as future direction.

Revised: B. A DDITIONAL STUDIES ON THE HYPER - PARAMETERS

Revised: The optimization for the augmentation network is a min-max game, which leads to hyperparameters to balance the contradicting loss.

Revised: Specifically, − w 1 ∇ L (ˆ y G ) + w 2 ∇ L consist (ˆ y, ˆ y G ) + w 3 ∇ L VAE , where L VAE refers to the KL divergence regularizer on the latent encoding distribution.

Revised: In our main experiment, we use w 1 = 0 .

Revised: 0001 , w 2 = 0 .

Revised: 1 , w 3 = 0 .

Revised: 1 on all datasets except Melbourne

Revised: Airbnb and SNLI-VE.

Revised: On Melbourne Airbnb and SNLI-VE, we use w 1 = 0 .

Revised: 001 , w 2 = 0 .

Revised: 1 , w 3 = 0 .

Revised: Note that the hyperparameters are relative consistent across datasets.

Revised: Further, we investigate the influence of the different combinations of w 1 , w 2 , and w 3 .

Revised: We summarize the result on Petfinder Table 7.

Revised: We observe consistent improvements over the original multimodal network across various combinations.

Revised: [Table]

Revised: C M OTIVATIONAL EXAMPLES WHEN ONLY AUGMENTING ONE MODALITY

Revised: We have performed an additional set of experiments to investigate the effect of augmenting a single modality on Hateful Memes using state-of-the-art augmentation techniques.

Revised: In Hateful Memes, both text and image are required to decide if the content is hateful.

Revised: Two modalities provide complementary information to each other.

Revised: We run baseline augmentation to only one modality or independently on two modalities.

Revised: We observe no consistent improvements.

Revised: Essentially, performing augmentation naively to one modality or jointly without considering cross-modality relationships won’t lead to effective augmentation.

Revised: [Table]

Revised: Trivial Augment 0.

Revised: MixUp 0.

Revised: Manifold Mixup 0.

Revised: [Table]