Data Augmentation for Compositional Data: Advancing Predictive Models of the Microbiome
Abstract
Data augmentation plays a key role in modern machine learning pipelines.
While numerous augmentation strategies have been studied in the context of computer vision and natural language processing, less is known for other data modalities.
Our work extends the success of data augmentation to compositional data , i.e., simplex-valued data, which is of particular interest in microbiology, geochemistry, and other applications.
Drawing on key principles from compositional data analysis, such as the Aitchison geometry of the simplex and subcompositions , we deﬁne novel augmentation strategies for this data modality.
Incorporating our data augmentations into standard supervised learning pipelines results in consistent performance gains across a wide range of standard benchmark datasets.
In particular, we set a new state-of-the-art for key disease prediction tasks including colorectal cancer, type 2 diabetes, and Crohn’s disease.
In addition, our data augmentations enable us to deﬁne a novel contrastive learning model, which improves on previous representation learning approaches for microbiome compositional data.
Introduction
Data augmentation, i e., generating synthetic training examples to be used in model ﬁtting, is a core component of modern deep learning pipelines [55, 18].
In computer vision, augmentations such as image resizing and shifting have been used since as early as LeNet-5 [36].
These and many other augmentations have become essential to highly successful state-of-the-art architectures, ranging from AlexNet [35] and ResNet [30] to recent contrastive learning models such as SimCLR [10] and MoCo [29].
As such, a growing body of literature has emerged to develop and characterize data augmentation techniques, particularly in computer vision [46, 67, 14, 11, 12, 66], as well as natural language processing [54, 13, 18].
However, deﬁning useful data augmentations is highly domain-dependent, and fewer works have studied augmentations for more general data modalities, such as tabular data [67, 65].
The goal of our work is to extend the success of data augmentation to a previously unexplored data modality; namely, compositional data.
Compositional data (CoDa) are those that represent the parts of a whole, and therefore carry only relative information.
Equivalently, we can think of CoDa as a set of datapoints living in the simplex:
[Equation]
1 , where
[Equation]
Examples of CoDa arise throughout the sciences, including microbiology (proportions of different species in a bacterial community)
[24, 23, 51, 4], geochemistry (chemical composition of a geological sample)
[6, 5, 43], materials science (components of matter)
[41, 2], and many more.
As a result, starting with the seminal work of Aitchison [1], numerous bespoke techniques have been developed for the statistical analysis of CoDa [45, 44, 19].
A rapidly growing area of application for CoDa is the human microbiome, which comprises the populations of microorganisms that reside inside and on the human body [60].
Microbiome data arise from an inexhaustive sampling procedure as a result of high-throughput sequencing [24, 52].
In particular, each feature typically represents the relative abundance of some species of microorganism; as such, each observation can be normalized to the simplex prior to downstream analyses [24, 23, 51, 38, 42].
The microorganisms that constitute the microbiome are known to impact human physiology, both in health and in disease [22, 39, 21].
Thus, a central problem in CoDa is to learn the association x i (cid:55)→ y i , where x i ∈ ∆ p − 1 denotes the microbial composition, and y i denotes the disease status of the i th subject in a clinical study.
For example, the composition of the gut microbiome, as recorded from a stool sample, can be predictive of colorectal cancer, which is the third most prevalent form of cancer [17, 34].
This observation offers the potential for a noninvasive alternative to traditional colonoscopy procedures used for early detection of colorectal cancer [57].
In turn, accurate predictive models for microbiome CoDa are a key stepping stone towards achieving this potential [40, 34].
Note that cancer is just one application; the demand for improved predictive models holds more broadly across medicine and biological science [34, 22].
For example, the microbiome has been linked to type 2 diabetes, Crohn’s disease, obesity, and others [28, 63, 47, 21].
Classical techniques including logistic regression, support vector machines, and random forests have been widely used as predictive models for microbiome data [61, 7, 37].
More recently, specialized deep learning architectures such as DeepCoDa [49], MetaNN [38], and DeepMicro [42] have been developed.
However, the capacity of these deep networks and other expressive models has been limited by the low sample size and high dimensionality of typical microbiome studies.
These characteristics have also spurred the use of strong regularization through early stopping, weight decay and dropout, among others [58, 49, 38, 42].
However, no previous works have explored the use of data augmentation for CoDa, 2 which, as we shall demonstrate, provides a cheap and simple technique for boosting the performance of predictive models for this data modality.
Careful consideration of the sample space will motivate our novel data augmentation strategies for
CoDa.
Our work draws on foundational principles from the ﬁeld of CoDa, such as the Aitchison geometry of the simplex [1], which we combine with popular techniques from the literature on data augmentation, such as Mixup [67] and CutMix [66].
This combination will lead us to deﬁne custom data augmentation strategies for CoDa, such as Aitchison Mixup and Compositional CutMix .
In turn, incorporating these novel augmentations into existing supervised learning pipelines will result in consistent performance gains across a wide range of microbiome datasets, including a dozen standard benchmarks from the Microbiome Learning Repo [61].
These performance gains are particularly large for some deep models, for example, DeepCoDa gains more than 10% in test
AUC for discriminating colorectal cancer from healthy controls, and over 20% for type 2 diabetes.
The gains are also signiﬁcant across other expressive model families, including random forests and gradient boosting machines.
Overall, we set a new state-of-the-art on 8 out of 12 benchmark learning tasks, including clinically relevant disease prediction tasks.
Of the remaining 4 tasks, 2 were easily separable, with 100% test accuracy irrespective of whether data augmentation was used.
Importantly, our augmentations rarely hurt model performance, and in the few instances that this was the case, the drops were typically of only 1% in test AUC.
In addition to supervised learning, our novel augmentations will allow us to deﬁne contrastive representation learning for CoDa , which to the best of our knowledge, represents the ﬁrst contrastive learning model for compositional data.
Our novel data augmentations are at the core of this approach; the contrastive loss uses randomly augmented training examples to deﬁne a self-supervised optimization objective whose labels are generated from unlabelled data [27, 15, 10, 29].
In particular, our contrastive model is trained to discriminate between compositions that were generated as random augmentations of the same training example, and those that were generated as random augmentations of different training examples.
Our implementation is adapted from SimCLR [10], but using a smaller network architecture together with our novel augmentations.
Unlike SimCLR, we use our individual augmentation strategies in isolation, and ﬁnd them sufﬁcient to surpass previous representation learning approaches for microbiome CoDa [4, 42].
Altogether, we expect our data augmentations will enable signiﬁcant future progress, possibly in combination with novel architectures, in both supervised and representation learning pipelines for microbiome CoDa.
Related Work
Data augmentation: numerous data augmentation strategies have been proposed in the context of image data [55] and text data [18].
LeNet-5 [36] used random shifts and resizing, and AlexNet [35] used reﬂections and color distortions, which have become widely adopted in computer vision [30].
More general purpose data augmentations include Mixup, which generates synthetic samples by taking convex combinations of training examples, and has been applied to both image and tabular data [67].
Mixup is related to an earlier method called SMOTE [9], which aims to address class imbalance by generating synthetic samples from the minority class, taken as convex combinations between nearest neighbors within the minority class.
Random masking, whereby random input features are hidden during training, has also seen a diverse range of applications, including natural language processing (e.g., BERT [13]), computer vision (e.g., Cutout [14]), and tabular data (e.g.,
VIME [65]).
In the context of CoDa, masking is related to subcompositions , which we will use to deﬁne an analogous augmentation on the simplex.
CutMix [66] is akin to both masking and Mixup, in that random patches from different images are pasted together to generate new synthetic images.
We note that a subsequent line of work has developed techniques to automatically select optimal data augmentations for a given dataset [11, 12], somewhat akin to hyperparameter optimization.
While the data augmentation strategies that we present in the next Section could also be combined and ﬁnetuned in similar ways, such experimentation falls outside the scope of our work and is left to future research.
Microbiome models: random forests remain a strong baseline across many microbiome studies, due to their expressivity and robustness [58].
Gradient boosting machines such as XGBoost provide similarly strong performance and have also enjoyed signiﬁcant adoption [58].
AutoML [31] has shown potential for microbiome data, with a recent approach called mAML achieving state-of-theart results on several benchmarks [64].
mAML uses cross-validation to automatically select the best model class and hyperparameters for each learning task.
In addition, several specialized deep learning architectures have been proposed for microbiome CoDa.
DeepCoDa [49] introduced the log-bottleneck layer, which ensures the hidden units remain scale-invariant, a key desiderata in CoDa models.
This architecture obtained strong predictive performance using weight decay regularization.
MetaNN [38] employs a multilayer architecture regularized using dropout [56].
The authors of
MetaNN also considered generating synthetic data from a negative binomial distribution, but found no performance improvements beyond the use of dropout.
DeepMicro [42] uses a deep autoencoder architecture to learn low-dimensional representations of the microbial composition.
In turn, these representations are fed to a downstream classiﬁer trained with a supervised objective.
Contrastive learning: the goal of contrastive learning is to learn low-dimensional representations by optimizing some pretext task , where the objective function is similar to those used for supervised learning, but using labels that are derived from unlabelled data only [27].
Commonly, the pretext task is to discriminate augmented instances of the same training example from augmented instances of different training examples [15, 10, 29].
Recent contrastive learning architectures have enjoyed tremendous success, setting the state-of-the-art across a range of computer vision benchmarks [10, 29, 32, 8].
We highlight SimCLR [10] for its relative simplicity and strong empirical performance.
This method uses a nonlinear projection head between the representations and the contrastive loss computed with normalized embeddings.
Our own experiments on contrastive learning borrow these elements from SimCLR, but use a smaller network architecture together with our specialized data augmentations for CoDa.
Methods
In the following subsections, we introduce 3 novel data augmentation techniques for CoDa.
Note that many variations of our augmentation schemes could be considered; the versions we present here are intended to be as concise as possible.
Our goal is not to design augmentations that are empirically or theoretically optimal, but rather to demonstrate the effectiveness of our novel methodology by establishing simple and performant baselines.
For instance, whenever a random mixing parameter is required, we use a U (0 , 1) distribution, even though other choices can likely result in increased performance.
For clarity, we will focus on the classiﬁcation setting; generalizing to regression problems is straightforward.
We will use the notation v , x ∈ ∆ p − 1 for simplex-valued vectors, λ ∈ R a scalar, and D = { x i , y i } our training data.
3.1 Aitchison Mixup
Aitchison [1] deﬁned a Hilbert space structure on the simplex, known as the Aitchison geometry , with the following vector addition, scalar multiplication, and inner product:
[Equation]
This geometry provides a principled deﬁnition of linear combinations and a distance metric on the simplex; parallel and orthogonal lines are illustrated in Figure 1.
The
Aitchison geometry is closely related to the centered-logratio transformation [1], which is an inverse of the softmax and deﬁnes an isometry between ∆ p − 1 and Euclidean space [45].
Taken together, these properties form the basis of much of CoDa methodology [1, 45, 44, 19].
Our ﬁrst data augmentation strategy, which we call Aitchison Mixup , is based on taking linear combinations of the training points, in the Aitchison sense.
While general linear combinations may be used, for simplicity we focus on intra-class pairwise convex combinations.
Namely, each new datapoint is sampled as follows:
Draw a class c from the class prior and draw λ ∼ U (0 , 1) .
Draw two training points i 1 , i 2 such that y i
=
y
i
= c , uniformly at random.
Set x aug = ( λ (cid:12) x i 1 ) ⊕ ((1 − λ ) (cid:12) x i 2 ) and y aug
= c .
Put more succinctly, for each class we generate convex combinations of the points in that class, in the Aitchison sense.
Note this augmentation strategy is a CoDa analogue of Mixup, which generates
Euclidean convex combinations of images and tabular data [67].
Much like Mixup is capable of boosting predictive accuracy by regularizing models towards a linear decision boundary in the regions between training examples, Aitchison Mixup aims to enforce the equivalent regularization in the
Aitchison geometry of the simplex.
Note that, once we restrict to convex combinations (as opposed to general linear combinations), our augmented data will remain inside the simplex regardless of whether we operate in the Euclidean or the Aitchison geometry.
However, we found the latter to be more empirically effective, perhaps unsurprisingly given the associated vector space structure of the simplex.
We also note that the authors of Mixup [67] sampled λ ∼ Beta ( α, α ) , where α is a hyperparameter, and took inter-class as well as intra-class convex combinations, setting y aug = λy i 1 + (1 − λ )
y
i 2 .
While Aitchison Mixup can also incorporate such generalizations, we chose not to do so in our implementation because, as discussed above, our focus was on ﬁnding the simplest performant augmentation strategies for our new data modality.
3.2 Compositional Feature Dropout
In CoDa, a subcomposition of x ∈ ∆ p − 1 refers to a lower-dimensional composition, x sub ∈ ∆ k −with k < p , formed by taking a subset of the elements of x and renormalizing to a unit total.
Namely, if j 1 , . . .
, j k denotes a subset of the indices { 1 , . . .
, p } , the corresponding subcomposition is deﬁned as:
[Equation]
We can generate augmented data by taking a random subcomposition of the training points; such a strategy is analogous to masking in language data or cropping in image data, or more generally, feature dropout [56].
Since our predictive models require inputs of ﬁxed dimension p , rather than discarding elements of x we simply zero them out.
Thus, each new datapoint is generated as follows:
Draw λ ∼ U (0 , 1) .
Draw a training point i uniformly at random and set ˜x = x i .
For each j ∈ { 1 , . . .
, p } , draw I j iid ∼ Bernoulli ( λ ) , and set ˜ x
j = 0 if I j = 0 .
Set x
aug =
˜x / ( (cid:80) pj
=1 ˜ x j ) and y
aug
= y i .
In short, we zero out random entries of the training points and renormalize.
Intuitively, this strategy encourages our predictive models to become robust to partially observed inputs.
Note that, distinctively from feature dropout, our augmentation includes an additional renormalization to a unit total, and we therefore use the term Compositional Feature Dropout .
Importantly, this renormalization ensures that the augmented samples remain in the support of the training data, i.e., the simplex.
Note that many CoDa models, including mAML and DeepCoDa, apply log transformations to their inputs, and therefore require that these be non-zero.
For this reason, we implement a zero-replacement step where we add a small positive quantity to all the parts of each composition and renormalize.
This small quantity is set to 1 /L
i , where L i corresponds to the library size from the high-throughput sequencing procedure.
L i can simply be thought of as a large number such that
/L
i is smaller than the non-zero components of x i .
Note that this transformation is a standard preprocessing step in
CoDa [51].
3.3 Compositional CutMix
Our third augmentation scheme, which we call Compositional CutMix , combines elements of our previous 2 augmentations.
Like in Mixup, we generate new datapoints by combining pairs of training points from the same class.
However, instead of combining these training points linearly (in the
Aitchison sense), we take complementary subcompositions and renormalize.
Namely, we generate each new datapoint as follows:
Draw a class c from the class prior and draw λ ∼ U (0 , 1) .
Draw two training points i 1 , i 2 such that y i
=
y
i
= c , uniformly at random.
For each j ∈ { 1 , . . .
, p } , draw I j iid ∼ Bernoulli ( λ ) , and set ˜ x
j = x i 1 j if I j = 0 or ˜ x
j = x i 2 j if I j = 1 . 4.
Set x
aug =
˜x / ( (cid:80) pj
=1 ˜ x j ) and y aug = c .
Note this strategy can be thought of as a CoDa analogue of CutMix , whereby new images are formed by pasting together patches from different training images [66].
Note also that in computer vision,
CutMix and related methods take local image patches as opposed to randomly sampled pixels.
Likewise, one could sample microbial subcompositions according to biologically relevant groupings, for example using a phylogenetic tree [51].
Such a strategy would require incorporating additional domain knowledge and is left to future work, but we expect it would further increase the quality of our data augmentations.
This remark applies both to Compositional Feature Dropout and Compositional
CutMix.
With regard to inter-class combinations, a similar discussion as in Section 3.1 applies.
Experiments 4.1 Supervised learning
We evaluate our augmentation strategies on 12 standard binary classiﬁcation tasks taken from the
Microbiome Learning Repo [61].
These datasets comprise various disease and phenotype prediction
[Table]
tasks, including colorectal cancer [34], type 2 diabetes [47], Crohn’s disease [21]
[Table]
and cirrhosis [48]
[Table]
as well as multiple body sites including the gut, skin, oral cavity, airways, and vagina [39, 53].
As such [Table] this benchmark provides a comprehensive evaluation for predictive models of the human microbiome [64, 49, 25].
More details on these 12 learning tasks can be found in Table 1; note this benchmark is constructed from the original Microbiome Learning Repo by ﬁltering datasets that contain a minimum sample size of 100, with at least 50 in either class.
For each learning task, we take 20 independent 80/20 train/test splits and we ﬁt Random Forest,
XGBoost, mAML [64], DeepCoDa [49], and MetaNN [38], ﬁrst to the original training data, then on 3 augmented training sets obtained using our 3 augmentation strategies.
Thus, we train a total of 12 × 20 × 5 × 4 = 4 800 models.
5 We evaluate test performance using ROC AUC, and we note that our datasets do not present severe class imbalance (with the exception of tasks 3 and 11, both of which were well separated by all our classiﬁers and therefore had no effect on the overall comparison).
For the augmented training sets, we generated 10 times as many synthetic samples as there were original training examples.
The factor of 10 was chosen so as to obtain a relatively large augmented sample, in order to reduce the sampling variance from our random augmentations.
In turn, we compensate for the fact that our augmented data is then much more numerous than our original training data, by downweighting the synthetic samples by a factor of 10; the total weight of the original and synthetic data is then equal to 1 / 2 each.
Our results are shown in Table 2 (Aitchison Mixup) in the main text, as well as Tables 5 (Compositional Feature Dropout), and 6 (Compositional CutMix) in the Appendix:
• Table 2 shows that Aitchison Mixup improved the test performance of existing models across a large majority of learning tasks.
On average, we obtained a 5% gain in test AUC for DeepCoDa, 2% for random forests and XGBoost, and 1% for mAML.
MetaNN remained ﬂat, but this was also the worst performing model overall.
Importantly, in the few instances where Aitchison Mixup hurt the performance of a model, the loss was typically of just 1%.
• Table 5 shows that Compositional Feature Dropout also result in consistent performance improvements across most models and tasks, though to a slightly lesser degree than Aitchison Mixup.
On average, DeepCoDa enjoyed a 3% gain in test AUC, random forests 2%, and XGBoost and MetaNN both 1% (mAML remained ﬂat). • Table 6 shows that Compositional CutMix obtained the strongest classiﬁcation performance out of our 3 augmentation strategies on microbiome CoDa.
On average, DeepCoDa saw a 5% gain in test AUC, with 2% for random forests and XGBoost, and 1% for mAML and MetaNN.
Moreover, models trained with Compositional CutMix set a new state-of-the-art on 8 out of 12 tasks, including colorectal cancer (0.76), type 2 diabetes (0.72), and Crohn’s
[Table]
[Figure]
Figure 2 shows the effect of data augmentation on test performance as a function of the number of samples in the training set, for Compositional CutMix.
We note that datasets with smaller sample sizes tended to beneﬁt more from data augmentation.
This suggests that our methodology may prove beneﬁcial to many microbiome research projects, where datasets with low-hundreds of samples are commonplace.
Figure 3 shows a similar scatterplot, as a function of the dataset dimensionality rather than sample size; the gains from augmentation appear consistent across lower- and higher-dimensional microbiome CoDa (we observe similar trends Aitchison Mixup and Compositional Feature Dropout, and we omit the corresponding scatterplots).
In addition, we evaluated the impact of our data augmentations on the expected calibration error (ECE) [26] of our models.
Previous works have noted that increasingly ﬂexible predictive models such as deep neural networks, while enabling greater predictive accuracy, tend to become overconﬁdent in their predictions, degrading uncertainty quantiﬁcation [26].
In Appendix A, we show the ECE obtained by our models, and we verify that our data augmentations do not hurt model calibration overall (in fact, some modest improvements are obtained).
[Figure]
4.2 Contrastive representation learning
We have shown that our novel data augmentation strategies provide consistent performance gains across supervised training pipelines.
Next, we use our data augmentations in order to apply contrastive representation learning to compositional data for the ﬁrst time (to the best of our knowledge).
Note that our experiment provides a simple proof of concept showing that contrastive learning is capable of learning useful representations of microbiome CoDa, which result in better downstream classiﬁcation performance than previous deep representation learning approaches.
However, more extensive experiments will be required to validate the performance of contrastive learning across a wider range of model architectures for CoDa.
As a point of comparison we take DeepMicro [42], a deep autoencoder designed speciﬁcally for microbiome data. DeepMicro includes an encoder and a decoder that are trained jointly by minimizing reconstruction error.
The encoder outputs are used as a low-dimensional representation that is passed to a downstream classiﬁer. Our contrastive model discards the decoder and trains the encoder using the contrastive loss from SimCLR [10].
This loss function requires pairs of positive and negative datapoints, where positive pairs are constructed by taking synthetic augmentations of the same training example and negative pairs are constructed by taking synthetic augmentations of different training examples. In the context of microbiome CoDa, it is our novel augmentations that allow us to deﬁne positive and negative pairs required for contrastive learning.
Full detail on model architecture and implementation is provided in Appendix B.
After applying unsupervised pretraining (reconstructive or contrastive), we evaluate the quality of learned representations under two standard protocols: linear evaluation and ﬁnetuning.
In linear evaluation, we freeze the encoder weights and train a linear classiﬁcation head using the supervised cross-entropy loss on the original training set.
In ﬁnetuning, the linear classiﬁcation head is trained jointly with the encoder network, again using the supervised cross-entropy loss.
The ﬁnetuning protocol is evaluated both with and without using additional data augmentation (i.e., passing additional synthetic examples to the supervised loss).
These evaluation protocols are applied for both DeepMicro and our contrastive model, in addition to a “no pretraining” control.
This control is a randomly initialized encoder network and linear head with the same architecture, trained only on the supervised objective (i.e., a neural network that did not enjoy the beneﬁt of unsupervised pretraining).
The results are shown in Table 3; the representations learned by our contrastive model using Compositional Feature Dropout consistently outperform those learned by DeepMicro, both in linear evaluation and ﬁnetuning.
Note that ﬁnetuning with augmentation performs slightly better than without, suggesting that the beneﬁts of supervised training with our data augmentations can be combined with the beneﬁts of augmentation as part of contrastive learning.
Importantly, note that the comparison to DeepMicro is conservative, in the sense that we replicated the architecture and simply changed the pretraining objective from reconstruction error to contrastive loss; tuning the encoder architecture itself under a contrastive objective would likely lead to further performance improvements.
We conclude that Compositional Feature Dropout provides a valuable data augmentation for contrastive learning on the simplex.
Note that our other 2 augmentation strategies, Aitchison Mixup and Compositional CutMix, may also be used for contrastive learning.
However, these augmentations require paired training examples to generate each synthetic sample, and the implementation is therefore slightly more involved. In
[Table]
particular, we generate each pair of positive samples by taking one pair of training examples
[Table] and drawing two random combinations (in the Aitchison Mixup or the Compositional CutMix sense) of that one pair.
The training examples are randomly partitioned into pairs at each epoch [Table] and negative samples correspond to those that originate from separate pairs.
The results are provided in Appendix
A and show similarly strong performance to Table 3.
4.3 Additional benchmarks
Further to the 12 microbiome classiﬁcation tasks from the previous sections
[Table] we analyze 8 additional compositional datasets [Table] which were selected to represent a diverse set of applications that are commonly encountered by CoDa practitioners.
Note that, while these datasets correspond to a wide range of scientiﬁc domains [Table] they have all been analyzed as case studies speciﬁcally in the CoDa literature. The datasets can be summarized as follows:
• Glass : a dataset from the UCI ML repo, on the elemental composition of glass particles of different types of origin.
Such particles are routinely recovered from the clothing of criminal suspects; the goal is to provide forensic evidence by classifying the type of origin (broken window vs. other) given the composition recorded in a multielement analysis [16, 59].
• Bayesite : a classical materials science dataset due to Aitchison [2].
The goal is to predict the permeability of ﬁbreboard from the mix of its ingredients.
• Serum : another classical dataset due to Aitchison [2], where the composition of blood serum protein is used to discriminate between two known diseases.
• Hydrochem : a hydrochemical dataset measuring the composition of water samples across sites in the Llobregat river basin [43].
Classiﬁcation targets are formed by separating upstream and downstream sites.
• Jura : a geochemical dataset from the Swiss Jura, where rock types (Kimmeridgian vs. Sequanian) are classiﬁed from the chemical composition of soil samples [3].
• Metabolites : gut metabolomic proﬁles for subjects with and without inﬂammatory bowel disease [20, 50].
The goal is to diagnose disease from the metabolite composition.
• MicroRNA : contains microRNA expression data for different types of primary breast cancer [62, 50].
The goal is to discriminate between breast cancer subtypes (Luminal A vs. Luminal B) from the molecular proﬁle of each tumor.
[Table]
• Coffee : a nutrition dataset describing the chemical composition of commercially available coffee samples [33].
Classiﬁcation targets are taken as different blends of coffee.
In modeling these datasets, we follow the experimental protocol described in Section 4.1, except we no longer test mAML and MetaNN (which were not intended for non-microbiome data), and we replace the latter with a small neural network with one hidden layer of 32 units and ReLU activations, denoted as NN (note we choose to keep this architecture very simple as it is trained on several heterogeneous dataset).
The results are shown in Table 4 for Aitchison Mixup.
As was the case for our microbiome benchmark, we again see that our data augmentation results in consistent performance gains across a wide range of classiﬁers.
Indeed, on most datasets (6 out of 8) the best performing model was one that used
Aitchison Mixup, and only on a single dataset (Serum) was the best model (NN) one trained without augmentation.
Note that, in contrast to our microbiome benchmarks, Table 4 shows a strong outperformance for tree-based models.
Importantly, Aitchison Mixup never hurt the performance of any of our tree-based models, and only did so rarely for our network models.
We observe similarly positive results (albeit slightly weaker) for Compositional Feature Dropout and Compositional
CutMix, provided in Tables 7 and 8 from the Appendix, respectively.
Conclusion
By combining ideas from data augmentation with the principles of CoDa, we have deﬁned 3 novel augmentation strategies for CoDa: Aitchison Mixup, Compositional Feature Dropout, and Compositional CutMix.
Our augmentations offer cheap performance gains across a wide range of benchmark datasets, advancing the state-of-the-art on standard classiﬁcation benchmarks for colorectal cancer, type 2 diabetes, and Crohn’s disease.
While we ﬁnd all 3 strategies performant across a wide range of benchmark datasets, Aitchison Mixup and Compositional CutMix performed best on microbiome
CoDa (with a slight outperformance for the latter), and Aitchison Mixup performed best on nonmicrobiome CoDa.
In addition, our data augmentations allowed us to deﬁne the ﬁrst contrastive learning model for CoDa, which we show improves on existing representation learning frameworks for microbiome data.
Data augmentation and contrastive learning have enjoyed tremendous success in other application domains such as computer vision; our novel methodology aims to help spur similar developments in the ﬁelds of CoDa and the microbiome. Acknowledgments and Disclosure of Funding
We thank Samuel Lippl, Richard Zemel, and the anonymous reviewers for helpful discussions, and the Simons Foundation 542963, Sloan Foundation, McKnight Endowment Fund, NSF DBI-1707398, and the Gatsby Charitable Foundation for support.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] Compare Abstract/Intro to Methods/Experiments (b) Did you describe the limitations of your work?
[Yes] See Section 3 and Section D in the Appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Not the focus of our paper, but brieﬂy discussed model calibration in Section 4.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] No theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A] No theoretical results.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4 or code in supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See appendix A.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[No] Assets are open-source.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Code provided in supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] Datasets used are in the public domain.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
No identiﬁable or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] Not used.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] Not used.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] Not used.
A Additional experimental results
Tables 5 and 6 show the performance of Compositional Feature Dropout and Compositional CutMix on our microbiome benchmarks and are discussed in the main text.
Tables 7 and 8 show the performance of Compositional Feature Dropout and Compositional CutMix on our non-microbiome benchmarks.
Tables 9, 10, and 11 show the effect on expected calibration error (ECE) of adding data augmentation to our various models.
Notice that overall calibration error stays the same or improves slightly after incorporating data augmentation.
Tables 14, 15, and 16, show error bars for Tables 2, 5, and 6, respectively.
In Tables 12 and [Table] we show the performance of our contrastive models deﬁned with Aitchison Mixup and Compositional CutMix, respectively.
The latter performs equally strongly as Compositional
Feature Dropout, as shown in Section 4.2, the former performs noticeably worse, however still no worse than DeepMicro, thus demonstrating the robustness of our contrastive framework to multiple augmentation strategies.
Further gains may be obtained by combining augmentations in a contrastive loss, a direction which is left to future work.
Table 3 shows error bars for Table 17, respectively.
[Table]
Mean 0.
DeepMicro [42] is composed of an encoder and a decoder; the encoder has 2 hidden layers with 256 and 128 units respectively, and ReLU nonlinearities, and a 64-dimensional output. The decoder architecture is a mirror image of the encoder. The decoder output is of the same dimension as the encoder input, and the model is trained to minimize the mean squared error between the two, using the Adam optimizer with default parameters for 2,000 epochs.
To ensure a fair comparison, our contrastive model uses the same encoder architecture from DeepMicro. The decoder layers are discarded; instead, the 64-dimensional latent representation is passed to a multilayer projection head that is trained jointly with the encoder using a contrastive loss, as is done in SimCLR [10]. Our projection head contains one 32-dimensional hidden layer with ReLU activations, and a 16-dimensional projection output, which is normalized to 1 unit in L2 norm and passed to a temperature-scaled cross-entropy loss. Intuitively, this loss function is designed to draw the representations of positive pairs of examples close together, and push those of negative pairs far apart.
Positive pairs refer to two synthetic samples generated by random data augmentations of the same training example; negative pairs refer to synthetic samples generated from different training examples. In our implementation, at each training step we sample two Compositional Feature Dropout from each training example, and compute the contrastive loss over all such pairs. In particular, our batch size corresponds to the entire training set, which is reasonable given the small sample sizes in our data. We again used the Adam optimizer with default parameters for 2,000 epochs.
[Table]
[Table]
C Other augmentation strategies
We propose an additional augmentation strategy, Multinomial Resampling , which can be thought of as a CoDa analogue of image blurring.
The goal of this augmentation is to inject noise across the coordinates of x in a principled manner.
Additive Gaussian noise is clearly ill-suited for CoDa, as it is not constrained to the simplex.
Instead, inspired by the recording mechanism for microbiome CoDa, we propose generating new datapoints from a multinomial distribution.
Note that high-throughput sequencing technologies record the microbial composition present in a specimen by subsampling the larger population.
The total number of reads in this subsample, known as the sequencing depth, is an artifact of the measurement process.
Assuming the subsample is small relative to the population, the multinomial distribution provides a crude approximation for this generative process, with each trial being drawn according to the true proportions in the underlying population.
In this way, we can sample new datapoints from a multinomial distribution where the number of trials corresponds to the sequencing depth.
Namely, each new datapoint is generated as follows:
Draw a training point i uniformly at random.
Draw ˜x ∼ Multinomial
( L i , x i ) , where L i is the sequencing depth (
a.k.a. library size).
Set x aug =
˜x / ( (cid:80) pj
=1 ˜ x j ) and y
aug
= y i .
[Table]
Task
RF
Aug
XGB
Aug mAML
Aug
DeepCoDa
Aug
MetaNN
Aug 0.
[Table]
Notice that the number of trials L i controls the noise level; as L i → ∞ , x aug → x i .
When applied to
CoDa that does not arise from high-throughput sequencing, L i can be speciﬁed arbitrarily, or treated as a hyperparameter.
In our microbiome datasets, the sequencing depth is typically on the order of
L ∼ 10 000 .
D Limitations of our work
In addition to those mentioned in Section 3, we note the following limitations of our work:
• Although rare, there were some datasets where data augmentation hurt the classiﬁcation performance for certain models.
• We did not study the effect of combining multiple augmentations in a single pipeline, which is left to future study.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Mean 0.
[Table]
Mean 0.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Mean 0.