Original: A UTO B AYES : A UTOMATED B AYESIAN G RAPH E XPLORATION FOR N UISANCE -R OBUST I NFERENCE
Revised: A UTO B AYES : A UTOMATED B AYESIAN G RAPH E XPLORATION FOR N UISANCE -R OBUST I NFERENCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning data representations that capture task-related features, but are invariant to nuisance variations 1 remains a key challenge in machine learning
Revised: Learning data representations that capture task-related features, but are invariant to nuisance variations 1 remains a key challenge in machine learning.

Original: We introduce an automated Bayesian inference framework, called AutoBayes , that explores different graphical models linking classiﬁer encoder decoder estimator and adversary network blocks to optimize nuisance -invariant machine learning pipelines
Revised: We introduce an automated Bayesian inference framework, called AutoBayes, that explores different graphical models linking classiﬁer, encoder, decoder, estimator and adversarial network blocks to optimize nuisance-invariant machine learning pipelines.

Original: AutoBayes also enables learning disentangled representations, where the latent variable is split into multiple pieces to impose various relationships with the nuisance variation and task labels
Revised: AutoBayes also enables learning disentangled representations, where the latent variable is split into multiple pieces to impose various relationships with the nuisance variation and task labels.

Original: We benchmark the framework on several public datasets, where we have access to subject and class labels during training, and provide analysis of its capability for subject-transfer learning with/without variational modeling and adversarial training
Revised: We benchmark the framework on several public datasets and provide analysis of its capability for subject-transfer learning with/without variational modeling and adversarial training.

Original: We demonstrate a signiﬁcant performance improvement with ensemble learning across explored graphical models
Revised: We demonstrate a signiﬁcant performance improvement with ensemble learning across explored graphical models

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The great advancement of deep learning techniques based on deep neural networks (DNN) has enabled more practical design of human-machine interfaces (HMI) through the analysis of the user’s physiological data (Faust et al., 2018), such as electroencephalogram (EEG) (Lawhern et al., 2018) and electromyogram (EMG) (Atzori et al., 2016).
Revised: The great advancement of deep learning techniques based on deep neural networks (DNN) has enabled more practical design of human-machine interfaces (HMI) through the analysis of the user’s physiological data (Faust et al., 2018), such as electroencephalogram (EEG) (Lawhern et al., 2018) and electromyogram (EMG) (Atzori et al., 2016).

Original: However, such biosignals are highly prone to variation depending on the biological states of each subject (Christoforou et al., 2010).
Revised: However, such biosignals are highly prone to variation depending on the biological states of each subject (Christoforou et al., 2010).

Original: Hence, frequent calibration is often required in typical HMI systems.
Revised: Hence, frequent calibration is often required in typical HMI systems.

Original: Toward resolving this issue, subject-invariant methods (Özdenizci et al., 2019b), employing adversarial training (Makhzani et al., 2015; Lample et al., 2017; Creswell et al., 2017) with the Conditional
Revised: Toward resolving this issue, subject-invariant methods (Özdenizci et al., 2019b), employing adversarial training (Makhzani et al., 2015; Lample et al., 2017; Creswell et al., 2017) with the Conditional

Original: Variational AutoEncoder (A-CVAE)
Revised: Variational AutoEncoder (A-CVAE)

Original: (Louizos et al., 2015; Sohn et al., 2015) shown in Fig. 1(b), have emerged to reduce user calibration for realizing successful HMI systems.
Revised: (Louizos et al., 2015; Sohn et al., 2015) shown in Fig. 1(b), have emerged to reduce user calibration for realizing successful HMI systems.

Original: Compared to a standard
Revised: Compared to a standard

Original: DNN classiﬁer C in Fig.
Revised: DNN classiﬁer C in Fig.

Original: 1(a), integrating additional functional blocks for encoder E , nuisanceconditional decoder D , and adversary A networks offers excellent subject-invariant performance.
Revised: 1(a), integrating additional functional blocks for encoder E , nuisanceconditional decoder D , and adversary A networks offers excellent subject-invariant performance.

Original: The DNN structure may be potentially extended with more functional blocks and more latent nodes as shown in Fig.
Revised: The DNN structure may be potentially extended with more functional blocks and more latent nodes as shown in Fig.

Original: 1(c).
Revised: 1(c).

Original: However, most works rely on human design and insight to determine the block connectivityand architecture of DNNs.
Revised: However, such a DNN architecture design may rely on human effort and insight to determine the block connectivity of DNNs.

Original: Automation of hyperparameter and architecture exploration in the context of AutoML (Ashok et al., 2017; Brock et al., 2017; Cai et al., 2017; He et al., 2018; Miikkulainen et al., 2019; Real et al., 2017; 2020; Stanley & Miikkulainen, 2002; Zoph et al., 2018) can facilitate DNN design suited for nuisance-invariant inference.
Revised: Automation of hyperparameter and architecture exploration in the context of AutoML (Ashok et al., 2017; Brock et al., 2017; Cai et al., 2017; He et al., 2018; Miikkulainen et al., 2019; Real et al., 2017; 2020; Stanley & Miikkulainen, 2002; Zoph et al., 2018) can facilitate DNN design suited for nuisance-invariant inference.

Original: Nevertheless, without proper reasoning, most of the search space for link connectivity will be pointless.
Revised: Nevertheless, without proper reasoning, most of the search space for link connectivity will be pointless.

Original: In this paper, we propose a systematic automation framework called AutoBayes, which searches for the best inference graph model associated with a Bayesian graph model  well-suited to reproduce the training datasets.
Revised: In this paper, we propose a systematic automation framework called AutoBayes, which searches for the best inference graph model associated with a Bayesian graph model (also a.k.a. Bayesian network) well-suited to reproduce the training datasets.

Original: The proposed method automatically formulates various different Bayesian graphs by factorizing the joint probability distribution in terms of data, class label, subject identiﬁcation (ID), and inherent latent representations.
Revised: The proposed method automatically formulates various different Bayesian graphs by factorizing the joint probability distribution in terms of data, class label, subject identiﬁcation (ID), and inherent latent representations.

Original: Given Bayesian graphs, some meaningful inference graphs are generated through the Bayes-Ball algorithm (Shachter, 2013) for pruning redundant links to achieve 
Revised: Given Bayesian graphs, some meaningful inference graphs are generated through the Bayes-Ball algorithm (Shachter, 2013) for pruning redundant links to achieve high-accuracy estimation.

Revised: In order to promote robustness against nuisance variations such as inter-subject/session factors, the explored Bayesian graphs can provide

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: high-accuracy estimation. In order to promote robustness against nuisance variations such as intersubject/session factors, the explored Bayesian graphs can provide reasoning to use adversarial training with/without variational modeling and latent disentanglement.
Revised: reasoning to use adversarial training with/without variational modeling and latent disentanglement.

Original: We demonstrate that AutoBayes can achieve excellent performance across various public datasets,and in particular with an ensemble stacking of multiple explored graphical models.
Revised: We demonstrate that AutoBayes can achieve excellent performance across various public datasets, in particular with an ensemble stacking of multiple explored graphical models.

Original:  K EY C ONTRIBUTIONS
Revised:  K EY C ONTRIBUTIONS

Original: At the core of our methodology is the consideration of graphicalBayesian models that capture the probabilistic relationship between random variables representing the data features X , task labels Y , nuisance variation labels S , and (potential) latent representations Z .
Revised: At the core of our methodology is the consideration of graphical models that capture the probabilistic relationship between random variables representing the data features X , task labels Y , nuisance variation labels S , and (potential) latent representations Z .

Original: The ultimate goal is to infer the task label Y from the measured data feature X , which is hindered by the presence of nuisance variations (e.g., inter-subject/session variations) that are (partially) labelled by S .
Revised: The ultimate goal is to infer the task label Y from the measured data feature X , which is hindered by the presence of nuisance variations (e.g., inter-subject/session variations) that are (partially) labelled by S .

Original: One may use a standard DNN to classify Y given X as shown in Fig. 1(a), without explicitly involving S or Z .
Revised: One may use a standard DNN to classify Y given X as shown in Fig. 1(a), without explicitly involving S or Z .

Original: Although A-CVAE in Fig. 1(b) may offer nuisance-robust performance through adversarial disentanglement of S from latent Z , there is no guarantee that such a model can perform well across different datasets.
Revised: Although A-CVAE in Fig. 1(b) may offer nuisance-robust performance through adversarial disentanglement of S from latent Z , there is no guarantee that such a model can perform well across different datasets.

Original: It is exempliﬁed in Fig.
Revised: It is exempliﬁed in Fig.

Original: 2 where A-CVAE outperforms the standard DNN model for some datasets (QMNIST, Stress, ErrP) while it does not for the other cases.
Revised: 2 where A-CVAE outperforms the standard DNN model for some datasets (QMNIST, Stress, ErrP) while it does not for the other cases.

Original: This may be due to the underlying probabilistic relationship of the data varying across datasets.
Revised: This may be due to the underlying probabilistic relationship of the data varying across datasets.

Original: Our proposed framework can construct justiﬁable models, achieving higher performance for every dataset, as demonstrated in Fig.
Revised: Our proposed framework can construct justiﬁable models, achieving higher performance for every dataset, as demonstrated in Fig.

Original: It is veriﬁed that signiﬁcant gain can be realized with ensemble methods of different Bayesian graphs which are explored in our AutoBayes.
Revised: It is veriﬁed that signiﬁcant gain is attainable with ensemble methods of different Bayesian graphs which are explored in our AutoBayes.

Original: For example, our method with a relatively shallow[Figure] architecture achieves 99 .
Revised: For example, our method with a relatively shallow architecture achieves 99 .

Original: 61 % accuracy which is close to state-of-the-art performance in QMNIST dataset.
Revised: 61 % accuracy which is close to state-of-the-art performance in QMNIST dataset.

Original: The main contributions of this paper over the existing works are ﬁve-fold as follows:
Revised: The main contributions of this paper over the existing works are ﬁve-fold as follows:

Original: • AutoBayes automatically explores potential graphical models inherent to the data by combinatorial pruning of dependency assumptions (edges) and then applies Bayes-Ball to examine various inference strategies, rather than blindly exploring hyperparameters of DNN blocks.

Original: • AutoBayes offers a solid reason of how to connect multiple DNN blocks to impose conditioning and adversary censoring for the task classiﬁer, feature encoder, decoder, nuisance indicator and adversary networks, based on an explored Bayesian graph.

Original: • The framework is also extensible to multiple latent representations and nuisances factors.

Original: • Besides fully-supervised training, AutoBayes can automatically build some relevant graphical models suited for semi-supervised learning.

Original: • Multiple graphical models explored in AugoBayes can be efﬁciently exploited to improve performance by ensemble stacking.

Original: We note that this paper relates to some existing literature in AutoML, variational Bayesian inference (Kingma & Welling, 2013;

Original: Sohn et al., 2015; Louizos et al., 2015), and adversarial training (Goodfellow et al., 2014; Dumoulin et al., 2016; Donahue et al., 2016; Makhzani et al., 2015;

Original: Lample et al., 2017; Creswell et al., 2017), as addressed in Appendix A.1.

Original: Nonetheless, AutoBayes is a novel framework that diverges from AutoML, which is mostly employed to architecture tuning at a micro level.

Original: Our work focuses on exploring neural architectures at a macro level, which is not an arbitrary diversion, but a necessary interlude.

Original: To make further progress in AutoML, our method focuses on the relationships between the connections in a neural network’s architecture and the characteristics of the data that is being presented to it (Minsky & Papert, 2017).

Original: In addition, our method provides a new perspective in how to involve the adversarial blocks and to exploit multiple models for ensemble stacking.

Original: A UTO B AYES

Original: AutoBayes Algorithm:

Original: The overall procedure of the AutoBayes algorithm is described in the pseudocode of Algorithm 1.

Original: The AutoBayes automatically constructs non-redundant inference factor graphs given a hypothetical Bayesian graph assumption, through the use of the Bayes-Ball algorithm.

Original: Depending on the derived conditional independency and pruned factor graphs, DNN blocks for encoder E , decoder D , classiﬁer C , nuisance estimator N and adversary A are reasonably connected.

Original: The whole DNN blocks are trained with adversary learning in a variational Bayesian inference.

Original: The Bayes-Ball algorithm (Shachter, 2013) facilitates an automatic pruning of redundant links in inference factor graphs through the analysis of conditional independency.

Original: Fig.

Original: 3 shows ten Bayes-Ball rules to identify conditional independency.

Original: Given a Bayesian graph, we can determine whether two disjoint sets of nodes are independent conditionally on other nodes through a graph separation criterion.

Original: Speciﬁcally, an undirected path is activated if a Bayes ball can travel along without encountering a stop symbol: in Fig.

Original: If there are no active paths between two nodes when some conditioning nodes are shaded, then those random variables are conditionally independent.

Original: Graphical Models: We here focus on 4-node graphs.

Original: Let p ( y, s, z, x ) denote the joint probability distribution underlying the datasets for the four random variables, i.e., Y , S , Z , and X .

Original: The chain

Original: Require: Nodes set V =
Revised: Require: Nodes set V =

Original: [ Y, X, S 1 , S 2 , . . .
Revised: [ Y, X, S 1 , S 2 , . . .

Original: , S n , Z 1 , Z 2 , . . .
Revised: , S n , Z 1 , Z 2 , . . .

Original: , Z m ] , where Y denotes task labels, X is a measurement data, S
Revised: , Z m ] , where Y denotes task labels, X is a measurement data, S

Original: =
Revised: =

Original: [ S 1 , S 2 , . . .
Revised: [ S 1 , S 2 , . . .

Original: , S n ] are (potentially multiple) semi-supervised nuisance variations, and Z =
Revised: , S n ] are (potentially multiple) semi-supervised nuisance variations, and Z =

Original: [ Z 1 , Z 2 , . .
Revised: [ Z 1 , Z 2 , . .

Original: , Z m ] are (potentially multiple) latent vectors
Revised: , Z m ] are (potentially multiple) latent vectors

Original: Ensure: Semi-supervised training/validation datasets 1: for all permutations of node factorization from Y to X do 2:
Revised: Ensure: Semi-supervised training/validation datasets 1: for all permutations of node factorization from Y to X do 2:

Original: Let B 0 be the corresponding Bayesian graph for the permuted full-chain factorization 3: for all combinations of link pruning on the full-chain Bayesian graph B 0 do 4:
Revised: Let B 0 be the corresponding Bayesian graph for the permuted full-chain factorization 3: for all combinations of link pruning on the full-chain Bayesian graph B 0 do 4:

Original: Let B be the corresponding pruned Bayesian graph 5:
Revised: Let B be the corresponding pruned Bayesian graph 5:

Original: Apply the Bayes-Ball algorithm on B to build a conditional independency list I 6: for all permutations of node factorization from X to Y do 7:
Revised: Apply the Bayes-Ball algorithm on B to build a conditional independency list I 6: for all permutations of node factorization from X to Y do 7:

Original: Let F 0 be the factor graph corresponding to a full-chain conditional probability 8:
Revised: Let F 0 be the factor graph corresponding to a full-chain conditional probability 8:

Original: Prune all redundant links in F 0 based on conditional independency
Revised: Prune all redundant links in F 0 based on conditional independency

Original: I 9:
Revised: I 9:

Original: Let F be the pruned factor graph 10:
Revised: Let F be the pruned factor graph 10:

Original: Merge the pruned Bayesian graph B into the pruned factor graph F 11:
Revised: Merge the pruned Bayesian graph B into the pruned factor graph F 11:

Original: Attach an adversary network A to latent nodes Z for Z k ⊥ S ∈
Revised: Attach an adversary network A to latent nodes Z for Z k ⊥ S ∈

Original: I 12:
Revised: I 12:

Original: Assign an encoder network E for p ( Z| · · · ) in the merged factor graph 13:
Revised: Assign an encoder network E for p ( Z| · · · ) in the merged factor graph 13:

Original: Assign a decoder network D for p ( x | · · · ) in the merged factor graph 14:
Revised: Assign a decoder network D for p ( x | · · · ) in the merged factor graph 14:

Original: Assign a nuisance indicator network N for p ( S| · · · ) in the merged factor graph 15:
Revised: Assign a nuisance indicator network N for p ( S| · · · ) in the merged factor graph 15:

Original: Assign a classiﬁer network C for p ( y | · · · ) in the merged factor graph 16:
Revised: Assign a classiﬁer network C for p ( y | · · · ) in the merged factor graph 16:

Original: Adversary train the whole DNN structure to minimize a loss function in (5) 17: end for (cid:46)
Revised: Adversary train the whole DNN structure to minimize a loss function in (5) 17: end for (cid:46)

Original: At most ( |V| − 2)!
Revised: At most ( |V| − 2)!

Original: combinations 18: end for (cid:46)
Revised: combinations 18: end for (cid:46)

Original: At most 2 |V| ( |V|− 1) / 2 combinations 19: end for (cid:46)
Revised: At most 2 | • AutoBayes automatically explores potential graphical models inherent to the data by combinatorial pruning of dependency assumptions (edges) and then applies Bayes-Ball to examine various inference strategies, rather than blindly exploring hyperparameters of DNN blocks. • AutoBayes offers a solid reason of how to connect multiple DNN blocks to impose conditioning and adversary censoring for the task classiﬁer, feature encoder, decoder, nuisance indicator and adversary networks, based on an explored Bayesian graph. • The framework is also extensible to multiple latent representations and nuisances factors.

Original: At most ( |V| − 2)!
Revised: • Besides fully-supervised training, AutoBayes can automatically build some relevant graphical models suited for semi-supervised learning. • Multiple graphical models explored in AutoBayes can be efﬁciently exploited to improve performance by ensemble stacking.

Original: combinations 20: return the best model having highest task accuracy in validation sets
Revised: We note that this paper relates to some existing literature in AutoML, variational Bayesian inference (Kingma & Welling, 2013; Sohn et al., 2015; Louizos et al., 2015), adversarial training (Goodfellow et al., 2014; Dumoulin et al., 2016; Donahue et al., 2016; Makhzani et al., 2015; Lample et al., 2017; Creswell et al., 2017), and Bayesian network (Nie et al., 2018; Njah et al., 2019; Rohekar et al., 2018) as addressed in Appendix A.1 in more detail. Nonetheless, AutoBayes is a novel framework that diverges from AutoML, which mostly employs architecture tuning at a micro level.

Revised: Our work focuses on exploring neural architectures at a macro level, which is not an arbitrary diversion, but a necessary interlude.

Revised: Our method focuses on the relationships between the connections in a neural network’s architecture and the characteristics of the data (Minsky & Papert, 2017).

Revised: In addition to the macro-level structure learning of Bayesian network, our approach provides a new perspective in how to involve the adversarial blocks and to exploit multiple models for ensemble stacking.

Revised:  A UTO B AYES

Revised: AutoBayes Algorithm:

Revised: The overall procedure of the AutoBayes algorithm is described in the pseudocode of Algorithm 1.

Revised: The AutoBayes automatically constructs non-redundant inference factor graphs given a hypothetical Bayesian graph assumption, through the use of the Bayes-Ball algorithm.

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: Depending on the derived conditional independency and pruned factor graphs, DNN blocks for encoder E , decoder D , classiﬁer C , nuisance estimator N and adversary A are reasonably connected.

Revised: The entire network is trained with variational Bayesian inference and adversarial training.

Revised: The Bayes-Ball algorithm (Shachter, 2013) facilitates an automatic pruning of redundant links in inference factor graphs through the analysis of conditional independency.

Revised: Fig.

Revised: 3 shows ten Bayes-Ball rules to identify conditional independency.

Revised: Given a Bayesian graph, we can determine whether two disjoint sets of nodes are independent conditionally on other nodes through a graph separation criterion.

Revised: Speciﬁcally, an undirected path is activated if a Bayes ball can travel along without encountering a stop symbol: in Fig.

Revised: If there are no active paths between two nodes when some conditioning nodes are shaded, then those random variables are conditionally independent.

Revised: Graphical Models: We here focus on 4 -node graphs.

Revised: Let p ( y, s, z, x ) denote the joint probability distribution underlying the datasets for the four random variables, i.e., Y , S , Z , and X .

Original:  rule can yield the following factorization for a generative model from Y to X (note that at most 4!
Revised: The chain rule can yield the following factorization for a generative model from Y to X (note that at most 4!

Original: factorization orders exist including useless ones such as reverse direction from X to Y ):
Revised: factorization orders exist including useless ones such as with reversed direction from X to Y ):

Original: [Equation]
Revised: [Equation]

Original: which is visualized in Bayesian graph of Fig. 4(a)
Revised: which is visualized in Bayesian graph of Fig. 4(a)

Original: The probability conditioned on X can  be factorized, e.g., as follows (among 3!
Revised: The probability conditioned on X can then be factorized, e.g., as follows (among 3!

Original: different orders of inference factorization for four-node graphs):
Revised: different orders of inference factorization for four-node graphs):

Original: [Equation]
Revised: [Equation]

Original: which are marginalized to obtain the likelihood: p ( y | x )
Revised: which are marginalized to obtain the likelihood: p ( y | x )

Original: =
Revised: =

Original: E s,z | x (cid:2) p ( y, s, z | x ) (cid:3) .
Revised: E s,z | x (cid:2) p ( y, s, z | x ) (cid:3) .

Original: The above two scheduling strategies in (2) are illustrated in factor graph models as in Figs. 4(b) and (c), respectively.
Revised: The above two scheduling strategies in (2) are illustrated in factor graph models as in Figs. 4(b) and (c), respectively.

Original: The graphical models in Fig.
Revised: The graphical models in Fig.

Original: 4 do not impose any assumption of potentially inherent independency in datasets and thus most generic.
Revised: 4 do not impose any assumption of potentially inherent independency in datasets and hence are most generic.

Original: However, depending on  underlying independency in datasets, we may be able to prune some edges in those graphs.
Revised: However, depending on the underlying independency in datasets, we may be able to prune some edges in those graphs.

Original: For example, if the data only follows the simple
Revised: For example, if the data only follows the simple

Original: Markov chain of Y − X , while being independent of S and Z , as shown in Fig.
Revised: Markov chain of Y − X , while being independent of S and Z , as shown in Fig.

Original: 5(a), all links except one between X and Y will be unreasonable in inference graphs of Figs. 4(b) and (c), that justiﬁes the standard classiﬁer model in Fig. 1(a).
Revised: 5(a), all links except one between X and Y will be unreasonable in inference graphs of Figs. 4(b) and (c), that justiﬁes the standard classiﬁer model in Fig. 1(a).

Original: This implies that more complicated inference models such as
Revised: This implies that more complicated inference models such as

Original: A-CVAE can be unnecessarily redundant depending on the dataset.
Revised: A-CVAE can be unnecessarily redundant depending on the dataset.

Original: This motivates us to consider an extended AutoML framework which automatically explores the best pair of inference factor graph and corresponding Bayesian graph models matching dataset statistics besides the micro-scale hyperparameter tuning.
Revised: This motivates us to consider an extended AutoML framework which automatically explores the best pair of inference factor graph and corresponding Bayesian graph models matching dataset statistics besides the micro-scale hyperparameter tuning.

Revised: Methodology:

Revised: AutoBayes begins with exploring any potential Bayesian graphs by cutting links of the full-chain graph in Fig. 4(a), imposing possible (conditional) independence.

Revised: We then adopt the

Revised: Bayes-Ball algorithm on each hypothetical Bayesian graph to examine conditional independence over

Original: Y
Revised: Y

Original: X (a) Model A
Revised: X (a) Model A

Original: Y
Revised: Y

Original: Z
Revised: Z

Original: X (b) Model B
Revised: X (b) Model B

Original: Y S
Revised: Y S

Original: X (c) Model C
Revised: X (c) Model C

Original: Y S
Revised: Y S

Original: Z
Revised: Z

Original: X (d) Model D
Revised: X (d) Model D

Original: Y S
Revised: Y S

Original: Z
Revised: Z

Original: X (e) Model E
Revised: X (e) Model E

Original: Y S
Revised: Y S

Original: Z
Revised: Z

Original: X (f) Model F
Revised: X (f) Model F

Original: Y S
Revised: Y S

Original: Z
Revised: Z

Original: X (g) Model G
Revised: X (g) Model G

Original: Y S
Revised: Y S

Original: Z
Revised: Z

Original: X (h) Model H
Revised: X (h) Model H

Original: Y S
Revised: Y S

Original: Z
Revised: Z

Original: X (i) Model I
Revised: X (i) Model I

Original: S
Revised: S

Original: Y
Revised: Y

Original: Z Z
Revised: Z Z

Original: X (j) Model J
Revised: X (j) Model J

Original: S
Revised: S

Original: Y
Revised: Y

Original: Z Z
Revised: Z Z

Original: X (k) Model K

Original: Methodology:

Original: AutoBayes begins with exploring any potential Bayesian graphs by cutting links of the full-chain graph in Fig. 4(a), imposing possible (conditional) independence.

Original: We then adopt the

Original: Bayes-Ball algorithm on each hypothetical Bayesian graph to examine conditional independence over different inference strategies, e.g., full-chain Z -/ S -ﬁrst inference graphs in Figs. 4(b)/(c).
Revised: X (k) Model K different inference strategies, e.g., full-chain Z -/ S -ﬁrst inference graphs in Figs. 4(b)/(c).

Original: Applying
Revised: Applying

Original: Bayes-Ball justiﬁes the reasonable pruning of the links in the full-chain inference graphs, and also the potential adversarial censoring when Z is independent of S .
Revised: Bayes-Ball justiﬁes the reasonable pruning of the links in the full-chain inference graphs, and also the potential adversarial censoring when Z is independent of S .

Original: This process automatically constructs a connectivity of inference, generative, and adversary blocks with sound reasoning.
Revised: This process automatically constructs the connectivity of inference, generative, and adversary blocks with sound reasoning.

Original: Consider an example case when the data adheres to the following factorization:
Revised: Consider an example case when the data adheres to the following factorization:

Original: [Equation]
Revised: [Equation]

Original: where we explicitly indicate conditional independence by slash-cancellation from the full-chain case in (1).
Revised: where we explicitly indicate conditional independence by slash-cancellation from the full-chain case in (1).

Original: This corresponds to a Bayesian graphical model illustrated in Fig. 5(e).
Revised: This corresponds to a Bayesian graphical model illustrated in Fig. 5(e).

Original: Applying the
Revised: Applying the

Original: Bayes-Ball algorithm to the Bayesian graph yields the following conditional probability:
Revised: Bayes-Ball algorithm to the Bayesian graph yields the following conditional probability:

Original: [Equation]
Revised: [Equation]

Original: for the Z-ﬁrst inference strategy in (2).
Revised: for the Z-ﬁrst inference strategy in (2).

Original: The corresponding factor graph is then given in Fig. 6(c).
Revised: The corresponding factor graph is then given in Fig. 6(c).

Original: Note that the Bayes-Ball also reveals that there is no marginal dependency between Z and S , which provides the reason to use adversarial censoring to suppress nuisance information S in the latent space Z .
Revised: Note that the Bayes-Ball also reveals that there is no marginal dependency between Z and S , which provides the reason to use adversarial censoring to suppress nuisance information S in the latent space Z .

Original: In consequence, by combining the Bayesian graph and factor graph, we automatically obtain
Revised: In consequence, by combining the Bayesian graph and factor graph, we automatically obtain

Original: A-CVAE model in Fig. 1(b).
Revised: A-CVAE model in Fig. 1(b).

Original: AutoBayes justiﬁes A-CVAE under the assumption that the data follows the Bayesian model E in Fig. 5(e).
Revised: AutoBayes justiﬁes A-CVAE under the assumption that the data follows the Bayesian model E in Fig. 5(e).

Original: As the true generative model is unknown, AutoBayes explores different Bayesian graphs like in Fig.
Revised: As the true generative model is unknown, AutoBayes explores different Bayesian graphs like in Fig.

Original: 5 to search for the most relevant model.
Revised: 5 to search for the most relevant model.

Revised: Our framework is readily applicable to graphs with more than 4 nodes to represent multiple Y , S , and Z .

Revised: Models J and

Revised: K in Fig.

Revised: 5 are such examples having multiple latent factors Z 1 and Z 2 .

Revised: Despite the search space for AutoBayes will rapidly grow with the number of nodes, most realistic datasets do not require a large number of neural network blocks for macro-level optimization.

Original: See Appendix A.2 for more detail descriptions for each Bayesian graph model to construct factor graphs like in Fig.
Revised: See Appendix A.2 for more detailed descriptions for some Bayesian graph models to construct factor graphs like in Fig.

Original: Also refer discussions of graphical models suitable for semi-supervised learning in Appendix A.4.
Revised: Also see discussions of graphical models suitable for semi-supervised learning in Appendix A.4.

Original: Training:
Revised: Training:

Original: Given a pair of generative graph and inference graph, the corresponding DNN structures will be trained.
Revised: Given a pair of generative graph and inference graph, the corresponding DNN structures will be trained.

Original: For example of the generative graph model K in Fig. 5(k), one relevant inference graph Kz in Fig.
Revised: For example of the generative graph model K in Fig. 5(k), one relevant inference graph Kz in Fig.

Original: 6(k) will result in the overall network structure as shown in Fig.
Revised: 6(k) will result in the overall network structure as shown in Fig.

Original: 7, where adversary network is attached as Z 2 is (conditionally) independent of S .
Revised: 7, where adversary network is attached as Z 2 is (conditionally) independent of S .

Original: This 5 -node graph model justiﬁes a recent work on partially disentanged A-CVAE by Han et al.
Revised: This 5 -node graph model justiﬁes a recent work on partially disentanged A-CVAE by Han et al.

Original: Each factor block is realized by a DNN, e.g., parameterized by θ for p
Revised: Each factor block is realized by a DNN, e.g., parameterized by θ for p

Original: θ ( z 1 , z 2 | x ) , and all of the networks except for adversarial network are optimized to minimize corresponding loss functions including L (ˆ y, y ) as follows:
Revised: θ ( z 1 , z 2 | x ) , and all of the networks except for adversarial network are optimized to minimize corresponding loss functions including L (ˆ y, y ) as follows:

Original: [Equation]
Revised: [Equation]

Original:  where λ ∗ denotes a regularization coefﬁcient, KL is the Kullback–Leibler divergence, and the adversary network p η ( s (cid:48) | z 2 ) is trained to minimize L (ˆ s (cid:48) , s ) in an alternating fashion (see the Adversarial
Revised: [Equation] where λ ∗ denotes a regularization coefﬁcient, KL is the Kullback–Leibler divergence, and the adversary network p η ( s (cid:48) | z 2 ) is trained to minimize L (ˆ s (cid:48) , s ) in an alternating fashion (see the Adversarial

Original: Regularization paragraph below).
Revised: Regularization paragraph below).

Original: [Figure]
Revised: [Figure]

Original: X

Original: [Figure]

Original: Z
Revised: Z

Original: Y
Revised: Y

Original: [Figure]
Revised: [Figure]

Original: Ds

Original: X
Revised: X

Original: Z
Revised: Z

Original: S
Revised: S

Original: Y (c) Model Ez
Revised: Y ( [Figure] Model Ez

Original: X
Revised: X

Original: Z
Revised: Z

Original: S
Revised: S

Original: Y (
Revised: Y (

Original: [Figure]
Revised: [Figure]

Original: ) Model Es X
Revised: [Figure] Es X

Original: Z
Revised: Z

Original: S
Revised: S

Original: Y (e) Model
Revised: Y (e) Model

Original: Fz
Revised: Fz

Original: X
Revised: X

Original: S
Revised: S

Original: Z
Revised: Z

Original: Y (f) Model Fs
Revised: Y ( [Figure] ) Model Fs

Original: [Figure]
Revised: [Figure]

Original: Gz

Original: X
Revised: X

Original: S Z
Revised: [Figure]

Original: Y (
Revised: Y (

Original: [Figure]
Revised: [Figure]

Original: Gs
Revised: Jz

Original: X
Revised: X

Original: Z Z S 
Revised: S Z

Original:  Y (i)
Revised: Z Y (

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Jz
Revised: Js

Original: X
Revised: X

Original: S Z Z
Revised: Z

Original: Y (j) Model Js X Z Z
Revised: Z

Original: S
Revised: S

Original: Y (k) Model Kz
Revised: Y ( [Figure] Kz

Original: X
Revised: X

Original: S
Revised: S

Original: ZZ
Revised: Z

Original:  Y ( [Figure]
Revised: Z Y ( [Figure]

Original: Ks

Original: Inference
Revised: Inference

Original: Adversary
Revised: Adversary

Original: [Figure]
Revised: [Figure]

Original: The training objective can be formally understood from a likelihood maximization perspective, in manner that can be seen as a generalization of the VAE Evidence Lower Bound (ELBO) concept.
Revised: The training objective can be formally understood from a likelihood maximization perspective, in manner that can be seen as a generalization of the VAE Evidence Lower Bound (ELBO) concept (Kingma & Welling, 2013).

Original: Speciﬁcally, it can be viewed as the maximization of a variational lower bound of the likelihood p
Revised: Speciﬁcally, it can be viewed as the maximization of a variational lower bound of the likelihood p

Original: Φ ( x, y, s ) that is implicitly deﬁned and parameterized by the networks, where Φ represents the collective parameters of the network modules (e.g., Φ = ( φ, ψ, µ ) in the example of equation 5) that specify the generative model p Φ ( x, y, s | z ) , which implies the likelihood p
Revised: Φ ( x, y, s ) that is implicitly deﬁned and parameterized by the networks, where Φ represents the collective parameters of the network modules (e [Equation] g., Φ = ( φ, ψ, µ ) in the example of equation 5) that specify the generative model p Φ ( x, y, s | z ) , which implies the likelihood p

Original: Φ ( x, y, s ) , as given by
Revised: Φ ( x, y, s ) , as given by

Original: [Equation]
Revised: [Equation]

Original: However, since this expression is generally intractable, we introduce q θ ( z | x, y, s ) as a variational approximation of the posterior p Φ ( z | x, y, s ) implied by the generative model (Kingma & Welling, 2013; Ranganath et al., 2014):
Revised: However, since this expression is generally intractable [Equation] we introduce q θ ( z | x, y, s ) as a variational approximation of the posterior p Φ ( z | x, y, s ) implied by the generative model (Kingma & Welling, 2013; Ranganath et al., 2014):

Original: [Equation]
Revised: [Equation]

Original: where the samples z i ∼ q θ ( z | x i , y i , s i ) are drawn for each training tuple ( x i , y i , s i ) , and the ﬁnal 
Revised: where the samples z i ∼ q θ ( z | x i , y i , s i ) are drawn for each training tuple ( x i , y i , s i ) , and the ﬁnal inequality follows from the non-negativity of KL divergence.

Original: [Table]
Revised: [Table]

Original: Ultimately, the minimization of our training loss function corresponds to the maximization of the lower bound in (7), which corresponds to maximizing the likelihood of our implicit generative model, while also optimizing the variational posterior q θ ( z | x, y, s ) toward the actual posterior for the latent representation p Φ ( z | x, y, s ) , since the gap in the bound is given by KL ( q θ ( z | x, y, s ) (cid:107) p Φ ( z | x, y, s )) .
Revised: Ultimately, the minimization of our training loss function corresponds to the maximization of the lower bound in (7), which corresponds to maximizing the likelihood of our implicit generative model, while also optimizing the variational posterior q θ ( z | x, y, s ) toward the actual posterior for the latent representation p Φ ( z | x, y, s ) , since the gap in the bound is given by KL ( q θ ( z | x, y, s ) (cid:107) p Φ ( z | x, y, s )) .

Original: Further factoring of log p Φ ( x, y, s | z ) yields the multiple loss-terms and network modules.
Revised: Further factoring of log p Φ ( x, y, s | z ) yields the multiple loss-terms and network modules.

Original: Adversarial Regularization: We can utilize adversarial censoring when Z and S should be marginally independent, e.g., such as in Fig. 1(b) and Fig.
Revised: Adversarial Regularization: We can utilize adversarial censoring when Z and S should be marginally independent, e.g., such as in Fig. 1(b) and Fig.

Original: 7, in order to reinforce the learning of a representation Z that is disentangled from the nuisance variations S .
Revised: 7, in order to reinforce the learning of a representation Z that is disentangled from the nuisance variations S .

Original: This is accomplished by introducing an adversarial network that aims to maximize a parameterized approximation q ( s | z ) of the likelihood p ( s | z ) , while this likelihood is also incorporated into the loss for the other modules with a negative weight.
Revised: This is accomplished by introducing an adversarial network that aims to maximize a parameterized approximation q ( s | z ) of the likelihood p ( s | z ) , while this likelihood is also incorporated into the loss for the other modules with a negative weight.

Original: The adversarial network, by maximizing the log likelihood log q ( s | z ) , essentially maximizes a lower-bound of the mutual information I ( S ; Z ) , and hence the main network is regularized with the additional term that corresponds to minimizing this estimate of mutual information.
Revised: The adversarial network, by maximizing the log likelihood log q ( s | z ) , essentially maximizes a lower-bound of the mutual information I ( S ; Z ) , and hence the main network is regularized with the additional term that corresponds to minimizing this estimate of mutual information.

Original: This follows since the log-likelihood maximized by the adversarial network is given by
Revised: This follows since the log-likelihood maximized by the adversarial network is given by

Original: [Equation]
Revised: [Equation]

Original: where the entropy H ( S ) is constant.
Revised: where the entropy H ( S ) is constant.

Original: Ensemble Learning: We further introduce ensemble methods to make best use of all Bayesian graph models explored by the AutoBayes framework without wasting lower-performance models.
Revised: Ensemble Learning: We further introduce ensemble methods to make best use of all Bayesian graph models explored by the AutoBayes framework without wasting lower-performance models.

Original: Ensemble stacked generalization works by stacking the predictions of the base learners in a higher level learning space, where a meta learner corrects the predictions of base learners (Wolpert, 1992).
Revised: Ensemble stacked generalization works by stacking the predictions of the base learners in a higher level learning space, where a meta learner corrects the predictions of base learners (Wolpert, 1992).

Original: Subsequent to training base learners, we assemble the posterior probability vectors of all base learners together to improve the prediction.
Revised: Subsequent to training base learners, we assemble the posterior probability vectors of all base learners together to improve the prediction.

Original: We compare the predictive performance of a logistic regression (LR) and a shallow multi-layer perceptron (MLP) as an ensemble meta learner to aggregate all inference models.
Revised: We compare the predictive performance of a logistic regression (LR) and a shallow multi-layer perceptron (MLP) as an ensemble meta learner to aggregate all inference models.

Original: See Appendix A.5 for more detailed description of the stacked generalization.
Revised: See Appendix A.5 for more detailed description of the stacked generalization.

Original:  E XPERIMENTAL E VALUATION
Revised:  E XPERIMENTAL E VALUATION

Original: Datasets: We experimentally demonstrate the performance of AutoBayes for publicly available datasets as listed in Table 1.
Revised: Datasets: We experimentally demonstrate the performance of AutoBayes for publicly available datasets as listed in Table 1.

Original: Note that they cover a wide variety of data size, dimensionality, subject scale, and class levels as well as sensor modalities including image, EEG, EMG, and electrocorticography (ECoG).
Revised: Note that they cover a wide variety of data size, dimensionality, subject scale, and class levels as well as sensor modalities including image, EEG, EMG, and electrocorticography (ECoG).

Original: See more detailed information of each dataset in Appendix A.6.
Revised: See more detailed information of each dataset in Appendix A.6.

Original: Model Implementation: All models were trained with a minibatch size of 32 and using the Adam optimizer with an initial learning rate of 0 .
Revised: Model Implementation: All models were trained with a minibatch size of 32 and using the Adam optimizer with an initial learning rate of 0 .

Original: The learning rate is halved whenever the validation loss plateaus.
Revised: The learning rate is halved whenever the validation loss plateaus.

Original: A compact convolutional neural network (CNN) with 4 layers is employed as an encoder network E to extract features from C × T data.
Revised: A compact convolutional neural network (CNN) with 4 layers is employed as an encoder network E to extract features from C × T data.

Original: Each convolution is followed by batch normalization (BN) and rectiﬁed linear unit (ReLU) activation.
Revised: Each convolution is followed by batch normalization (BN) and rectiﬁed linear unit (ReLU) activation.

Original: The AutoBayes chooses either a deterministic latent encoder or variational latent encoder under Gaussian prior.
Revised: The AutoBayes chooses either a deterministic latent encoder or variational latent encoder under Gaussian prior.

Original: The original data is reconstructed by a decoder network D that applies transposed convolutions.
Revised: The original data is reconstructed by a decoder network D that applies transposed convolutions.

Original: All of our experiments were run for 20 epochs on NVIDIA Tesla K80 12GB GPU.
Revised: All of our experiments were run for 20 epochs on NVIDIA Tesla K80 12GB GPU.

Original: See Appendix A.7 for more details.
Revised: See Appendix A.7 for more details.

Original: Results: Fig. 8(a) shows the performance of QMNIST across 39 different inference models explored in AutoBayes including 2 ensemble models.
Revised: Results: Fig. 8(a) shows the performance of QMNIST across 39 different inference models explored in AutoBayes including 2 ensemble models.

Original: Over 37 base models
Revised: Over 37 base models

Original: , some outperforms the standard
Revised: , some outperforms the standard

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: classiﬁer model A, whereas the rest of the models underperform.
Revised: classiﬁer model A, whereas the rest of the models underperform.

Original: We observe a large gap of 1 . 0 % between the best and worst models with a standard deviation of 0 .
Revised: We observe a large gap of 1 . 0 % between the best and worst models with a standard deviation of 0 .

Original: 23 % across all Bayesian  models.
Revised: 23 % across all Bayesian graph models.

Original: This indicates that we may have a potential risk that one particular model may lose up to 1 . 0 % accuracy if we do not explore different models.
Revised: This indicates that we may have a potential risk that one particular model may lose up to 1 . 0 % accuracy if we do not explore different models.

Original: Similar behaviors with a huge deviation can be seen for different datasets as shown in Fig. 8(b).
Revised: Similar behaviors with a huge deviation can be seen for different datasets as shown in Fig. 8(b).

Original: It was shown that the best inference strategy highly depends on datasets.
Revised: It was shown that the best inference strategy highly depends on datasets.

Original: Speciﬁcally, the best model at one dataset does not perform best for different datasets.
Revised: Speciﬁcally, the best model at one dataset does not perform best for different datasets.

Original: This suggests that we must consider different inference strategies for each target dataset and our AutoBayes provides such an adaptive framework across datasets.
Revised: This suggests that we must consider different inference strategies for each target dataset and our AutoBayes provides such an adaptive framework across datasets.

Original: More detailed results are found in Appendix A.8.
Revised: More detailed results are found in Appendix A.8.

Original: Remarkably, the ensemble of base learners further enhances the performance regardless of the choice from LR or MLP as the meta learner, as illustrated in Fig.
Revised: Remarkably, the ensemble of base learners further enhances the performance regardless of the choice from LR or MLP as the meta learner, as illustrated in Fig.

Original: 2 across all the datasets.
Revised: 2 across all the datasets.

Original: For some lowperforming datasets such as ErrP, MI and Faces (Noisy), ensemble learning signiﬁcantly improves the accuracy by 15 .
Revised: For some lowperforming datasets such as ErrP, MI and Faces (Noisy), ensemble learning signiﬁcantly improves the accuracy by 15 .

Original: 3 % and 13 .
Revised: 3 % and 13 .

Original: 2 % at the expense of more storage and computational resources.
Revised: 2 % at the expense of more storage and computational resources.

Original: Exploring different models has actually a signiﬁcant beneﬁt in improving nuisance robustness as shown in Fig.
Revised: Exploring different models has actually a signiﬁcant beneﬁt in improving nuisance robustness as shown in Fig.

Original: 9(a), where box-whisker plots are present to show the quartile distribution of the subject variation for the Stress dataset having | S | = 20 users.
Revised: 9(a), where box-whisker plots are present to show the quartile distribution of the subject variation for the Stress dataset having | S | = 20 users.

Original: We can observe that the standard classiﬁcation (Model A) has a wider distribution; the best subject achieves an accuracy grater than 96% , whereas the worst-case user has lower than 82% accuracy.
Revised: We can observe that the standard classiﬁcation (Model A) has a wider distribution; the best subject achieves an accuracy grater than 96% , whereas the worst-case user has lower than 82% accuracy.

Original: Except for model A, the other models from B to Kz take the subject ID ( S ) into consideration to extract nuisance-robust feature, which leads to signiﬁcant improvement for the worst-case user performance not only for the mean or median.
Revised: Except for model A, the other models from B to Kz take the subject ID ( S ) into consideration to extract nuisance-robust feature, which leads to signiﬁcant improvement for the worst-case user performance not only for the mean or median.

Original: The
Revised: The

Original: [Table]
Revised: [Table]

Original: ensemble stacking further improves the subject variation robustness, achieving the worst-case user performance of at least 94% .
Revised: ensemble stacking further improves the subject variation robustness, achieving the worst-case user performance of at least 94% .

Original: Additional results per user are found in Appendix A.9.
Revised: Additional results per user are found in Appendix A.9.

Original: Despite the performance gain, the nuisance-robust models tend to have higher complexity.
Revised: Despite the performance gain, the nuisance-robust models tend to have higher complexity.

Original: Fig. 9(b) shows the trade-off between the accuracy and the space complexity.
Revised: Fig. 9(b) shows the trade-off between the accuracy and the space complexity.

Original: Here, we varied the number of hidden layers and hidden nodes for the models A, B, and Js to adjust the space complexity.
Revised: Here, we varied the number of hidden layers and hidden nodes for the models A, B, and Js to adjust the space complexity.

Original: The
Revised: The

Original: Pareto front over the ﬁnite set of DNN conﬁgurations is indicated with lines.
Revised: Pareto front over the ﬁnite set of DNN conﬁgurations is indicated with lines.

Original: It is observed that the standard classiﬁer model A has superior performance only at low complexity regimes, while it does not improve performance beyond 95% accuracy even with increased complexity.
Revised: It is observed that the standard classiﬁer model A has superior performance only at low complexity regimes, while it does not improve performance beyond 95% accuracy even with increased complexity.

Original: The Pareto front of
Revised: The Pareto front of

Original: AutoBayes is thus better than the individual models at higher accuracy regimes.
Revised: AutoBayes is thus better than the individual models at higher accuracy regimes.

Original: See Appendix A. for an additional analysis of time complexity.
Revised: See Appendix A. for an additional analysis of time complexity.

Original: We benchmark the performance of AutoBayes against the benchmark competitor models from (Byerly et al., 2020; Han et al., 2020; Özdenizci et al., 2019c;b;a) in Table 2.
Revised: We ﬁnally compare the performance of AutoBayes with the benchmark competitor models from (Byerly et al., 2020; Han et al., 2020; Özdenizci et al., 2019c;b;a) in Table 2.

Original: It can be seen that AutoBayes outperforms the state-of-the-art in all datasets except QMNIST.
Revised: It can be seen that AutoBayes outperforms the state-of-the-art in all datasets except QMNIST.

Original: Consequently, we can see a great advantage of AutoBayes with exploring different graphical models.
Revised: Consequently, we can see a great advantage of AutoBayes with exploring different graphical models.

Original: Even forQMNIST,
Revised: Even for

Original:  AutoBayes meta-MLP model, achieving 99 .
Revised: QMNIST, AutoBayes meta-MLP model, achieving 99 .

Original: 61 % accuracy, ranks 17 in the published leaderboard.
Revised: 61 % accuracy, ranks 17 in the published leaderboard.

Original: Note that performing better than 99 .
Revised: Note that performing better than 99 .

Original: 84 % is nearly impossible, since some numbers are illegible or mislabeled.
Revised: 84 % is nearly impossible, since some numbers are illegible or mislabeled.

Original: Also note that we have not speciﬁcally designed AutoBayes architecture for image classiﬁcation but for spatio-temporal signal applications andmicro-scale hyper-parameters were not fully optimized yet.
Revised: Also note that we have not speciﬁcally designed AutoBayes architecture for image classiﬁcation but for spatio-temporal signal applications and hyper-parameters were not fully optimized yet.

Original: AutoBayes can be readily integrated with AutoML to optimize any hyperparameters of individual
Revised: AutoBayes can be readily integrated with AutoML to optimize any hyperparameters of individual

Original: DNN blocks.
Revised: DNN blocks.

Original: Nevertheless, as our primary objective was to show a proof-of-concept beneﬁt from solely graphical model exploration of AutoBayes, we leave more rigorous analysis to optimize DNN parameters such as network depths, widths, activation, augmentation, etc.
Revised: Nevertheless, as our primary objective was to show a proof-of-concept beneﬁt from solely graphical model exploration of AutoBayes, we leave more rigorous analysis to optimize DNN parameters such as network depths, widths, activation, augmentation, etc.

Original: as a future work.
Revised: as a future work.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We proposed a new concept called AutoBayes which explores various different Bayesian graph models to facilitate searching for the best inference strategy, suited for nuisance-robust deep learning.
Revised: We proposed a new concept called AutoBayes which explores various different Bayesian graph models to facilitate searching for the best inference strategy, suited for nuisance-robust deep learning.

Original: With the Bayes-Ball algorithm, our method can automatically construct reasonable link connections among classiﬁer, encoder, decoder, nuisance estimator and adversary DNN blocks.
Revised: With the Bayes-Ball algorithm, our method can automatically construct reasonable link connections among classiﬁer, encoder, decoder, nuisance estimator and adversary DNN blocks.

Original: As a proofof-concept analysis, we demonstrated the beneﬁt of AutoBayes for various public datasets.
Revised: As a proofof-concept analysis, we demonstrated the beneﬁt of AutoBayes for various public datasets.

Original: We observed a huge performance gap between the best and worst graph models, implying that the use of one deterministic model without graph exploration can potentially suffer a poor classiﬁcation result.
Revised: We observed a huge performance gap between the best and worst graph models, implying that the use of one particular model without graph exploration can potentially suffer a poor classiﬁcation result.

Original: In addition, the best model for one dataset does not always perform best for different data, which encourages us to use AutoBayes for adaptive model generation given target datasets.
Revised: In addition, the best model for one dataset does not always perform best for different data, which encourages us to use AutoBayes for adaptive model generation given target datasets.

Original: We further improved the performance approaching the state-of-the-art accuracy by exploiting multiple graphical models explored in AutoBayes through the use of ensemble stacking.
Revised: We further improved the performance approaching the state-of-the-art accuracy by exploiting multiple graphical models explored in AutoBayes through the use of ensemble stacking.

Original: We are extending the 
Revised: The ensemble AutoBayes offers signiﬁcant gain in nuisance robustness by improving the worst-case user performance. Even though additional computations are required, we showed that AutoBayes can still achieve the superior Pareto front in the trade-off between complexity and accuracy.

Original:  AutoBayes framework to integrate AutoML to optimize hyperparameters of each DNN block.
Revised: We are extending the AutoBayes framework to integrate AutoML to optimize hyperparameters of each DNN block.

Original: How to handle the exponentially growing search space of possible Bayesian graphs along with the number of random variables remains a challenging future work.
Revised: How to handle the exponentially growing search space of possible Bayesian graphs along with the number of random variables remains a challenging future work.

Original: It should require more sophisticated criteria of efﬁcient  exploration.
Revised: It should require more sophisticated metrics like Bayesian information criterion for efﬁcient graph exploration.

Original: A PPENDICES A. R ELATED W ORK
Revised: A PPENDICES A. R ELATED W ORK

Original: We note that this paper relates to some existing literature as follows.
Revised: We note that this paper relates to some existing literature as follows.

Original: • AutoML: Searching DNN models with hyperparameter optimization has been intensively investigated in a framework called AutoML (Ashok et al., 2017;
Revised: • AutoML: Searching DNN models with hyperparameter optimization has been intensively investigated in a framework called AutoML (Ashok et al., 2017;

Original: Brock et al., 2017; Cai et al., 2017; He et al., 2018; Miikkulainen et al., 2019; Real et al.,2017; 2020; Stanley & Miikkulainen, 2002;
Revised: Brock et al., 2017; Cai et al., 2017; He et al., 2018; Miikkulainen et al., 2019; Real et al.,

Revised: ; Stanley & Miikkulainen

Original:  Zoph
Revised: ; Zoph

Original: et al
Revised: et al.,

Original: The automated methods include architecture search (Zoph et al., 2018;
Revised: The automated methods include architecture search (Zoph et al., 2018;

Original: Real et al., 2017; He et al., 2018; Real et al., 2020), learning rule design (Bayer et al., 2009; Jozefowicz et al., 2015), and augmentation exploration (Cubuk et al., 2019; Park et al., 2019).
Revised: Real et al., 2017; He et al., 2018; Real et al., 2020), learning rule design (Bayer et al., 2009; Jozefowicz et al., 2015), and augmentation exploration (Cubuk et al., 2019; Park et al., 2019).

Original: Most work used either evolutionary optimization or reinforcement learning framework to adjust hyperparameters or to construct network architecture from pre-selected building blocks.
Revised: Most work used either evolutionary optimization or reinforcement learning framework to adjust hyperparameters or to construct network architecture from pre-selected building blocks.

Original: (Miconi, 2016) gradually increases the size of an RNN starting from only one node by incorporating structural parameters into model training, which are optimized along with the model weights.
Revised: (Miconi, 2016) gradually increases the size of an RNN starting from only one node by incorporating structural parameters into model training, which are optimized along with the model weights.

Original: (Zoph & Le, 2016) uses reinforcement learning to ﬁnd the optimal neural network architecture based on actor-critic framework.
Revised: (Zoph & Le, 2016) uses reinforcement learning to ﬁnd the optimal neural network architecture based on actor-critic framework.

Original: The method uses an LSTM as a controller and critic to explore the hyperparameter conﬁgurations for each layer (number of ﬁlters, kernel size and stride) based on the validation error of the output architecture that corresponds to reward.
Revised: The method uses an LSTM as a controller and critic to explore the hyperparameter conﬁgurations for each layer (number of ﬁlters, kernel size and stride) based on the validation error of the output architecture that corresponds to reward.

Original: The recent AutoML-Zero (Real et al., 2020) considers an extension to preclude human knowledge and insights for fully automated designs from scratch.
Revised: The recent AutoML-Zero (Real et al., 2020) considers an extension to preclude human knowledge and insights for fully automated designs from scratch.

Original: • Variational Bayesian Inference:Deep Bayesian network (Nie et al., 2018; Njah et al., 2019; Rohekar et al., 2018) has been studied to learn probabilistic relation between random variables. The VAE (Kingma & Welling, 2013) introduced variational Bayesian inference methods, incorporating autoassociative architectures, where generative and inference models can be learned jointly.
Revised: • Variational Bayesian Inference: The VAE (Kingma & Welling, 2013) introduced variational Bayesian inference methods, incorporating autoassociative architectures, where generative and inference models can be learned jointly.

Original: This method was extended with the CVAE (Sohn et al., 2015), which introduces a conditioning variable that could be used to represent nuisance variations, and a regularized VAE in (Louizos et al., 2015), which considers disentangling the nuisance variable from the latent representation.
Revised: This method was extended with the CVAE (Sohn et al., 2015), which introduces a conditioning variable that could be used to represent nuisance variations, and a regularized VAE in (Louizos et al., 2015), which considers disentangling the nuisance variable from the latent representation.

Original: • Adversarial Training:
Revised: • Adversarial Training:

Original: The concept of adversarial networks was introduced with Generative Adversarial Networks (GAN) (Goodfellow et al., 2014), and has been adopted into myriad applications.
Revised: The concept of adversarial networks was introduced with Generative Adversarial Networks (GAN) (Goodfellow et al., 2014), and has been adopted into myriad applications.

Original: The simultaneously discovered Adversarially Learned Inference (ALI) (Dumoulin et al., 2016) and Bidirectional GAN (BiGAN) (Donahue et al., 2016) propose an adversarial approach toward training an autoencoder.
Revised: The simultaneously discovered Adversarially Learned Inference (ALI) (Dumoulin et al., 2016) and Bidirectional GAN (BiGAN) (Donahue et al., 2016) propose an adversarial approach toward training an autoencoder.

Original: Adversarial training has also been combined with VAE to regularize and/or disentangle the latent representations (Makhzani et al., 2015;
Revised: Adversarial training has also been combined with VAE to regularize and/or disentangle the latent representations (Makhzani et al., 2015;

Original: Lample et al., 2017;
Revised: Lample et al., 2017;

Original: Creswell et al., 2017).
Revised: Creswell et al., 2017).

Original: • Bayesian Network Structure Learning: Learning model structure of a Bayesian network (BN) is a problem that haslong been studied in the probabilistic graphical models domain. The simplest approach was constructing a BN driven by a human expert’s knowledge in the dataset domain.
Revised: • Bayesian Network Structure Learning: Deep Bayesian network (Nie et al., 2018; Njah et al., 2019; Rohekar et al., 2018) has been studied to learn probabilistic relationships between random variables.

Original: One of the simplest methods is  recovery algorithmdeveloped by (Rebane & Pearl, 2013), which relies on the distinction between the three possible patterns in a 3 -node DAG. Recently, most common approaches are scoring-based methods  and constraint-based methods. Scoring-based methods require a scoring function and a search strategy.
Revised: Learning model structure of a Bayesian network is a problem that has long been studied, e.g., recovery algorithm (Rebane & Pearl, 2013), scoring methods (Campos, 2006), and constraint methods (Scutari, 2014; Pearl et al., 2000).

Original: Scoring function is commonly  the posterior probability of the BN given training data, such as BIC or BDeu (Campos, 2006). A score assigned to the DAG of a BN reﬂects its goodness of ﬁt, which is maximized by a search strategy.
Revised: Scoring methods commonly use the posterior probability of the Bayesian network given training data, such as Bayesian information criterion (BIC).

Original: The time complexity of an exhaustive search is superexponential in the number of variables,but recent approaches handle structure learning as an optimization problem and can learn structure of BNs with up to 100 variables using integer programming (Cussens et al., 2017).
Revised: Although the complexity of an exhaustive search is superexponential in the number of variables, recent approaches (Cussens et al., 2017) showed capability to learn structure of Bayesian network with up to 100 variables using integer programming.

Original: Constraint-based methods use conditional independence tests between pairs of variables, commonly mutual information test (for discrete BNs) and theexact Student’s t test for correlation (for GBNs) (Scutari, 2014). They are particularly faster than scoring-based methods, but sensitive to errors in conditional independence tests.
Revised: Constraint-based methods use conditional independence tests between pairs of variables, commonly mutual information test or the Student’s t-test for correlation.

Original: All constraint-based methods entail three phases: i.e., (i) learning Markov blankets of each variable. , (ii) learning neighbors (parents and children) of each variable that identiﬁes which arcs are present inthe DAG of a BN., and (iii) establishing arc directions (Pearl et al., 2000).
Revised: All constraint-based methods entail three phases: i.e., (i) learning Markov blankets of each variable, (ii) learning neighbors (parents and children) of each variable that identiﬁes which arcs are present in a Bayesian network, and (iii) establishing arc directions.

Original: Compared to the existing AutoML literature, our method provides more systematic framework to explore justiﬁable network architectures from a macro view.
Revised: Compared to the existing AutoML literature, our method provides more systematic framework to explore justiﬁable network architectures from a macro view.

Original: Although related Bayesian network was studied to design DNN architecture, our method extends it to realize nuisance robustness by reasonably involving adversarial networks.
Revised: Although related Bayesian network was studied to design DNN architecture, our method extends it to realize nuisance robustness by reasonably involving adversarial networks.

Original: In addition, ensemble stacking was ﬁrst introduced in
Revised: In addition, ensemble stacking was ﬁrst introduced in

Original: AutoML framework where multiple architectures can be reused to improve the performance over every individual model.
Revised: AutoML framework where multiple architectures can be reused to improve the performance over every individual model.

Original: A. B AYESIAN G RAPH AND I NFERENCE M ODELS
Revised: A. B AYESIAN G RAPH AND I NFERENCE M ODELS

Original: Given measurement data, we never know the true joint probability beforehand, and therefore we shall assume one of several possible generative models.
Revised: Given measurement data, we never know the true joint probability beforehand, and therefore we shall assume one of several possible generative models.

Original: AutoBayes aims to explore such potential graph models to match the measurement distributions.
Revised: AutoBayes aims to explore such potential graph models to match the measurement distributions.

Original: As the maximum possible number of graphical models is huge even for a four-node case involving Y , S , Z and X , we restrict our focus to a few meaningful graphs-of-interest shown in Fig.
Revised: As the maximum possible number of graphical models is huge even for a four-node case involving Y , S , Z and X , we restrict our focus to a few meaningful graphs-of-interest shown in Fig.

Original: Each Bayesian graph corresponds to the following assumption of the joint probability factorization ( p ( x | · · · ) term speciﬁes a generative model of X ): p ( y, s, z, x ) =
Revised: Each Bayesian graph corresponds to the following assumption of the joint probability factorization ( p ( x | · · · ) term speciﬁes a generative model of X ): p ( y, s, z, x ) =

Original:      p ( y ) p ( s | (cid:1) y ) p ( z | (cid:1) s, (cid:1) y )
Revised:      p ( y ) p ( s | (cid:1) y ) p ( z | (cid:1) s, (cid:1) y )

Original: p ( x | (cid:1) z, (cid:1) s, y ) ,
Revised: p ( x | (cid:1) z, (cid:1) s, y ) ,

Original: Model-A p ( y ) p ( s | (cid:1) y ) p ( z | (cid:1) s, y ) p ( x | z, (cid:1) s, (cid:1) y ) ,
Revised: Model-A p ( y ) p ( s | (cid:1) y ) p ( z | (cid:1) s, y ) p ( x | z, (cid:1) s, (cid:1) y ) ,

Original: Model-B p ( y ) p ( s | (cid:1) y ) p ( z | (cid:1) s, (cid:1) y )
Revised: Model-B p ( y ) p ( s | (cid:1) y ) p ( z | (cid:1) s, (cid:1) y )

Original: p ( x | (cid:1) z, s, y ) ,
Revised: p ( x | (cid:1) z, s, y ) ,

Original: Model-C p ( y ) p ( s | (cid:1) y ) p ( z | s, y ) p ( x | z, (cid:1) s, (cid:1) y ) ,
Revised: Model-C p ( y ) p ( s | (cid:1) y ) p ( z | s, y ) p ( x | z, (cid:1) s, (cid:1) y ) ,

Original: Model-D p ( y ) p ( s | (cid:1) y ) p ( z | (cid:1) s, y ) p ( x | z, s, (cid:1) y ) ,Model-E p ( y ) p ( s | (cid:1) y ) p ( z | s, (cid:1) y ) p ( x | z, (cid:1) s, y ) , Model-F p ( y ) p ( s | (cid:1) y ) p ( z | s, y ) p ( x | z, s, (cid:1) y ) , Model-G p ( y ) p ( s | (cid:1) y ) p ( z | s, y ) p ( x | z, (cid:1) s, y ) ,
Revised: Model-D p ( y ) p ( s | (cid:1) y ) p ( z | (cid:1) s, y ) p ( x | z, s, (cid:1) y ) ,

Original: Model-H p ( y ) p ( s | (cid:1) y ) p ( z | s,  y ) p ( x | z,  s, y ) , Model-I p ( y ) p ( s | (cid:1) y ) p( z 1 | s, (cid:1) y )
Revised: Model-E p ( y ) p ( s | (cid:1) y ) p ( z | s, (cid:1) y ) p ( x | z, (cid:1) s, y ) , Model-F p ( y ) p ( s | (cid:1) y ) p

Original: p ( z 2 | (cid:26)(cid:26) z 1 , (cid:1) s, y ) p ( x | z 2 , z 1 , (cid:1) s, (cid:1) y ) ,
Revised: [Equation]

Original: Model-J p ( y ) p ( s | (cid:1) y ) p ( z 1 | s, (cid:1) y )
Revised: Model-G [Equation] Model-H

Original: p ( z 2 | z 1 , (cid:1) s, y ) p ( x
Revised: [Equation]

Original: | z 2 , z 1 , (cid:1) s, (cid:1) y ) , Model-K (9) where we explicitly indicate independence by slash-cancelled factors from the full-chain case in equation 1.
Revised: Model-I [Equation] where we explicitly indicate independence by slash-cancelled factors from the full-chain case in equation 1.

Original: Blue-colored terms correspond to the blue arrows in Figs.
Revised: Blue-colored terms correspond to the blue arrows in Figs.

Original: 5 for generative graph of decoder networks.
Revised: 5 for generative graph of decoder networks.

Original: Depending on the assumed Bayesian graph, the relevant inference strategy will vary as some variables may be conditionally independent, which enables pruning links in the inference factor graphs.
Revised: Depending on the assumed Bayesian graph, the relevant inference strategy will vary as some variables may be conditionally independent, which enables pruning links in the inference factor graphs.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6, the reasonable inference graph model can be automatically generated by the Bayes-Ball algorithm (Shachter, 2013) on each Bayesian graph hypothesis inherent in datasets.
Revised: 6, the reasonable inference graph model can be automatically generated by the Bayes-Ball algorithm (Shachter, 2013) on each Bayesian graph hypothesis inherent in datasets.

Original: Speciﬁcally, the conditional probability p ( y, s, z | x ) can be obtained for each model as below.
Revised: Speciﬁcally, the conditional probability p ( y, s, z | x ) can be obtained for each model as below.

Original: Bayesian Graph Model A (Direct Markov): The simplest model between X and Y would be single Markov chain without any dependency of S and Z , shown in Bayesian graph of Fig. 5(a).
Revised: Bayesian Graph Model A (Direct Markov): The simplest model between X and Y would be single Markov chain without any dependency of S and Z , shown in Bayesian graph of Fig. 5(a).

Original: This model puts an assumption that the data are nuisance-invariant.
Revised: This model puts an assumption that the data are nuisance-invariant.

Original: For this case, there is no reason to employ complicated inference models such as A-CVAE since most factors will be independent as p ( y, s, z | x ) =
Revised: For this case, there is no reason to employ complicated inference models such as A-CVAE since most factors will be independent as p ( y, s, z | x ) =

Original: p ( z | (cid:1) x ) p ( s | (cid:1)
Revised: p ( z | (cid:1) x ) p ( s | (cid:1)

Original: z, (cid:1) x ) p ( y | (cid:1) s, (cid:1)
Revised: z, (cid:1) x ) p ( y | (cid:1) s, (cid:1)

Original: z, x ) .
Revised: z, x ) .

Original: We hence should use a standard classiﬁcation method, as in Fig. 1(a), to infer Y given X , based on the inference model p ( y | x ) without involving S and Z .
Revised: We hence should use a standard classiﬁcation method, as in Fig. 1(a), to infer Y given X , based on the inference model p ( y | x ) without involving S and Z .

Original: Bayesian Graph Model B (Markov Latent): Assuming a latent Z can work in a Markov chain of Y − Z − X shown in Fig. 5(b), we obtain a simple inference model: p ( y, s, z | x ) =
Revised: Bayesian Graph Model B (Markov Latent): Assuming a latent Z can work in a Markov chain of Y − Z − X shown in Fig. 5(b), we obtain a simple inference model: p ( y, s, z | x ) =

Original: p ( z | x ) p ( s | (cid:1)
Revised: p ( z | x ) p ( s | (cid:1)

Original: z, (cid:1) x ) p ( y | (cid:1) s, z, (cid:1) x ) .
Revised: z, (cid:1) x ) p ( y | (cid:1) s, z, (cid:1) x ) .

Original: Note that this model assumes independence between Z and S , and thus adversarial censoring (Makhzani et al., 2015;
Revised: Note that this model assumes independence between Z and S , and thus adversarial censoring (Makhzani et al., 2015;

Original: Creswell et al., 2017; Lample et al., 2017) can make it more robust against nuisance.
Revised: Creswell et al., 2017; Lample et al., 2017) can make it more robust against nuisance.

Original: This model is hence based on A-VAE.
Revised: This model is hence based on A-VAE.

Original: Bayesian Graph Model C (Subject-Dependent): We may model the case when the data X directly depends on subject S and task Y , shown in Fig. 5(c).
Revised: Bayesian Graph Model C (Subject-Dependent): We may model the case when the data X directly depends on subject S and task Y , shown in Fig. 5(c).

Original: For this case, we may consider the corresponding inference models due to the Bayes-Ball:
Revised: For this case, we may consider the corresponding inference models due to the Bayes-Ball:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that this model does not depend on Z , and thus Z -ﬁrst inference strategy reduces to S -ﬁrst model.
Revised: Note that this model does not depend on Z , and thus Z -ﬁrst inference strategy reduces to S -ﬁrst model.

Original: As a reference, we here consider additional Y -ﬁrst inference strategy to evaluate the difference.
Revised: As a reference, we here consider additional Y -ﬁrst inference strategy to evaluate the difference.

Original: Bayesian Graph Model D (Latent Summary): Another graphical model is shown in Fig. 5(d), where a latent space bridges all other random variables.
Revised: Bayesian Graph Model D (Latent Summary): Another graphical model is shown in Fig. 5(d), where a latent space bridges all other random variables.

Original: Bayes-Ball yields the following models:
Revised: Bayes-Ball yields the following models:

Original: [Equation]
Revised: [Equation]

Original: whose graphical models are depicted in Figs. 6(a) and (b), respectively.
Revised: whose graphical models are depicted in Figs. 6(a) and (b), respectively.

Original: Bayesian Graph Model E (Task-Summary Latent): Another graphical model involving latent variables is shown in Fig. 5(e), where a latent space only summarizes Y .
Revised: Bayesian Graph Model E (Task-Summary Latent): Another graphical model involving latent variables is shown in Fig. 5(e), where a latent space only summarizes Y .

Original: Bayes-Ball yields the following inference models:
Revised: Bayes-Ball yields the following inference models:

Original: [Equation]
Revised: [Equation]

Original: which are illustrated in Figs. 6(c) and (d).
Revised: which are illustrated in Figs. 6(c) and (d).

Original: Note that the generative model E has no marginal dependency between Z and S , which provides the reason to use adversarial censoring to suppress nuisance information S in the latent space Z .
Revised: Note that the generative model E has no marginal dependency between Z and S , which provides the reason to use adversarial censoring to suppress nuisance information S in the latent space Z .

Original: In addition, because the generative model of X is dependent on both Z and S , it is justiﬁed to employ the A-CVAE classiﬁer shown in Fig. 1(b).
Revised: In addition, because the generative model of X is dependent on both Z and S , it is justiﬁed to employ the A-CVAE classiﬁer shown in Fig. 1(b).

Original: Bayesian Graph Model F (Subject-Summary Latent): Consider Fig. 5(f), where a latent variable summarizes subject information S .
Revised: Bayesian Graph Model F (Subject-Summary Latent): Consider Fig. 5(f), where a latent variable summarizes subject information S .

Original: The Bayes-Ball provides the inference graphs shown in Figs. 6(e) and (f), which respectively correspond to:
Revised: The Bayes-Ball provides the inference graphs shown in Figs. 6(e) and (f), which respectively correspond to:

Original: [Equation]
Revised: [Equation]

Original: Bayesian Graph Model G: Letting the joint distribution follow the model G in Fig. 5(g), we obtain the following inference models via the Bayes-Ball:
Revised: Bayesian Graph Model G: Letting the joint distribution follow the model G in Fig. 5(g), we obtain the following inference models via the Bayes-Ball:

Original: [Equation]
Revised: [Equation]

Original: whose graphical models are described in Figs. 6(g) and (h).
Revised: whose graphical models are described in Figs. 6(g) and (h).

Original: Note that the inference model Gs in
Revised: Note that the inference model Gs in

Original: Fig. 6(h) is identical to the inference model Ds in Fig. 6(b).
Revised: Fig. 6(h) is identical to the inference model Ds in Fig. 6(b).

Original: Although the inference graphs Gs and Ds are identical, the generative model of X is different as shown in Figs. 5(g) and (d).
Revised: Although the inference graphs Gs and Ds are identical, the generative model of X is different as shown in Figs. 5(g) and (d).

Original: Speciﬁcally, VAE decoder for the model G should feed S along with variational latent space Z , and thus using CVAE is justiﬁed for the model G but D.
Revised: Speciﬁcally, VAE decoder for the model G should feed S along with variational latent space Z , and thus using CVAE is justiﬁed for the model G but D.

Original: This difference of the generative models can potentially make a different impact on the performance of inference despite the inference graph alone is identical.
Revised: This difference of the generative models can potentially make a different impact on the performance of inference despite the inference graph alone is identical.

Original: Bayesian Graph Models H and I: Both the generative models H
Revised: Bayesian Graph Models H and I: Both the generative models H

Original: and I shown in Figs. 5(h) and (i) have the fully-connected inference strategies as given in (2), whose graphs are shown in Figs. 4(b) and (c), respectively, since no useful conditional independency can be found with the Bayes-Ball.
Revised: and I shown in Figs. 5(h) and (i) have the fully-connected inference strategies as given in (2), whose graphs are shown in Figs. 4(b) and (c), respectively, since no useful conditional independency can be found with the Bayes-Ball.

Original: Analogous to the relation of models Ds and Gs, the inference graph can be identical for Bayesian graphs H and I, whereas the generative model of X is different as shown in Figs. 5(h) and (i).
Revised: Analogous to the relation of models Ds and Gs, the inference graph can be identical for Bayesian graphs H and I, whereas the generative model of X is different as shown in Figs. 5(h) and (i).

Original: Bayesian Graph Model J (Disentangled Latent): We can also consider multiple latent vectors to generalize the Bayesian graph with more vertices.
Revised: Bayesian Graph Model J (Disentangled Latent): We can also consider multiple latent vectors to generalize the Bayesian graph with more vertices.

Original: We here focus on two such examples of graph models with two-latent spaces as shown in Figs. 5(j) and (k).
Revised: We here focus on two such examples of graph models with two-latent spaces as shown in Figs. 5(j) and (k).

Original: Those models are identical class of the model D, except that a single latent Z is disentangled into two parts Z 1 and Z 2 , respectively associated with S and Y .
Revised: Those models are identical class of the model D, except that a single latent Z is disentangled into two parts Z 1 and Z 2 , respectively associated with S and Y .

Original: Given the Bayesian graph of Fig. 5(j), the Bayes-Ball yields some inference strategies including the following two models:[Equation] Model
Revised: Given the Bayesian graph of Fig. 5(j), the Bayes-Ball yields some inference strategies including the following two models:

Original: [Equation]
Revised: [Equation]

Original: which are shown in Figs. 6(i) and (j).
Revised: which are shown in Figs. 6(i) and (j).

Original: Note that Z 2 is marginally independent of the nuisance variable
Revised: Note that Z 2 is marginally independent of the nuisance variable

Original: S , which encourages the use of adversarial training to be robust against subject/session variations.
Revised: S , which encourages the use of adversarial training to be robust against subject/session variations

Revised: [Equation]

Original: Bayesian Graph Model K (Conditionally Disentangled Latent): Another modiﬁed model in
Revised: Bayesian Graph Model K (Conditionally Disentangled Latent): Another modiﬁed model in

Original: Fig. 5(k) linking Z 1 and Z 2 yields the following inference models: 
Revised: Fig. 5(k) linking Z 1 and Z 2 yields the following inference models: [Equation] Model-Kz

Original: [Equation]
Revised: [Equation]

Original: as shown in Figs. 6(k) and (l).
Revised: as shown in Figs. 6(k) and (l).

Original: The major difference from the model J lies in the fact that the inference graph should use Z 1 along with Z 2 to infer Y .
Revised: The major difference from the model J lies in the fact that the inference graph should use Z 1 along with Z 2 to infer Y [Equation]

Original: A. B ACKGROUND ON V ARIATIONAL B AYESIAN I NFERENCE
Revised: A. B ACKGROUND ON V ARIATIONAL B AYESIAN I NFERENCE

Original: Variational AE AutoBayes may automatically construct autoencoder architecture when latent variables are involved, e.g., for the model E in Fig. 5(e).
Revised: Variational AE AutoBayes may automatically construct autoencoder architecture when latent variables are involved, e.g., for the model E in Fig. 5(e).

Original: For this case, Z represents a stochastic node to marginalize out for X reconstruction and Y inference, and hence VAE will be required.
Revised: For this case, Z represents a stochastic node to marginalize out for X reconstruction and Y inference, and hence VAE will be required.

Original: In contrast to vanilla autoencoders, VAE uses variational inference by assuming a marginal distribution for latent p ( z ) .
Revised: In contrast to vanilla autoencoders, VAE uses variational inference by assuming a marginal distribution for latent p ( z ) .

Original: In variational approach, we reparameterize Z from a prior distribution such as the normal distirbution to marginalize.
Revised: In variational approach, we reparameterize Z from a prior distribution such as the normal distirbution to marginalize.

Original: Depending on the Bayesian graph models, we can also consider reparametering semi-supervision on S (i.e., incorporating a reconstruction loss for S ) as a conditioning variable.
Revised: Depending on the Bayesian graph models, we can also consider reparametering semi-supervision on S (i.e., incorporating a reconstruction loss for S ) as a conditioning variable.

Original: Conditioning on Y and/or S should depend on consistency with the graphical model assumptions.
Revised: Conditioning on Y and/or S should depend on consistency with the graphical model assumptions.

Original: Since VAE is a special case of CVAE, we will go into further detail about the more general CVAE below.
Revised: Since VAE is a special case of CVAE, we will go into further detail about the more general CVAE below.

Original: Conditional VAE When X is directly dependent on S or Y along with Z in the Bayesian graph, the AutoBayes gives rise the CVAE architecture, e.g., for the models E/F/G/H/I in Fig.
Revised: Conditional VAE When X is directly dependent on S or Y along with Z in the Bayesian graph, the AutoBayes gives rise the CVAE architecture, e.g., for the models E/F/G/H/I in Fig.

Original: For those generative models, the decoder DNN needs to feed S or Y as a conditioning parameter.
Revised: For those generative models, the decoder DNN needs to feed S or Y as a conditioning parameter.

Original: Even for other Bayesian graphs, the S -ﬁrst inference strategy will require conditional encoder in CVAE,
Revised: Even for other Bayesian graphs, the S -ﬁrst inference strategy will require conditional encoder in CVAE,

Original: e.g., the models Ds/Es/Fs/Gs/Js/Ks in Fig.
Revised: e.g., the models Ds/Es/Fs/Gs/Js/Ks in Fig.

Original: 6, where latent Z depends on S .
Revised: 6, where latent Z depends on S .

Original: Consider the case when S plays as the conditioning variable in a data model with the factorization:
Revised: Consider the case when S plays as the conditioning variable in a data model with the factorization:

Original: [Equation]
Revised: [Equation]

Original: 17) where we directly parameterize p ( x | s, z ) , set p ( z ) to something simple (e.g., isotropic Gaussian), and leave p ( s ) arbitrary (since it will not be directly used).
Revised: (17) where we directly parameterize p ( x | s, z ) , set p ( z ) to something simple (e.g., isotropic Gaussian), and leave p ( s ) arbitrary (since it will not be directly used).

Original: The CVAE is trained according to maximizing the likelihood of data tuples ( s, x ) with respect to p ( x | s ) , which is given by
Revised: The CVAE is trained according to maximizing the likelihood of data tuples ( s, x ) with respect to p ( x | s ) , which is given by

Original: [Equation]
Revised: [Equation]

Original: which is intractable to compute exactly given the potential complexity of the parameterization of p ( x | s, z ) .
Revised: which is intractable to compute exactly given the potential complexity of the parameterization of p ( x | s, z ) .

Original: While it could be possible to approximate the integration with sampling of Z , the crux of the VAE approach is to utilize a variational lower-bound of the likelihood that involves a variational approximation of the posterior p ( z | s, x ) implied by the generative model.
Revised: While it could be possible to approximate the integration with sampling of Z , the crux of the VAE approach is to utilize a variational lower-bound of the likelihood that involves a variational approximation of the posterior p ( z | s, x ) implied by the generative model.

Original: With q ( z | s, x ) representing the variational approximation of the posterior, the Evidence Lower-Bound (ELBO) is given by
Revised: With q ( z | s, x ) representing the variational approximation of the posterior, the Evidence Lower-Bound (ELBO) is given by

Original: [Equation]
Revised: [Equation]

Original: The parameterization of the variational posterior q ( z | s, x ) may also be decomposed into parameterized components, e.g., q ( z | s, x ) =
Revised: The parameterization of the variational posterior q ( z | s, x ) may also be decomposed into parameterized components, e.g., q ( z | s, x ) =

Original: q ( s | x ) q
Revised: q ( s | x ) q

Original: ( z | s, x ) such as in the S -ﬁrst models shown in Fig.
Revised: ( z | s, x ) such as in the S -ﬁrst models shown in Fig.

Original: Such decomposition also enables the possibility of semi-supervised training, which can be convenient when some of the variables, such as the nuisances variations, are not always labeled.
Revised: Such decomposition also enables the possibility of semi-supervised training, which can be convenient when some of the variables, such as the nuisances variations, are not always labeled.

Original: For data tuples that include s , the likelihood q ( s | x ) can also be directly optimized, and the given value for s is used an input to the computation of q ( z | s, x ) .
Revised: For data tuples that include s , the likelihood q ( s | x ) can also be directly optimized, and the given value for s is used an input to the computation of q ( z | s, x ) .

Original: However, for tuples where s is missing, the component q ( s | x ) can be used to generate an estimate of s to be input to q ( z | s, x ) .
Revised: However, for tuples where s is missing, the component q ( s | x ) can be used to generate an estimate of s to be input to q ( z | s, x ) .

Original: We further discuss semi-supervised learning and the sampling methods for categorical nuisance variables in Appendix A.4 below.
Revised: We further discuss semi-supervised learning and the sampling methods for categorical nuisance variables in Appendix A.4 below.

Original: A. S EMI -S UPERVISED L EARNING : C ATEGORICAL S AMPLING
Revised: A. S EMI -S UPERVISED L EARNING : C ATEGORICAL S AMPLING

Original: Graphical Models for Semi-Supervised Learning Nuisance values S such as subject ID or session ID may not be always available for typical physiological datasets, in particular for the testing phase of an HMI system deployment with new users, requiring semi-supervised methods.
Revised: Graphical Models for Semi-Supervised Learning Nuisance values S such as subject ID or session ID may not be always available for typical physiological datasets, in particular for the testing phase of an HMI system deployment with new users, requiring semi-supervised methods.

Original: We note that some graphical models are well-suited for such semi-supervised training.
Revised: We note that some graphical models are well-suited for such semi-supervised training.

Original: For example, among the Bayesian graph models in Fig. 5, the models C/E/G/I require the nuisance S to reproduce
Revised: For example, among the Bayesian graph models in Fig. 5, the models C/E/G/I require the nuisance S to reproduce

Original: X .
Revised: X .

Original: If no ground-truth labels of S are available, we need to marginalize S across all possible categories for the decoder DNN D .
Revised: If no ground-truth labels of S are available, we need to marginalize S across all possible categories for the decoder DNN D .

Original: Even for other Bayesian graphs, the corresponding inference factor graphs in Fig.
Revised: Even for other Bayesian graphs, the corresponding inference factor graphs in Fig.

Original: 6 may not be convenient for the semi-supervised settings.
Revised: 6 may not be convenient for the semi-supervised settings.

Original: Speciﬁcally, for models
Revised: Speciﬁcally, for models

Original: Ez/Fz/Jz/Kz have an inference of S at the end node, whereas the other inference models use inferred
Revised: Ez/Fz/Jz/Kz have an inference of S at the end node, whereas the other inference models use inferred

Original: S for subsequent inference of other parameters.
Revised: S for subsequent inference of other parameters.

Original: If S is missing or unknown as a semi-supervised setting, those inference graphs having S in a middle node are inconvenient as we need sampling over all possible nuisance categories.
Revised: If S is missing or unknown as a semi-supervised setting, those inference graphs having S in a middle node are inconvenient as we need sampling over all possible nuisance categories.

Original: For instance, the model Kz shown in Fig.
Revised: For instance, the model Kz shown in Fig.

Original: 7 does not need S marginalization, and thus readily applicable to semi-supervised datasets.
Revised: 7 does not need S marginalization, and thus readily applicable to semi-supervised datasets.

Original: Variational Categorical Reparameterization In order to deal with the issue of categorical sampling, we can use the Gumbel-Softmax reparameterization trick (Jang et al., 2016), which enables differentiable approximation of one-hot encoding.
Revised: Variational Categorical Reparameterization In order to deal with the issue of categorical sampling, we can use the Gumbel-Softmax reparameterization trick (Jang et al., 2016), which enables differentiable approximation of one-hot encoding.

Original: Let [ π 1 , π 2 , . . .
Revised: Let [ π 1 , π 2 , . . .

Original: , π | S | ] denote a target probability mass function for the categorical variable S .
Revised: , π | S | ] denote a target probability mass function for the categorical variable S .

Original: Let g 1 , g 2 , . .
Revised: Let g 1 , g 2 , . .

Original: . , g | S | be independent and identically distributed samples drawn from the Gumbel distribution Gumbel(0 , 1) .
Revised: . , g | S | be independent and identically distributed samples drawn from the Gumbel distribution Gumbel(0 , 1) .

Original: Then, generate an | S | - dimensional vector ˆ s =
Revised: Then, generate an | S | - dimensional vector ˆ s =

Original: [ˆ s 1 , ˆ s 2 , . .
Revised: [ˆ s 1 , ˆ s 2 , . .

Original: , ˆ s | S | ] according to
Revised: , ˆ s | S | ] according to

Original: [Equation]
Revised: [Equation]

Original: where τ > 0 is a softmax temperature.
Revised: where τ > 0 is a softmax temperature.

Original: As the softmax temperature τ approaches 0 , samples from the
Revised: As the softmax temperature τ approaches 0 , samples from the

Original: Gumbel-Softmax distribution become one-hot and the distribution becomes identical to the target categorical distribution.
Revised: Gumbel-Softmax distribution become one-hot and the distribution becomes identical to the target categorical distribution.

Original: The temperature τ is usually decreased across training epochs as an annealing technique, e.g., with exponential decaying.
Revised: The temperature τ is usually decreased across training epochs as an annealing technique, e.g., with exponential decaying.

Original: A. E NSEMBLE L EARNING : S TACKED G ENERALIZATION
Revised: A. E NSEMBLE L EARNING : S TACKED G ENERALIZATION

Original: To achieve higher predictive performance, we construct ensembles from the output posterior class probabilities of all graphical models.
Revised: To achieve higher predictive performance, we construct ensembles from the output posterior class probabilities of all graphical models.

Original: Let D 

Revised: Let D

Original: = { ( x n , y n , s n ) | n = 1 : N } denote a data set, where x n is a data instance, y n is the task label, s n is the nuisance (subject) label and N is the number of samples in the dataset.
Revised: = { ( x n , y n , s n ) | n = 1 : N } denote a data set, where x n is a data instance, y n is the task label, s n is the nuisance (subject) label and N is the number of samples in the dataset.

Original: We randomly split the data into training set D train and validation set D test .
Revised: We randomly split the data into training set D train and validation set D test .

Original: Given 37 graphical models, which we call base learners, we induce a decision algorithm M k , for k = 1 , . .
Revised: Given 37 graphical models, which we call base learners, we induce a decision algorithm M k , for k = 1 , . .

Original: . , 37 by invoking the k th graphical model on the data in D train .
Revised: . , 37 by invoking the k th graphical model on the data in D train .

Original: For each x n in D train , graphical model M k generates a class probability vector for task and nuisance label prediction.
Revised: For each x n in D train , graphical model M k generates a class probability vector for task and nuisance label prediction.

Original: Let
Revised: Let

Original: P ky ( x n ) =
Revised: P ky ( x n ) =

Original: { P ( y 1 | x n ) , . . .
Revised: { P ( y 1 | x n ) , . . .

Original: , P ( y i | x n ) , . . .
Revised: , P ( y i | x n ) , . . .

Original: , P ( y N y | x n ) } denote the posterior probability distribution over N y task labels and P ks ( x n ) =
Revised: , P ( y N y | x n ) } denote the posterior probability distribution over N y task labels and P ks ( x n ) =

Original: { P ( s 1 | x n ) , . . .
Revised: { P ( s 1 | x n ) , . . .

Original: , P ( s i | x n ) , . . .
Revised: , P ( s i | x n ) , . . .

Original: , P ( s N s | x n ) } denote the posterior probability distribution over N s nuisance labels produced by model M k given data instance x n .
Revised: , P ( s N s | x n ) } denote the posterior probability distribution over N s nuisance labels produced by model M k given data instance x n .

Original: Ensemble generalization works by stacking the predictions of the base learners in a higher level learning space, where meta learner, denoted as ˜ M k , corrects the predictions of base learners (Wolpert, 1992).
Revised: Ensemble generalization works by stacking the predictions of the base learners in a higher level learning space, where meta learner, denoted as ˜ M k , corrects the predictions of base learners (Wolpert, 1992).

Original: Subsequent to training base learners, we assemble the posterior probability vectors of all base learners together: P y ( x n ) =
Revised: Subsequent to training base learners, we assemble the posterior probability vectors of all base learners together: P y ( x n ) =

Original: { P ky ( x n ) } and P s ( x n ) =
Revised: { P ky ( x n ) } and P s ( x n ) =

Original: { P ks ( x n ) } , where k = 1 : 37 .
Revised: { P ks ( x n ) } , where k = 1 : 37 .

Original: ˜ M k is trained using the predictions from all base learners as input attributes: ˜ D intrain =
Revised: ˜ M k is trained using the predictions from all base learners as input attributes: ˜ D intrain =

Original: { ( P y ( x n ) , P s ( x n )) } and correct labels as output: ˜ D outtrain =
Revised: { ( P y ( x n ) , P s ( x n )) } and correct labels as output: ˜ D outtrain =

Original: { ( y n , s n ) } , where n = 1 : N train .
Revised: { ( y n , s n ) } , where n = 1 : N train .

Original: Hold-out D test is used to measure the classiﬁcation performance of both base and meta learners.
Revised: Hold-out D test is used to measure the classiﬁcation performance of both base and meta learners.

Original: To make best use of the base learners, we compare the predictive performance of a LR model and a shallow MLP as a meta learner in Table 2.
Revised: To make best use of the base learners, we compare the predictive performance of a LR model and a shallow MLP as a meta learner in Table 2.

Original: A. D ATASETS D ESCRIPTION
Revised: A. D ATASETS D ESCRIPTION

Original: We used publicly available physiological datasets as well as a benchmark MNIST as follows.
Revised: We used publicly available physiological datasets as well as a benchmark MNIST as follows.

Original: The parameters of datasets are also summarized in Table 1.
Revised: The parameters of datasets are also summarized in Table 1.

Original: • QMNIST:
Revised: • QMNIST:

Original: A hand-written digit image MNIST with extended label information including a writer ID number (Yadav & Bottou, 2019).
Revised: A hand-written digit image MNIST with extended label information including a writer ID number (Yadav & Bottou, 2019).

Original: 3 There are | S | =
Revised: 3 There are | S | =

Original: 539 writers for classify- The ing | Y | = 10 digits from grayscale 28 × 28 pixel images over 60 , 000 training samples.
Revised: 539 writers for classifying | Y | = 10 digits from grayscale 28 × 28 pixel images over 60 , 000 training samples.

Original: Additional 297 writers provide 10 , 000 test samples.
Revised: Additional 297 writers provide 10 , 000 test samples.

Original: • Stress:
Revised: • Stress:

Original: A physiological dataset considering neurological stress level (Birjandtalab et al., 2016).
Revised: A physiological dataset considering neurological stress level (Birjandtalab et al., 2016).

Original: It consists of multi-modal biosignals for | Y | = 4 discrete stress states from | S | =healthy subjects, including physical/cognitive/emotional stresses as well as relaxation.
Revised: It consists of multi-modal biosignals for | Y | = 4 discrete stress states from | S | =healthy subjects, including physical/cognitive/emotional stresses as well as relaxation.

Original: The data were collected by C = 7 sensors, i.e., electrodermal activity, temperature, threedimensional acceleration, heart rate, and arterial oxygen level.
Revised: The data were collected by C = 7 sensors, i.e., electrodermal activity, temperature, threedimensional acceleration, heart rate, and arterial oxygen level.

Original: For each stress status, a corresponding task of 5 minutes long (i.e., T = 300 time samples with 1 Hz down-sampling) wasassigned to subjects for a total of 4 trials.
Revised: For each stress status, a corresponding task of 5 minutes long (i.e., T = 300 time samples with 1 Hz down-sampling) was

Original: • RSVP:
Revised: • RSVP:

Original: An EEG-based typing interface using rapid serial visual presentation (RSVP) paradigm (Orhan et al., 2012).
Revised: An EEG-based typing interface using rapid serial visual presentation (RSVP) paradigm (Orhan et al., 2012).

Original: 5 | S | = 10 healthy subjects participated in the experiments at three sessions performed on different days.
Revised: 5 | S | = 10 healthy subjects participated in the experiments at three sessions performed on different days.

Original: The dataset consists of 41 , 400 epochs of C = 16 channel EEG data for T = 128 samples, which were collected by g.USBamp biosignal ampliﬁer with active electrodes during RSVP keyboard operations.
Revised: The dataset consists of 41 , 400 epochs of C = 16 channel EEG data for T = 128 samples, which were collected by g.USBamp biosignal ampliﬁer with active electrodes during RSVP keyboard operations.

Original: | Y | = 4 labels for emotion elicitation, resting-state, or motor imagery/execution task.
Revised: | Y | = 4 labels for emotion elicitation, resting-state, or motor imagery/execution task.

Original: • MI: The PhysioNet EEG Motor Imagery (MI) dataset (Goldberger et al., 2000).
Revised: • MI: The PhysioNet EEG Motor Imagery (MI) dataset (Goldberger et al., 2000).

Original: 6 Excluding irregular timestamp, the dataset consists of | S | = 106 subjects’ EEG data.
Revised: 6 Excluding irregular timestamp, the dataset consists of | S | = 106 subjects’ EEG data.

Original: During the experiments, subjects were instructed to perform cue-based motor execution/imagery tasks while C = 64 channels were recorded at a sampling rate of 160 Hz.
Revised: During the experiments, subjects were instructed to perform cue-based motor execution/imagery tasks while C = 64 channels were recorded at a sampling rate of 160 Hz.

Original: Focusing on motor imagery tasks, we use the EEG data for three seconds of post-cue interval data (i.e., T =time samples).
Revised: Focusing on motor imagery tasks, we use the EEG data for three seconds of post-cue interval data (i.e., T =time samples).

Original: The subject performed | Y | = 4 -class tasks; either right hand motor imagery, left hand motor imagery, both hands motor imagery, or both feet motor imagery.
Revised: The subject performed | Y | = 4 -class tasks; either right hand motor imagery, left hand motor imagery, both hands motor imagery, or both feet motor imagery.

Original: This resulted in a total of 90 trials per subject.
Revised: This resulted in a total of 90 trials per subject.

Original: • ErrP:
Revised: • ErrP:

Original: An error-related potential (ErrP) of front-central EEG dataset (Margaux et al.,
Revised: An error-related potential (ErrP) of front-central EEG dataset (Margaux et al.,

Original: 2012).The dataset consists of EEG data recorded from | S | = 16 healthy subjects participating in an ofﬂine P300 spelling task, where visual feedback of the inferred letter is provided to the user at the end of each trial for 1 . 3 seconds to monitor evoked brain responses for erroneous decisions made by the system.
Revised: 2012).The dataset consists of EEG data recorded from | S | = 16 healthy subjects participating in an ofﬂine P300 spelling task, where visual feedback of the inferred letter is provided to the user at the end of each trial for 1 . 3 seconds to monitor evoked brain responses for erroneous decisions made by the system.

Original: EEG data were recorded from C = 56 channels for epoched 1 .
Revised: EEG data were recorded from C = 56 channels for epoched 1 .

Original: 25 seconds at a sampling rate of 200 Hz (i.e., T = 250 ).
Revised: 25 seconds at a sampling rate of 200 Hz (i.e., T = 250 ).

Original: Across ﬁve recording sessions, each subject performed a total of 340 trials.
Revised: Across ﬁve recording sessions, each subject performed a total of 340 trials.

Original: Since it was an ofﬂine copy spell task, binary | Y | = 2 labels were provided as erroneous or correct feedback.
Revised: Since it was an ofﬂine copy spell task, binary | Y | = 2 labels were provided as erroneous or correct feedback.

Original: • Faces Basic: An implanted electrocorticography (ECoG) array dataset for visual stimulus experiments (Miller et al., 2015; 2016).
Revised: • Faces Basic: An implanted electrocorticography (ECoG) array dataset for visual stimulus experiments (Miller et al., 2015; 2016).

Original: 8 ECoG arrays were implanted on the subtemporal cortical surface of | S | = 14 epilepsy patients.
Revised: 8 ECoG arrays were implanted on the subtemporal cortical surface of | S | = 14 epilepsy patients.

Original: | Y | = 2 classes of grayscale images, either faces or houses, were displayed rapidly in random sequence for 400 ms each with blackscreen intervals of 400 ms.
Revised: | Y | = 2 classes of grayscale images, either faces or houses, were displayed rapidly in random sequence for 400 ms each with blackscreen intervals of 400 ms.

Original: The ECoG potentials were measured with respect to a scalp reference and ground, at a sampling rate of 1000 Hz.
Revised: The ECoG potentials were measured with respect to a scalp reference and ground, at a sampling rate of 1000 Hz.

Original: Subjects performed a basic face and house discrimination task.
Revised: Subjects performed a basic face and house discrimination task.

Original: There were 3 sessions for each patient, with 50 house pictures and 50 face pictures in each run, in total 4 , 100 samples.
Revised: There were 3 sessions for each patient, with 50 house pictures and 50 face pictures in each run, in total 4 , 100 samples.

Original: We use the ﬁrst C = 31 channels to analyze for T = 400 .
Revised: We use the ﬁrst C = 31 channels to analyze for T = 400 .

Original: Reusing the public dataset requires the ethics statement information.• Faces Noisy: The implanted ECoG arrays dataset for visual stimulus experiments (Miller et al., 2015; 2017).
Revised: Reusing the public dataset requires the ethics statement information.• Faces Noisy: The implanted ECoG arrays dataset for visual stimulus experiments (Miller et al., 2015; 2017).

Original: The experiment is similar to Faces Basic dataset, while pictures of faces Faces
Revised: The experiment is similar to Faces Basic dataset, while pictures of faces and houses are randomly scrambled.

Original: Ethics statement : All patients participated in a purely voluntary manner, after providing informed written consent, under experimental protocols approved by the Institutional Review Board of the University of Washington (#12193).

Original: All patient data was anonymized according to IRB protocol, in accordance with HIPAA mandate.

Original: These data originally appeared in the manuscript “Spontaneous Decoding of the Timing and Content of Human

Original: Object Perception from Cortical Surface Recordings Reveals Complementary Information in the Event-Related Potential and Broadband Spectral Change” published in PLoS Computational Biology in 2016 (Miller et al., 2016).

Original: [Table]

Original: and houses are randomly scrambled.

Original: There are | S | = 7 subjects with C = 39 channels.
Revised: There are | S | = 7 subjects with C = 39 channels.

Original: Refer ethics  the [Table] • ASL: An EMG dataset for ﬁnger gesture identiﬁcation for American Sign Language (ASL) (Günay et al., 2019).
Revised: Refer ethics statement to reuse the dataset.• ASL: An EMG dataset for ﬁnger gesture identiﬁcation for American Sign Language (ASL) (Günay et al., 2019).

Original: 11 | S | = 5 healthy, right-handed, subjects participated in experiments with surface EMG (
Revised: 11 | S | = 5 healthy, right-handed, subjects participated in experiments with surface EMG (

Original: Delsys Inc.
Revised: Delsys Inc.

Original: Trigno) recorded at 2 kHz from | C | =lower-arm muscles.
Revised: Trigno) recorded at 2 kHz from | C | =lower-arm muscles.

Original: Subjects shaped their right hand into letters and numbers of the ASL posture set presented as pictures on a computer screen ( | Y | = 33 postures, 3 trials perposture).
Revised: Subjects shaped their right hand into letters and numbers of the ASL posture set presented as pictures on a computer screen ( | Y | = 33 postures, 3 trials per

Original: Dynamic letters ‘J’ and ‘Z’ were omitted, along with the number ‘0’, which is visually the same as the letter ‘O’.
Revised: [Table]

Original: The participants were given 2 seconds to form the posture, 6 seconds to maintain it, and 2 seconds to rest between trials.
Revised: posture)

Original: The signal is decimated to be T = 100 .
Revised: [Table]

Original: A. DNN M ODEL P ARAMETERS
Revised: A. DNN M ODEL P ARAMETERS

Original: For 2D datasets, we use deep CNN for the encoder E and decoder D blocks.
Revised: For 2D datasets, we use deep CNN for the encoder E and decoder D blocks.

Original: For the classiﬁer C , nuisance estimator N , and adversary A , we use a multi-layer perceptron (MLP) having three layers, whose hidden nodes are doubled from the input dimension.
Revised: For the classiﬁer C , nuisance estimator N , and adversary A , we use a multi-layer perceptron (MLP) having three layers, whose hidden nodes are doubled from the input dimension.

Original: We also use batch normalization (BN) and ReLU activation as listed in Table 3.
Revised: We also use batch normalization (BN) and ReLU activation as listed in Table 3.

Original: Note that for a tabular data such as Stress datasets, CNN was replaced with 3-layer MLP having ReLU activation and dropout with a ratio of 20% .
Revised: Note that for a tabular data such as Stress datasets, CNN was replaced with 3-layer MLP having ReLU activation and dropout with a ratio of 20% .

Original: Also the
Revised: Also the

Original: MLP classiﬁer was replaced with CNN for 2D input dimension cases such as in the model A.
Revised: MLP classiﬁer was replaced with CNN for 2D input dimension cases such as in the model A.

Original: The number of latent dimensions was chosen | Z | = 64 .
Revised: The number of latent dimensions was chosen | Z | = 64 .

Original: When we need to feed S along with 2D data of X into the CNN encoder such as in the model Ds, dimension mismatch poses a problem.
Revised: When we need to feed S along with 2D data of X into the CNN encoder such as in the model Ds, dimension mismatch poses a problem.

Original: We address this issue by using one linear layer to project S into the temporal dimensional space of X and another linear layer to project it into the spatial dimensional space of X .
Revised: We address this issue by using one linear layer to project S into the temporal dimensional space of X and another linear layer to project it into the spatial dimensional space of X .

Original: The dot product of those two projected vectors is concatenated as additional channel input.
Revised: The dot product of those two projected vectors is concatenated as additional channel input.

Original: We use λ ∗ = 0 .
Revised: We use λ ∗ = 0 .

Original: 01 for the regularization coefﬁcient.
Revised: 01 for the regularization coefﬁcient.

Original: We leave hyperparameter exploration to integrate AutoML and AutoBayes as a remaining future work.
Revised: We leave hyperparameter exploration to integrate AutoML and AutoBayes as a remaining future work.

Original: A. P ERFORMANCE R ESULTS
Revised: A. P ERFORMANCE R ESULTS

Original: The additional results for the all datasets are listed in Table 4.
Revised: The additional results for the all datasets are listed in Table 4.

Original: The results suggest that the best inference strategy highly depends on datasets.
Revised: The results suggest that the best inference strategy highly depends on datasets.

Original: Speciﬁcally, the best model at one dataset does not perform best for different datasets; e.g., the model non-variational Is was best for ASL dataset, while the model variational Ds was best for RSVP dataset.
Revised: Speciﬁcally, the best model at one dataset does not perform best for different datasets; e.g., the model non-variational Is was best for ASL dataset, while the model variational Ds was best for RSVP dataset.

Original: It suggests that we shall consider different inference strategies for each target dataset and AutoBayes provides such an adaptive framework.
Revised: It suggests that we shall consider different inference strategies for each target dataset and AutoBayes provides such an adaptive framework.

Original: Also note that reconstruction loss may not be a good indicator to select the graph model.
Revised: Also note that reconstruction loss may not be a good indicator to select the graph model.

Original: In addition, a huge performance gap between the best and worst models was observed for some datasets.
Revised: In addition, a huge performance gap between the best and worst models was observed for some datasets.

Original: For example, the task accuracy of 76 .
Revised: For example, the task accuracy of 76 .

Original: 4 % was achieved with model non-variational Dz for Faces (Noisy) dataset, whereas the model variational B offers 51 .
Revised: 4 % was achieved with model non-variational Dz for Faces (Noisy) dataset, whereas the model variational B offers 51 .

Original: This implies that we may have a potential risk that one particular model cannot achieve good performance if we do not explore different models.
Revised: This implies that we may have a potential risk that one particular model cannot achieve good performance if we do not explore different models.

Original: A. P ER -S UBJECT  P ERFORMANCE
Revised: A. S UBJECT V ARIATION P ERFORMANCE

Original: For Stress dataset, there are | S | = 20 subjects.
Revised: For Stress dataset, there are | S | = 20 subjects.

Original: As we have shown in Fig.
Revised: As we have shown in Fig.

Original: 9(a), we demonstrated that
Revised: 9(a), we demonstrated that

Original: AutoBayes can improve robustness against the nuisance variation, i.e., subject ID S .
Revised: AutoBayes can improve robustness against the nuisance variation, i.e., subject ID S .

Original: In Fig.
Revised: In Fig.

Original: 10, we show that the task classiﬁcation accuracy highly depends on the subject ID S .
Revised: 10, we show that the task classiﬁcation accuracy highly depends on the subject ID S .

Original: Here, the box-whisker plots shows the accuracy distribution over different models from A to Kz.
Revised: Here, the box-whisker plots shows the accuracy distribution over different models from A to Kz.

Original: The outliers are identiﬁed by a whisker factor of 2 . 4 with respective to an inter-quartile range.
Revised: The outliers are identiﬁed by a whisker factor of 2 . 4 with respective to an inter-quartile range.

Original: It is seen that some users (e.g.,
Revised: It is seen that some users (e.g.,

Original: S = 8 ) have superior performance whereas classiﬁcation task is harder for some other users (e.g.,

Original: S = 6) .

Original: Our AutoBayes can well resolve the issues of such a nuisance variation by linking the adversarial block for S -independent latent variables Z to generate subject-invariant feature.

Original: [Figure]

Original: A. T

Original: [Figure]

Original: C OMPLEXITY A NALYSIS

Original: In Fig. 9(b), we have shown the accuracy vs. the space complexity.

Original: In this section, we evaluate the time complexity in Figs.

Original: 11(a)

Original: and (b), which show the task classiﬁcation accuracy as a function of computation time for training and testing, respectively, for the Stress dataset.

Original: As in the same setting of Fig. 9(b), we explored different DNN conﬁgurations for the models A, B, and Js, by sweeping the number of hidden layers and hidden nodes.

Original: Some Pareto-front DNN conﬁgurations having lower complexity and higher accuracy are connected with lines.

Original: We used pytorch on NVIDIA Tesla K

Original: GPU under CUDA 10.1.

Original: It is seen that the standard classiﬁer model A outperforms the other models in lower complexity regimes, whereas our AutoBayes can achieve better Pareto front for higher accuracy regimes.

Original: It should be also noted that the increase of the time complexity is not so signiﬁcant in comparison to that of the space complexity in Fig. 9(b).

Original: [Table]
Revised: [Table]

Original: − 61 .−
Revised: − 61 .−

Original: 3 K 00 :
Revised: 3 K 00 :

Original: 01 : 02 ModelDz − 66 .− 48 .81 .

Revised: 01 : 02 ModelDz − 66 .− 48 .81 .

Original: K 00 : 01 : 03 ModelEs − 66 .−
Revised: K 00 : 01 : 03 ModelEs − 66 .−

Original: 3 K 00 : 01 : 08 ModelEz − 64 .− 59 .

Revised: 3 K 00 : 01 : 08 ModelEz − 64 .− 59 .

Original: 7 K 00 : 

Revised: 7 K 00 :

Original: : 46 ModelFs − 64 .−
Revised: : 46 ModelFs − 64 .−

Original: 4 K 00 : 01 : 04 ModelFz − 66 .−
Revised: 4 K 00 : 01 : 04 ModelFz − 66 .−

Original: 1 K 00 :
Revised: 1 K 00 :

Original: [Table]
Revised: [Table]

Original: 40 ModelGs − 67 .− 64 .
Revised: 40 ModelGs − 67 .− 64 .

Original: [Table]
Revised: [Table]

Original: 3 K 00 :
Revised: 3 K 00 :

Original: [Table]
Revised: [Table]

Original: 11 ModelGz − 65 .− 

Revised: 11 ModelGz − 65 .−

Original: [Table]
Revised: [Table]

Original: 102 K
Revised: 102 K

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 01 ModelHs − 65 .− 38 .78 .72 .94 .
Revised: 01 ModelHs − 65 .− 38 .78 .72 .94 .

Original: [Table]
Revised: [Table]

Original: 4 K 00 : 01 : 04 ModelHz − 66 .−
Revised: 4 K 00 : 01 : 04 ModelHz − 66 .−

Original: K 00 : 01 : 06 ModelIs − 66 .− 

Revised: K 00 : 01 : 06 ModelIs − 66 .−

Original: 4 K 00 : 01 : 04 ModelIz − 67 .−
Revised: 4 K 00 : 01 : 04 ModelIz − 67 .−

Original: .103 K 00 :
Revised: .103 K 00 :

Original: 01 : 04 ModelJs − 67 .−
Revised: 01 : 04 ModelJs − 67 .−

Original: .95 .30 .140 K 00 : 

Revised: .95 .30 .140 K 00 :

Original: : 21 ModelJz − 66 .− 54 .86 .

Revised: : 21 ModelJz − 66 .− 54 .86 .

Original: .95 .86 .135 K 00 : 02 : 07 ModelKs − 68 .− 

Revised: .95 .86 .135 K 00 : 02 : 07 ModelKs − 68 .−

Original: .94 .86 .146 K 00 : 01 : 20 ModelKz − 66 .−
Revised: .94 .86 .146 K 00 : 01 : 20 ModelKz − 66 .−

Original: 51 .85 .61 .94 .86 .141 K 00 : 02 :
Revised: 51 .85 .61 .94 .86 .141 K 00 : 02 :

Original: RSVP
Revised: RSVP

Original: ModelA − 30 .
Revised: ModelA − 30 .

Original: — — — 93 .— 268 K 00 : 48 : 25 ModelB − 34 .− 35 .
Revised: — — — 93 .— 268 K 00 : 48 : 25 ModelB − 34 .− 35 .

Original: 87 M 01 : 00 : 35 ModelCs − 31 .— 90 .— 91 .
Revised: 87 M 01 : 00 : 35 ModelCs − 31 .— 90 .— 91 .

Original: — 437 K 00 : 55 : 35 ModelCy − 31 .— 90 .— 91 .— 435 K 00 : 54 : 29 ModelDs − 35 .− 30 .91 .

Revised: — 437 K 00 : 55 : 35 ModelCy − 31 .— 90 .— 91 .— 435 K 00 : 54 : 29 ModelDs − 35 .− 30 .91 .

Original: 01 M 00 : 56 : 05 ModelDz − 35 .− 35 .92 .86 .92 .92 .1 .
Revised: 01 M 00 : 56 : 05 ModelDz − 35 .− 35 .92 .86 .92 .92 .1 .

Original: 87 M 00 : 48 : 35 ModelEs − 35 .− 31 .

Revised: 87 M 00 : 48 : 35 ModelEs − 35 .− 31 .

Original: 02 M 00 : 54 : 43 ModelEz − 35 .−
Revised: 02 M 00 : 54 : 43 ModelEz − 35 .−

Original: 03 M 01 : 16 : 32 ModelFs − 35 .− 30 .

Revised: 03 M 01 : 16 : 32 ModelFs − 35 .− 30 .

Original: 06 M 01 : 

Revised: 06 M 01 :

Original: : 52 ModelFz − 32 .− 30 .9 .

Revised: : 52 ModelFz − 32 .− 30 .9 .

Original: 93 M 01 : 08 : 24 ModelGs − 35 .− 31 .

Revised: 93 M 01 : 08 : 24 ModelGs − 35 .− 31 .

Original: 03 M 00 : 57 : 25 ModelGz
Revised: 03 M 00 : 57 : 25 ModelGz

Original: [Table]
Revised: [Table]

Original: ModelHs
Revised: ModelHs

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: − 42 .−
Revised: − 42 .−

Original: 41 M 00 : 47 : 42 ModelEs − 39 .−
Revised: 41 M 00 : 47 : 42 ModelEs − 39 .−

Original: 14 M 00 : 59 : 27 ModelEz − 41 .−
Revised: 14 M 00 : 59 : 27 ModelEz − 41 .−

Original: 15 M 01 : 18 : 58 ModelFs − 39 .−
Revised: 15 M 01 : 18 : 58 ModelFs − 39 .−

Original: 13 M 01 : 09 : 08 ModelFz − 41 .−
Revised: 13 M 01 : 09 : 08 ModelFz − 41 .−

Original: 50 M 01 : 17 : 28 ModelGs −
Revised: 50 M 01 : 17 : 28 ModelGs −

Original: [Table]
Revised: [Table]

Original: −
Revised: −

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 14 M 00 :
Revised: 14 M 00 :

Original: [Table]
Revised: [Table]

Original: 57 ModelGz − 41 .−
Revised: 57 ModelGz − 41 .−

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 15 M 00 : 57 : 12 ModelHs − 40 .−
Revised: 15 M 00 : 57 : 12 ModelHs − 40 .−

Original: [Table]
Revised: [Table]

Original: 13 M 01 : 10 : 47 ModelHz − 41 .−
Revised: 13 M 01 : 10 : 47 ModelHz − 41 .−

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 13 M 01 : 05 : 37 ModelIs − 38 .−
Revised: 13 M 01 : 05 : 37 ModelIs − 38 .−

Original: 23 M 01 : 07 : 48 ModelIz − 40 .− 

Revised: 23 M 01 : 07 : 48 ModelIz − 40 .−

Original: 23 M 01 : 10 : 42 ModelJs − 40 .−
Revised: 23 M 01 : 10 : 42 ModelJs − 40 .−

Original: 21 M 01 : 11 : 08 ModelJz − 42 .−
Revised: 21 M 01 : 11 : 08 ModelJz − 42 .−

Original: 54 M 01 : 06 : 01 ModelKs
Revised: 54 M 01 : 06 : 01 ModelKs

Original: [Table] [Table]
Revised: [Table] [Table]

Original: FacesNoisy
Revised: FacesNoisy

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 30 .− 30 .96 .14 .76 .53 .5 .
Revised: 30 .− 30 .96 .14 .76 .53 .5 .

Original: 28 M 00 : 24 : 18 ModelEs − 30 .− 

Revised: 28 M 00 : 24 : 18 ModelEs − 30 .−

Original: 59 M 00 : 27 : 45 ModelEz − 30 .−
Revised: 59 M 00 : 27 : 45 ModelEz − 30 .−

Original: 61 M 00 : 40 : 16 ModelFs − 30 .− 30 .95 .13 .71 .75 .5 .
Revised: 61 M 00 : 40 : 16 ModelFs − 30 .− 30 .95 .13 .71 .75 .5 .

Original: 68 M 00 :
Revised: 68 M 00 :

Original: 32 : 51 ModelFz − 29 .− 27 .27 .17 .72 .72 .5 .
Revised: 32 : 51 ModelFz − 29 .− 27 .27 .17 .72 .72 .5 .

Original: 40 M 00 : 40 : 20 ModelGs − 28 .−
Revised: 40 M 00 : 40 : 20 ModelGs − 28 .−

Original: 59 M 00 : 30 : 07 ModelGz − 30 .− 30 .94 .80 .75 .75 .5 .
Revised: 59 M 00 : 30 : 07 ModelGz − 30 .− 30 .94 .80 .75 .75 .5 .

Original: 61 M 00 : 27 : 52 ModelHs − 30 .− 30 .

Revised: 61 M 00 : 27 : 52 ModelHs − 30 .− 30 .

Original: 68 M 00 : 31 : 14 ModelHz − 30 .− 

Revised: 68 M 00 : 31 : 14 ModelHz − 30 .−

Original: 68 M 00 : 32 : 58 ModelIs − 30 .−
Revised: 68 M 00 : 32 : 58 ModelIs − 30 .−

Original: 72 M 00 : 31 :
Revised: 72 M 00 : 31 :

Original: 27 ModelIz − 30 .− 29 .88 .87 .74 .72 .5 .
Revised: 27 ModelIz − 30 .− 29 .88 .87 .74 .72 .5 .

Original: 72 M 00 : 33 : 43 ModelJs − 30 .− 

Revised: 72 M 00 : 33 : 43 ModelJs − 30 .−

Original: 6 M 00 : 33 : 16 ModelJz − 30 .− 30 .

Revised: 6 M 00 : 33 : 16 ModelJz − 30 .− 30 .

Original: 3 M 00 : 35 : 01 ModelKs − 30 .− 

Revised: 3 M 00 : 35 : 01 ModelKs − 30 .−

Original: 6 M 00 : 31 : 34 ModelKz − 30 .− 28 .

Revised: 6 M 00 : 31 : 34 ModelKz − 30 .− 28 .

Original: 3 M 00 : 34 :
Revised: 3 M 00 : 34 :

Original: ASL
Revised: ASL

Original: ModelA − 24 .
Revised: ModelA − 24 .

Original: — 588 K 01 : 18 : 06 ModelB − 23 .−
Revised: — 588 K 01 : 18 : 06 ModelB − 23 .−

Original: 53 M 01 : 34 : 55 ModelCs − 24 .
Revised: 53 M 01 : 34 : 55 ModelCs − 24 .

Original: — 726 K 01 : 26 : 27 ModelCy − 24 .— 94 .— 38 .— 729 K 01 : 26 : 31 ModelDs − 24 .−
Revised: — 726 K 01 : 26 : 27 ModelCy − 24 .— 94 .— 38 .— 729 K 01 : 26 : 31 ModelDs − 24 .−

Original: 63 M 01 : 32 : 02 ModelDz − 24 .−
Revised: 63 M 01 : 32 : 02 ModelDz − 24 .−

Original: 53 M 01 : 16 : 50 ModelEs − 24 .− 

Revised: 53 M 01 : 16 : 50 ModelEs − 24 .−

Original: 65 M 01 : 32 : 04 ModelEz − 24 .− 

Revised: 65 M 01 : 32 : 04 ModelEz − 24 .−

Original: 65 M 01 : 55 : 58 ModelFs − 24 .−
Revised: 65 M 01 : 55 : 58 ModelFs − 24 .−

Original: M 01 : 39 : 40 ModelFz − 24 .− 24 .

Revised: M 01 : 39 : 40 ModelFz − 24 .− 24 .

Original: 89 M 01 : 50 : 56 ModelGs − 24 .− 

Revised: 89 M 01 : 50 : 56 ModelGs − 24 .−

Original: 65 M 01 : 29 : 42 ModelGz − 24 .− 

Revised: 65 M 01 : 29 : 42 ModelGz − 24 .−

Original: 65 M 01 : 27 : 01 ModelHs − 25 .− 

Revised: 65 M 01 : 27 : 01 ModelHs − 25 .−

Original: M 01 : 39 : 54 ModelHz − 24 .−
Revised: M 01 : 39 : 54 ModelHz − 24 .−

Original: M 01 : 45 : 32 ModelIs − 24 .− 

Revised: M 01 : 45 : 32 ModelIs − 24 .−

Original: 01 M 01 : 39 : 47 ModelIz − 24 .− 
 01 M 01 : 45 : 43 ModelJs − 24 .−
Revised: 01 M 01 : 39 : 47 ModelIz − 24 .− 
 01 M 01 : 45 : 43 ModelJs − 24 .−

Original: 92 M 01 : 47 : 38 ModelJz − 24 .− 

Revised: 92 M 01 : 47 : 38 ModelJz − 24 .−

Original: 79 M 01 : 35 : 45 ModelKs − 24 .−
Revised: 79 M 01 : 35 : 45 ModelKs − 24 .−

Original: 93 M 01 : 39 : 54 ModelKz − 24 .− 

Revised: 93 M 01 : 39 : 54 ModelKz − 24 .−

Original: 80 M 01 : 40 :
Revised: 80 M 01 : 40 :

Revised: S = 8 ) have superior performance whereas classiﬁcation task is harder for some other users (e.g.,

Revised: S = 6) .

Revised: Our AutoBayes can well resolve the issues of such a nuisance variation by linking the adversarial block for S -independent latent variables Z to generate subject-invariant feature.

Original: [Figure]
Revised: [Figure]

Revised: A. T IME C OMPLEXITY A NALYSIS

Revised: In Fig. 9(b), we have shown the accuracy vs. the space complexity.

Revised: In this section, we evaluate the time complexity in Figs.

Revised: 11(a)

Revised: and (b), which show the task classiﬁcation accuracy as a function of computation time for training and testing, respectively, for the Stress dataset.

Revised: As in the same setting of Fig. 9(b), we explored different DNN conﬁgurations for the models A, B, and Js, by sweeping the number of hidden layers and hidden nodes.

Revised: Some Pareto-front DNN conﬁgurations having lower complexity and higher accuracy are connected with lines.

Revised: We used pytorch on NVIDIA Tesla K

Revised: GPU with CUDA 10.1.

Revised: It is seen that the standard classiﬁer model A outperforms the other models in lower complexity regimes, whereas our AutoBayes can achieve better Pareto front for higher accuracy regimes.

Revised: It should be also noted that the increase of the time complexity is not so signiﬁcant (by a few folds) in comparison to that of the space complexity (by a few magnitudes) in Fig. 9(b).

Revised: [Figure]