How to modify the training data with bounded transferable perturbation that can lead to the largest generalization gap?

we consider the task of adding imperceivable noises to the training data, hoping to maximally confuse any corresponding classifier so as to make wrong predictions as much as possible when facing clean test data.


For instance, the attacker wants the classifier to wrongly recognize the pattern from class A specifically to Class B (thus not to Class C).

by teaching the perturbation generator to hijacking the training trajectory of the victim classifier, the generator can thus learn to move against the victim classifier step by step.


Mathematically, in the case of transformation sets, the latter can be modeled by a property of the true distribution.

[39,26], the S-PGD mechanism uses projected gradient descent with respect to the translation and rotation parameters with projection on the constrained set S of transformations.


Out of curiosity we also trained a "two-stage" STN where we train the localization network separately in a supervised fashion.

we use a randomly transformed version of the training data, treating the transformation parameters as prediction targets.


Unfortunately, the relationship between power and these parameters changes with the underlying distribution of the null and alternate p-values, as well as their relative frequency.

below, we only provide a heuristic argument about how to tune these parameters for ADDIS * .


In this section, we numerically compare the performance of ADDIS against the previous state-ofthe-art algorithm SAFFRON [5], and other well-studied algorithms like LORD++ [4], LOND [10] and Alpha-investing [2].

we use ADDIS * defined in Algorithm 1 as the representative of our ADDIS algorithm.


In Section 2, we derive the ADDIS algorithm and state its guarantees (FDR and mFDR control), deferring proofs to the supplement.

in Section 2.4, we discuss how to choose the hyperparameters in ADDIS to balance adaptivity and discarding for optimal power.


ADDIS is based on a new serial estimate of the false discovery proportion, having adaptivity to both fraction of nulls (like SAFFRON) and the conservativeness of nulls (unlike SAFFRON).

we present the following generalizations and leave the details (formal setup, proofs) to supplement for interested readers.


Doing the computation in an ASIC would drastically reduce the latency of kinematics inference and eliminate a large power draw for the gigabytes of neural data that must be transferred otherwise.

we plan to create an ASIC that can be implanted in the brain to perform inference of kinematics from neural signals.


Moreover, because of state feedback, a small recurrent network can be equivalent to a large feed-forward network.

a recurrent network will be computationally efficient, especially for the applications that require hardware implementation [19].


During inference, since the ground truth values are unavailable, the feedback, z k , has to be replaced by the previous network predictions.

the same approach cannot be applied during training since the DRNN has not been trained yet and it may cause poor performance of the DRNN.


Because of this disparity between training and testing, the DRNN may enter unseen regions of the state-space, leading to mistakes at the beginning of the sequence prediction process.

we should find a strategy to start from the ground truth distribution and move toward the predictions' distribution slowly as the DRNN learns.


These characteristics of PPC differentiate it from other brain areas and, while providing a large amount of information to the decoder, also require new paradigms, such as those discussed here, to extract useful information.

extracting appropriate neural features and designing a robust decoder that can model this relationship in an actual BMI setting is required.


6-HALF MOONS interpolates the original half moons dataset using 6 points with added noise.


This means one needs to store O(#nodes X #message passing steps) states, which can be costly for large graphs.

gRevNets can reconstruct hidden states in lower layers from higher layers during backpropagation, meaning one only needs to store O(#nodes) states.


Machine learning has been quite successful at generative modeling of complex domains such as images, audio, and text.

relational data poses new and interesting challenges such as permutation invariance, as permuting the nodes results in the same underlying graph.


Our GNF consists of 10 MP steps with attention and an MLP for each of F 1 , F 2 , G 1 , and G 2 .

this poses a problem for certain aggregation functions like sum and mean.


This allows the model to learn how to organize nodes in order to match a specific distribution.


The result of using a fully connected graph is that the computational cost of message passing is O(N 2 ), similar to the GraphRNN.

each step of the GNF is expressible in terms of matrix operations, making it more amenable to parallel architectures.


The initial values for v i (0) and u i (0) are set to -65 and -13 respectively for all nodes.

biological neural networks in the brain grow architecture that can generalize very well to innumerable datasets.


Among all the short sequences, we randomly choose 100,000 sequences for each class for training, validation, and test.

we feed the one-hot encoded DNA sequences into an LSTM layer, followed by a dense layer and a softmax function to predict the probability distribution over the 4 letters of {A, C, G, T }, and train the model using only the in-distribution training data.


Please see supplementary for the algorithms and time complexities of all the proposed methods and HGNN.

a simple yet effective way to overcome the limitations is to introduce hyperedge-dependent vertex weights [14].


They study sketching algorithms from both statistical and algorithmic perspectives.

they focus on a different setting, where n >> r, and prove bounds on RE and P E. For instance, they discover that RE can be bounded even when r is not too large, proving bounds such as RE ≤ 1 + 44p/r for subsampling and subgaussian projections


(Drineas et al., 2012), show furthermore that leverage scores can be approximated fast using the Hadamard transform.

suppose also that X is a deterministic matrix s.t. the esd. of XTX converges weakly to some fixed probability distribution with compact support bounded away from the origin.


Various versions of this fundamental problem have been studied before (see e.g., Drineas et al., 2006Drineas et al., , 2011Dhillon et al., 2013;Ma et al., 2015;Raskutti and Mahoney, 2016;Thanei et al., 2017, and the references therein).

in a generative model where the data are sampled from a linear regression model, Raskutti and Mahoney (2016) have recently compared the statistical performance of various sketching algorithms, such as Gaussian projections and subsampled randomized Hadamard transforms (SRHT) (introduced earlier in Sarlos (2006); Ailon and Chazelle (2006)).


this is because iid projections distort the geometry of Euclidean space due to their non-orthogonality.


We observe that our results are accurate, both in simulations and in two empirical data analysis examples, see Section 3.

they go beyond earlier work (Raskutti and Mahoney, 2016) because they are accurate not just up to the rate, but also down to the precise constants, even in relatively small samples (see Section A.16 in the supplemental for a comparison).


Fast orthogonal transforms such as the Hadamard transforms are considered as a baseline for sketching methods, because they are efficient and work well quite generally.

if the data are very uniform, for instance if the data matrix can be assumed to be nearly rotationally invariant, then sampling methods can work just as well, as will be shown below.


To encode this form of equivariance for human pose regression tasks, we propose "chirality nets."

the output of a chirality net is guaranteed to be equivariant w.r.t. a transformation composed of reflections and label switching.


For instance, collaborative virtual reality applications rely on accurate pose estimation for which significant advances have been reported in recent years.

recent state-of-the-art approaches use supervised learning to address pose regression and employ deep nets.


additionally, when comparing without test-time augmentation, our approach outperforms by 1 mm.


Pose symmetric 1D convolution layers can be based on fully connected layers.

a 1D convolution is a fully connected layer with shared parameters across the time dimension, i.e., at each time step the computation is the sum of fully connected layers over a window: we enforce equivariance at each time step by employing the symmetry pattern of fully connected layers at each time slice.


The temporal convolution considers the intra-frame information while the spatial convolution considers the inter-frame information.


If both gate and cell preserve the negation then the product will not.

we change the weight sharing scheme for the gates.


The proposed model, which achieves compelling results, leverages those cues by combining a long-short-term-memory (LSTM) module based deep net with attention over objects to obtain grounding and context.

the proposed model is also very intricate.


Moreover, notice that the elements of diagonal confusions (d's) and off-diagonal confusions (c's) reflect correct and incorrect classification, respectively.

according to standard practice, wlog., we focus on eliciting monotonically increasing DLPMs and monotonically decreasing LPMs in their respective arguments.


For any query based approach, it is important to understand the structure of the query space.

we first study the properties of the query spaces and then develop parametrizations required for efficient elicitation.


Instead, we provide novel strategies for eliciting linear functions of the multiclass confusion matrix and extend elicitation to more complicated yet popular functional forms such as linear-fractional functions of the confusion matrix elements [14].

the elicitation procedures involve binary-search type algorithms that are robust to both finite sample and oracle feedback noise.


Finite samples may affect the size of the sphere Sλ in LPM elicitation, but we observe that as long as λ is greater than Ω LPMs can be elicited (Appendix F.2).

here we emprically validate only DLPM elicitation with finite samples.


Metric Elicitation is a principled framework for selecting the performance metric that best reflects implicit user preferences.

available strategies have so far been limited to binary classification.


By using Lemma 1, obtain its BO off-diagonal confusions over the sphere Sλ , which clearly lies on the lower boundary.

varying θ, in this procedure, parametrizes the lower boundary ∂Sλ .


The term z 1 log(z 2 /(q 2 )) may attribute to the number of cycles in Algorithm 2, but due to the curvature of the sphere, we observe that it is not a dominating factor in the query complexity.

intuitively, this weak assumption ensures that when the cost or reward tradeoffs for the classes change, the preferred confusion matrices for those cost or reward tradeoffs also change (and vice-versa).


We propose to assess bias at the contextual word level.

this is true of all instances of significant effect sizes on test +I5 except for the c-word encoding of BERT (bbc).


This seems to suggest a degree of anti-stereotypical associations, but none of the tests which have negative effect sizes are significant Table 5: Intersectional embedding association tests and effect sizes, for word encodings (word), sentence encoding (sent) and contextual word representation (c-word).

there are only 2 significant positive associations for test +I4, but there are 9 such associations for test +I3 across all models.


Since the WebText dataset has not been released fully by [27], we only use the partially released version with 250K documents 2 .

barring a more robust study with more model sizes in consideration, we caution against a definitive conclusion.


We depart from previous work by including the nominative (she), accusative (her), prenominal possessive (her) and predicative possessive (hers) inflections of personal pronouns, and we also include the non-gendered or collective pronoun they.

for each sentence, we increment a count for female pronoun occurrence if any female pronoun is in the sentence, and then count the number pro-stereotypical and anti-stereotypical associations with occupation words (same for male and non-gendered pronouns).


Although these methods are promising, Gonen and Goldberg [13] show that the gender bias encoded in word embeddings is mostly hidden from the defined metric of projecting onto the "he-she" vector direction.

words still receive implicit gender from their associations (e.g. "receptionist" is no longer gendered with respect to "he", but is still gendered with respect to "captain").


Moreover, we find stronger evidence for bias when comparing against race (test +I3) than when comparing against gender (test +I4).


This indicates that when assessing social bias, the type of encoding matters determines whether the bias can be measured.

contextual word representations can be used in conjunction with sentence encodings to determine bias in a given model.


Fifth, we introduce a method of comparison that anchors at the most or least privileged group to show that intersectional identities suffer from such bias as well, and more so than their minority identities.

we show that the effect of race on intersectional identities seems to be larger than the effect of gender.


Emerging Convolutions use "causal masks"  whose inverse function falls into a complete autoregressive transformation.

mACOW achieves significantly more efficient inference and sampling ( §4.3), due to the carefully designed masks (Figure 1).


Compared with Theorem 2, the lower bounds in Theorem 3 lose a polynomial factor in M due to a larger policy space.

since the number of batches M of interest is at most O(log T ) (otherwise by Corollary 1 we effectively arrive at the fully adaptive case with M = T ), this penalty is at most poly-logarithmic in T .


In the case of variable noise, our method remains roughly as good when the noise parameters are known, but starts to have trouble when they need to be estimated from data.

with additive Gaussian noise the product involves two Gaussians, and because both distributions are functions of x, we have where we have exploited the symmetry of Gaussian distribution in the first term to swap x and y.


For example, at x-axis position marked "1" they have been trained for 2M minibatches compared to 0.5M minibatches for our method.

the receptive field of the resulting network includes the center pixel, so we offset the feature maps by one pixel before combining them.


Unfortunately, this does not imply that our self-supervised technique would be guaranteed to find the same optimum: we approximate the prior distribution with a Gaussian, whereas standard supervised training corresponds to a Gaussian approximation of the posterior.

a single-pixel offset at the end of each branch separates the receptive field from the center pixel.


The approximate distribution p(x|â„¦ y ) allows us to now apply Bayesian reasoning to include information from y at test-time.

the (unnormalized) posterior probability of the clean value x given observations of both the noisy pixel y and its context is given by Bayes' rule as follows: From this point of view, the distribution p(x|â„¦ y ) takes the role of the prior, encoding our beliefs on the possible xs before observing y.


They are endowed with an LSH family via the Hellinger approximation.

let P and Q be two different distributions on Ω.


Other examples are time series distributions, content of documents, or images that can be represented as histograms.

analysis of similarities in time series distributions or documents can be used in the context of attacks, and spam detection.


All the intermediate activations were then "re-computed" by solving the ODE layers backwards.

it has been recently shown that such an approach could lead to incorrect gradients, due both to numerical instability and also to inconsistencies that relate to optimizing infinite dimensional operators (the so called Discretize-Then-Optimize vs. Optimize-Then-Discretize issue) [9].


In configuration one, a residual block is created for each of the three time steps.

in configuration two, we only apply the first and last time evolutions of the parameters (i.e., we only use w 0 and w 1 to apply to activations).


We derive the optimality conditions for how backpropagation should be performed for the coupled ODE formulation using by imposing the standard Karush-Kuhn-Tucker conditions.

we implement the corresponding Discretize-Then-Optimize (DTO) approach, along with a checkpointing scheme presented in [9].


The governing equation here is dz/dt = c(t) dz/dx, where c(t) is variable velocity, and z is the signal that changes in time.

as discussed above, an alternative view of a residual network is the following continuous-time formulation: dz dt = f (z(t); âœ“), with z(t = 0) = z 0 and z(t = 1) = z 1 (we will use both z(t) and z t to denote activation at time t).


Here we perform an ablation study in which we remove the evolution of the model parameters, and instead we fix them to stale values in time (which is the configuration used in [8,9]), and test with a case where the model parameters are indeed evolved in time, which corresponds to results of Table 2.

we use two time steps for activation ODE (Eq. . 5b) and ten time steps for the evolution of the model parameters (Eq. 5c).


On the other hand, truly Bayesian inference would instead put a posterior distribution over model weights to characterize the uncertainty during training [2,20,30].

due to the complexity of nonlinear neural networks, analytic posterior is not available, hence strong independent assumptions over model weight have to be made in order to achieve computationally tractable variational solution.


While most previous works aim to diversify the representations, we explore the complementary direction by performing an adaptive and data-dependent regularization motivated by the empirical Bayes method.

we propose to construct a matrix-variate normal prior (on weights) whose covariance matrix has a Kronecker product structure.


Neural codes were shown to outperform existing state-of-the-art codes on the feedback channel [31].

equipping a decoder with a learnable neural network also allows fast adaptation via meta-learning [25].


The encoder and the decoder together can be naturally viewed as an over-complete autoencoder, where the noisy channel in the middle corrupts the hidden representation (codeword).

designing a reliable autoencoder can have a strong bearing on alternative ways of designing new encoding and decoding schemes for wireless communication systems.


There are two novel components of Turbo code which led to its success: an interleaved encoder and an iterative decoder.

in the canonical setting of AWGN channel, neural codes are still far from capacity-approaching performance due to the following challenges.


The population size |P| is 50 and the parents size |P| is 10.

there are 1000 networks evaluated in one search.


It can perform with a pre-trained backbone network and search with the previous NAS algorithm [31].

table 4 shows the computation cost for each steps.


To this end, DetNAS chooses the evolutionary search algorithm.

the search method and search space follow DetNAS.


The issue is that the batch statistics on one path should be independent of others.

we need to recompute batch statistics for each single path (child networks) before each evaluation.


However, in supernet pre-training, we adopt a path-wise [7] manner to ensure the trained supernet can reflect the relative performance of candidate networks.

in each iteration, only one single path is sampled for feedforward and backward propagation.


ShuffleNetv2-40 is inferior to ResNet-101 and DetNAS by 0.8% mmAP on COCO.

for RetinaNet, the training settings is similar to FPN, except that the initial learning rate is 0.01.


Comparisons to the network for ImageNet classification.

many object detectors directly use networks designed for Image classification as backbones.


We have shown that efficient coding models nonlinearly scale their resource allocation in sensory bottlenecks under nonuniform input densities.

the limited number of outputs neurons is not simply allocated proportional to receptor density.


Eigenvalues chosen from region H imply that the receptive field of the added output neuron also falls onto region H 1 .

the problem of how output neurons are allocated to either input region is solved by calculating and sorting the eigenvalues associated with each individual input region.


We show that different interactions between PMI vectors reflect semantic word relationships, such as similarity and paraphrasing, that are encoded in low dimensional word embeddings under a suitable projection, theoretically explaining why embeddings of W2V and GloVe work.

we also reveal an interesting mathematical interconnection between the considered semantic relationships themselves.


Interestingly, the difference between the associated PMI vectors: is a vector of un-weighted KL divergence components.

if dimensions were suitably weighted, the sum of difference components (comparable to Manhattan distance but directed) would equate to a KL divergence between induced distributions.


Whilst W2V and GloVe train two embedding matrices, typically only W is used and C discarded.

although relationships are learned between W and C, they are tested between W and W. If W and C are equal, the distinction falls away, but that is not found to be the case in practice.


Hashimoto et al. [14] and Arora et al. [4] propose generative language models to explain the structure found in word embeddings.

both contain strong a priori assumptions of an underlying geometry that we do not require (further, we find that several assumptions of [4] fail in practice (Appendix D)).


Intuitively, we can check if our alignment is correct via these reference measurements.

we can obtain aligned, denoised inner products with each of the two parameter vectors.


Instead, we use a new method motivated by error-correcting codes.

we perform several redundant queries, that help us to do this alignment.


Since there are L different vectors in the mixture, with O(L log L) measurements with a row we will be able to see the samples corresponding to each of the L vectors with that row.

even if this is true for measurements with each rows, we will still not be able to align measurements across the rows.


For ReLU networks (piece-wise linear networks in general), exact verifiers solve the robustness verification problem (1) by typically employing MILP solvers [Cheng et al., 2017, Lomuscio and Maganti, 2017, Dutta et al., 2018, Fischetti and Jo, 2017 or Satisfiability Modulo Theories (SMT) solvers [Scheibler et al., 2015, Carlini et al., 2017, Ehlers, 2017.

due to the NP-completeness for solving such a problem , it can be really challenging to scale these to large networks.


We do this by setting in (C) and computing the exact optimum.

yet this improvement is not significant enough to close the gap with the lower bounds.


Over different models, sizes, training methods, and datasets (MNIST and CIFAR-10), we find that (i) in terms of lower bounding the minimum l âˆž adversarial distortion 3 , the optimal layer-wise convex relaxation only slightly improves the lower bound found by , especially when compared with the upper bound provided by the PGD attack, which is consistently 1.5 to 5 times larger; (ii) in terms of upper bounding the robust error, the optimal layer-wise convex relaxation does not significantly close the gap between the PGD lower bound (or MILP exact answer) and the upper bound from .

there seems to be an inherent barrier blocking our progress on this road of layer-wise convex relaxation, and we hope this work provokes much thought in the community on how to bypass it.


These establish the conditions, in the context of the variational problem of Section 2, under which each of these algorithms are optimal.

this allows us to understand the prior assumptions which these algorithms make on the gradients of the objective function they are trying to minimize, and the way noise is introduced in the sampling of stochastic gradients, (gt)t>0.


The fact that this framework can naturally recover these algorithms begs further study.

it is still an open question whether it is possible to recover other stochastic algorithms via this framework, particularly those with second-order scaling adjustments such as ADAM or AdaGrad.


As a result, we can recover algorithms from this framework which have built-in online learning properties.

these algorithms use an online Bayesian filter on the stream of noisy gradient samples, g t , to compute estimates off (x t ).


This derivation demonstrates that the solution to the variational problem described in Section 2, under the assumption of a Gaussian model for the evolution of gradients, recovers mirror descent and SGD.

the martingale gradient model proposed in this section can be roughly interpreted as assuming that gradients behave as random walks over the path of the optimizer.


The choice of h determines the way we measure distance, and is typically chosen so that it mimics features of the loss function f .

this quantity plays a central role in mirror descent and non-linear sub-gradient algorithms.


(2017a), the structure of the manifold interacts with the local approximation of the objective function in a complicated way.

we provide convergence guarantees for perturbed first order Riemannian optimization methods to seond-order stationary points (local minima).


It is well-known that convexity/concavity does not imply the PL condition and PL condition does not imply convexity/concavity [30].

the problems we consider in the next section are neither restriction nor extension of our results on PL games.


Experimental Setup: The recent work in [42] observed that training a logisitic regression model to classify the images of the Fashion MNIST dataset can be biased against certain categories.

the function gλ(·)  becomes smooth with Lipschitz gradient; see Lemma B.1 in the supplementary.


Also note that the notion of FNE does not depend on the ordering of the min and max.

to be consistent with our notion of PL-games, we can formulate the problem as generative adversarial imitation learning of linear quadratic regulators is an example of finding a FNE for a min-max PL-game.


In fact, even finding local Nash equilibria is NP-hard in the general non-convex non-concave regime.In addition, as shown by [28, Proposition 10], local Nash equilibria for general non-convex non-concave games may not exist.

consider the game min Then (0, 0) is the only first-order Nash equilibrium and is not a second-order Nash equilibrium.


Similarly, one can show that even second-order Nash equilibrium may not exist for non-convex games, see Section 2 for more details.

a well-justified objective is to find first order Nash equilibria of such games [48]; see definitions and discussion in Section 2.


Given an `2 N, let m `be the sum of the multiplicities of the first `nonzero top eigenvalues of L K .

the following corollary follows immediately from Corollary 2.


This problem is reminiscent of the "class imbalance" problem in discriminatory modeling -the challenge however is exacerbated since there is not a single column to balance and the real data distribution should be kept intact.

the goal is to resample efficiently in a way that all the categories from discrete attributes are sampled evenly (but not necessary uniformly) during the training process, and to recover the (not-resampled) real data distribution during test.


Since columns in a row do not have local structure, we use fully-connected networks in generator and critic to capture all possible correlations between columns.

we use two fully-connected hidden layers in both generator and critic.


Recent works have shown that stochastic gradient descent (SGD) achieves the fast convergence rates of full-batch gradient descent for over-parameterized models satisfying certain interpolation conditions.

the step-size used in these works depends on unknown quantities and SGD's practical performance heavily relies on the choice of this step-size.


(ii) The performance of SGD with line-search and Polyak momentum is always better than "tuned" constant step-size SGD and Adam, whereas that of SGD with Goldstein line-search is competitive across datasets.

compared to SGD, checking this condition only makes use of additional mini-batch function (and not gradient) evaluations.


Note that the above line-search condition uses the function and gradient values of the mini-batch at the current iterate w k .


The simulation results can be found in Figure 1, and clearly indicate that using greedy planning leads to negligible degradation in the performance.

the simulations verify our claim that greedy policy updates greatly improve the efficiency of the algorithm while maintaining the same performance.


In many applications, the high computational complexity of model-based RL makes them infeasible.

practical model-based approaches alleviate this computational burden by using short-term planning e.g., Dyna [Sutton, 1991], instead of full-planning.


Note that Theorem 9 exhibits similar problem-dependent regret-bounds as in Theorem 1 of [Zanette and Brunskill, 2019].

the same corollaries derived in [Zanette and Brunskill, 2019] for EULER can also be applied to EULER-GP.


Surprisingly, we find that both generalized algorithms do not suffer from performance degradation, up to numerical constants and logarithmic factors.

we conclude that there exists an RL algorithm that achieves the minimax regret bound, while acting according to greedy policies.


In this work, we established that tabular model-based RL algorithms can explore by 1-step planning instead of full-planning, without suffering from performance degradation.

the value V is initialized optimistically and the algorithm interacts with the unknown environment in an episodic manner.


exploring with model-based greedy policies can be minimax optimal in terms of regret.


Using Lemma 1, we observe that the sequence of values is decreasing and bounded from below.

intuitively, the decrements of the values cannot be indefinitely large.


Note that the counterexamples are simply state vectors without labels, and their Lyapunov risk will be determined by the learner, not the falsifier.

although it is possible to have spurious counterexamples due to the Î´ error, they are used as extra samples and do not harm correctness of the end result.


[42] proposes to overcome some of these limitations by combining the UVFA approach with GPI.

it doesn't formulate a general framework for choosing base policies and their embeddings when learning a particular task space, or for sampling these policies, or addresses the issue of task boundaries and online adjustment of policy use.


Our algorithm draws inspiration from biology, where animals face similar continual learning tasks while foraging or evading predators in familiar environments.

we combine the successor representation (SR), that factors the value of actions into expected outcomes and corresponding rewards, with evaluating task similarity through clustering the space of rewards.


We investigated two approaches that could enable this flexibility: factorized representations, which abstract away general aspects of a task from those prone to change, and nonparametric, memory-based approaches, which can provide a principled way of using similarity to past experiences to guide current behaviour.

a number of recent or concurrent papers have proposed algorithms for introducing transfer into RL/deep RL settings, by using multiple policies in some way, though none of them use an inferential framework similar to ours, provides a principled way to deal with unsignalled task boundaries, or explains biological phenomena.


A sharply tuned policy for a previous reward setting with reward locations close to current rewards could lose out to a broadly tuned, but mostly unhelpful competing policy.

keeping all stored policies diffuse, or otherwise regularising them can be costly, as it can hinder exploitation or the fast instantiation of optimal policies given new rewards.


The tabular setting enables us to test many components of the algorithm and compare the emerging representations to their biological counterparts.

it is important to validate that these can be scaled up and used with function approximators, allowing the use of continuous state and action spaces and more complex tasks.


In this paper, we show that such independence is in fact not needed for such results which continue to hold under fairly general dependence structures.

we present uniform bounds on random quadratic forms of stochastic processes which are conditionally independent and sub-Gaussian given another (latent) process.


In this paper, we consider a generalization of such existing results by allowing for statistical dependence in  ξ.

we assume  ξ = { ξj } to be a stochastic process where the marginal random variables  ξ j are conditionally independent and sub-Gaussian given some other (latent) process F = {F j }.


For the uniform bound, the two terms can be separated using Jensen's inequality, the first term can be bounded using Theorem 1 and the second term can be bounded using a standard application of generic chaining using (SP-1) and (SP-2).

mean shifted versions of our results also hold.


In the learning process of neural networks, their weight parameters are updated iteratively so that the loss decreases.

in some settings the loss does not decrease simply, but its decreasing speed slows down significantly partway through learning, and then it speeds up again after a long period of time.


Deep learning, and neural network as its essential component, has come to be applied to various fields.

these still remain unclear in various points theoretically.


This is contrast to the case of linear networks which have no activation; in that case, as µ1 decreases the speed of learning gets exactly 1/µ1 -times larger.

this phenomenon is peculiar to nonlinear networks.


D, E, F is the second layers' counterpart of Q, R, T .

we consider a two-layer perceptron which has N input units, K hidden units and 1 output unit.


These indicate that the length of the plateau shortens as µ2 becomes large.

the more the distribution of nonzero eigenvalues gets broaden, the more the plateau gets alleviated.


We clarified the relationship between the input data statistics and plateau phenomenon.

it is shown that the data whose covariance matrix has small and disparsed eigenvalues tend to make the phenomenon inconspicuous, by numerically analyzing the macroscopic system.


the generated data (Î¾, t) is then fed to the student network stated above and learned by it in the on-line manner (see Figure 3).


It indicates that the plateau length and height heavily depend on µ1 , the input scale.

as µ1 decreases, the plateau rapidly becomes longer and lower.


We applied our test to check if state-of-the-art classifiers for the ImageNet dataset [8] have been overfitted to the test set.

we use the VGG16 classifier of [27] and the Resnet50 classifier of [16].


The generalization bounds are also applicable when the model and the data are dependent (e.g., for cross validation or for error estimates based on the training data or the reused test data), but they usually lead to loose error bounds.

although much tighter bounds are available if the test data and the model are independent, comparing confidence intervals constructed around the training and test error rates leads to an underpowered test for detecting the dependence of a model on the test set.


Thus, the test error rate and the adversarial error estimate (calculated based on the same test set) must be close if the test set and the model are independent, and are expected to be different in the opposite case.

if the gap between the two error estimates is large, the independence hypothesis (i.e., that the model and the test set are independent) is dubious and will be rejected.


More importantly, since it is based on adversarially generated data points, the adversarial estimator is expected to differ significantly from the test error rate if the model is overfitted to the test set, providing a way to detect test set overfitting.

the test error rate and the adversarial error estimate (calculated based on the same test set) must be close if the test set and the model are independent, and are expected to be different in the opposite case.


Under (H), standard concentration inequalities, such as the Chernoff or empirical Bernstein bounds [3], can be used to quantify how fast R S and R g (f ) concentrate around the expected error R(f ).

we assume that the data is linearly separable with a margin and the density  ρ is known.


Spatially-varying regularization has also been addressed from a Bayesian view in [34] by putting priors on B-spline transformation parameters.

metric estimation in [34] is in a fixed atlas-space, whereas RDMM addresses general pairwise image registration.


In [23] both the velocity field and the regularizer are assumed to be constant in time.


Theorem 1 (Image-based RDMM optimality conditions).

regularizer values indicate effective local standard deviations of a local multi-Gaussian regularizer.


However, static quadratures do not account for the error made during the integration.

the quadrature is inaccurate when the integrand is not smooth enough and the number of integration steps is too small.


We observed in our experiments that sharing the same integrand function does not impact performance.

the neural embedding function h i must produce a fixed size output for i ∈ {1, . . . , d}.


In this paper, we investigate graphical conditions to allow efficient identification in arbitrary linear structural causal models (SCMs).

we develop a method to efficiently find unconditioned instrumental subsets, which are generalizations of IVs that can be used to tame the complexity of many canonical algorithms found in the literature.


Each element of Z has edges to all parents of y, meaning that any scIS existing in the graph must have as many instruments as there are clauses.

finding an scIS for y in this graph corresponds to finding a satisfying assignment for the formula.


[38] performs a thorough investigation of various persistent and non-persistent, as well as convergent and non-convergent learning schemes.

the emphasis is on learning proper energy function with persistent and convergent Markov chains.


We hypothesize that an ideal unsupervised learning algorithm should use past observations to create a stable representation of the environment.

a representation that captures the global factors of variation of the environment in a temporally coherent way.


The challenge of model-based RL in rich 3D environments has compelled some researchers to use privileged information such as camera-locations [15], depth information [53], and other ground-truth state variables of the state simulator [54,49].

some work has provided evidence that we may not need very expressive models to benefit to some degree [30].


This state is a belief state and is used to calculate policy and value as well as being the starting point for predictions of the future.


For both datasets, as the number of layers (hence number of parameters) increases, overfitting becomes more severe for no tying.

in the above regression tasks, the performance metric is the MLL of the test data (or test MLL).


Additional results for IPVI with and without parameter tying are found in Appendix C.3.

parameter tying alleviates the overfitting considerably.


Since then, several extensions to the general methodology have been proposed (Ahmed et al., 2017;Shen et al., 2018).

it is still not entirely clear: what do the multiple heads in these models buy us?


As a first step to understand whether some heads are universally important, we perform the same ablation study on a second, out-of-domain test set.

we consider the MNLI "mismatched" validation set for BERT and the MTNT English to French test set (Michel and Neubig, 2018) for the WMT model, both of which have been assembled for the very purpose of providing contrastive, out-of-domain test suites for their respective tasks.


Attention is a powerful and ubiquitous mechanism for allowing neural models to focus on particular salient pieces of information by taking their weighted average when making predictions.

multi-headed attention is a driving force behind many recent state-of-the-art natural language processing (NLP) models such as Transformer-based MT models and BERT.


This observation begets the question: is more than one head even needed?

we compute the difference in performance when all heads except one are removed, within a single layer.


Figure 4 shows that performance drops much more rapidly when heads are pruned from the Enc-Dec attention layers.

pruning more than 60% of the Enc-Dec attention heads will result in catastrophic performance degradation, while the encoder and decoder self-attention layers can still produce reasonable translations (with BLEU scores around 30) with only 20% of the original attention heads.


In particular, pruning more than 60% of the Enc-Dec attention heads will result in catastrophic performance degradation, while the encoder and decoder self-attention layers can still produce reasonable translations (with BLEU scores around 30) with only 20% of the original attention heads.

encoder-decoder attention is much more dependent on multi-headedness than self-attention.


this suggests that the important heads are determined early (but not immediately) during the training process.


Unfortunately, in our case, this projection is quite complicated, since the intersection of the sparsity constraint and the quantization constraint is complicated.

we notice that the projection onto the feasible set defined by each individual constraint is doable (the projection onto the sparsity constraint is quite standard, how to do efficient projection onto the quantization constraint defined set will be clear soon).


The enormous complexity of CNNs remains a major inhibitor for their more extensive applications in resource-constrained IoT systems.

model compression [5] is becoming increasingly demanded and studied [6][7][8][9].


Meanwhile, ATMC also achieves very close, sometimes better accuracy-compression ratio trade-offs on benign testing sets than NAP, with much enhanced robustness.

it has indeed combined the best of both worlds.


We next briefly review three mainstream compression methods: pruning, factorization, and quantization.

we recognize that using group sparsity norms in (3) might potentially be a preferred option if ATMC will be adapted for model acceleration.


Next, since the target model needs to defend against the attacker, it requires to suppress the worst risk.

the overall objective for the target model to gain adversarial robustness could be expressed as Z denotes the training data set: As we reviewed previously, typical CNN model compression strategies include pruning (element-level [10], or channel-level [8]), low-rank factorization [16,13,17], and quantization [23,19,20].


To demonstrate that ATMC achieves remarkably favorable trade-offs between robustness and model compactness, we carefully design experiments on a variety of popular datasets and models as summarized in Section 3.1.

since no algorithm with exactly the same goal (adversarially robust compression) exists off-the-shelf, we craft various ablation baselines, by sequentially composing different compression strategies with the state-of-the-art adversarial training [32].


On all three victim models, our method significantly outperforms NES and Bandits-TD in both query efficiency and success rates.


As introduced in the previous section, we reckon that it is promising to apply the gradient of some reference models to span the search subspace for mounting black-box attacks.

there remain some challenges in doing so.


As suggested in Figure 2(b), one way of guaranteeing a low failure rate in our method is to collect adequate reference models.

we know from prior works that even adversarial examples crafted using some single-step attacks like the fast gradient (sign) [18] can transfer [28,22], hence one can hypothesize that the gradients of some "substitute" models are more helpful in spanning the search subspaces with reduced dimensionalities.


it is usually troublesome in practice, if not infeasible.


Our solution to resolve this issue is inspired by the dropout [35] and "droplayer" (a.k.a., stochastic depth) [12] techniques.


Additionally, the proposed transformation of the covariance function seems to be better suited for ordinal-valued variables rather than categorical variables, further restricting the utility of this approach.

we propose a method that can deal with high-dimensional combinatorial (categorical and/or ordinal) spaces.


If two vertices are connected by an edge, then their respective set of combinatorial choices differ only by a single combinatorial choice.

we can now revisit the notion of smoothness on combinatorial structures as smoothness of a graph signal [8,35] defined on the combinatorial graph.


We define then smoothness of functions on combinatorial structures to be the smoothness of graph signals using the Graph Fourier Transform (GFT) [35].

we propose as our GP kernel on the graph a variant of the diffusion kernel, the automatic relevance determination(ARD) diffusion kernel, for which computing the GFT is computationally tractable via a decomposition of the eigensystem.


We circumvent this requirement by the notion of the combinatorial graph defined as a graph, which contains all possible combinatorial choices as its vertices for a given combinatorial problem.

each vertex corresponds to a different joint assignment of categorical or ordinal variables.


When comparing min(ESS)/s, SA outperforms AM by 31x on covertype and 11x on MiniBoonE and outperforms NUTS by 24x on covertype and 147x on MiniBoonE.

sA samples effectively from this high-dimensional, challenging posterior distribution without requiring any tuning of the initialization distribution.


There are a few work related to our considered class of OT with tree metrics [35,62].

kloeckner [35] studied geometric properties of OT space for measures on an ultrametric space, and Sommerfeld and Munk [62] focused on statistical inference for empirical OT on finite spaces including tree metrics.


We use the farthest-point clustering due to its fast computation.

the complexity of the farthest-point clustering into κ clusters for n data points is O(n log κ) using the algorithm in [23].


As in [42], we generated 1000 orbits for each class where each orbit contains 1000 points.

we propose a positive definite tree-(sliced-)Wasserstein kernel that generalizes the sliced-Wasserstein kernel [11,36].


Following [33, Â§1, p.245-247], an ultrametric implies a tree structure which can be constructed by hierarchical clustering schemes.


The TW distance, as well as its average over several trees, can be shown to be negative definite 1 .


Louizos et al. [2017] studied scale mixtures of Gaussian priors and half-Cauchy scale priors for the hidden units of VGG models [Simonyan and Zisserman, 2014] and achieved good model compression performance on CIFAR10 [Krizhevsky, 2009] using VI.

due to the limitation of VI in non-convex optimization, the compression is still not sparse enough and can be further optimized.


Ghosh et al. [2018] proposed to use variational inference (VI) based on regularized horseshoe priors to obtain a compact model.

all the analyses fail when DNN has too many parameters, and the over-specified model tends to have a large prediction variance, resulting in poor generalization and causing over-fitting.


However, all the analyses fail when DNN has too many parameters, and the over-specified model tends to have a large prediction variance, resulting in poor generalization and causing over-fitting.

a proper model selection is on demand at this situation.


In all of these applications, because a sample is needed at every epoch t, it is desirable to have a fast online sampling algorithm.

the ultimate goal is to design an algorithm for Problem 1.1 such that the number of gradient evaluations is almost constant at each epoch t, so that the computational requirements at each epoch do not increase over time.


To overcome this, we could modify our algorithm by including an adaptive pre-conditioner which changes along with the target distribution.

there are many streaming algorithms that are used in practice which lack provable guarantees, or which rely on properties of the data (such as compressibility [HCB16,CB19]).


Motivated by real-world applications, we assume that functions are smooth, their associated distributions have a bounded second moment, and their minimizer drifts in a bounded manner, but do not assume they are strongly convex.

our assumptions hold for online Bayesian logistic regression, when the data satisfy natural regularity properties, giving a sampling algorithm with updates that are poly-logarithmic in T .


These problems lead us to a relaxed implementation of (Obj 0 ) , bringing us possibilities to embrace a practical and a more flexible solution.

the solution produced by the optimization method shows a close connection between our method and the optimal transport problem, which brings new insight into how negative transfer could be prevented across latent tasks and output tasks.


From a non-asymptotic perspective, Prop.2-(b) shows how fast this approximation will take place.

we will get a reasonable approximation of the original OT problem with a small ϑ.


The N-gram approach constructs a representation vector c (n) for the sentence, whose coordinates correspond to all n-grams and the value of a coordinate is the number of times the corresponding n-gram shows up in the sentence.

the dimension of an n-gram vector is |V | n for a vocabulary V , and the vector c (1) is just the count vector of the words in the sentence.


In contrast, virtual screening (i.e., selecting molecules based on predicted properties via machine learning methods) can be done in minutes for predicting millions of molecules.

it can be a good filtering step before the physical experiments, to help accelerate the drug discovery process and significantly reduce resource requirements.


At iteration n, each vertex updates its latent vector by element-wise multiplying it with the sum of the latent vectors of its neighbors.

at the end of iteration n, the latent vector on vertex i is the sum of the embeddings of the walks that end at i and have length n, and the sum of the all latent vectors is the embedding of the n-gram walk set (with proper scaling).


Since our focus is to compare the representations of the graphs, no transfer learning or multi-task learning is considered.

we are comparing each task independently, which gives us 28 regression tasks and 32 classification tasks in total.


Deep Tensor Neural Networks (DTNN) [47] and Message-Passing Neural Networks (MPNN) [26] are two graph neural networks that are able to utilize 3D information encoded in the datasets.

we further compare our method to these two most advanced GNN models, on the two datasets QM8 and QM9 that have 3D information.


Finally as has been done in previous works, e.g. .g. [5, 8–10] for compressed sensing and denoising, translating our results to practical situations in designing an AMP algorithm that takes care of correlated GAN or VAE weights is still under investigatio

we solve the fixed point equations (12) and plot the MMSE obtained from the fixed point in a heat map, for the linear, sign and relu activations.


We analyze the Bayesoptimal performance under specific generative models for the spike.

as a unique fixed point is found, the Bayes optimal errors are continuous and we did not observe any algorithmic gap.


These results demonstrate that the exponential scheme convicingly outperforms the polynomial step-size schemes.

we consider the use of: Where, we perform a systematic grid search on the parameters Î· 0 and b.


In contrast to the non-smooth setting, the state of our understanding of SGD's final iterate for smooth stochastic convex optimization, or, say, the streaming least squares regression setting is far less mature âˆ’ this gap motivates our paper's contributions.

practical SGD implementations typically return the final iterate of a stochastic gradient procedure.


In contrast, this paper shows that Step Decay schedules, which cut the learning rate by a constant factor every constant number of epochs (i.e., the learning rate decays geometrically) offer significant improvements over any polynomially decaying step size schedule.

the behavior of the final iterate with step decay schedules is off from the statistical minimax rate by only log factors (in the condition number for the strongly convex case, and in T in the non-strongly convex case).


First introduced by Ioffe and Szegedy [2015], BatchNorm fixes layer distributions to reduce ICS (Internal Covariate Shift), a phenomenon that the upper layers need to continuously adapt to the new distributions of lower layers.

eq. ( 3) shows that normalization re-centers and re-scales input vector x.


The improvement mainly comes from the stronger expressive power of deep layers.

with the increase of depth, the network training process becomes complicated and requires advanced architectural techniques.


Initially, considering that input information may be lost when normalizing input distributions, the bias and gain are designed for affine transformation on normalized vectors to enhance the expressive power.

since the bias and gain are learned from the training set and they ignore the input distributions of the testing data, the risk of over-fitting may increase in LayerNorm.


a typical example is its application in the state-of-the-art framework, Transformer [Vaswani et al., 2017].


Next we focus on the theoretical lower bound for the efficiency-fairness tradeoff that any algorithm could achieve.

we show that the tradeoff achieved in Theorem 3.1 is actually tight in this model.


Leaving the system as it is would affect the sustainability of the ridesourcing business model in the long run, as unsatisfied drivers will not renew their memberships and new drivers will be deterred from signing up.

fairness on the drivers' side should be assessed more carefully and should receive more attention.


Efficiency and fairness are often competing objectives such that in most cases, the optimum of both cannot be achieved simultaneously.

we are naturally led to the question of how to reconcile system efficiency and drivers' fairness in a ridesourcing assignment.


In this paper, we focus on both the system efficiency and the fairness among drivers and quantitatively analyze the tradeoffs between these two objectives.

we give an explicit answer to the question of whether there always exists an assignment that achieves any target efficiency and fairness.


Note that such changes are two-way (i.e., efficiency or fairness-oriented), gradual, and dependent on dynamically changing demand conditions.

to provide better managerial flexibilities to decision makers, we need to provide a full set of candidate allocation solutions and characterize the trade-offs inherent in applying these concepts.


For instance, a central decision maker may choose a mild strategy that balances the tension between efficiency and fairness in normal traffic conditions.

as demand increases and traffic conditions worsen, it may be desirable to move to a strategy that puts higher attention to efficiency to quickly serve the waiting passengers.


To reflect real road conditions and traveling time, we construct a road network of Manhattan with 3,671 nodes and 7,674 edges.

it is evident that our algorithm manages to redistribute the trip utility increments to the vehicles with low historical utilities, without sacrificing too much on the efficiency.


[5,6] has specifically studied fairness and the corresponding efficiency loss in a general divisible resource allocation framework and applied their results to a case study in the context of air traffic management.

none of these works can easily incorporate additional waiting time and pick-up distance constraints because these are unique to the ridesourcing problem.


(i), (ii), and (iii) are intricately connected and may have competing effect up to some extent on the efficiency loss.

we can conclude that the theoretical worst-case efficiency loss does not necessarily arise even in artificial examples; here, we have seen the problem instances that show much more benign behaviour.


While we focused on the correspondence problem between transcriptomics and electrophysiology(k = 2), we presented the technical development of k-coupled autoencoders in full generality.

our method is applicable to the joint alignment of additional modalities.


2B suggests that the individual cell types are well approximated by hyperellipsoids.

fitting a Gaussian mixture model to the encodings provides an efficient prior distribution for p(zse) (or p(zse)), and produces a generative model for multi-modal datasets.


The trained network is expected to align the representations without any explicit transformation matrices.

in the absence of any normalization of the representations, the individual agents can arbitrarily scale down their representations to minimize the coupling cost without a penalty on reconstruction accuracy.


Parameters of the resulting autoencoding architectures are fitted with stochastic mini-batch training and the Adam optimizer [24].


We present a rigorous analysis of these problems, and show that normalization with the full covariance matrix of the mini-batch is sufficient, as expected [8], to obtain reasonable latent space representations.

this calculation can be prohibitively inaccurate depending on the latent space dimensionality and batch size ("curse of dimensionality").


However, this calculation can be prohibitively inaccurate depending on the latent space dimensionality and batch size ("curse of dimensionality").

we propose an alternative normalization that relies only on estimating the minimum eigenvalue of this covariance matrix.


In this study, we have demonstrated a surprising level of cross-modal predictive ability across transcriptomic and electrophysiological recordings.

we showed that the transcriptomic class can be predicted with ∼80% accuracy from electrophysiological recordings when the transcriptomic hierarchy is resolved into 15 classes, and with ∼70% accuracy when it is resolved into 25 classes (λ = 10 results).


Xu et al. [36] and Gleave and Habryka [12] have proposed approaches that combine IRL and gradient-based meta-learning [9], which provide promising results on deriving generalizable reward functions.

they have been limited to tabular MDPs [36] or settings with provided task distributions [12], which are challenging to gather in real-world applications.


Without further constraints over m, directly applying AIRL to learning the reward function (by augmenting each component of AIRL with an additional context variable m inferred by q Ïˆ ) could simply ignore m, which is similar to the case of InfoGAIL [21].

some connection between the reward function and the latent context variable m need to be established.


The primary contribution of this paper is a new framework, termed Probabilistic Embeddings for Meta-Inverse Reinforcement Learning (PEMIRL), which enables meta-learning of rewards from unstructured multi-task demonstrations.

pEMIRL combines and integrates ideas from context-based meta-learning [5,26], deep latent variable generative models [17], and maximum entropy inverse RL [42,41], into a unified graphical model (see Figure 4 in Appendix D) that bridges the gap between few-shot reward inference and learning from unstructured, heterogeneous demonstrations.


The coherence curves follow similar trend for other values for the threshold 2 ≤ T ≤ 20.

the measure is sensitive to topic sparsity; if the support of the topic distribution is smaller than T the measure becomes more noisy and less meaningful.


The topics for our model are semantically meaningful, capturing certain intuitive themes, as desired.

the model applies KLdivergences between the observed similarities (distributions) and latent distributions as likelihoods.


The saturating distance curves also show an effective range for λ values; for λ >> 0.2 (not shown), the computations eventually become more unstable, because more and more terms are assigned to the empirical distributions and the topics become too sparse complicating posterior inference.

the entropies show how the topics become (on average) more sparse for increasing λ.


The K-divergence is always well-defined for all values for qm ∈ ∆; this is especially relevant at the boundaries of ∆.

the K-divergence (4) is not as sensitive to misses as KL(p m , q m ), which approaches infinity close to the boundaries imposing infinite penalty for misses, essentially, imposing a barrier function.


Indexing starts at 1, with A i,: and A :,i indicating the i-th row and i-th column of A respectively.

now the entries of vector w + are in between 0 and 1.


That is, if M = L in Theorem 1 then it becomes the known Cheeger's inequality.

lemma 2 (Matrix Bernstein inequality, Theorem 1.4 in (Tropp 2012)).


In contrast to (Globerson et al. 2015, Foster et al. 2018), we study the sufficient conditions for exact recovery in polynomial time, and provide high probability results for general families of undirected connected graphs, which we consider to be a novel result to the best of our knowledge.

we show that weak-expander graphs (e.g., grids) can be exactly recovered by adding small perturbations (edges coming from the Erdos-Rényi model with small probability).


Since our prediction is a column vector y, we have that y y is rank-1 and symmetric, which implies that Y is a positive semidefinite matrix.

our relaxation to the combinatorial problem (3) results in the following primal formulation 2 : We will make use of the following matrix concentration inequality for our main proof.


This fact combined with complementary slackness, primal and dual feasibility, entail that Y is a multiple of y y .

we must have that Y = y y because Y i,i = 1.


Since we have that y is an eigenvector of V âˆ’ X with eigenvalue zero, showing that Î» 2 (V âˆ’ X) > 0 will imply that V âˆ’ X is positive semidefinite.

we focus on controlling its second smallest eigenvalue.


our result in Theorem 1 apply for more general matrices and is of use for our next result.


Our results show that exact recovery is possible and achievable in polynomial time for a large class of graphs.

we show that graphs that are bad expanders can be exactly recovered by adding small edge perturbations coming from the Erdős-Rényi model.


Finally, from Theorem 3, it is clear that since the parameter q ∈ (0, 0., for a sufficiently large n we have an exponential decay of the probability of error 2 .

we focus on the conditions of the first stage and provide examples in the next section.


When keeping more keypoints, poor local maxima starts to get selected for these models (e.g in the sky or the river in Figure 4) and the matching performance drops.

having numerous keypoints is important for many applications such as visual localization because it augments the chance that at least a few of them will be correctly matched despite occlusions or other noise sources.


After learning the keypoint detector, a deep descriptor is trained using a second network branch, sharing most of the computation.

our approach learns both of them jointly from scratch and without introducing any artificial bias in the keypoint detector, which is also achieved by Georgakis et al.


We note that the Baseline model does not locate any keypoints in the arms area.

when the pose difference with the initial pose increases, the model cannot reconstruct the video (columns 3,4 and 5).


Once trained on a set of videos depicting objects of the same category (e.g. faces, human bodies), our method can be applied to any object of this class.

we note that, with respect to the Baseline model, the MKR of the full model is smaller by the factor of 2.75.


At test time we apply our model to pairs composed of the source image and of each frame of the driving video and perform image animation of the source object.

we use equal loss weights in all our experiments.


Recently, deep generative models have emerged as effective techniques for image animation and video retargeting [2,41,3,42,27,28,37,40,31,21].

generative Adversarial Networks (GANs) [14] and Variational Auto-Encoders (VAEs) [20] have been used to transfer facial expressions [37] or motion patterns [3] between human subjects in videos.


In the case of Tai-Chi-HD, the human-pose estimator returns an additional binary label for each keypoint indicating whether or not the keypoints were successfully detected.

we also report the MKR defined as the percentage of keypoints that are detected in the ground truth frame but not in the generated one.


Finally, in our preliminary experiments, we observed that our model shows low sensitivity to the relative weights of the reconstruction and the two equivariance losses.


In generating adversarial examples, the gradient is usually normalized [13,24], such that the direction of the gradient estimator, instead of the magnitude, will affect the performance of attacks.

we also estimate the gradient norm every 10 attack iterations in all experiments to reduce the number of queries, since usually its value is relatively stable in the optimization process.


These methods all use queries to obtain knowledge of the black-box model, and train/find surrogate models to generate adversarial examples, with the purpose of improving the transferability.

we do not optimize the surrogate model, but focus on utilizing the gradient of a fixed surrogate model to obtain a more accurate gradient estimate.


we incorporate a scaling factor b in Eq. (7) and minimize the error w.r.t. b, which can neglect the impact of the magnitude on the loss of the estimator gˆ.


In [25], communities are used to re-identify multiple addresses belonging to a same user in Bitcoin trading networks.

there is a need to hide the community affiliations in order to preserve the privacy of online users.


Discovering hidden patterns in this network structure is a compelling application of graph data mining algorithms.

community detection stands out as one of the most important graph mining methods [11,16,23,26].


Our experimental results obtained on three datasets show that this principle works.

our segmentation model is competitive with supervised approaches trained on a few hundred labeled examples.


State-of-the-art methods use clever architectural choices or pipelines tailored to the challenges of the task [5,20,58].

most of those models use pixel-level supervision, which can be unavailable in some settings, or time-consuming to acquire in any case.


Note that our objective is not to generate appealing images but to learn an object segmentation function.

reDO generates images that are less realistic than the ones generated by state-of-the-art GANs.


If A t (defined in Section 4) is small then it is likely that A t+1 is also small.

our algorithm is prone to select the last part of iterates.


A basic and representative method of this type is the gradient descent, which iteratively moves iterates along the minus gradient direction of the current iterate.

gradient descent applied to machine learning problems requires to go through all training examples at each iteration, which is not efficient when the sample size is large.


Indeed, one can randomly draw the output according to a distribution over the iterate sequence with the probability mass function determined by any optimal weighted averaging scheme [20,31,37].

this scheme introduces new variances due to the choice of a random iterate as the output.


Intuitively, A t is related to the distance between w t and w t+1 .

our selection of the iterate shares some spirit with the widely used heuristic of terminating the algorithm when the successive iterates are close.


The choice of algorithms would affect the implicit regularization introduced in the learned models.

the optimization technique itself "biases" towards a certain model in an implicit way (Soudry et al.  [2018]).


Moreover, [28,27] take a wide spectrum by considering both the social preference and individual variations simultaneously.

it designs a basic linear mixed-effect model which not only can derive the common preference on population-level, but also can estimate user's preference/utility deviation in an individual-level.


If a voter thinks college V 3 is better than college V 6 , a solid arrowed line from V 3 to V 6 occurs (i.e., superiority).

when a voter thinks the two colleges (i.e., V 1 and V 3 ) listed are incomparable and difficult to judge, he may click the button "I can't decide", then a dotted line connecting V 1 and V 3 happens (i.e., tie).


In this paper, instead of learning a global ranking which is agreed with the consensus, we pursue the tie-aware partial ranking from an individualized perspective.

we formulate a unified framework which not only can be used for individualized partial ranking prediction, but also be helpful for abnormal user selection.


This is realized by a variable splittingbased algorithm called iSplitLBI.

our algorithm generates a sequence of estimations with a regularization path, where both the hyperparameters and model parameters are updated.


Since α-cut is an ensemble-based algorithm, its performance depends on the choice of weak learners.

we compare our proposed algorithm with the α-cut algorithm where different types of such weak learners and regularization schemes are adopted.


Clearly one can see that users jumped out earlier (i.e., the top 10% marked with pink) show larger L 2 -distance, thus are those with large deviation from the population's opinion and can be treated as abnormal users.

users jumped out later (i.e., the bottom 10% marked with blue) tend to have smaller or even zero L 2 -distance.


Our approach ideally requires identifying important regions that a human considers most critical in answering the question.

the base system utilizes a Faster R-CNN head [22] in conjunction with a ResNet-101 base network [10] as the object detection module.


In this section, we quantitatively evaluate the effectiveness of the proposed self-critical objective.

we evaluate the fraction of false sensitivity where the predicted incorrect answer's sensitivity to the influential object (to which the correct answer is most sensitive) is greater than the Figure 4: Positive examples are showing that our self-critical reasoning approach prevents the incorrectly predicted answer in the UpDn baseline system from being sensitive to the most influential object.


The state-of-the-art VQA systems [8,2,1,3,12,33,25,29,14,15,21] achieve high performance when the training and test question-answer (QA) pairs are sampled from the same distribution.

most of these systems fail to generalize to test data with a substantially different QA distribution.


For question embedding, following [2], we perform standard text pre-processing and tokenization.

questions are first converted to lower case and then trimmed to a maximum of 14 words, and the words that appear less than 5 times are replaced with an "<unk>" token.


This is because negative gradients on the inputs to a ReLU are valuable evidence against the current prediction.

there is no need to zero them out with a ReLU.


In order to ensure the training and test data have different category distributions, we intentionally assign different weights to the two components.

during training, the examples are drawn from N 1 with probability p, and during test, the examples are drawn from N 1 with probability 1 âˆ’ p. We examine the effectiveness of our self-critical approach varying p from 0.05 to 0.5 (i.e. 0.05, 0.1, 0.2, 0.5) (0.5 means no train/test difference).


The number around the bounding box is the answer's sensitivity to the object.

the work most related to ours is HINT [25], which enforces the system's gradient-based importance scores for each detected object to have the same rankings as its human importance scores.


First, like the previous work [28,35,25,20], we used regions that humans have explicitly marked as important.

this requires a fair bit of extra human effort to provide such detailed annotations.


For example, in VQA-CP, bedrooms are the most common room type.

during testing, systems frequently incorrectly classify bathrooms (which are rare in the training data) as bedrooms.


These systems are trained to encourage the network to focus on regions in the image that humans have somehow annotated as important (which we will refer to as "important regions.").

many times, the network also focuses on these important regions even when it produces a wrong answer.


We showed that by correctly incorporating an estimate of the propensity score into the covariance kernel, one can substantially improve the precision of both the posterior mean and posterior uncertainty quantification.

this makes the modified GP method highly competitive with state-of-the-art methods.


Acknowledgements: Botond Szabó received funding from the Netherlands Organization for Scientific Research (NWO) under Project number: 639.031.654.

more concretely, one can sample from the marginal posterior for ψ by drawing a full posterior sample (F, π, m) and computing the corresponding draw Ïˆ according to the formula (2).


Then, invariant descriptors can be extracted on the detected local patches.

finally, GAS uses the average of 128-dimensional embedded group features weighted by the attention weights as descriptors.


The robustness of detectors can be guaranteed theoretically, e.g., by the scale-space theory [34].

a typical image often have very few pixels for which viewpoint covariant patches may be reliably detected [22].


The Universal Correspondence Network (UCN) [7] uses a convolutional spatial transformer [26] in the network to normalize the local patches to a canonical shape.

learning an invariant spatial transformer is as difficult as learning a viewpoint covariant detector.


Such a permutation preserves local structures of the group features.

we propose to use group convolutions to encode the local structures of the group features, resulting in feature representations that are not only discriminative but equivariant to the transformations in the group.


We further demonstrate the robustness of GIFT to extremely large scale and orientation changes on several new datasets.

this transformation-invariant dense descriptor simplifies correspondence estimation as detecting covariant patches can be avoided.


Some recent works [45,61,29,41,64,12,62] try to learn such viewpoint covariant patch detectors by CNNs.

the definition of a canonical scale or orientation is ambiguous.


Instead of relying on a sparse set of covariant patches, some recent works [20,7,11] propose to extract dense descriptors by feeding the whole image into a convolutional neural network (CNN) and constructing pixel-wise descriptors from the feature maps of the CNN.

the CNN-based descriptors are usually sensitive to viewpoint changes as convolutions are inherently not invariant to geometric transformations.


The main idea is to add adversarial examples into the training set to improve the robustness.

earlier work usually only adds adversarial example once or only few times during the training phase.


Recall that L A (W) is the loss suffered with respect to the perturbation function A.

this makes f (W, x) = 0 at initialization, which helps eliminate some unnecessary technical nuisance.


An adversarial example can be generated by maximizing the loss function within an -ball around a natural sample.

generating adversarial examples can be viewed as solving a constrained optimization problem and can be (approximately) solved by a projected gradient descent (PGD) method [31].


Our proof idea utilizes the same high-level intuition as [1,27,18,55,10,11] that near the initialization the network is linear.

compared with Equation ( 7) which shows an n-interpolation class can be realized by a network architecture with VC-Dimension O(n log n), we can conclude that robust interpolation by neural networks needs more capacity, so increasing the width of neural network is indeed in some sense necessary.


As an example, using quadratic ReLU as activation function, we solve the explicit dependency on in Appendix C.2 that doesn't rely on Assumption 5.2.

adversarial training is guaranteed to find a robust classifier under a given attack algorithm when the network width is sufficiently large.


In contrast, there are neural net architectures that can interpolate n samples with only O(n) parameters and VC-Dimension at most O(n log n).

the capacity required for robust learning is higher.


The second approach is to instead learn a representation that is faithful to the underlying data structure, hoping that this is sufficient to disentangle the representation.

there is currently little to no agreement in the literature on how to learn such representations [Locatello et al., 2019].


Inspired by Detlefsen et al. [2018], we also consider transformations T using the highly expressive diffiomorphic transformations CPAB from Freifeld et al. [2015].

the naming comes from Eq. 5 and 6, where z A is respectively unconditioned and conditioned on z P .


For Gradient Descent, we make use of this property by showing divergence increases as the strategies move from one pure strategy to another.

strategies will never reach the boundary for some variants of FTRL.


These algorithms again have interesting properties in zero-sum games.

in two-by-two zero-sum games, both steps (1) and (3) trivially extend for other variants of FTRL.


The time-average strategy converges to a O(η)-approximate Nash equilibrium (Cesa-Bianchi and Lugoisi [2006]).

bailey and Piliouras [2018] show that the day-to-day behavior diverges away from interior Nash equilibria.


Despite the well established nature of these results recent work has revealed some surprising insights that come to challenge the traditional ways of thinking in this area.

in the case of zero-sum games what is referred to as "convergence" to equilibrium, is the fact that when both agent apply regret-minimizing algorithms, both the time-average of the mixed strategy profiles as well as the utilities of the agents converge approximately to their Nash equilibrium values, where the approximation error can become arbitrarily close to zero by choosing a sufficiently small step-size.


While our approach draws high-level inspiration from MTL, we highlight key differences: whereas tasks are disjoint in MTL, slice tasks are formulated as micro-tasks that are direct extensions of a base task-they are designed specifically to learn deviations from the base-task representation.

hPS: In the style of multi-task learning, we model slices as separate task heads with a shared backbone trained via hard parameter sharing.


Intuitively, if the model knows "where" in the 2-dim data space an example lives (as defined by SFs), it can condition on slice-specific features as it makes a final, slice-aware prediction.

we consider two text-based relation extraction datasets: Chemical-Disease Relations (CDR), [41], in which we identify causal links between chemical and disease entities in a dataset of PubMed abstracts, and Spouses [9], in which we identify mentions of spousal relationships using preprocessed pairs of person mentions from news articles (via Spacy [17]).


A related line of work [KLS09, ABL17, DKK + 16, LRV16, DKK + 17, DKK + 18, DKS18, KKM18, DKS19, DKK + 19] has given polynomial time robust estimators for a range of learning tasks.

[KLS09, ABL17, DKS18, DKK + 19] obtained efficient PAC learning algorithms for halfspaces with malicious noise [Val85,KL93], under the assumption that the uncorrupted data comes from a "tame" distribution, e.g., Gaussian or isotropic log-concave.


Instead of trying to guess the projection of w * onto the space of large eigenvectors all at once, we will do so in stages.

we study the problem of properly learning large margin halfspaces in the agnostic PAC model.


We note that the inapproximability ratio of [DLS14] is close to being tight for a natural, yet restricted, family of improper learners.

our proper hardness result holds against all proper learners under a widely believed worst-case complexity assumption.


Our algorithm tries to simulate the above described procedure by making appropriate guesses.

we start by guessing a sequence of positive integers k (i) whose sum is at most 8/(δγ2).


We give learning algorithms and computational hardness results for this problem, for all values of the approximation ratio α ≥ 1, that are nearly-matching for a range of parameters.

for the natural setting that α is any constant bigger than one, we provide an essentially tight complexity characterization.


We borrow an idea from [BFKV96] that in some sense allows us to "reduce" the general case to the large margin case.

[BFKV96] (see also [DV04a]) developed a pre-processing routine that slightly modifies the distribution on the unlabeled points and guarantees the following weak margin property: After preprocessing, there exists an explicit margin parameter Ïƒ = â„¦(1/poly(d, b)), such that any hyperplane through the origin has at least a non-trivial mass of the distribution at distance at least Ïƒ from it.


Moreover, this region is efficiently identifiable by a simple thresholding rule.

we show that there exists a threshold T > 0 (which can be found algorithmically) such that the hypothesis sign(hwb , xi) has error bounded by η +  in the region RT = {x : |hwb , xi| ≥ T}.


In the RCN model, the large margin case is easy because the learning problem is essentially convex.

there is a convex surrogate that allows us to formulate the problem as a convex program.


As a corollary of our main result (Theorem 1.2), we answer this question in the affirmative.

we obtain an efficient algorithm that achieves misclassification error arbitrarily close to η for all LTFs.


Our algorithm circumvents this difficulty by approaching the problem indirectly to find a nonproper classifier.

our algorithm works in multiple rounds, where within each round only points with high value of | w, x | are considered.


We study the problem of distribution-independent PAC learning of halfspaces in the presence of Massart noise.

we are given a set of labeled examples (x, y) drawn from a distribution D on R d+1 such that the marginal distribution on the unlabeled points x is arbitrary and the labels y are generated by an unknown halfspace corrupted with Massart noise at noise rate Î· < 1/2.


In fact one might not have access to the true distribution of the mapping, but just to the numerical outputs.

one needs to consider more sophisticated metrics/divergences, such as the total variation distance The total variation distance is one of the most broadly used probability metrics.


Overconfidence alone decreases monotonically as we increase â†µ as shown in Figure 2i.

we note that this approach is similar to the recent work described in [11] that utilizes mixup for improving sentence classification which is among the few works, besides ours, studying the effects of mixup in the NLP domain.


Both the confidence (captured by the winning score) as well as accuracy start out low and gradually increase as the network learns.

what is interestingand concerning -is that the confidence always leads accuracy in the later stages of training; accuracy saturates while confidence continues to increase resulting in a very sharply peaked distribution of winning scores and an overconfident model.


This definition of SB-disentangled representations does not make any assumptions on what form the group action should take when acting on the relevant disentangled vector subspace.

many subsequent tasks may benefit from a SB-disentangled representation where the group actions transform their corresponding disentangled subspace linearly.


One may utilize the measure to quantify better models [29] or directly use it as an objective function to optimize [37,1].

measuring the discrepancy of GANs (and cGANs) is another challenging problem, since the data distribution remains unknown and the distribution GANs learn is implicit [35].


We observe that the random samples (without rejection) often contain low-quality samples with uncertain and/or wrong classes.

samples with high marginal values improve the quality (or vividness), and samples with high conditional values improve the class accuracy (but loses the diversity).


While both of the two approaches above, (a) and (b), can be used in a semi-supervised setting, cGANs of (b) provide a more natural framework for using both labeled and unlabeled data; 3 one can use the unlabeled data to learn p(x), and the labeled data to learn both p(x) and p(c|x).

we focus on evaluating the second type of architectures, e.g., the auxiliary classifier GAN (ACGAN) [39].


The generative adversarial network (GAN) [15] is arguably the most successful generative model in recent years, which have shown a remarkable progress across a broad range of applications, e.g., image synthesis [5,21,40], data augmentation [49,18] and style transfer [58,10,34].

as its advanced variant, the conditional GANs (cGANs) [31] have gained a considerable attention due to its class-wise controllability [9,42,10] and superior quality for complex generation tasks [39,33,5].


However, because the labels are biased, we must immediately assume that the corresponding accuracy measurements are also biased.

we are crucially interested in evaluating accuracy on the unbiased ground truth labels, which are devoid of any such label bias.


It may be the case that label-bias is so obvious to most authors that it does not even occur to them to mention it; howbeit, the conspicuous absence of label-bias from papers on fairness perniciously pervades real-world discussions underlying the decisions about how to balance the trade-off between fairness and accuracy.

we believe this finding to be of practical importance and worthy of highlighting.


We focused on demographic parity in this paper, but the ideas emphasized in this work, especially label bias, have potentially serious implications for other notions of fairness that go beyond even their relationship with accuracy.

recent ways of assessing fairness such as disparate mistreatment, equal odds and equal oppurtunity involve error rates as measured against labeled data.


In this typical evaluation setting, if we train a set of classifiers that differ only in the extent to which their training objective functions enforce fairness, and then record their respective fairness and accuracy scores on a test set with such label bias, we see that increased fairness is achieved at the expense of accuracy (Figure 3a).

because the labels are biased, we must immediately assume that the corresponding accuracy measurements are also biased.


While uncommon, some papers do indeed mention label-bias, including recent work that considers the largely hypothetical case: if we have access to unbiased labels, then we can propose a better way of evaluating fairness with "disparate mistreatment" [20].

their emphasis is on new fairness metrics, not on its tradeoff with accuracy.


Critically, SSE progresses by stochastically transitioning between embeddings as opposed to a more brute-force regularization such as graph-based Laplacian regularization and ridge regularization.

sSE integrates seamlessly with existing stochastic optimization methods and the resulting regularization is data-driven.


In stochastic optimization, we can replace the loss gradient for one movie's embedding with the other similar movie's embedding, and this will not significantly bias the gradient if the prior belief is accurate.

if this exchange is stochastic, then it will act to smooth the gradient steps in the long run, thus regularizing the gradient updates.


We summarize our results in Table 5 and find that SSE-SE helps improving accuracy and BLEU scores on both dev and test sets in 10 out of 11 years from 2008 to 2018.

on the last 5 years' test sets from 2014 to 2018, the transformer model with SSE-SE improves BLEU scores by 0.92 on average when compared to the baseline model without SSE-SE.


in [3], the authors masked 15% of words and 10% of the time replaced the [mask] token with a random token.


We continue to pre-train Google pre-trained BERT model on our crawled IMDB movie reviews with and without SSE-SE and compare downstream tasks performances.


Our analysis then consists of two main parts where each part bounds one of the terms in the RHS above.

if we only want to achieve zero inversions with high probability, how large does m need to be?


we first prove that given the partial comparisons seen so far, we can obtain a relatively good estimation to the rank of the arrived element, and then in the second part, we show that we can typically find an unassigned position in the close proximity of this estimated rank to assign to it.


Our algorithm maintains an ordering of the elements seen so far.

the remainder of the paper is organized as follows.


Two important applications of this problem are search engines for document retrieval [L + 09, RJ05, LXQ + 07, CXL + 06, XL07] and collaborative filtering approaches to recommender systems [SLH10, SKB + 12, LY08, WRdVR08].

our main result is a matching upper and lower bound for the secretary ranking problem.


Consider a university department that would like to assign the best scholarships available to the best students.

scholarships arrive one at a time and the school needs to decide which student is assigned that scholarship knowing only the relative quality of the scholarships arrived so far.


when a new element arrives, the algorithm can find the position of this new element in the current ordering by binary search.


The total number of inversions can be approximated within a factor of 2 by the Spearman's footrule.

we can write the cost of Algorithm 1 (up to a factor 2) as follows: This basically breaks the cost of the algorithm in two parts: one is the cost incurred by the estimation step and the other one is the cost of the assignment step.


In later time steps, we observe a large number of comparisons and using the randomness of elements arrival, the true rank of the elements can be estimated well.

the main difficulty is that at these time steps many of the positions have already been assigned to some element arrived earlier and are hence not available.


Differently, HER opens up a new way to learn more from failures, assigning hindsight credit to primal experiences.

it is limited by only applicable when combined with off-policy algorithms [3].


A straightforward way to utilize self-imitate learning is to adopt the inverse dynamics.

in most cases the actions stored in inverse dynamics are irrelevant to the goals.


PCHID greatly improves the learning efficiency of PPO.

in general the efficiency of finding a transition sequence that is i-step solvable decreases as i increases.


In practice the k-step solvability can be treated as an evolving concept that changes gradually as the learning goes.

at the beginning, an agent can only walk with small paces as it has learned from experiences collected by random movements.


However, the problems and models they study are very different.

they assume that a decision-maker shows a subset of products to a user.


(2) Implementation: Our theoretical analysis does not require that each matrix estimation phase in REAL-Bandit is solved to completion.

rEAL-Bandit does not need finding a global minimum of the relevant estimator's optimization (penalized maximum likelihood) problem.


However, existing matrix estimation results provide a bound on the estimation error of the whole matrix, which would be a crude upper bound for the estimation error of each single row.

rEAL-Bandit includes a subroutine (called Row Enhancement) that refines the estimates in order to establish stronger row-wise guarantees that may be of independent interest (see Â§3 for details).


The aim of this paper is to reduce this dependence under a low-rank assumption on the k × d matrix of parameters of the reward functions.

each of the k reward functions is represented by a d-vector of coefficients (one coefficient per covariate) that is a row of the parameter matrix.


For example, the model in [4] assumed that the conditional intensity function exponentially decreases or increases with the elapsed time from the most recent event until the next event.

using such an assumption can limit the expressive ability of the model and potentially deteriorate the predictive skill if the employed assumption is incorrect.


The conditional intensity function is then modeled as a function of the hidden state of the RNN.

the RNN based models outperform the parametric models in prediction performance.


RNN based models usually assume a specific functional form for the time course of the intensity function of a point process (e.g., exponentially decreasing or increasing with the time since the most recent event).

such an assumption can restrict the expressive power of the model.


In many cases, the occurrences of the event are correlated to each other in a certain manner, and information on future events may be extracted from the information of past events.

the appropriate modeling of the dependence of the event occurrence on the history of past events is important for understanding the system and predicting future events.


In contrast to these studies, the log-likelihood function of our general model can be exactly evaluated without any numerical approximations because the integral of the hazard function is modeled by a feedforward neural network in our approach.

a more accurate estimate can be efficiently obtained by our approach.


To evaluate the predictive performances based on a metric other than the log-likelihood, we also carry out the time prediction experiments.

we use the median of the predictive distri-bution to predict the timing of the coming event and evaluate the prediction by the mean absolute error.


However, computing the assignments requires access to the optimal representation, which is not available.

we propose an optimization scheme that alternates between updating the representation by minimizing our proposed loss given the current assignments of points to ground-truth representatives and updating the assignments given the current representation.


Moreover, the nature of our conditions, as opposed to asymptotic results, allowed to design the loss in (7).

we use Euclidean distance for pairwise dissimilarities.


Notice that the F1 score is relatively stable with respect to the hyperparameter change.

changing λ from 0.001 to 0.1 the performance over the dataset changes by at most 1.2% in F1 score, while changing ρinter and ρintra from 0.01 to 10, the performance changes by at most 0.6% and 2.1%, respectively.


This has the following effect: when a cluster has a small number of points, it could be considered as under-sampled, hence, to generalize better to test data, we need to have a better separation from other clusters, i.e., larger margin.

for a cluster with a large number of points, the margin could be smaller as the chance of changing the distances between and within clusters by adding more samples to it would be low.


To address the problem, [50,52] try to learn a combination of different criteria, i.e., weights of a mixture of submodular functions.

chengguang Xu would like to thank Dat Huynh and Zwe Naing for their help and advice with some of the implementations during his research assistantship at MCADS lab, which resulted in this work.


We draw data from two distribution families, the Normal family with mean a and variance b 2 , and the log-Normal family, with log-mean a log and log-variance b 2 log , under multiple parameter settings.

we consider the impact of shifting the distribution location over [âˆ’40.0, 40.0], with small and large variance settings.


While this approach lets us deal with unbounded losses, naturally the statistical error guarantees are only as good as the confidence intervals available for the empirical mean deviations.

strong assumptions on all of the moments of the loss are essentially unavoidable using the traditional tools espoused by Bégin et al. [3], which means the “heavy-tailed” regime cannot be handled, where all we assume is that a few higher-order moments are finite (say finite variance and/or finite kurtosis).


For most gradient-based NAS methods, an over-parameterized super-network is constructed firstly with all candidates paths included and one superior path is selected on each edge with the other candidates removed.

signals in network often contain numerous channels during forward propagation, which means that a path is not the minimum separable structure unit in network and path-level search methods [3,24,22] limit the granularity of architecture search.


Recently, Neural Architecture Search has achieved great success in large-scale image classification.

there have been limited works focusing on architecture search for object detection, mainly because the costly ImageNet pretraining is always required for detectors.


Qualitatively, we found that the estimates of feature importance provided by CXPlain were more focused on the subjectively more important semantic regions of the sample images from both MNIST and ImageNet (Figures 4 and 5; more in Appendix D).

providing fast and accurate estimates of feature importance for high-dimensional data, and quantifying the uncertainty of such estimates remain open challenges.


Absent any prior knowledge about the structure of the input data, multilayer perceptrons (MLPs) are likely a sensible default choice.

since architectures that exploit the spatial or temporal structure of input data have been shown to be efficacious, we reason that, depending on the data modality of the input features of the model to be explained, special-purpose architectures, such as convolutional neural networks [42] for images and attentive neural networks for texts [43], could perform better than MLPs.