In one-shot imitation learning, the agent must perform a task conditioned on one reference example of the task.

unlike our method, these approach does not easily allow for sequencing skills into longer horizon tasks or composing tasks via arithmetic operations on the latent representation.


The success of machine learning (ML) during the past decade has relied heavily on the rapid growth of computational power, new techniques training larger and more representative neural networks, and critically, the availability of enormous amounts of annotated data.

new challenges have arisen with the move from cloud computing to edge computing and Internet of Things (IoT), and demands for customized models and local data privacy are increasing, which raise the question: how can a powerful model be trained for a specific user using only a limited number of local data?


classification that aims to distinguish among cat, dog and others.

in each step t + 1, each class y aggregates prototypes from its neighbors (parents and children) by multi-head attention, and chooses between the aggregated message or the message from itself by a gate g to update its prototype.


Hence, tasks can be related through the paths on the graph that links their nodes even when they share few output classes.

we use linear transformation for g(·) and h(·).


Experiments show that our algorithm achieves a strong performance over several image classification datasets.

it obtains an error rate of 1.6% for CIFAR-10, 23.9% for ImageNet under mobile settings, and achieves state-of-the-art results on three additional datasets.


Unlike conventional deep networks that come with a fixed number L of layers, the runtime of DEQ depends strongly on the number of Broyden steps to reach the equilibrium.

it's challenging to fairly compare the runtimes of implicit-depth models like DEQ with those of corresponding weight-tied deep networks (e.g., using higher depth necessarily takes longer to run).


Such weight-tying is generally considered to come with four major benefits: 1) it acts as a form of regularization that stabilizes training and supports generalization; 2) it significantly reduces the model size; 3) it is trivial to show that any deep network can be represented by a weight-tied deep network of equal depth and only a linear increase in width (see Appendix C); and 4) the network can be unrolled to any depth, typically with improved feature abstractions as depth increases [8,18].

in practice almost all such models (and deep nets in general) are stacked, trained and evaluated by unrolling a pre-determined, fixed number of layers.


Importantly, we show that DEQ can directly differentiate through the fixed point equations via implicit differentation, which does not require storing any intermediate activation values.

we can backpropagate through the infinite-depth network while using only constant memory, equivalent to a single layer's activations.


Unlike a conventional network where the output is the activations from the L th layer, the output of a DEQ is the equilibrium point itself.

the forward evaluation could be any procedure that solves for this equilibrium point.


Among the many applications of deep networks, sequence modeling has witnessed continuous advances in model architectures.

while recurrent networks have long been the dominant model for sequences [21,26,14,34], deep feedforward architectures based on temporal convolutions [49,47,7] and self-attention [48,16,13] have (re-)emerged to claim superior performance on a variety of sequence prediction tasks.


It is noted that the optimizationbased methods need fine-tuning on the target task, making the classification time consuming.

our method requires no model updating solves the tasks in an feed-forward manner, which is much faster and simpler than above methods and has better results.


A straightforward approach is fine-tuning a pretrained model using the few labeled samples from the unseen classes.


These approaches generate the parameters of the feature extractor based on the support set and extract the query features adaptively.

in this way, we can progressively enrich the class features to be more representative and robust.


There are m machines, each observing n i.i.d sample functions from P .

in this work, we present an algorithm with O log(mn) bits per message, which we call Multi-Resolution Estimator for Convex landscapes and log mn bits communication budget (MRE-C-log) algorithm.


The expected loss of this modified algorithm can be shown to still matches the existing lower bounds up to logarithmic factors.

in the n = 1 regime, the expected error of MRE-C-log algorithm goes to zero as the number of machines increases, while the expected errors of the previously available estimators remain lower bounded by a constant.


The observations are also in line with the expected error bounds we give in this paper and those previously available.


Conjugate gradients and related MVM-based algorithms [9,13,40] have been used in certain settings throughout the GP literature.

although exact GPs take longer to train, we find that their speed is comparable to approximate methods at test time.


There may be some regimes where other approximate methods or combinations of methods outperform these two approximations.

for SGPR, we perform 100 iterations of Adam with a learning rate of 0.1.


Many approximate methods have been introduced to improve scalability, relying on mixture-of-experts models [7], inducing points [12,37,39,42], random feature expansions [20,30,47], or stochastic variational optimization [2,16,17,36,46].

due to the historical intractability of training exact GPs on large datasets, it has been an open question how approximate methods compare to an exact approach when much more data is available.


Although forming K XX requires O(n 2 ) memory, the result of the MVM K XX v requires only O(n) memory.

we reduce the memory requirement to O(n) by computing K XX v in separate constant-sized pieces.


While each iteration of CG requires computing each kernel matrix partition from scratch, the preconditioner is computed once before any iterations of CG are performed.

it can be efficient to increase the size of the preconditioner to an extent if it reduces the number of CG iterations.


For n≥100,000, we must use kernel partitioning as discussed above, which significantly increases the training time for exact GPs.

if the necessary computational resources are available, these experiments show that it may be preferable to train an exact GP to make more accurate predictions in exchange for longer training times.


A natural question to ask is whether we can consider our approach "exact" in light of the fact that CG perform solves only up to a pre-specified error tolerance.

unlike the approximate methods presented in this paper, the difference between a CG-based model and a theoretical model with "perfect" solves can be precisely controlled by this error tolerance.


Gaussian processes (GPs) are flexible non-parametric models, with a capacity that grows with the available data.

computational constraints with standard inference procedures have limited exact GPs to problems with fewer than about ten thousand training points, necessitating approximations for larger datasets.


The proof of Theorem 2.7 closely follows that of [43,Theorem 9].

the size of the ambiguity set and the regularization parameter are selected using stratified 5-fold cross validation.


The generic perturbed-iterate framework is given in Section 4.

aSYNCADA as given in Algorithm 1 maintains the exact global clock t. this option may not be desirable (or available) in certain asynchronous computing scenarios.


We further use this framework to derive the first asynchronous online and stochastic optimization algorithm with non-box constraints that uses non-Euclidean regularizers.

we present and analyze HEDGEHOG, the parallel asynchronous variant of the EG algorithm, also known as Hedge in online linear optimization [34,14], 2 In this paper, we do not further consider BCD-based methods, for two main reasons: a) in general, a BCD update may unnecessarily slow down the convergence of the algorithm by focusing only on a single coordinate of the gradient information, especially in the sparse-data problems we consider in this paper (see, e.g., Pedregosa et al.


The exact way this is done depends on the specific step-size schedule.

we consider two situations: First, when the step-size is either constant or a simple function of t (or tt in case of ASYNCADA(â‡¢)), and second, when diagonal ADA-GRAD step-sizes are used.


Of particular interest are asynchronous lock-free algorithms, where Recht et al. [30] demostrated first that linear speed-ups are possible: They showed that if ⌧ processes run stochastic gradient descent (SGD) and apply their updates to the same shared iterate without locking, then the overall algorithm (called Hogwild!) converges after the same amount of work as serial SGD, up to a multiplicative factor that increases with the number of concurrent processes and decreases with the sparsity of the problem.

if the problem is sparse enough, this penalty can be considered a constant, and the algorithm achieves linear speed-up.


Instead, a somewhat weaker goal is to ensure that the price of parallelism is at most a constant factor: that is, the parallel variant needs at most constant-times more updates (or work).

using t parallel process requires a wall-clock running time that is only O(1/t)-times that of the serial variant.


When measuring the quality of a large population of generated images, these underrepresented regions have little impact as it is unlikely that too many generated samples land there -even though the hyperspheres may be large, they are sparsely located and cover a small volume of space in total.

we see that FID favors configurations with high recall (A, F) over the ones with high precision (B, C), and the same is also true for the individual snapshots.


In order to approximate these quantities directly, we construct adaptive-resolution finite approximations to the real and generated manifolds that are able to answer binary membership queries: "does sample x lie in the support of distribution P ?".

they make effects in the "null space" of FID clearly visible.


As shown in Figure 3, each input node ν ∈ V is represented by concatenating its corresponding semantic embedding a Î½ with a random Gaussian sample zν .

early approaches address it as an embedding problem [1,2,6,8,16,24,32,36,39,42,43,48].


In hyperspherical prototype networks, classification and regression are optimized in the same manner based on a cosine similarity loss.

both tasks can be optimized not only with the same base network, as is common in multi-task learning [6], but even in the same output space.


We found that a direct alignment impedes separation, since word embedding representations do not fully incorporate a notion of separation.

we use a ranking-based loss function, which incorporates similarity order instead of direct similarities.


As illustrated in Figure 1, a comparison of optimization landscapes without and with preconditioning shows that preconditioning symmetrifies the optimization landscapes and eliminates spurious local minimizers.

it makes the problem more amendable for optimization.


As demonstrated in Figure 6, optimizing Huber-loss using vanilla RGD can near perfectly recover both the underlying Bessel PSF and point-sources, producing accurate high resolution image.

optimizing l4 -loss [21] fails to recover the PSF, resulting in some aliasing effects of the recovered image.


First, GMF-Q converges after about 10 outer iterations; secondly, as the number of inner iterations increases, the error decreases (Figure 2); and finally, the convergence is robust with respect to both the change of number of states and the initial population distribution (Figure 3).

the Naive algorithm does not converge even with 10000 inner iterations, and the joint distribution L t keeps fluctuating (Figure 4).


As |S| and |A| increase, GMF-Q is robust with respect to this increase of dimensionality, while both MF-Q and IL clearly suffer from the increase of the dimensionality with decreased convergence rate and accuracy.

gMF-Q is more scalable than IL and MF-Q, when the system is complex and the number of players N is large.


Even Density Functional Theory (DFT) [Hohenberg and Kohn, 1964], a widely used approximation to the equations of quantum mechanics, has trouble scaling to more than a few hundred atoms.

the majority of practical work in molecular dynamics today falls back on fundamentally classical models, where the atoms are essentially treated as solid balls and the forces between them are given by pre-defined formulae called atomic force fields or empirical potentials, such as the CHARMM family of models [Brooks et al., 1983[Brooks et al., , 2009.


Every neuron n i corresponds to some subset S i of the atoms.

each input neuron corresponds to a single atom.


In principle, quantum mechanics provides a perfect description of the forces governing the behavior of atoms, molecules and crystalline materials such as metals.

for systems larger than a few dozen atoms, solving the Schrödinger equation explicitly at every timestep is not a feasible proposition on present day computers.


Our Cormorant neural network has invariance to rotations baked into its architecture in a way that is similar to the physical equations of the previous section: the internal activations are all spherical tensors, which are then combined at the top of the network in such a way as to guarantee that the final output is a scalar (i.e., is invariant).

to allow the network to learn interactions that are more complicated than classical interatomic forces, we allow each neuron to output not just a single spherical tensor, but a combination of spherical tensors of different orders.


However it does have a well studied decomposition into irreducibles, called the Clebsch-Gordan decomposition: ) is an 'th order spherical tensor.

we note that similar Clebsch-Gordan nonlinearities were used in , and that the Clebsch-Gordan product is also an essential part of Tensor Field Networks [Thomas et al., 2018].


Given T , a bottom-up path is any path connecting a node with one of its ancestors in T .

we call a backbone path any bottom up path having maximal length.


A distinctive feature of these algorithms is that they are rather ad hoc in the way they deal with the structure of our problem.

they cannot be seen as finding the query that splits the version space as evenly as possible, a common approach in many active learning papers (e.g., [12,25,15,16,27,24], and references therein).


Here, for a given prior P(c * ), the goal of learning is to identify C * either exactly (when Î» = 0) or approximately (when Î» > 0), while bounding the expected number of queries (x it , x jt ) made to the ground-truth matrix Î£, the expectation being over the noise, and possibly over P(c * ).

if C is the clustering produced by the algorithm after it stops, we would like to prove upper bounds on E[d H (Î£ * , C)], as related to the number of active learning rounds, as well as to the properties of the prior distribution.


Extending this result to deep ReLU networks requires understanding their behaviour under composition.

we have ridge functions which vanish on some half space, i.e. colloquially speaking each neuron may “discard half the information” it receives from the previous layer.


Note, however, that each realization has many different, possibly degenerate, parametrizations.

a local minimum in the parametrization space needs not correspond to a local minimum in the realization space.


We proceed by demonstrating how inverse stability opens up new perspectives on the optimization problem which arises in neural network training.

consider a loss function on the space of continuous functions.


Moreover, local minima of (8) in the parametrization space must correspond to local minima of ( 9) in the realization space if and only if we have inverse stability.

furthermore, we show that by optimizing over such restricted sets, it is still possible to learn any function which can be learned by optimization over unrestricted sets.


Inverse stability is also prevented by redundant directions as the following example illustrates.


The design of our generative model is inspired by the way humans write articles: Each word should be semantically consistent with not only its surrounding words, but also the entire paragraph/document.

we assume text generation is a two-step process: A center word is first generated according to the semantics of the paragraph, and then the surrounding words are generated based on the center word's semantics 2 .


Since the optimization problem is constrained on the unit sphere, the Euclidean space optimization methods such as SGD cannot be used to optimize our objective, because the Euclidean gradient provides the update direction in a non-curvature space, while the parameters in our model must be updated on a surface with constant positive curvature.

we need to base our embedding training problem on Riemannian optimization.


This is probably because negative samples are randomly sampled from the vocabulary and most of them are semantically irrelevant with the center word.

their ideal embeddings should be orthogonal to the center word's embedding.


Importantly, previous NAS approaches have focused on searching for E p directly by using |E| as one of the optimization objectives [14].


Let H be the function class: In contrast, a privacy definition is said to compose sequentially if the privacy properties of algorithms that satisfy it degrade gracefully as the same dataset is used in multiple private releases.

renyi differential privacy is said to satisfy sequential additive composition -if multiple private algorithms are used on the same dataset, then their privacy parameters add up.


Let f be a lower semi-continuous convex function such that f(1) = 0, and let P and Q be two distributions over a probability space (Ω, Σ) such that P is absolutely continuous with respect to Q.

we can show that for any H and for any Î“, (H, Î“)-capacity bounded differential privacy is preserved after post-processing if certain conditions about the function classes hold: Theorem 1.


Let H 1 , H 2 be two function classes that are convex and translation invariant.

motivated by these concerns, a recent line of work in machine learning investigates properties that algorithms and methods should possess so that they can automatically guarantee generalization [27,10,6,29].


We note that there also exists a conceptually simpler smoothing technique based indirect algorithm, which prefixes the tolerance of Îµ (Appendix D).

our goal is to find a direct algorithm which does note require prefixing the tolerance at Îµ.


Remark 2: Unlike standard AGD for h(y), which only updates y k in the outer-loop, DIAG's outerstep updates both x k and y k thus allowing us to better track the primal-dual gap.

dIAG's dependence on the condition number L/Ïƒ seems sub-optimal and can perhaps be improved if we do not compute Imp-STEP nearly optimally allowing for inexact updates; we leave further investigation into improved dependence on the condition number for future work.


Now, first order stationary point of a non-smooth nonconvex function is welldefined, i.e., x * is a first order stationary point (FOSP) of a function f (x) if, 0 âˆˆ âˆ‚f (x * ) (see Definition 3).

unlike smooth functions, it is nontrivial to define an approximate FOSP.


Recently, [12] showed linear convergence of gradient descent ascent for strongly-convex-concave problem with bilinear coupling when A has full row rank.

it has remained an open question if the fast rate of O(1/k 2 ) can be achieved for general strongly-convex-concave minimax problems.


Additionally, we also examine another quantity as an alternative to distance from initialization: the 2 diameter of the parameter space explored by SGD.

for a fixed initialization and data distribution, we consider the set of all parameters learned by SGD across all draws of a dataset of size m; we then consider the diameter of the smallest ball enclosing this set.


This called for a "rethinking" of conventional, algorithm-independent techniques to explain generalization.

it was argued that learning-theoretic approaches must be reformed by identifying and incorporating the implicit bias/regularization of stochastic gradient descent (SGD) [6,35,30].


For our argument, we will rely on a classifier trained empirically, in contrast to our linear examples where we rely on an analytically derived expression for the learned classifier.

this section illustrates that the effects we modeled theoretically in the linear classifier are indeed reflected in typical training settings, even though here it is difficult to precisely analyze the learning process.


Traditional wisdom is that uniform convergence bounds are a bad choice for complex classifiers like k-nearest neighbors because these hypotheses classes have infinite VC-dimension (which motivated the need for stability based generalization bounds in these cases [33,5]).

consider a scenario where the algorithm generalizes well i.e., for every training set S, h S has zero error on S and has small test error.


Unfortunately, uniform convergence runs into trouble while dealing with such bad datasets.

as we can see from the above definition, uniform convergence demands that |L D (h S ) âˆ’ LS (h S )| be small on all datasets in S Î´ , which excludes a Î´ fraction of the datasets.


this sort of an argument against uniform convergence may still leave one with the faint hope that, by aggressively pruning the hypothesis class (depending on the algorithm and the data distribution), one can achieve meaningful uniform convergence.


Inspired by our linear model in Section 3.1, we conjecture that the weights w learned on a dataset S can be expressed as w 1 +w 2 , where w T 1 Ï†(x) dominates the output on most test inputs and induces a simple decision boundary.

it may be possible to apply uniform convergence on the function w T 1 Ï†(x) to obtain a small generalization bound.


Motivated by the seemingly insurmountable hurdles towards developing bounds satisfying all the above five necessary criteria, we take a step back and examine how the underlying technique of uniform convergence may itself be inherently limited in the overparameterized regime.

we present examples of overparameterized linear classifiers and neural networks trained by GD (or SGD) where uniform convergence can provably fail to explain generalization.


To make this more concrete, for illustration, consider the high-dimensional linear model that sufficiently wide networks have been shown to converge to [17].

roughly, these networks can be written as h(x) = w T Ï†(x) where Ï†(x) is a rich high-dimensional representation of x computed from many random features (chosen independent of training data).


It is important to recognize that the second objective (exploration) is important only in so far as it facilitates the first (maximizing reward); there is no intrinsic value in reducing uncertainty.

exploration should be targeted or application specific; it should not excite the system arbitrarily, but rather in such a way that the information gathered is useful for achieving the goal.


Concerning worst-case performance, nom attains the lowest cost at the initial epoch, as it does no explicit exploration.

methods that incorporate exploration achieve better performance at subsequent epochs (and in terms of total cost) due to greater reduction in uncertainty.


In this section we present: (i) some qualitative insights of our meta-model by showing how it is able to coherently represent a sets of tasks in its latent space, (ii) an illustration of why PROFET helps to obtain statistically meaningful results and (iii) a comparison of various methods from the literature on our new benchmark suite.

we show results for the following state-of-the-art BO methods as well as two popular evolutionary algorithms: â€¢ BO with Gaussian processes (BO-GP) [22].


Notice that dropping a coordinate j ∈ S from the input samples makes the labels of the resulting training set totally independent from the input.

dropping j ∈ S results in a training set that is still labeled by a noisy parity.


Nyström approximation: Unlike the QFF approximation where the basis functions (cosine and sine) do not depend on the data, the basis functions used by the Nyström method are data dependent

we took the adjusted closing price of 29 stocks from January 4th, 2016 to April 10th, 2019.


In this paper, we assume smoothness for f induced by the structure of a kernel on X .

we make the standard assumption of a p.s.d. kernel k : X × X → R such that k(x, x) ≤ 1 for all x ∈ X , and f being an element of the reproducing kernel Hilbert space (RKHS) Hk(X ) of smooth real valued functions on X .


The proof is based on constructing a finite subset of "difficult" functions in H k (X ).

we choose f as a uniformly sampled function from a finite set {f1, . . . , fM}, where each fj is obtained by shifting a common function g ∈ Hk(Rd) by a different amount such that each of these has a unique maximum, and then cropping to X = [0, 1].


We clearly see that for the standard VAE, the variance is low where we have data and arbitrary away from data.

more precisely, we consider weather data from over 130 years.


Note that this is a more informative sample, as all observations in the sample are likely to influence the same subset of parameters in Î¸, effectively increasing the degrees of freedom 4 , hence the quality of variance estimation.

if the variance network is sufficiently expressive, our Monte Carlo gradients under this sampling scheme are of smaller variation and more sparse.


On these datasets our model is the best-performing neural network.

the local log-likelihood is approximately 0 regardless of the observed value y(x 0 ), which should be interpreted as a large entropy of y(x 0 ).


GPs are robust in settings with a low amount of data, and can model a rich class of functions with few hyperparameters.

gPs are computationally intractable for large amounts of data and limited by the expressiveness of a chosen kernel.


Experimentally, we have demonstrated that proposed methods are complementary and provide significant improvements over state-of-the-art.

on benchmark data we have shown that our method improves upon the test set log-likelihood without improving the RMSE, which demonstrate that the uncertainty is a significant improvement over current methods.


Closest to our setting is the recent work of Lee et al.  [32], which uses implicit differentiation for quadratic programs in a final SVM layer.

our formulation allows for adapting the full network for generic objectives (beyond hinge-loss), thereby allowing for wider applications.


While the broad idea of implicit differentiation is well known, it has not been empirically demonstrated in the past for learning more than a few parameters (e.g., hyperparameters), or highly structured settings such as quadratic programs [4].

our method meta-trains deep neural networks with thousands of parameters.


Let Note that the derivative (Jacobian) depends only on the final result of the algorithm, and not the path taken by the algorithm.

in principle any approach of algorithm can be used to compute Alg i (Î¸), thereby decoupling meta-gradient computation from choice of inner level optimizer.


The computational cost of 1 CG step is comparable to 1 inner GD step with the MAML algorithm, since both require 1 hessianvector product (see section C for discussion).

the computational cost as well as memory of iMAML with 100 inner GD steps is significantly smaller than MAML with 100 GD steps.


The memory use is also independent of the number of CG iterations, since the intermediate computations need not be stored in memory.

memory for MAML grows linearly in grad steps, reaching the capacity of a 12 GB GPU in approximately 16 steps.


This straightforward application of the Laplace mechanism was mentioned at least by Dwork and Smith [2009] and has been widely applied since by several authors [e.g. Zhang et al., 2016, Foulds et al., 2016, Park et al., 2016, Honkela et al., 2018, Bernstein and Sheldon, 2018.

Foulds et al. [2016]  show that the sufficient statistics perturbation is more efficient than OPS for models where both are applicable.


The mean absolute difference between the Hellinger squared estimates is It is worth pointing out that although the above statistical properties of the Hellinger distance make it well-suited as a variational objective, it does pose computational issues during optimization.

to avoid numerically unstable gradient estimation, one must transform Hellingerbased objectives such as Eq. (5).


The parameters of the intrinsic reward functions are updated to maximize the standard accumulated discounted team return from the environment.

the objective of the entire procedure is consistent with that of the original MARL problem.


Our approach builds a connection between reward shaping and critic learning.

we propose to learn each agent a parameterized individual intrinsic reward function by maximizing a centralized critic.


In order to figure out what has been learned in the intrinsic reward function, we propose to explicitly visualize these rewards.

we plot the learned intrinsic reward of each agent at each time step in a complete trajectory during testing.


We compare LIIR with a number of state-of-the-art MARL methods on battle games in StarCraft II.

4(b) and (c), at time step 6, all the three agents focus fire on one of the enemy Marine, while agent 1 has the lowest HP; after that, agent 1 still keeps firing instead of running away from the enemies and the intrinsic reward function predicts a low r in 1 , indicating that u 1 = attack is not a good action at that time; finally, agent 1 dies at time step 9 and the corresponding intrinsic reward is very low.


In this paper, we propose to merge the two directions and learn each agent an intrinsic reward function which diversely stimulates the agents at each time step.

the intrinsic reward for a specific agent will be involved in computing a distinct proxy critic for the agent to direct the updating of its individual policy.


Inspired by the concept, we propose to introduce the intrinsic reward design into multi-agent systems to distinguish the contributions of the agents when the environment only returns a team reward.

we learn each agent a parameterized intrinsic reward function that outputs an intrinsic reward for that agent at each time step to induce diversified behaviors.


such memories are likely mediated by horizontal or recurrent feedback mechanisms or both.


HPNet in Frame-Frame scheme performs better than PredNet, suggesting that a feature hierarchy is better than a prediction error hierarchy for long-range prediction.

hPNet in Frame-Frame scheme does not perform as well as PredRNN++ on long range video prediction.


Visual neurons' receptive fields are spatiotemporal 3D kernels, rather than 2D.

we use spatiotemporal block of the input video sequence as input to the neuron in our neural network.


It has been demonstrated to be effective in explaining the predictive suppression phenomena in the inferotemporal cortex [63].

predNet only builds a hierarchical representation of errors, where the model of a higher layer learns to predict the prediction errors of the lower layer.


For HPNet, semantic clustering and decoding accuracy improves progressively as one moves up the hierarchy, from 26% in the first layer (module) to 63% for the top module of the 4-module network.

the better performance of HPNet in long range video predictions might be attributed to its ability to learn semantically meaningful hierarchical spatiotemporal feature representations and movement to movement relationships (see also [51]).


We also performed the same experiment on PredNet and PredRNN++ and found that their corresponding R neurons would also exhibit predictive suppression effect to a certain extent, but much smaller in magnitude.

we can only claim that the neurophysiological finding is consistent with this genre of self-supervised predictive learning models, though HPnet might be a better approximation.


These memories, together with the generic statistical priors encoded in receptive fields and connectivity of neurons, serve as internal models of the world for predicting incoming visual experiences.

it is not clear why early visual cortex needs to be involved in the encoding of spatiotemporal memories and what computational roles it might play.


What is novel and unanticipated about our finding is that V2 neurons' receptive fields are very local, yet this "memory effect" depends on the presence of the global context of the entire movie -reducing the movie aperture from 8 o to 3 o , barely larger than the receptive fields of the individual neurons, would annihilate the predictive suppression effects.

this predictive suppression effects were not due to adaptation of local receptive field features, but reflected a sensitivity to the memory of the global context of the movies or movement.


We ask for algorithms that enforce well-studied statistical fairness constraints across two protected populations (we focus on the "equal opportunity" constraint of [18], which enforces equalized false positive rates or false negative rates, but our techniques also apply to other statistical fairness constraints like "statistical parity" [12]).

we ask for algorithms that satisfy these constraints (with respect to the unknown underlying distribution) at every round of the learning procedure.


train a model on one half and predict on the other and vice versa).

hence, double robustness is used mostly as a variance reduction technique.


Based on this result, we manage to achieve variance-based regret bounds without the need for variance or moment penalization [15,20,9] used in prior work and which can render a computationally tractable policy learning problem, non-convex.

our method provides a computationally efficient alternative to the variance penalization when the original ERM problem is convex.


We empirically evaluate our framework on the personalized pricing application with synthetic data.

we use simulations to assess our estimator's ability to evaluate and optimize personalized pricing functions.


We note that we present our theoretical results for the simpler case where the nuisance estimates are trained on a separate split of the data.

our results qualitatively extend to the case where we use the cross-fitting idea of [5] (i.e. train a model on one half and predict on the other and vice versa).


However, we show that at least asymptotically and without further assumptions on the functions θ0(z) and Σ0(z), this cannot be the case.

we show that our estimator achieves what is known as the semi-parametric efficient variance limit for our setting.


This sequence of operations is illustrated in Figure 2c.

we used a fixed vocabulary size of 32k, sequence length 1024 and batch size 32.


GPipe also introduces low communication overhead, given that we only need to pass activation tensors at the partition boundaries between accelerators.

we can achieve efficient scaling performance even on accelerators without high-speed interconnects.


Figure 2c assumes partitions are evenly balanced.

memory requirements and computation flops at different layers are often quite imbalanced.


We apply synchronous mini-batch gradient descent for training, where gradients are accumulated across all micro-batches in a mini-batch and applied at the end of a mini-batch.

gradient updates using GPipe are consistent regardless of the number of partitions, allowing researchers to easily train increasingly large models by deploying more accelerators.


On the one hand, controlling Term I, is more or less standard and closely follows previous such bounds on UCB-type algorithms (e.g., Abbasi-Yadkori et al. (2011));  see Appendix B.2 for details.

controlling Term II, which we call the regret of safety is more delicate.


As such, it allows us to quantify tradeoffs between learning the safe set and minimizing the regret.

we propose Safe-LUCB which is comprised of two phases: (i) a pure-exploration phase that speeds up learning the safe set; (ii) a safe exploration-exploitation phase that optimizes  minimizing the regret.


For example, when managing demand to minimize costs in a power system, it is required that the operational constraints of the power grid are not violated in response to our actions (these can be formulated as linear constraints that depend on the demand).

for such systems, it becomes important to develop new bandit algorithms that account for critical safety requirements.


we propose a variation of Safe-LUCB refered to as generalized safe linear upper confidence bound (GSLUCB).


However, here we introduce necessary modifications to account for the safety constraint (1).

we choose the actions with the following two principles.


Note that for notational simplicity our linear regression model is multiplied by âˆš n compared to standard scaling in high-dimensional linear regression [BRT09].

a second contribution is to propose a new knockoffs mechanism, called Conditionally Independent Knockoffs (CIK), which possesses both simple analytic expressions and excellent experimental performance.


We introduce a simple functional called effective signal deficiency (ESD) of the covariance matrix of the predictors that predicts consistency of various variable selection methods.

eSD reveals that the structure of the precision matrix plays a central role in consistency and therefore, so does the conditional independence structure of the predictors.


While some recent work has begun to consider stability properties of neural networks [5,17,19], it has typically done so by ("softly") enforcing stability as an additional loss term on the training data.

they can say little about the stability of the system in unseen states.


Notably, our proposed approach here is able to learn provably stable systems without solving this (generally hard) problem.

while it is difficult to find a Lyapunov function that certifies the stability of some known system, we exploit the fact that it is relatively much easier to enforce some function to behave in a stable manner according to a Lyapunov function.


We also show how such dynamics models can be integrated into larger network systems to learn dynamics over complex output spaces.

despite the predictive power of these systems, it has been difficult to make formal claims about the basic properties of the learned systems.


The intuition of the approach we propose in this paper is straightforward: instead of learning a dynamics function and attempting to separately verify its stability via a Lyapunov function, we propose to jointly learn a dynamics model and Lyapunov function, where the dynamics is inherently constrained to be stable (everywhere in the state space) according to the Lyapunov function.

a damped, rigid n-link pendulum’s state x can be described by the angular position θi and angular velocity θi of each link i.


For comparison, we also include an example trajectory and resulting frames when the dynamics are modelled without the stability constraint  (i.e. letting f in the above loss be a generic neural network).

lyapunov functions in deep learning Finally, there has been a small set of recent work exploring the intersection of deep learning and Lyapunov analysis [5,17,19].


we show how to combine the model with a variational auto-encoder (VAE) [11] to learn dynamic "video textures" [18].


Owing to their representational power, neural networks have long been a natural choice for modeling the function f [7,14,13,6].

when using a generic neural network to model dynamics in this setting, very little can be guaranteed about the behavior of the learned system.


Although the treatment above seems to make the problem of learning stable systems quite straightforward, the sublety of the approach lies in the choice of the function V .

as mentioned previously, V needs to be positive definite, but additionally V needs to have no local optima except 0.


However, because the input polyhedron P 1-ReLU is a hyperrectangle (when projected to X ), it does not capture relational constraints between different x i 's in X (meaning it typically has to substantially over-approximate the set S).

as expected, the computed result S 1-ReLU of the 1-ReLU method will incur significant imprecision when compared with the S best result.


We can observe that the volume of the output from 3-ReLU and 2-ReLU is respectively 9 and 7 orders of magnitude smaller than from 1-ReLU.


We now show, on a simple example, that the k-ReLU concept can be used to improve the results of state-of-the-art verifiers.

we illustrate how the output of our verifier kPoly instantiated with 1-ReLU is refined by instantiating it with 2-ReLU.


Note that P 1-ReLU only contains interval constraints whereas P k-ReLU contains both, the same interval constraints and extra relational constraints.

any convex relaxation obtained using k-ReLU is typically strictly more precise than a 1-ReLU one.


Neural networks are being increasingly used in many safety critical domains including autonomous driving, medical devices, and face recognition.

it is important to ensure they are provably robust and cannot be fooled by adversarial examples [1]: small changes to a given image that can fool the network into making a wrong classification.


In principle, the high-level policy, πh(g|s), can be trained with any reinforcement learning algorithm, given a suitable way to generate sentences for the goals.

generating coherent sequences of discrete tokens is difficult, particularly when combined with existing reinforcement learning algorithms.


In all cases, the agent receives a binary reward only if all constraints are satisfied.

this makes obtaining meaningful signal in these tasks extremely challenging as only a very small number of action sequences will yield non-zero signal.


The second approach instead augments the low-level learning with auxiliary rewards that can bring better inductive bias.

we open the exposition with formalizing the problem of solving temporally extended task with language, including our assumptions regarding the availability of supervision.


We are interested in learning temporally-extended tasks by leveraging the compositionality of language.

in addition to the standard reinforcement learning assumptions laid out in Section 3, we also need some form of grounded language supervision in the environemnt E during training.


Since our low-level policy training is related to goal-conditioned HRL, we can benefit from algorithmic advances in multi-goal reinforcement learning [29,55,4,50].

we extend the recently popularized goal relabeling strategy [29,4] to instructions, allowing us to relabel based on achieving a language statement that describes a region of state space, rather than relabeling based on reaching an individual state.


Finally, in the comparison to a bag-of-words representation (BOW), we observe that, while the BOW agent's return increases at a faster rate than that of the language agent at the beginning of traininglikely due to the difficulty of optimizing recurrent neural network in language agent -the language agent achieves significantly better final performance.

the performance of the BOW agent plateaus at around 8 instructions per episode.


We use the non-parametric approach of [18] proposed in the context of estimating high-dimensional distributions with sparse covariance structure.

they use the empirical CDF of the marginal distributions, where m observations are considered.


We can clearly see that increasing the size of the orientation set in OMP results in a larger angular distance since more dissimilar orientations are included.

the angular distance of candidate sets chosen by GreedyOrientation decreases fast and then stabilized, which indicates that GreedyOrientation forward selection criterion is defined well so that the best candidate orientations approximate the ground truth are among the immediate ones.


This optimization, however, is highly under-constrained, with many possible (dense) solutions.

this objective alone does not enforce a biologically plausible fascicle structure in Φ.


First of all, we notice that our method in general provides slightly less accurate solutions than the method in [DSSW18], i.e., r e > r e in this case.

comparing to the brute force algorithm, our method still generates relatively accurate solutions, especially when m is large, e.g., the relative residual percentage w.r.t. the optimal solution is around 1% when m ≈ 16000.


For the Kronecker Product Low-Rank Approximation (LRA) problem, we give an input sparsity O( q i=1 nnz(A i ) + poly(dk/ ))-time algorithm which computes a rank-k matrix B such that down the Kronecker Product A, which is also the running time of state-of-the-art general purpose LRA algorithms [CW13, MM13, NN13].

our results demonstrate that substantially faster algorithms for approximate LRA are possible for inputs with a Kronecker product structure.


Our algorithm draws from a large number of disparate sketching techniques, such as the dyadic trick for quickly finding heavy hitters [CM05, KNPW11, LNNT16, NS19], and the precision sampling framework from the streaming literature [AKO11].

compressing a matrix A to a low-trank approximation yields many of the same benefits as LRA, such as compact representation, fast matrix-vector product, and fast matrix multiplication, and thus is applicable in many of the settings where LRA is used.


A key contribution of our analysis is to quantify the impact of the part-based structure of the problem on the learning rates of the proposed estimator.

we prove that under natural assumptions on 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.


We identify two main directions for future work: 1) consider settings where the parts are unknown (or "latent") and need to be discovered/learned from data; 2) Consider more general locality assumptions.

for a smaller , such advantage becomes less prominent even when the number of parts is large.


Efficient inference for gradient approximation: Since R 2 P(M) is a combinatorial space, generally the expensive MCMC algorithm is required for sampling from p (R|T, O) to approximate (16).

this can be largely accelerated by scrutinizing the logic property in the proposed model.


In medical trials [11], to identify the best drug for a disease, one can conduct tests in rounds, such that each round involves testing multiple candidate drugs on multiple clinical subjects (e.g., mice) simultaneously.

after each round of testing, there is typically a waiting time (e.g., days) before the effects of drugs become observable to guide the design of the next round of testing.


According to Lemma 2, in each iteration, we can bound the total number of arms in S r+1 without double test.

we go through the algorithm first and then explain why.


In many real applications, not only the query complexity but also the round complexity need to be optimized.

another metric was considered in [3], where the identified k arms can have at most ke regret in total.


directly using such indicator without double test, the indicator may be positively biased.


Then encoding them with the same representation h a = h b can later help the agent extrapolate that if m a is associated with a previously unseen sensory state s a , then m b will also be.

we propose to train a neural network to perform sensorimotor prediction, and to analyze how it learns to encode motor states depending on the type of exploration that generates the sensorimotor data.


For example, by taking fewer gradient steps to compute adversarial examples during training.

this can produce models which are robust against weak attacks, but break down under strong attacks -often due to gradient obfuscation.


Adversarial training is an effective methodology to train deep neural networks which are robust against adversarial, norm-bounded perturbations.

the computational cost of adversarial training grows prohibitively as the size of the model and number of input dimensions increase.


We perform a large scale evaluation of existing methods for adversarially robust training under consistent, strong, white-box attacks.

the work presented here is a regularization technique which encourages the loss function to be well approximated by its linear Taylor expansion in a sufficiently small neighbourhood.


In other words, the absolute difference between these two values, is an indicator of how linear the surface is.

we consider the quantity to be a measure of how linear the surface is within a neighbourhood B( ).


The cost of solving Eq (3) is dominated by the cost of solving the inner maximization problem.

the inner maximization should be performed efficiently to reduce the overall cost of training.


This non-linearity prevents gradient based optimization methods from finding an adversarial perturbation within a small number of iterations [4,24].

if the loss surface was linear in the vicinity of the training examples, which is to say well-predicted by local gradient information, gradient obfuscation cannot occur.


The resulting model trained using LLR degrades gracefully in terms of adversarial accuracy when we increase the strength of attack, as shown in Fig 3 .

fig 3a shows that, for CIFAR-10, when the attack changes from Untargeted to Multi-Targeted, the LLR's accuracy remains similar with only a 2.18% drop in accuracy.


The corresponding adversarial loss is given by (x + Î´).

we also see similar trends in accuracy in Table 2.


The original policy gradient theorem is on-policy and used to optimize the on-policy objective.

in many cases, we would prefer to learn off-policy to improve data efficiency (Lin, 1992) and exploration (Osband et al., 2018).


To learn c via stochastic approximation, Hallak and Mannor (2017) propose the COP operator.

the COP operator does not have a unique fixed point.


By contrast, evaluating the alternative life objective requires samples from the stationary distribution of the target policy, to which the agent does not have access.

it is Jπ, not Jµ, that correctly reflects the deploy-time performance of π, as the behavior policy will no longer matter when we deploy the off-policy learned π in a continuing task.


The data generated by Âµ is significantly different from any meaningful policy in those tasks.

this setting exhibits a high degree of off-policyness.


The data generated by µ is significantly different from any meaningful policy in those tasks.

our experiments aim to answer the following questions.


Gradient TD methods are true stochastic gradient methods and enjoy convergence guarantees.

they are usually two-time-scale, involving two sets of parameters and two learning rates, which makes it hard to use in practice (Sutton et al., 2016).


Under this task specification (White, 2017), the optimal policy under the alternative life objective Jπ, which is equivalent to the average reward objective as γ and i are constant, is to stay in the outer circle

to maximize Jµ, the agent prefers the inner circle


In a recent paper, Li et al. [2019] study a harder variant of bandits, where the delays d t remain unknown.

if an action is played at time s and then more times in between time steps s and s + d s , the learner cannot tell which specific round the loss observed at time s + d s refers to.


In addition to photoreceptors and bipolar cells in the retina [17], ribbon synapses are featured in many other sensory systems, such as in auditory hair cells and the vestibular system [18].

our proposed Bayesian framework links stimulus-response modeling to a biophysically inspired, mechanistic model of the ribbon synapse.


Many mechanistic models in computational neuroscience only provide means to simulate data and do not yield an explicit likelihood function.

their parameters cannot be inferred easily.


In principle, inference via rejection sampling could be applied, but is often inefficient.

recently proposed methods use parametric models (like a mixture of Gaussian) to approximate the posterior over several sampling rounds [6].


The activity of sensory neurons is noisy -a central goal of systems neuroscience has therefore been to devise probabilistic models that allow to model the stimulus-response relationship of such neurons while capturing their variability [1].

linear-nonlinear (LN) models and their generalizations have been used extensively to describe neural activity in the retina [2,3].


In contrast to purely statistical models, the parameters of the LNR model are readily interpretable in terms of properties of the ribbon synapase.

we show that its parameters can be fitted efficiently on synthetic data and two-photon imaging measurements.


In Section H, we provide a sketch of this argument and obtain a bound for relu networks that is polynomial in hidden layer and Jacobian norms and inverse preactivations.

it is not obvious how to adapt the argument of Nagarajan and Kolter [2019] to activation functions whose derivatives are not piecewise-constant.


Now, following the standard technique of bounding Rademacher complexity via covering numbers, we can obtain generalization error bounds for augmented loss.

neural nets are complex, so there remain many other data-dependent properties which could potentially lead to better generalization.


More recently, a semi-supervised deep generative model for attributed network embedding has been proposed [14], which applies the generative adversarial nets (GANs) to generates fake samples in low-density areas in networks and leverages clustering property to help classification.

it still learns embeddings for nodes only.


Many efforts have been paid to co-embedding multiple entities for heterogeneous systems in a fully unsupervised learning procedure [24,30].

to our knowledge, no model in the literature can learn embeddings for heterogeneous data with multiple entities in a semi-supervised way.


Is it beneficial to adapt the auxiliary task weight based on its longer term effect, i.e. N-step update (Section 4.2) compared to the 1-step update (Section 4.1)?

for all the manipulation environments, the goal is an RGBD image with objects in the desired configuration.


Transferring knowledge from other auxiliary tasks is a powerful tool for improving the learning efficiency.

the usage of auxiliary tasks has been limited so far due to the difficulty in selecting and combining different auxiliary tasks.


Deep reinforcement learning has enjoyed recent success in domains like games [1,2], robotic manipulation, and locomotion tasks [3,4].

most of these applications are either limited to simulation or require a large number of samples collected from real-world experiences.


Previous work manually tunes the auxiliary task weights w i [9].

a number of issues arise when we try to scale the number of auxiliary tasks.


However, using gradient-based optimization with only the main task gradient ∇θLmain is often slow and unstable, due to the high variance of reinforcement learning.

auxiliary tasks are commonly used, especially for image based tasks, to help to learn a good feature representation.


The supervised baselines use a training set of ground-truth images and generate observations with random parameters on the fly in each epoch, to create a much larger number of effective image-measurement pairs.

our method is trained with only two measurements per image from the same training set (but not the image itself), with the pairs kept fixed through all epochs of training.


We begin with the simpler case of non-blind estimation, when the parameter Î¸ for a given measurement y is known, both during inference and training.

we focus on a more general class of observation models, which requires injecting the measurement process in loss computation.


However, unlike shallow networks, the training process of RNN often runs into the trouble of vanishing or exploding gradient [53].

the value of the gradient becomes exponentially small or large in the time horizon, even when the training objective is still constant.


One may also want to study whether RNN can be trained close to zero test error.

unlike feedforward networks, the training error, or the ability to memorize examples, may actually be desirable for RNN.


It is expected that if one could sample a training set ad libitum and measure the learner's true performance over all data, such behavior disappears.

for the log-likelihood, we basically prove the same: there are standard learners for which the (negative) log-likelihood is not weakly (Z , l, N)-monotonic for any N.


The hinge loss is appropriate only for the classification setting.

it should be clear that this paper does not get to the bottom of the learning-curve issue.


Around the same time, Opper and Kinzel [1996] showed that in the context of neural networks a similar behavior is observed for small samples.

the error rate for the single layer perceptron is demonstrated to increase when the training set size goes towards the dimensionality of the data [Opper, 2001].


First of all, we show that nonmonotonic behavior can occur in the setting where the complexity of the learner is small compared to the training set size.

the reported behavior is not due to jamming or peaking.


though the rest of the setup remains the same, outputs are limited to the set Y = {−1,+1}  for the hinge loss.


In other words, there is no discrepancy between the objective used during training and the loss used at test time.

possibly odd behavior cannot be explained by dipping.


if one could indeed get to the performance in expectation over all test data and over all training samples of a particular size, performance supposedly improves with more data.


What this work does convey is that learning curves can (provably) show some rather counterintuitive and surprising behavior.

we have demonstrated that least squares regression, regression with the absolute loss, linear models trained with the hinge loss, and likelihood estimation of the variance of a normal distribution can all suffer from nonmonotonic behavior, even when evaluated with the loss they optimize for.


Most optimization methods for solving problems with the Schatten-p norm perform SVD on X at every iteration, with time complexity of O(m 2 n) (supposing m ≤ n) [21,22].

the natural algorithm to minimize FGSR 2/3 and FGSR 1/2 does not use the SVD, as the regularizers are simple (not spectral) functions of the factors.


Previous work addresses this issue by adding a variance regularizer [17,22,18] and clipping/truncating the importance weight [6,22].

the variance regularizer is challenging to use in interactive learning when data arrives sequentially, and it is unclear how the clipping/truncating threshold should be chosen to yield good theoretical guarantees.


In this set of experiments, we inject faults to CNN models and report the accuracy drops of CNN models protected using different strategies.

let X be a floating-point tensor and Xq be the 8-bit quantized version.


Because of its flexibility, RL can encode such a vast array of different problem settings -many of which are entirely intractable.

it is crucial to understand what conditions enable an RL agent to effectively learn about its environment, and to account for the success of RL methods in practice.


In inference, to realize the distribution in Equation (5), Ct ∪ Dt is provided like in GQN (see Appendix C.2).

we see this through the poorer reconstruction and generation quality in the 3D tasks.


By taking the meta-learning framework, Neural Processes learn to learn a stochastic process quickly from observations while experiencing multiple tasks of stochastic process modeling.

in Neural Processes, unlike typical neural networks, learning a function is fast and uncertaintyaware while, unlike Gaussian processes, prediction at test time is still efficient.


Although SSMs have good properties like modularity and interpretability due to the Markovian assumption, the closed-form solution is only available for simple cases like the linear Gaussian SSMs.

in many applications, SSMs show difficulties in capturing nonlinear non-Markovian long-term dependencies  (Auger-Méthé et al., 2016).


In general, the use of PD improves generation quality in all the explored cases.

the action at each time-step is chosen uniformly.


In task (a), we are interested in how the transition model generalizes over the time steps.

we provide context points only in the first 10 time-steps out of 20.


In GQN, query x corresponds to a camera viewpoint in a 3D space, and output y is an image taken from the camera viewpoint.

the problem in GQN is cast as: given a context set of viewpoint-image pairs, (i) to infer the representation of the full 3D space and then (ii) to generate an observation image corresponding to a given query viewpoint.


Neural Processes combine the strengths of neural networks and Gaussian processes to achieve both flexible learning and fast prediction in stochastic processes.

a large class of problems comprises underlying temporal dependency structures in a sequence of stochastic processes that Neural Processes (NP) do not explicitly consider.


The term "pruning" implies the dropping of connections by setting weights to zero, and these weights are thought of as unimportant.

if the value of zero for the pruned weights is not important to the performance of the network, we should expect that we can set pruned weights to some other value, such as leaving them frozen at  their initial values, without hurting the trainability of the network.


This yields fixed-cost adaptation mechanisms, and enables greater sharing across training tasks.

it may under-fit if the function approximation is not sufficiently flexible.


Whereas previous work on CNPs has focused on homogeneous regression and classification datasets and fairly simple models, here we study multiple heterogeneous classification datasets and use a more complex model to handle this scenario.

whereas the original CNP approach to classification required pre-specifying the number of classes in advance, CNAPS handles varying way classification tasks, which is required for e.g.


However, the dimensionality of this subspace is very low compared to the dimensionality of PreResNet-164 CIFAR-10â†’STL-10  the weight space, and we can not guarantee that SWAG variance estimates are adequate along all directions in weight space.

we would expect SWAG to under-estimate the variances along random directions, as the SGD trajectory is in a low-dimensional subspace of the weight space, and a random vector has a close-to-zero projection on this subspace with high probability.


Bayesian methods provide a natural probabilistic representation of uncertainty in deep learning [e.g., 3, 24, 5], and previously had been a gold standard for inference with neural networks [38].

existing approaches are often highly sensitive to hyperparameter choices, and hard to scale to modern datasets and architectures, which limits their general applicability in modern deep learning.


We note that the combination of SWA and temperature scaling presents a competitive baseline.

unlike SWAG it requires using a validation set to tune the temperature; further, temperature scaling is not effective when the test data distribution differs from train, as we observe in experiments on transfer learning from CIFAR-10 to STL-10.


For the multi-view representation, CPM-Nets jointly considers multi-view complementarity and class distribution, making them mutually improve each other to obtain the representation reflecting the underlying patterns.

the encoded latent representation from observations is complete and versatile thus promotes the prediction performance, while the clustering-like classification schema in turn enhances the separability for latent representation.


For each observation, we retrain the classifier using all past observations-a step requiring samples from the two populations.

we predict with a uniform distribution (or P (L) when known) until at least one instance of each label has been observed.)


In this work, we examined the inductive bias of CNNs, and challenged the accepted view that FCNs are unable to generalize as well as CNNs on visual tasks.

we showed that the CNN prior is mainly useful during the early stages of training, to prevent the unconstrained FCN from falling prey of spurious solutions with poor generalization too early.


The practical interest of our protocol is limited by the very large size of the highly sparse eFCN.

it offers interesting insights into the persistence of architectural bias under stochastic gradient dynamics.


We then use these approximate gradient oracles to devise approximate gradient descent algorithms.

we use the stable manifold theorem to prove that zero order methods can almost surely avoid saddle points.


While local optima may be satisfactory for some applications in machine learning [CHM + 15], saddle points can make high dimensional non convex optimization tasks significantly more difficult [DPG + 14,SQW18].

researchers have focused their efforts on functions possessing the strict saddle property.


A recent line of work has shown that gradient descent and variations of it can actually converge to SOSPs .

[LPP + 19] shows that gradient descent starting from a random point can eventually converge to SOSPs with probability one.


In the previous subsections we provided sufficient conditions for approximate gradient descent to avoid strict saddle points.

intuitively, SMT formalizes why convergence to unstable fixed points is unlikely starting from a local region around an unstable fixed point.


The key insight provided by the SMT is that the all the initialization points that eventually converge to an unstable fixed point lie in a low dimensional manifold.

to obtain a stronger result we have to understand how SMT restricts the dimensionality of this stable manifold for a fixed h0 .


In this case our approximate gradient cannot guarantee a substantial decrease of f .

we know that the Hessian has a substantially negative eigenvalue and therefore a direction of steep decrease of f must exist.


In all our experiments, we assume that the weightfunction comes from the class F of mixture of m 2D non-negatively weighted Gaussians as described in the end of Section 3.2.

the persistent homology is one of the most important developments in the field of topological data analysis, and there have been fundamental developments both on the theoretical front (e.g, [23,10,13,8,14,5]), and on algorithms / implementations (e.g, [43,4,15,20,29,3]).


There are natural distances defined for persistence diagrams, including the bottleneck distance and the Wasserstein distance, both of which have been well studied (e.g, stability under them [16,18,14]) with efficient implementations available [27,28].

to facilitate downstream machine learning tasks, it is desirable to further map the persistence diagrams to another "vector" representation.


For example, for the persistence diagrams computed from atomic configurations of molecules, features with small persistence could capture the local packing patterns which are of utmost importance and thus should be given a larger weight; while in many other scenarios, small persistence leads to noise with low importance.

the 10 * 10-fold nested cross validation are applied to evaluate our algorithm: There are 10 folds in outer loop for evaluation of the model with selected hyperparameters and 10 folds in inner loop for hyperparameter tuning.


They still lack the investigations on synthesizing the images in a more controllable way, like finely manipulating the visual appearance of every object.

to generate the images with preferred objects and rich interactions, we propose a semi-parametric method, PasteGAN, for generating the image from the scene graph and the image crops, where spatial arrangements of the objects and their pair-wise relationships are defined by the scene graph and the object appearances are determined by the given object crops.


In order to integrate the objects in the expected way defined by the scene graph as well as maintaining the visual appearance of the objects, we designed a Crop Refining Network as well as an attentionbased Object-Image Fuser, which can encode the spatial arrangements and visual appearance of the objects as well as their pair-wise interactions into one scene canvas.

it can encode the complicated interactions between the objects.


With such algorithms, everyone can become an artist: you just need to define the objects and how they interact with each other, and then the machine will produce the image following your descriptions.

it is a challenging problem as it requires the model to have a deep visual understanding of the objects as well as how they interact with each other.


A good crop is not simply matching the category, but also of the similar scene.

to retrieve the good crop, we should also consider the context information of the object, i.e. e. the entire scene graph it belongs to.


Policies that aim for high empowerment can lead to complex behavior, e.g. balancing a pole in the absence of any explicit reward signal [23].

the implicit dependency of p on the optimization argument Ï€ empower renders the problem non-trivial.


Note that our SAC baseline is comparable with the SAC from [22] on Hopper-v2, Walker2d-v2, Ant-v2 and Humanoid-v2 after 5 â€¢ 10 5 steps (the SAC from [21] uses the earlier v1-versions of Mujoco and is hence not an optimal reference).

there is a discrepancy on HalfCheetah-v2.


This motivates regarding Eq. (4) as a regularization term for learning H, which guides the learning representation process, forming the cluster structures.

visualization analysis illustrates the effectiveness of cluster-specific temporal representations and demonstrates the robustness of the learning process, even if K-means makes mistakes.


To further analyze the performance, we perform a pairwise comparison for each method against DTCR.

we conduct the Wilcoxon signed rank test [36] to measure the significance of the difference.


DTCR integrates temporal reconstruction and the K-means objective into a seq2seq model.

dTCR adapts bidirectional Dilated recurrent neural networks [17] as the encoder, enabling the learned representation to capture the temporal dynamics and multi-scale characteristics of time series.


Note that USSL depends on pseudo-labels to guide the learning, while there is no mechanism to reduce the negative impact when mistakes occur in the pseudo-labels.

dTCR is capable of correcting mistakes with the help of temporal reconstruction (for analysis see Section 4.3.3).


Zakaria et al. [24] proposed u-shapelet to learn local patterns.

the number of selected time steps is bα × Tc, where α ∈ (0, 1] is a hyper-parameter we set to 0.2.


More importantly, we establish the statistical-computational barrier under an oracle computational model [16,18,19,54], which is an abstraction of computations made by algorithms that interact with data.

we study the signal detection problem where the link function is defined as a continuous interpolation of two link functions of different types.


Scale estimation in object tracking is challenging and existing methods mainly enumerate possible scales [2,46] and select the optimal one.

the scale can be estimated by our proposed model.


Our model performs favorably against the self-supervised state-of-the-art methods.

our model outperforms Wang et al. [52] by 13.3% in J and 16.6% in F. and is even 6.9% better in J and 4.1% better in F than the ResNet-18 model [13] trained on ImageNet [9] with classification labels.


For dense correspondence, the inter-frame affinity needs to be sparse to ensure one-to-one mapping.

it is challenging to model a sparse matrix in a deep neural network.


the following theorem states that the sparsity constrained optimization problem in (2) is hard even to approximate, in the sense that no deterministic approximation algorithm exists that solves it in polynomial time.


We have shown in the proof of Theorem 2 that there always exist extreme examples that are hard to solve.

in the most general sense, and without further assumptions, one can find pathological cases which make the problem hard.


The projection (2) is iteratively solved in IHT (step 5 in Algorithm 1).

for the algorithm to be practical, it is important to study the tractability of the projection step.


Our result can be a significant improvement as m grows.

in our setting the individual machines need not be able to obtain any estimate of v i , but the corresponding average is still accurate.


Linear networks are simple since they can only represent linear transformation, but they preserve one of the most important aspects of deep neural networks, the layered structure.

analysis of linear networks will be helpful for understanding nonlinear cases.


We observe that the blue curve takes a long time in the neighborhood of saddle point (0, 0), however the red curve does not.

zAS is equivalent to initializing all the residual blocks and the output layer with zero in a linear residual network.


[10] provides a simpler proof of this result for deep residual networks, and shows that the he Polyak-Łojasiewic condition is satisfied in a neighborhood of a global minimum.

these results do not imply that gradient descent can find global minima, and also cannot tell us the number of iterations required for convergence.


For the case of known order of dying, we propose Algorithm 2, Hedge-Perm-Known (HPK), which is slightly different than HPU.

the weight redistribution (when an expert dies) and initialization of coefficient h i,1 is different.


For example, in online advertising, an ad broker has contracts with their providers and these contracts may expire in an order known to Learner.

we will study the problem in two different settings: when Learner is aware of this order and when it is not.


In [7], the comparator set is the set of all probability vectors over K experts, while we compare Learner's performance to the performance of the best ordering.

the problem considered in [7] aims to compare Learner's performance to the best mixture of actions, which also includes our comparator set (orderings).


We also performed a qualitative comparison of the latent structures inferred by the different models and found that the sparse variant of the PRGDS inferred some components that the other models did not.

the sparse variant of the PRGDS is uniquely capable of inferring bursty latent structures that are highly localized in time; we visualize examples in Fig. 5.


We recommend using these layers when generative modeling with normalizing flows (Dinh et al., 2017) or understanding how networks make predictions (Jacobsen et al., 2018).

our layers tie the model specification to the inference algorithm (typically, variational inference).


We initialized a random matrix B ∈ Rk⇥m, testing different values for m

we find that our trained probes are able to achieve slightly improved accuracy down to m = 128.


This dataset has the constituency grammar for the sentences, which was translated to a dependency grammar using the PyStanfordDependencies library [14].

we clamped the cosine similarity terms to within ±0.1 of the pre-training averages for same and different senses.


If we do not assume Assumption 1, the arguments in Theorem 2 easily extend to showing instead that we approach some identified âœ“ that satisfies all moment conditions.

using a rich class of moment conditions allows us to learn correspondingly a rich g0 .


In particular this means that if we parametrize f and g via neural networks where we can permute the parameter vector âœ“ and obtain an identical function, our result still holds.

as long as F has bounded complexity, even if its ambient dimension may be infinite, we can guarantee the consistency of our approach.


When g(x; θ) is a flexible model such as a high-capacity neural network, many -possibly infinitely many -moment conditions may be needed to identify θ 0 .

gMM and OWGMM algorithms fail when we use too many moment conditions.


Instead, our approach is to leverage our variational reformulation in Lemma 1 and replace the class of functions F with a rich (non-subspace) set in this new formulation, which is distinct from GMM and avoids these issues.


On the other hand, we also cannot hope to learn the optimal weighting: the matrix Co in Eq. (6) will necessarily be singular and using its pseudoinverse would mean deleting all but n moment conditions.

we cannot simply use infinite or even too many moment conditions in GMM or OWGMM.


Using first-order iterative algorithms for solving Eq. ( 9) enables us to effectively handle very large datasets.

we implement DeepGMM using PyTorch, which efficiently provides gradients for use in our descent algorithms [30].


The rationale behind the clustering penalty was to better disentangle the features in the latent space.

we obtain latent representations where the different clusters (i.e., classes) are better separated, thereby facilitating their modeling.


Alternatively, using a nonparametric bivariate copula estimator provides the required flexibility.

the bivariate Gaussian kernel estimator, targeted at densities of unbounded support, cannot be directly applied to pair-copulas, which are supported in the unit square.


It should be noted that, in both cases, the nonparametric estimator captures the fact that X 1 and X 2 are independent, as can be seen from the contour densities on the left panel.

adversarial approaches require multiple models to be trained, leading to difficulties and computational burden [62,26,24], and variational approaches make (strong) distributional assumptions, potentially detrimental to the generative model performance [64].


However, AEs simply learn the most informative features to minimize the reconstruction loss, and therefore cannot be considered as generative models.

since they do not learn the distributional properties of the latent features [5], they cannot be used to sample new data points.


However, expected future rewards equal to zero are in general not possible for MDPs.

we introduce sequence-Markov decision processes (SDPs), for which reward distributions need not to be Markov.


We can use any contribution analysis method, but we specifically consider three methods: (A) differences of return predictions, (B) integrated gradients (IG) [46], and (C) layer-wise relevance propagation (LRP) [3].

this task has probabilistic state transitions, which can be represented as a tree with states as nodes.


FNNs are prone to such prediction errors since they have to predict the expected return again and again from each different state-action pair and cannot use stored information.

the LSTM is less prone to produce spurious rewards: (i) The LSTM will only learn to store information if a state-action pair has a strong evidence for a change in the expected return.


The advantage function is the sales price minus the expected immediate repair costs minus the expected future delivery costs.

you want to know whether the advantage function is positive.


Learning with non-optimal reward redistributions does work since the optimal policies do not change according to Theorem 1.

reward redistributions that are optimal considerably speed up learning, since future expected rewards introduce biases in TD methods and high variances in MC methods.


For example, any contributions would be incorrect if the true expectation of the return did not change.

prediction errors might falsely cause contributions leading to spurious rewards.


The reward can be viewed to be probabilistic but is prone to have high variance.


The sum of the Q-value differences gives the difference between expected return at sequence begin and the expected return at sequence end (telescope sum).

q-value differences allow to predict the expected return of the whole state-action sequence.


These tasks are designed to show problems of TD, MC, and potential-based reward shaping.


Consecutive predictions share the same past and the same future contributions except for two immediate state-action pairs.

in the difference of consecutive predictions contributions cancel except for the two immediate state-action pairs.


The goal is to learn that a delayed reward is larger than a distracting immediate reward.

the correct expected future reward must be assigned to many state-action pairs.


According to Theorem 1, reward redistribution keeps the optimal policies.

even nonoptimal reward redistributions ensure correct learning.


In contrast to standard empirical risk minimization in supervised learning, where test data follow the same distribution as training data, minimax statistical learning arises in problems of distributionally robust learning [16,18,28,29,40] and minimizes the worst-case risk over a family of probability distributions.

it can be applied to the learning setting in which the test data distribution differs from that of the training data, such as in domain adaptation and transfer learning [12].


The bound may be loose in some cases, since we consider the worst case distribution in the Wasserstein ball to avoid computing the transport map.

for some problems, it may be possible to derive the transport map and thus get tighter bounds.


Machine learning models, especially deep neural networks, have achieved impressive performance across a variety of domains including image classification, natural language processing, and speech recognition.

these techniques can easily be fooled by adversarial examples, i.e., carefully perturbed input samples aimed to cause misclassification during the test phase.


Then, Lee & Raginsky [28] derived the risk bound for minimax learning by exploiting the dual representation of worst-case risk.

their minimax risk bound would go to infinity and thus become vacuous as eb-> 0.


Finally we note that the shape of M remains similar to tanh(E).

secondly, we introduce a secondary de-attention (deleted attention) matrix, finally learning a multiplicative composition of similarity and dissimilarity.


(2) As a result, while this robust prediction objective falls within our framework, and regularizes robust attributions, it allows a small regularization term where attributions actually change significantly but they cancel each other in summation.

the control over robust attributions can be weak.


Our theory is grounded in the recent work, Integrated Gradients (IG) [STY17], in axiomatically attributing a neural network's output change to its input change.

an emerging problem to tackle in this domain is to train models that produce reliable interpretations for their predictions.


While, in general, f H may be difficult to compute, we show settings in which it is tractable.

for kernel ridge regression with a Gaussian kernel, we prove a bound on f H that, as a byproduct, yields generalization bounds that match (up to a small constant) the standard ones.


To circumvent this issue, next we approach the DRO problem from a different angle: we directly search for the adversarial distribution Q.

for example, f and f may not belong to the same RKHS -it is not hard to construct counterexamples, even when is merely quadratic.


External memory may be helpful or even necessary for agents operating in partially observable environments, where current decision making depends on a sequence of past observations.

memory mechanisms also add complexity to an agent's learning process [19], since the agent must learn to access its memory during experience.


Because these feedback connections are not computed nor used sequentially, updates can be parallelized for speed.

fixed random feedback connections may be inefficient.


The optimal parameter ter can be obtained by maximizing the lower bound in (2), which are for the datapoints of seen classes.

by sampling the unseen datapoints while reducing the model uncertainty, the model better describes the target distribution of unseen classes.


To relieve the problem, SGAL-dropout uses dropout activation when sampling unseen datapoints and shows more robust classification results compared to that of SGAL's.


The combination of the large amount of data and deep learning finds the usage in various fields such as machine learning and artificial intelligence.

deep learning as a non-linear regression tool based on statistics mostly suffers from the insufficient or non-existing training data, which is the usual case and should be overcome for autonomous learning systems.


This optimal generative model can be obtained with (5) by trained on that sampled datapoints X u * of unseen classes, and existing datapoints of seen classes.

we can have a generative model which covers both seen and unseen classes by obtaining optimal parameters and sampled datapoints that satisfy (4) and (5).


In order to circumvent the need for samples of unseen classes, we treat the non-existing data as missing examples.

our network aims to find optimal unseen datapoints and model parameters, by iteratively following the generating and learning strategy.


In other words, while training our model iteratively generate unseen samples and use them as training datapoints to gradually update model parameters.

our model favorably attain both seen and unseen classes understanding.


Compared to the Gumbel-Softmax estimator, our approach relies on maximization over the latent variable assignments, rather than summation, which is computationally more efficient.

performing maximization exactly or approximately is possible in many structured cases, even when summation remains intractable.


The typical solution is to replace the desired objective with a surrogate differentiable loss, such as the cross-entropy loss between the targets and the predicted distribution over labels.

the direct loss minimization approach proposes to minimize the desired objective directly.


One can also see that the general structured encoder, with any αi,j (x), achieves better test loss than the super-modular structured encoder.

this comes with a computational price, as the maxflow algorithm is orders of magnitude faster than CPLEX, and structured encoder with CPLEX becomes better than maxflow only in epoch 85, see Figure 3.


[12] use the Gumbel-Max trick to reparameterize the discrete VAE objective, but, unlike our work, they relax the resulting formulation, replacing the arg max with a softmax operation.

they introduce the continuous Concrete (Gumbel-Softmax) distribution and replace the discrete random variables with continuous ones.


The Perturb-and-Parse approach [5] focuses on latent dependency parses, and iteratively replaces any arg max with a softmax operation in a spanning tree algorithm.

our framework is not restricted to a particular class of structures.


This unsatisfactory pattern of quantization error in Theorem 3 comes from the finite sample setting and proof methodology, since the bound is a worst case bound with n and k both finite.

this bound is less meaningful for practical purposes.


At this point, 1-bit LM quantizer has much smaller debiased variance than others.

we expect 1-bit LM to provide highest test accuracy on these two datasets, which is again consistent with Figure 4.


Throughout this paper, we assume that every sample in X is standardized to have unit Euclidean norm 2 .

the domain of X is the unit Euclidean sphere S d , which allows us to call "inner product" and "cosine" interchangeably.


Hence, for 1-NN classification, we should ideally choose quantizers with low debiased variance around p * = cos(x, x i ), provided that it can be known (or estimated) a priori.

quantizers with small debiased variance are favorable for NN classifier and linear classifier, in high similarity region and around ρ = 0, respectively.


Here, we look at this problem in the asymptotic domain.

intuitively, label flipping is much more likely to occur for the points near the boundary (i.e., with small Ä¥T x).


These results provide an encouraging case for disentanglement being helpful in finding fairer representations.

roadmap: In Section 2, we briefly review the state-of-the-art approaches to extract and evaluate disentangled representations.


As one builds machine learning models for different tasks on top of such general purpose representations, it is not clear how the properties of the representations relate to the fairness of the predictions.

for different downstream prediction tasks, there may be different sensitive variables that we would like to be fair to.


As a sanity check, we finally confirm by visual inspection that the adjusted metrics still measure disentanglement.

we believe that these results serve as a motivation for further investigation on the practical benefits of disentangled representations, especially in the context of fairness.


With this we compute the disentanglement metrics and use the following score to measure the unfairness of the predictions where T V is the total variation.

we compare the average total variation of the prediction after intervening on s, thus directly measuring the violation of demographic parity.


We clearly observe that learned representations can be unfair, even in the setting where the target variable and the sensitive variable are independent.

the total variation can reach as much as 15% 25% on five out of seven data sets.


We remark that this setup is different from what we consider in this paper, as we do not assume access to any labeled information when learning a representation.

we do not assume to know what the downstream task will be and what are the sensible variables (if any).


If we limit the memory usage, then this implies that the effective capacity of the replay is limited as any transitions we replace are forgotten completely.

parametric models may be able to achieve good accuracy with a fixed and comparatively small memory footprint.


Sometimes, there may be no practical differences between replay and models, depending on how they are used.

a replay memory is less flexible than a model, since we cannot query it at arbitrary states that are not present in the replay memory.


Although the distinction is not fully unambiguous, there exist two prototypical families of algorithms: those that learn without an explicit model of the environment (model free), and those that first learn a model and then use it to plan a solution (model based).

this has implications for Dyna-style learning, as well as for replay methods [cf.. van Hasselt et al., 2018].


Instead, planning backward when the model is inaccurate may lead to updates to fictional states that are unreachable, which may or may not be useful, but is less likely to be harmful than updating real states with fictional transitions.

if, however, we sample states according to a non-uniform distribution (e.g., using prioritised replay) this can make replay-based algorithms less stable and potentially divergent [cf.


Experience replay [Lin, 1992] refers to storing previously observed transitions to replay later for additional updates to the predictions and policy.

one of the main results by Kaiser et al. [2019] was to compare SimPLe to Rainbow DQN [Hessel et al., 2018a], which combines the DQN algorithm [Mnih et al., 2013[Mnih et al., , 2015 with double Q-learning [van Hasselt, 2010, dueling network architectures [Wang et al., 2016], prioritised experience replay , noisy networks for exploration [Fortunato et al., 2017], and distributional reinforcement learning [Bellemare et al., 2017].


We examine the question of when and how parametric models are most useful in reinforcement learning.

we look at commonalities and differences between parametric models and experience replay.


Both of these methods can only work on feed-forward networks that are separable between layers.

neither approach can parallelize Transformer-based language models, because the shared embeddings make the networks non-separable.


Empirical results demonstrate the superiority of Transformer networks and show that a larger model tends to yield better performance.

in the next section, we analyze the convergence rate of Algorithm 1, which is the basis of analysis for other variants of SGD.


It is an ideal case to obtain linear speedup, using K × machines to achieve K × speedup regarding time.

it is impossible to achieve even for data parallelism.


In the proposed algorithm, the backward computation in module k is always one time step behind module k + 1.

the computations in all modules can be parallelized.


We split an L-layer network into K modules so that the weights of the network are divided into K groups and each group is placed on a GPU.

we have w = [w G (1) , w G(2) , ..., w G(K) ] where G(k) denotes layer indices in group k.


when a model is so large that it has to be allocated on multiple GPUs, data parallelism over these GPUs is not applicable because it requires each GPU to have one copy of the whole model.


Consequently, we propose to store the input of each module as well as a random seed.

before computing activations, we initialize the random number generator in GPU with the stored seed.


Recent efforts in deep learning models interpretation allow us to understand the decision for particular examples Ribeiro et al. (26); Shrikumar et al. (27); Štrumbelj, Kononenko (31); Koh, Liang (15); Lundberg, Lee (19), but understanding the logic behind a complex model is still a challenging task.

symmetric trees outperform other tree shapes: they are coded by a decision table, thus evaluation for one tree requires several bit-wise operations and one look-up in the table and takes >10x less time for the same number of trees Dorogush et al. (6).


There is no universal strategy for tree induction: optimal one changes with dataset at hand.

symmetric trees outperform other tree shapes: they are coded by a decision table, thus evaluation for one tree requires several bit-wise operations and one look-up in the table and takes >10x less time for the same number of trees Dorogush et al.


CatBoost (25) uses a level-wise policy, similar to one used in XGBoost, but with one more restriction: CatBoost searches for just one single condition to split all leaves simultaneously.

the result is, effectively, a decision table.


This set optimization is clearly NP-hard: there is a set cover problem under the shell 3 .

the target of the optimization is submodular, so it is possible to find an approximately optimal solution in polynomial time Bach (1).


One desired property of such a probabilistic model is sharpness (or high accuracy), i.e., if possible, the model should assign the highest probability to the true tumour grade (which maybe can not be inferred from the image at hand but only by other means such as an additional immunohistochemical staining).

to be able to trust the predictions the probabilities should be calibrated (or reliable) as well (DeGroot and Fienberg, 1983;Murphy and Winkler, 1977).


While in tree-based methods, the tree index's hierarchy also affects the retrieval model training.

how to learn the tree index and user preference model jointly is an important problem.


A series of related work [30,6,18,16,32,33,34] has shown that the user's historical behaviors play a key role in predicting the user's interests.

the corpus size, and there is no restriction on the preference model structure.


This further decomposition is important in understanding how the ensemble model's predictive uncertainty changes by accounting for the fact that its prediction and distribution functions may be misspecified.

recall that an ensemble model's parametric uncertainty is the uncertainty about the ensemble weights under the current model specification (i.e. by assuming δ = 0,G = I).


We vary sample size between 100 and 1000, and repeat the simulation 50 times in each setting.

bAE's aleatoric uncertainty is still biased in that it is roughly constant throughout the range of x, failing to account for the heterogeneity in observation's variance, a pattern that is evident in data's empirical distribution.


As shown in Figure 1, the posterior uncertainty in each of a BNE's model parameters {ω,δ,G} accounts for an important source of model uncertainty.

both the aleatoric and epistemic uncertainties are quantified by the BNE's posterior distribution, and can be distinguished through a careful decomposition of the model posterior.


Aleatoric uncertainty arises due to the stochastic variability inherent in the data generating process, for example due to an imperfect sensor, and is described by the cumulative distribution function (CDF) F((y|x,Θ) of the data specified by a given model.

epistemic uncertainty arises due to our lack of knowledge about the data generating mechanism.


In practice, semi-supervised learning methods with many hyperparameters can be problematic because cross-validation is difficult with small validation sets [35,39,35].

we find in practice that most of MixMatch's hyperparameters can be fixed and do not need to be tuned on a per-experiment or per-dataset basis.


BCOP has all of its singular values at 1 throughout training by design due to its gradient norm preservation and orthogonality.

we empirically verify the downsides of other methods and show that our proposed method enables maximally expressive Lipschitz constrained convolutional layers with guaranteed gradient-norm-preservation.


The property above (Equation 2) allows one to upper-bound the Lipschitz constant of a network by the product of the Lipschitz constants of each layer.

as modern neural networks tend to possess many layers, the resultant upper-bound is likely to be very loose, and constraining it increases the risk of diminishing the Lipschitz constrained network capacity that can be utilized.


The Lipschitz constant (or spectral norm) of a convolution operator is bounded by a constant factor of the spectral norm of its reshaped matrix [12,48,40], which enables bounding of the convolution operator's Lipschitz constant by bounding that of the reshaped matrix.

this upper-bound can be conservative, causing a bias towards convolution operators with low Lipschitz constants, limiting the method's expressive power.


We also evaluate the empirical robustness of our model around under two gradient-based attacks and two decision-based attacks: (i) PGD attack with CW loss [34,7], (ii) FGSM [47], (iii) Boundary attack (BA) [6], (iv) Point-wise attack (PA) [44].

the gradient-based methods ((i) and (ii)) are done on the whole test dataset; the decision-based attacks ((iii) and (iv)) are done only on the first 100 test data points since they are expensive to run.