pwc-archive/papers-with-abstracts huggingface dataset:
Fair Resource Allocation in Federated Learning
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Fairness, Federated Learning

Federated learning involves jointly learning over massively distributed partitions of data generated on remote devices. Naively minimizing an aggregate loss function in such a network may disproportionately advantage or disadvantage some of the devices. In this work, we propose q-Fair Federated Learning (q-FFL), a novel optimization objective inspired by resource allocation strategies in wireless networks that encourages a more fair accuracy distribution across devices in federated networks. To solve q-FFL, we devise a scalable method, q-FedAvg, that can run in federated networks. We validate both the improved fairness and flexibility of q-FFL and the efficiency of q-FedAvg through simulations on federated datasets.
Model Aggregation via Good-Enough Model Spaces
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: model

  In many applications, the training data for a machine learning task is partitioned across multiple nodes, and aggregating this data may be infeasible due to storage, communication, or privacy constraints. In this work, we present Good-Enough Model Spaces (GEMS), a novel framework for learning a global satisficing (i.e. "good-enough") model within a few communication rounds by carefully combining the space of local nodes' satisficing models. In experiments on benchmark and medical datasets, our approach outperforms other baseline aggregation techniques such as ensembling or model averaging, and performs comparably to the ideal non-distributed models.
Adaptive Gradient-Based Meta-Learning Methods
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Meta-Learning

We build a theoretical framework for understanding practical meta-learning methods that enables the integration of sophisticated formalizations of task-similarity with the extensive literature on online convex optimization and sequential prediction algorithms in order to provide within-task performance guarantees. Our approach improves upon recent analyses of parameter-transfer by enabling the task-similarity to be learned adaptively and by improving transfer-risk bounds in the setting of statistical learning-to-learn. It also leads to straightforward derivations of average-case regret bounds for efficient algorithms in settings where the task-environment changes dynamically or the tasks share a certain geometric structure.
LEAF: A Benchmark for Federated Settings
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Autonomous Vehicles, Benchmarking, Federated Learning, Meta-Learning, Multi-Task Learning

Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts of data each day. This wealth of data can help to learn models that can improve the user experience on each device. However, the scale and heterogeneity of federated data presents new challenges in research areas such as federated learning, meta-learning, and multi-task learning. As the machine learning community begins to tackle these challenges, we are at a critical time to ensure that developments made in these areas are grounded with realistic benchmarks. To this end, we propose Leaf, a modular benchmarking framework for learning in federated settings. Leaf includes a suite of open-source federated datasets, a rigorous evaluation framework, and a set of reference implementations, all geared towards capturing the obstacles and intricacies of practical federated environments.
Error Correcting Algorithms for Sparsely Correlated Regressors
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: compressed sensing, regression

Autonomy and adaptation of machines requires that they be able to measure their own errors. We consider the advantages and limitations of such an approach when a machine has to measure the error in a regression task. How can a machine measure the error of regression sub-components when it does not have the ground truth for the correct predictions? A 
compressed sensing approach applied to the error signal of the regressors can recover their precision error without any ground truth. It allows for some regressors to be strongly correlated as long as not too many are so related.
Its solutions, however, are not unique - a property of ground truth inference solutions. Adding  $\ell_1$--minimization
as a condition can recover the correct solution in settings where error correction is possible. We briefly discuss the similarity of the mathematics of ground truth inference for regressors to that for classifiers.
Meta-Teaching: Curriculum Generation for Lifelong Learning
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Lifelong learning, Meta-Learning

Meta-learning will be crucial to creating lifelong, generalizable AI. In practice, however, it is hard to define the meta-training task distribution that is used to train meta-learners. If made too small, tasks are too similar for a model to meaningfully generalize. If made too large, generalization becomes incredibly difficult. We argue that both problems can be alleviated by introducing a teacher model that controls the sequence of tasks that a meta-learner is trained on. This teacher model is incentivized to start the student meta-learner on simple tasks then adaptively increase task difficulty in response to student progress. While this approach has been previously studied in curriculum generation, our main contribution is in extending it to meta-learning.
Deep Knowledge Based Agent: Learning to do tasks by self-thinking about imaginary worlds
Authors: Anonymous
Date: 2019-05-16 00:00:00
Proceeding: ICML Workshop AMTL 2019 6
Tasks: Knowledge Base Construction, Meta-Learning, Model-based Reinforcement Learning, reinforcement-learning, Reinforcement Learning (RL)

Lifelong and meta learning aim to learn quickly new skills by transferring knowledge from past experiences. Current algorithms learn a meta-policy from a set of hand-crafted training tasks and adapt their policy to new but similar problems. In contrast, human are able to learn from very sparse data and apply learned skills to very different situations. In this work, we  integrate the old-standing idea of knowledge-based agents with deep model-based reinforcement learning.  We show that our model can transfer knowledge from completely randomly generated environments to real tasks. We argue that the weights of forward models can be used as a knowledge-base by an inference engine to infer optimal policy in real environments. We demonstrate that this approach automatizes both task-specification and knowledge base construction.
The Effect of Network Depth on the Optimization Landscape
Authors: Behrooz Ghorbani, Ying Xiao, Shankar Krishnan
Date: 2019-05-28 00:00:00

It is well-known that deeper neural networks are harder to train than shallower ones. In this short paper, we use the (full) eigenvalue spectrum of the Hessian to explore how the loss landscape changes as the network gets deeper, and as residual connections are added to the architecture. Computing a series of quantitative measures on the Hessian spectrum, we show that the Hessian eigenvalue distribution in deeper networks has substantially heavier tails (equivalently, more outlier eigenvalues), which makes the network harder to optimize with first-order methods. We show that adding residual connections mitigates this effect substantially, suggesting a mechanism by which residual connections improve training.
In Support of Over-Parametrization in Deep Reinforcement Learning: an Empirical Study
Authors: Brady Neal, Ioannis Mitliagkas
Date: 2019-05-17 00:00:00
Proceeding: ICML Workshop Deep_Phenomen 2019 6
Tasks: Deep Reinforcement Learning, OpenAI Gym, reinforcement-learning, Reinforcement Learning (RL)

There is significant recent evidence in supervised learning that, in the over-parametrized setting, wider networks achieve better test error. In other words, the bias-variance tradeoff is not directly observable when increasing network width arbitrarily. We investigate whether a corresponding phenomenon is present in reinforcement learning. We experiment on four OpenAI Gym environments, increasing the width of the value and policy networks beyond their prescribed values. Our empirical results lend support to this hypothesis. However, tuning the hyperparameters of each network width separately remains as important future work in environments/algorithms where the optimal hyperparameters vary noticably across widths, confounding the results when the same hyperparameters are used for all widths.
Do deep neural networks learn shallow learnable examples first?
Authors: Karttikeya Mangalam, Vinay Uday Prabhu
Date: 2019-05-17 00:00:00
Proceeding: ICML Workshop Deep_Phenomen 2019 6

In this paper, we empirically investigate the training journey of deep neural networks relative to fully trained shallow machine learning models. We observe that the deep neural networks (DNNs) train by learning to correctly classify shallow-learnable examples in the early epochs before learning the harder examples. We build on this observation this to suggest a way for partitioning the dataset into hard and easy subsets that can be used for improving the overall training process. Incidentally, we also found evidence of a subset of intriguing examples across all the datasets we considered, that were shallow learnable but not deep-learnable. In order to aid reproducibility, we also duly release our code for this work at https://github.com/karttikeya/Shallow_to_Deep/
Line attractor dynamics in recurrent networks for sentiment classiﬁcation
Authors: Niru Maheswaranathan, Alex H. Williams, Matthew D. Golub, Surya Ganguli, David Sussillo
Date: 2019-05-17 00:00:00
Proceeding: ICML Workshop Deep_Phenomen 2019 6

Recurrent  neural  networks  (RNNs)  are  a  powerful tool for modeling sequential data. Despite their widespread usage, understanding how RNNs solve complex problems remains elusive.  Here, we characterize how popular RNN architectures perform document-level sentiment classification.Despite their theoretical capacity to implement complex, high-dimensional computations, we find that  trained  networks  converge  to  highly  interpretable, low-dimensional representations.  We identify a simple mechanism, integration along an approximate line attractor, and find this mechanism present across RNN architectures (including LSTMs, GRUs, and vanilla RNNs). Overall, these results demonstrate that surprisingly universal and human interpretable computations can arise across a range of recurrent networks.
Channel Normalization in Convolutional Neural Network avoids Vanishing Gradients
Authors: Zhenwei Dai and Reinhard Heckel
Date: 2019-05-17 00:00:00
Proceeding: ICML Workshop Deep_Phenomen 2019 6
Tasks: Decoder

Normalization layers are widely used in deep neural networks to stabilize training. 
In this paper, we consider the training of convolutional neural networks with gradient descent on a single training example.
This optimization problem arises in recent approaches for solving inverse problems such as the deep image prior or the deep decoder. 
We show that for this setup, channel normalization, which centers and normalizes each channel individually, avoids vanishing gradients, whereas without normalization, gradients vanish which prevents efficient optimization.
This effect prevails in deep single-channel linear convolutional networks, and we show that without channel normalization, gradient descent takes at least exponentially many steps to come close to an optimum. 
Contrary, with channel normalization, the gradients remain bounded, thus avoiding exploding gradients.
Using effective dimension to analyze feature transformations in deep neural networks
Authors: Kavya Ravichandran, Ajay Jain, Alexander Rakhlin
Date: 2019-05-28 00:00:00
Proceeding: ICML Workshop Deep_Phenomen 2019 6

In a typical deep learning approach to a computer vision task, Convolutional Neural Networks (CNNs) are used to extract features at varying levels of abstraction from an image and compress a high dimensional input into a lower dimensional decision space through a series of transformations. In this paper, we investigate how a class of input images is eventually compressed over the course of these transformations. In particular, we use singular value decomposition to analyze the relevant variations in feature space. These variations are formalized as the effective dimension of the embedding. We consider how the effective dimension varies across layers within class. We show that across datasets and architectures, the effective dimension of a class increases before decreasing further into the network, suggesting some sort of initial whitening transformation. Further, the decrease rate of the effective dimension deeper in the network corresponds with training performance of the model.
Layer rotation: a surprisingly simple indicator of generalization in deep networks?
Authors: Simon Carbonnelle, Christophe De Vleeschouwer
Date: 2019-05-17 00:00:00
Proceeding: ICML Workshop Deep_Phenomen 2019 6

Our work presents empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance. Compared to previously studied indicators of generalization, we show that layer rotation has the additional benefit of being easily monitored and controlled, as well as having a network-independent optimum: the training procedures during which all layers' weights reach a cosine distance of 1 from their initialization consistently outperform other configurations -by up to 20% test accuracy. Finally, our results also suggest that the study of layer rotation can provide a unified framework to explain the impact of weight decay and adaptive gradient methods on generalization.
Predicting the accuracy of neural networks from final and intermediate layer outputs
Authors: Chad DeChant, Seungwook Han, Hod Lipson
Date: 2019-05-17 00:00:00
Proceeding: ICML Workshop Deep_Phenomen 2019 6

We show that information about whether a neural network's output will be correct or incorrect is present in the outputs of the network's intermediate layers. To demonstrate this effect, we train a new "meta" network to predict from either the final output of the underlying "base" network or the output of one of the base network's intermediate layers whether the base network will be correct or incorrect for a particular input. We find that, over a wide range of tasks and base networks, the meta network can achieve accuracies ranging from 65% - 85% in making this determination.
Emergence of Implicit Filter Sparsity in Convolutional Neural Networks
Authors: Dushyant Mehta, Kwang In Kim, Christian Theobalt
Date: 2019-05-17 00:00:00
Proceeding: ICML Workshop Deep_Phenomen 2019 6
Tasks: L2 Regularization

We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained using adaptive gradient descent techniques with L2 regularization or weight decay. Through an extensive empirical study (Anonymous, 2019) we hypothesize the mechanism be hind the sparsification process. We find that the interplay  of  various  phenomena  influences  the strength of L2 and weight decay regularizers, leading the supposedly non sparsity inducing regularizers to induce filter sparsity.  In this workshop article we summarize some of our key findings and experiments, and present additional results on modern network architectures such as ResNet-50.
Measuring the Spectrum of Deepnet Hessians
Authors: Vardan Papyan
Date: 2019-05-17 00:00:00

We apply state-of-the-art tools in modern high-dimensional numerical linear algebra to approximate efficiently the spectrum of the Hessian of modern deepnets, with tens of millions of parameters, trained on real data. We decompose the Hessian into different components and study the dynamics with training and sample size of each term individually.
Function changes in the Backpropagation equation is equivalent to an implicit learning rate
Authors: Anonymous
Date: 2019-05-17 00:00:00

The backpropagation algorithm is the de-facto standard for credit assignment in artificial neural networks due to its empirical results. Since its conception, variants of the backpropagation algorithm have emerged. More specifically, variants that leverage function changes in the backpropagation equations to satisfy their specific requirements. Feedback Alignment is one such example, which replaces the weight transpose matrix in the backpropagation equations with a random matrix in search of a more biologically plausible credit assignment algorithm. In this work, we show that function changes in the  backpropagation procedure is equivalent to adding an implicit learning rate to an artificial neural network. Furthermore, we learn activation function derivatives in the backpropagation equations to demonstrate early convergence in these artificial neural networks. Our work reports competitive performances with early convergence on MNIST and CIFAR10 on sufficiently large deep neural network architectures.
On Adversarial Robustness of Small vs Large Batch Training
Authors: Sandesh Kamath, Amit Despande, K V Subrahmanyam
Date: 2019-05-17 00:00:00
Tasks: Adversarial Robustness

Large-batch training is known to incur poor generalization by Jastrzebski et al. (2017) as well as poor adversarial robustness by Yao et al. (2018b). Hessian-based analysis of large-batch training by Yao et al. (2018b) concludes that adversarial training as well as small-batch training leads to lower Hessian spectrum. They combine adversarial training and second order information to come up with a new large-batch training algorithm to obtain robust models with good generalization. In this paper, we empirically observe that networks trained with constant learning rate to batch size ratio as proposed by Jastrzebski et al. (2017) not only have better generalization but also have roughly constant adversarial robustness across all batch sizes.
Implicit bias of gradient descent on convolutional generators
Authors: Reinhard Heckel and Paul Hand
Date: 2019-05-17 00:00:00

The majority of image generating neural networks have a convolutional architecture, and 
this architecture has empirically been proven to be efficient at representing natural images. 
It is widely acknowledged that convolutional neural networks incorporate strong prior assumptions about natural images, sometimes referred to as an inductive or implicit bias. 
An instantiation of this bias towards simple, natural images, is that an un-trained overparmeterized convolutional network optimized with gradient descent fits a single natural image significantly faster than complex images such as noise. 
In this paper, we isolate this effect and demonstrate that it is most pronounced if the network incorporates convolutions with a fixed and smooth upsampling kernel. 
This effect can be understood as the network architecture imposing constraints so that an natural image is closer in parameter space to a random initialization than a highly complex image such as noise.
Stochastic algorithms under single spiked models
Authors: Emile Richard
Date: 2019-05-17 00:00:00

We study SGD and Adam for estimating a rank one signal planted in matrix or tensor noise. The extreme simplicity of the problem setup allows us to isolate the effects of various factors: signal to noise ratio, density of critical points, stochasticity and initialization. We observe a surprising phenomenon: Adam seems to get stuck in local minima as soon as polynomially many critical points appear (matrix case), while SGD escapes those. However, when the number of critical points degenerates to exponentials (tensor case), then both algorithms get trapped. Theory tells us that at fixed SNR the problem becomes intractable for large $d$ and in our experiments SGD does not escape this. We exhibit the benefits of warm starting in those situations. We conclude that in this class of problems, warm starting cannot be replaced by stochasticity in gradients to find the basin of attraction.
An Investigation of Memory in Recurrent Neural Networks
Authors: Aude Forcione-Lambert, Guy Wolf, Guillaume Lajoie
Date: 2019-05-17 00:00:00

We investigate the learned dynamical landscape of a recurrent neural network solving a simple task requiring the interaction of two memory mechanisms: long- and short-term. Our results show that while long-term memory is implemented by asymptotic attractors, sequential recall is now additionally implemented by oscillatory dynamics in a transverse subspace to the basins of attraction of these stable steady states. Based on our observations, we propose how different types of memory mechanisms can coexist and work together in a single neural network, and discuss possible applications to the fields of artificial intelligence and neuroscience.
Deep connectomics networks: Results from neural network architectures inspired from network neuroscience
Authors: Nicholas Roberts, Vinay Uday Prabhu
Date: 2019-05-17 00:00:00

Claims from the fields of network neuroscience and connectomics suggest that topological models of the brain involving complex networks are of particular use and interest. The field of deep neural networks has mostly left inspiration from these claims out. In this paper, we propose three architectures and use each of them to explore the intersection of network neuroscience and deep learning in an attempt to bridge the gap between the two fields. Using the teachings from network neuroscience and connectomics, we show improvements over the ResNet architecture, we show a possible connection between early training and the spectral properties of the network, and we show the trainability of a DNN based on the neuronal network of C.Elegans.
Universality Patterns in the Training of Neural Networks
Authors: Raghav Somani, Navin Goyal, Prateek Jain, Praneeth Netrapalli
Date: 2019-05-17 00:00:00

This paper proposes and demonstrates a surprising pattern in the training of neural networks: there is a one to one relation between the values of any pair of losses (such as cross entropy, mean squared error, 0/1 error etc.) evaluated for a model arising at (any point of) a training run. This pattern is universal in the sense that this one to one relationship is identical across architectures (such as VGG, Resnet, Densenet etc.), algorithms (SGD and SGD with momentum) and training loss functions (cross entropy and mean squared error).
On the Importance of Full Rank Initializations in Deep Neural Networks
Authors: Vaibhav B Sinha*, Vishwak Srinivasan*, Vineeth N B
Date: 2019-05-17 00:00:00

Several methods have been proposed over the last few years for initializing the weights of neural networks in order to converge to better solutions or to reduce the time taken for convergence. On the other hand, there have been recent efforts connecting the full rank nature of weight matrices with the optimality of the final converged solution. In this work, we study the connection between popular initialization methods and the conditions necessary at optimal solution using deep linear networks with the squared loss. Through this connection, we attempt to provide a new explanation as to why these different initialization methods work well in practice.