Deep learning's success has led to larger and larger models to handle more and more complex tasks; trained models can contain millions of parameters.These large models are compute- and memory-intensive, which makes it a challenge to deploy them with minimized latency, throughput, and storage requirements.Some model compression methods have been successfully applied on image classification and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks.In this paper, we show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods.We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator.We show that this framework has a compelling performance to high degrees of sparsity, generalizes well to new tasks and models, and enables meaningful comparisons between different pruning granularities.

Bestehende Pruning-Methoden versagen, wenn sie auf GANs angewendet werden, die komplexe Aufgaben bewältigen. Daher stellen wir eine einfache und robuste Methode zum Pruning von Generatoren vor, die für eine Vielzahl von Netzwerken und Aufgaben gut funktioniert.

Die Autoren schlagen eine Modifikation der klassischen Destillationsmethode für die Aufgabe der Komprimierung eines Netzwerks vor, um das Versagen früherer Lösungen bei der Anwendung auf generative adversarische Netzwerke zu beheben.


Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure.The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections.In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth.To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training.We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus.On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB.Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile.

Wir stellen fest, dass 99,9 % des Gradientenaustauschs bei verteiltem SGD überflüssig sind; wir reduzieren die Kommunikationsbandbreite um zwei Größenordnungen, ohne an Genauigkeit zu verlieren.

In dieser Arbeit wird eine zusätzliche Verbesserung gegenüber dem Gradient Dropping vorgeschlagen, um die Kommunikationseffizienz zu steigern.


Image-to-image translation has recently received significant attention due to advances in deep learning.Most works focus on learning either a one-to-one mapping in an unsupervised way or a many-to-many mapping in a supervised way.However, a more practical setting is many-to-many mapping in an unsupervised way, which is harder due to the lack of supervision and the complex inner- and cross-domain variations.To alleviate these issues, we propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain.We assume that an image comprises of a content component which is shared across domains, and a style component specific to each domain.Under the guidance of an exemplar from the target domain we apply Adaptive Instance Normalization to the shared content component, which allows us to transfer the style information of the target domain to the source domain.To avoid semantic inconsistencies during translation that naturally appear due to the large inner- and cross-domain variations, we introduce the concept of feature masks that provide coarse semantic guidance without requiring the use of any semantic labels.Experimental results on various datasets show that EGSC-IT does not only translate the source image to diverse instances in the target domain, but also preserves the semantic consistency during the process.

Wir schlagen das Netzwerk Exemplar Guided & Semantically Consistent Image-to-Image Translation (EGSC-IT) vor, das den Übersetzungsprozess an ein Beispielbild in der Zieldomäne koppelt.

Erörtert ein zentrales Versagen und die Notwendigkeit von I2I-Übersetzungsmodellen.

Die Arbeit untersucht die Idee, dass ein Bild zwei Komponenten hat, und wendet ein Aufmerksamkeitsmodell an, bei dem die Merkmalsmasken, die den Übersetzungsprozess steuern, keine semantischen Bezeichnungen benötigen.


Deep neural networks can learn meaningful representations of data.However, these representations are hard to interpret.For example, visualizing a latent layer is generally only possible for at most three dimensions.Neural networks are able to learn and benefit from much higher dimensional representations but these are not visually interpretable because nodes have arbitrary ordering within a layer.Here, we utilize the ability of the human observer to identify patterns in structured representations to visualize higher dimensions.To do so, we propose a class of regularizations we call \textit{Graph Spectral Regularizations} that impose graph-structure on latent layers.This is achieved by treating activations as signals on a predefined graph and constraining those activations using graph filters, such as low pass and wavelet-like filters.This framework allows for any kind of graph as well as filter to achieve a wide range of structured regularizations depending on the inference needs of the data.First, we show a synthetic example that the graph-structured layer can reveal topological features of the data.Next, we show that a smoothing regularization can impose semantically consistent ordering of nodes when applied to capsule nets.Further, we show that the graph-structured layer, using wavelet-like spatially localized filters, can form localized receptive fields for improved image and biomedical data interpretation.In other words, the mapping between latent layer, neurons and the output space becomes clear due to the localization of the activations.Finally, we show that when structured as a grid, the representations create coherent images that allow for image-processing techniques such as convolutions.

Einfügen von Graphenstrukturen in neuronale Netzschichten zur Verbesserung der visuellen Interpretierbarkeit.

Ein neuartiger Regularisierer, der den verborgenen Schichten eines Neuronalen Netzes eine Graphenstruktur auferlegt, um die Interpretierbarkeit der verborgenen Repräsentationen zu verbessern.

Hervorheben des Beitrags des graphischen spektralen Regularisierers zur Interpretierbarkeit neuronaler Netze.


Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation.The dominant parametric approach is based on locally normalized models which predict one word at a time.While these work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process.In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level.In order to make training tractable, we first work in the residual of a pretrained locally normalized language model and second we train using noise contrastive estimation.Furthermore, since the EBM works at the sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa.Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity compared to locally normalized baselines.Moreover, generation via importance sampling is very efficient and of higher quality than the baseline models according to human evaluation.

Wir zeigen, dass energiebasierte Modelle, die auf den Residuen eines autoregressiven Sprachmodells trainiert werden, effektiv und effizient zur Texterzeugung eingesetzt werden können.

Ein vorgeschlagenes, auf Restenergie basierendes Modell (EBM) für die Texterstellung, das auf Satzebene arbeitet und daher BERT nutzen kann, erreicht eine geringere Komplexität und wird bei der menschlichen Bewertung bevorzugt.


We investigate the robustness properties of image recognition models equipped with two features inspired by human vision, an explicit episodic memory and a shape bias, at the ImageNet scale.As reported in previous work, we show that an explicit episodic memory improves the robustness of image recognition models against small-norm adversarial perturbations under some threat models.It does not, however, improve the robustness against more natural, and typically larger, perturbations.Learning more robust features during training appears to be necessary for robustness in this second sense.We show that features derived from a model that was encouraged to learn global, shape-based representations (Geirhos et al., 2019) do not only improve the robustness against natural perturbations, but when used in conjunction with an episodic memory, they also provide additional robustness against adversarial perturbations.Finally, we address three important design choices for the episodic memory: memory size, dimensionality of the memories and the retrieval method.We show that to make the episodic memory more compact, it is preferable to reduce the number of memories by clustering them, instead of reducing their dimensionality.

Systematische Untersuchung großer cachebasierter Bilderkennungsmodelle mit besonderem Augenmerk auf deren Robustheitseigenschaften.

In dieser Arbeit wurde vorgeschlagen, den Cache-Speicher zu nutzen, um die Robustheit gegenüber ungünstigen Bildbeispielen zu verbessern, und man kam zu dem Schluss, dass die Verwendung eines großen kontinuierlichen Cachespeichers der harten Aufmerksamkeit nicht überlegen ist.


Group convolutional neural networks (G-CNNs) can be used to improve classical CNNs by equipping them with the geometric structure of groups.Central in the success of G-CNNs is the lifting of feature maps to higher dimensional disentangled representations, in which data characteristics are effectively learned, geometric data-augmentations are made obsolete, and predictable behavior under geometric transformations (equivariance) is guaranteed via group theory.Currently, however, the practical implementations of G-CNNs are limited to either discrete groups (that leave the grid intact) or continuous compact groups such as rotations (that enable the use of Fourier theory).In this paper we lift these limitations and propose a modular framework for the design and implementation of G-CNNs for arbitrary Lie groups.In our approach the differential structure of Lie groups is used to expand convolution kernels in a generic basis of B-splines that is defined on the Lie algebra.This leads to a flexible framework that enables localized, atrous, and deformable convolutions in G-CNNs by means of respectively localized, sparse and non-uniform B-spline expansions.The impact and potential of our approach is studied on two benchmark datasets: cancer detection in histopathology slides (PCam dataset) in which rotation equivariance plays a key role and facial landmark localization (CelebA dataset) in which scale equivariance is important.In both cases, G-CNN architectures outperform their classical 2D counterparts and the added value of atrous and localized group convolutions is studied in detail.

Das Papier beschreibt einen flexiblen Rahmen für den Aufbau von CNNs, die äquivariant zu einer großen Klasse von Transformationsgruppen sind.

Ein Rahmenwerk für den Aufbau von Gruppen-CNN mit einer beliebigen Lie Gruppe G, das bei der Klassifizierung von Tumoren und der Lokalisierung von Landmarken eine Überlegenheit gegenüber einem CNN zeigt.


Global feature pooling is a modern variant of feature pooling providing better interpretatability and regularization.Although alternative pooling methods exist (eg. max, lp norm, stochastic), the averaging operation is still the dominating global pooling scheme in popular models.As fine-grained recognition requires learning subtle, discriminative features, we consider the question: is average pooling the optimal strategy?We first ask: ``is there a difference between features learned by global average and max pooling?'' Visualization and quantitative analysis show that max pooling encourages learning features of different spatial scales.We then ask ``is there a single global feature pooling variant that's most suitable for fine-grained recognition?'' A thorough evaluation of nine representative pooling algorithms finds that: max pooling outperforms average pooling consistently across models, datasets, and image resolutions; it does so by reducing the generalization gap; and generalized pooling's performance increases almost monotonically as it changes from average to max.We finally ask: ``what's the best way to combine two heterogeneous pooling schemes?'' Common strategies struggle because of potential gradient conflict but the ``freeze-and-train'' trick works best.We also find that post-global batch normalization helps with faster convergence and improves model performance consistently.

Ein Benchmarking von neun repräsentativen globalen Pooling-Systemen zeigt einige interessante Ergebnisse.

Für feinkörnige Klassifizierungsaufgaben wurde in dieser Arbeit bestätigt, dass maxpooling spärlichere Merkmalskarten als avgpooling fördert und diese übertrifft.


We present a technique to improve the generalization of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions.Although recent research has shown benefits of self-supervised learning (SSL) on large unlabeled datasets, its utility on small datasets is unknown.We find that SSL reduces the relative error rate of few-shot meta-learners by 4%-27%, even when the datasets are small and only utilizing images within the datasets.The improvements are greater when the training set is smaller or the task is more challenging.Though the benefits of SSL may increase with larger training sets, we observe that SSL can have a negative impact on performance when there is a domain shift between distribution of images used for meta-learning and SSL.Based on this analysis we present a technique that automatically select images for SSL from a large, generic pool of unlabeled images for a given dataset using a domain classifier that provides further improvements.We present results using several meta-learners and self-supervised tasks across datasets with varying degrees of domain shifts and label sizes to characterize the effectiveness of SSL for few-shot learning.

Die Selbstüberwachung verbessert die Few-Shot Erkennung in kleinen und schwierigen Datensätzen, ohne auf zusätzliche Daten angewiesen zu sein; zusätzliche Daten helfen nur, wenn sie aus demselben oder einem ähnlichen Bereich stammen.

Eine empirische Studie verschiedener Methoden des selbstüberwachten Lernens (SSL), die zeigt, dass SSL mehr hilft, wenn der Datensatz schwieriger ist, dass der Bereich für das Training wichtig ist und eine Methode zur Auswahl von Proben aus einem unbeschrifteten Datensatz.


Abstraction of Markov Decision Processes is a useful tool for solving complex problems, as it can ignore unimportant aspects of an environment, simplifying the process of learning an optimal policy.In this paper, we propose a new algorithm for finding abstract MDPs in environments with continuous state spaces.It is based on MDP homomorphisms, a structure-preserving mapping between MDPs.We demonstrate our algorithm's ability to learns abstractions from collected experience and show how to reuse the abstractions to guide exploration in new tasks the agent encounters.Our novel task transfer method beats a baseline based on a deep Q-network.

Wir erstellen abstrakte Modelle von Umgebungen aus Erfahrung und nutzen sie, um neue Aufgaben schneller zu lernen.

Eine Methode, die die Idee der MDP-Homomorphismen nutzt, um ein komplexes MDP mit einem kontinuierlichen Zustandsraum in ein einfacheres zu transformieren.


A number of recent methods to understand neural networks have focused on quantifying the role of individual features.  One such method, NetDissect identifies interpretable features of a model using the Broden dataset of visual semantic labels (colors, materials, textures, objects and scenes).  Given the recent rise of a number of action recognition datasets, we propose extending the Broden dataset to include actions to better analyze learned action models.  We describe the annotation process, results from interpreting action recognition models on the extended Broden dataset and examine interpretable feature paths to help us understand the conceptual hierarchy used to classify an action.

Wir erweitern die Netzwerk-Dissektion um die Interpretation von Handlungen und untersuchen interpretierbare Merkmalspfade, um die konzeptuelle Hierarchie zu verstehen, die zur Klassifizierung einer Handlung verwendet wird.


Automatic melody generation for pop music has been a long-time aspiration forboth AI researchers and musicians.However, learning to generate euphoniousmelody has turned out to be highly challenging due to a number of factors.Representationof multivariate property of notes has been one of the primary challenges.It is also difficult to remain in the permissible spectrum of musical variety, outsideof which would be perceived as a plain random play without auditory pleasantness.Observing the conventional structure of pop music poses further challenges.In this paper, we propose to represent each note and its properties as a uniqueâ€˜word,â€™ thus lessening the prospect of misalignments between the properties, aswell as reducing the complexity of learning.We also enforce regularization policieson the range of notes, thus encouraging the generated melody to stay closeto what humans would find easy to follow.Furthermore, we generate melodyconditioned on song part information, thus replicating the overall structure of afull song.Experimental results demonstrate that our model can generate auditorilypleasant songs that are more indistinguishable from human-written ones thanprevious models.

Wir schlagen ein neues Modell zur Darstellung von Noten und ihren Eigenschaften vor, das die automatische Melodieerzeugung verbessern kann.

In diesem Beitrag wird ein generatives Modell der symbolischen (MIDI-)Melodie in der westlichen Populärmusik vorgeschlagen, das Notensymbole zusammen mit Zeit- und Dauerinformationen kodiert, um musikalische "Worte" zu bilden.

In der Arbeit wird vorgeschlagen, die Erzeugung von Melodien zu erleichtern, indem Noten als "Wörter" dargestellt werden, die alle Eigenschaften der Note repräsentieren und so die Erzeugung von musikalischen "Sätzen" ermöglichen.


Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts.We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth.We propose robust growing and stopping policies to generalize to different network architectures and datasets.Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet.For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts.Our AutoGrow is efficient.It discovers depth within similar time of training a single DNN.

Eine Methode, die automatisch Schichten in neuronalen Netzen wachsen lässt, um die optimale Tiefe zu ermitteln.

Ein Rahmen für das Training eines flachen Netzes und das Hinzufügen neuer Schichten, der Einblicke in das Paradigma der "wachsenden Netze" bietet.


Given the importance of remote sensing, surprisingly little attention has been paid to it by the representation learning community.To address it and to speed up innovation in this domain, we provide simplified access to 5 diverse remote sensing datasets in a standardized form.We specifically explore in-domain representation learning and address the question of "what characteristics should a dataset have to be a good source for remote sensing representation learning".The established baselines achieve state-of-the-art performance on these datasets.

Erforschung des bereichsinternen Repräsentationslernens für Fernerkundung von Datensätze.

In dieser Arbeit wurden mehrere standardisierte Fernerkundungsdatensätze bereitgestellt und es wurde gezeigt, dass die bereichsinterne Repräsentation bessere Basisergebnisse für die Fernerkundung liefern kann als das Fine-Tuning mit ImageNet oder das Lernen von Grund auf.


Generative seq2seq dialogue systems are trained to predict the next word in dialogues that have already occurred.They can learn from large unlabeled conversation datasets, build a deep understanding of conversational context, and generate a wide variety of responses.This flexibility comes at the cost of control.Undesirable responses in the training data will be reproduced by the model at inference time, and longer generations often donâ€™t make sense.Instead of generating responses one word at a time, we train a classifier to choose from a predefined list of full responses.The classifier is trained on (conversation context, response class) pairs, where each response class is a noisily labeled group of interchangeable responses.At inference, we generate the exemplar response associated with the predicted response class.Experts can edit and improve these exemplar responses over time without retraining the classifier or invalidating old training data.Human evaluation of 775 unseen doctor/patient conversations shows that this tradeoff improves responses.Only 12% of our discriminative approachâ€™s responses are worse than the doctorâ€™s response in the same conversational context, compared to 18% for the generative model.A discriminative model trained without any manual labeling of response classes achieves equal performance to the generative model.

Vermeiden Sie es, Antworten Wort für Wort zu generieren, indem Sie eine schwache Überwachung verwenden, um einen Klassifikator zu trainieren, der eine vollständige Antwort auswählt.

Eine Möglichkeit zur Generierung von Antworten für medizinische Dialoge unter Verwendung eines Klassifikators zur Auswahl aus von Experten kuratierten Antworten auf der Grundlage des Gesprächskontextes.


There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs).This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP.In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels.We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible. Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical.As a consequence, translation equivariance, beneficial in finite channel CNNs trained with stochastic gradient descent (SGD), is guaranteed to play no role in the Bayesian treatment of the infinite channel limit - a qualitative difference between the two regimes that is not present in the FCN case.We confirm experimentally, that while in some scenarios the performance of SGD-trained finite CNNs approaches that of the corresponding GPs as the channel count increases, with careful tuning SGD-trained CNNs can significantly outperform their corresponding GPs, suggesting advantages from SGD training compared to fully Bayesian parameter estimation.

SGD-trainierte CNNs mit endlicher Breite vs. unendlich breite, vollständig Bayes'sche CNNs. Wer gewinnt?

Die Arbeit stellt eine Verbindung zwischen einem Bayes'schen Convolutional Neural Network mit unendlich vielen Kanälen und Gauß'schen Prozessen her.


Bayesian inference promises to ground and improve the performance of deep neural networks.It promises to be robust to overfitting, to simplify the training procedure and the space of hyperparameters, and to provide a calibrated measure of uncertainty that can enhance decision making, agent exploration and prediction fairness.Markov Chain Monte Carlo (MCMC) methods enable Bayesian inference by generating samples from the posterior distribution over model parameters.Despite the theoretical advantages of Bayesian inference and the similarity between MCMC and optimization methods, the performance of sampling methods has so far lagged behind  optimization methods for large scale deep learning tasks.We aim to fill this gap and introduce ATMC, an adaptive noise MCMC algorithm that estimates and is able to sample from the posterior of a neural network.ATMC dynamically adjusts the amount of momentum and noise applied to each parameter update in order to compensate for the use of stochastic gradients.We use a ResNet architecture without batch normalization to test ATMC on the Cifar10 benchmark and the large scale ImageNet benchmark and show that, despite the  absence of batch normalization, ATMC outperforms a strong optimization baseline in terms of both classification accuracy and test log-likelihood.We show that ATMC is intrinsically robust to overfitting on the training data and that ATMC provides a better calibrated measure of uncertainty compared to the optimization baseline.

Wir skalieren die Bayes'sche Inferenz auf die ImageNet-Klassifikation und erzielen wettbewerbsfähige Ergebnisse hinsichtlich Genauigkeit und Unsicherheitskalibrierung.

Ein adaptiver Noise-MCMC Algorithmus für die Bildklassifikation, der den Impuls und die Störungen, die auf jede Parameteraktualisierung angewendet werden, dynamisch anpasst, robust gegenüber Overfitting ist und ein Unsicherheitsmaß mit Vorhersagen liefert.


Now GANs can generate more and more realistic face images that can easily fool human beings.  In contrast, a common convolutional neural network(CNN), e.g. ResNet-18, can achieve more than 99.9% accuracy in discerning fake/real faces if training and testing faces are from the same source.In this paper, we performed both human studies and CNN experiments, which led us to two important findings.One finding is that the textures of fake faces are substantially different from real ones.CNNs can capture local image texture information for recognizing fake/real face, while such cues are easily overlooked by humans.The other finding is that global image texture information is more robust to image editing and generalizable to fake faces from different GANs and datasets.Based on the above findings, we propose  a  novel  architecture  coined  as  Gram-Net,  which  incorporates  â€œGram Blockâ€ in multiple semantic levels to extract global image texture representations.Experimental results demonstrate that our Gram-Net performs better than existing approaches for fake face detection.  Especially, our Gram-Net is more robust to image editing, e.g.  downsampling, JPEG compression, blur, and noise.  More importantly, our Gram-Net generalizes significantly better in detecting fake faces from GAN models not seen in the training phase.

Eine empirische Studie über gefälschte Bilder zeigt, dass die Textur ein wichtiger Hinweis darauf ist, dass sich gefälschte Bilder von echten Bildern unterscheiden. Unser verbessertes Modell, das globale Texturstatistiken erfasst, zeigt eine bessere GAN-übergreifende Erkennungsleistung für gefälschte Bilder.

Die Arbeit schlägt einen Weg vor, um die Leistung des Modells für die Erkennung gefälschter Gesichter in Bildern, die von einem GAN generiert wurden, zu verbessern, damit es auf der Grundlage von Texturinformationen verallgemeinerbar ist.


The Wasserstein probability metric has received much attention from the machine learning community.Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes.The value of being sensitive to this geometry has been demonstrated, among others, in ordinal regression and generative modelling, and most recently in reinforcement learning.In this paper we describe three natural properties of probability divergences that we believe reflect requirements from machine learning: sum invariance, scale sensitivity, and unbiased sample gradients.The Wasserstein metric possesses the first two properties but, unlike the Kullback-Leibler divergence, does not possess the third.We provide empirical evidence suggesting this is a serious issue in practice.Leveraging insights from probabilistic forecasting we propose an alternative to the Wasserstein metric, the CramÃ©r distance.We show that the CramÃ©r distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences.We give empirical results on a number of domains comparing these three divergences.To illustrate the practical relevance of the CramÃ©r distance we design a new algorithm, the CramÃ©r Generative Adversarial Network (GAN), and show that it has a number of desirable properties over the related Wasserstein GAN.

Die Wasserstein-Distanz ist mit stochastischem Gradientenabstieg schwer zu minimieren, während die Cramer-Distanz leicht optimiert werden kann und genauso gut funktioniert.

In dem Manuskript wird vorgeschlagen, die Cramer-Distanz als Verlust bei der Optimierung einer Zielfunktion mit stochastischem Gradientenabstieg zu verwenden, da sie unverzerrte Stichprobengradienten aufweist.

Der Beitrag des Artikels bezieht sich auf Leistungskriterien, insbesondere auf die Wasserstein/Mallows-Metrik.


We humans have an innate understanding of the asymmetric progression of time, which we use to efficiently and safely perceive and manipulate our environment.Drawing inspiration from that, we approach the problem of learning an arrow of time in a Markov (Decision) Process.We illustrate how a learned arrow of time can capture salient information about the environment, which in turn can be used to measure reachability, detect side-effects and to obtain an intrinsic reward signal.Finally, we propose a simple yet effective algorithm to parameterize the problem at hand and learn an arrow of time with a function approximator (here, a deep neural network).Our empirical results span a selection of discrete and continuous environments, and demonstrate for a class of stochastic processes that the learned arrow of time agrees reasonably well with a well known notion of an arrow of time due to Jordan, Kinderlehrer and Otto (1998).

Wir lernen den Pfeil der Zeit für MDPs und nutzen ihn, um die Erreichbarkeit zu messen, Nebenwirkungen zu erkennen und ein Neugier-Belohnungssignal zu erhalten.

In dieser Arbeit wird das h-Potenzial als Lösung für ein Ziel vorgeschlagen, das die Asymmetrie der Zustandsübergänge in einem MDP misst.


We formulate stochastic gradient descent (SGD) as a novel factorised Bayesian filtering problem, in which each parameter is inferred separately, conditioned on the corresopnding backpropagated gradient.  Inference in this setting naturally gives rise to BRMSprop and BAdam: Bayesian variants of RMSprop and Adam.  Remarkably, the Bayesian approach recovers many features of state-of-the-art adaptive SGD methods, including amongst others root-mean-square normalization, Nesterov acceleration and AdamW.  As such, the Bayesian approach provides one explanation for the empirical effectiveness of state-of-the-art adaptive SGD algorithms.  Empirically comparing BRMSprop and BAdam with naive RMSprop and Adam on MNIST, we find that Bayesian methods have the potential to considerably reduce test loss and classification error.

Wir haben SGD als ein Bayes'sches Filterproblem formuliert und zeigen, dass dies zu RMSprop, Adam, AdamW, NAG und anderen Merkmalen moderner adaptiver Methoden führt.

Das Papier analysiert den stochastischen Gradientenabstieg durch Bayes'sche Filterung als Rahmen für die Analyse adaptiver Methoden.

Die Autoren versuchen, bestehende adaptive Gradientenmethoden im Rahmen der Bayes'schen Filterung mit dem dynamischen Prior zu vereinen.


Data augmentation (DA) has been widely utilized to improve generalization in training deep neural networks.Recently, human-designed data augmentation has been gradually replaced by automatically learned augmentation policy.Through finding the best policy in well-designed search space of data augmentation, AutoAugment (Cubuk et al., 2019) can significantly improve validation accuracy on image classification tasks.However, this approach is not computationally practical for large-scale problems.In this paper, we develop an adversarial method to arrive at a computationally-affordable solution called Adversarial AutoAugment, which can simultaneously optimize target related object and augmentation policy search loss.The augmentation policy network attempts to increase the training loss of a target network through generating adversarial augmentation policies, while the target network can learn more robust features from harder examples to improve the generalization.In contrast to prior work, we reuse the computation in target network training for policy evaluation, and dispense with the retraining of the target network.Compared to AutoAugment, this leads to about 12x reduction in computing cost and 11x shortening in time overhead on ImageNet.We show experimental results of our approach on CIFAR-10/CIFAR-100, ImageNet, and demonstrate significant performance improvements over state-of-the-art.On CIFAR-10, we achieve a top-1 test error of 1.36%, which is the currently best performing single model.On ImageNet, we achieve a leading performance of top-1 accuracy 79.40% on ResNet-50 and 80.00% on ResNet-50-D without extra data.

Wir führen die Idee des kontradiktorischen Lernens in die automatische Datenerweiterung ein, um die Generalisierung eines Zielnetzes zu verbessern.

Eine Technik namens Adversarial AutoAugment, die während des Trainings mit Hilfe eines adversariellen Ansatzes dynamisch gute Datenerweiterungsstrategien erlernt.


In this study we focus on first-order meta-learning algorithms that aim to learn a parameter initialization of a network which can quickly adapt to new concepts, given a few examples.We investigate two approaches to enhance generalization and speed of learning of such algorithms, particularly expanding on the Reptile (Nichol et al., 2018) algorithm.We introduce a novel regularization technique called meta-step gradient pruning and also investigate the effects of increasing the depth of network architectures in first-order meta-learning.We present an empirical evaluation of both approaches, where we match benchmark few-shot image classification results with 10 times fewer iterations using Mini-ImageNet dataset and with the use of deeper networks, we attain accuracies that surpass the current benchmarks of few-shot image classification using Omniglot dataset.

Die Studie stellt zwei Ansätze zur Verbesserung der Generalisierung von Meta-Lernen erster Ordnung vor und präsentiert eine empirische Evaluierung zur Few-Shot Klassifizierung von Bildern.

Die Arbeit präsentiert eine empirische Studie des Meta-Lernalgorithmus erster Ordnung Reptile Algorithmus, der eine vorgeschlagene Regularisierungstechnik und tiefere Netzwerke untersucht.


In this paper, we propose the use of in-training matrix factorization to reduce the model size for neural machine translation.Using in-training matrix factorization, parameter matrices may be decomposed into the products of smaller matrices, which can compress large machine translation architectures by vastly reducing the number of learnable parameters.We apply in-training matrix factorization to different layers of standard neural architectures and show that in-training factorization is capable of reducing nearly 50% of learnable parameters without any associated loss in BLEU score.Further, we find that in-training matrix factorization is especially powerful on embedding layers, providing a simple and effective method to curtail the number of parameters with minimal impact on model performance, and, at times, an increase in performance.

In dieser Arbeit wird vorgeschlagen, die Matrixfaktorisierung zur Trainingszeit für die neuronale maschinelle Übersetzung zu verwenden, wodurch die Modellgröße und die Trainingszeit ohne Leistungseinbußen verringert werden können.

In dieser Arbeit wird vorgeschlagen, Modelle mit Hilfe der Matrixfaktorisierung während des Trainings für tiefe neuronale Netze der maschinellen Übersetzung zu komprimieren.


Though state-of-the-art sentence representation models can perform tasks requiring significant knowledge of grammar, it is an open question how best to evaluate their grammatical knowledge.We explore five experimental methods inspired by prior work evaluating pretrained sentence representation models.We use a single linguistic phenomenon, negative polarity item (NPI) licensing, as a case study for our experiments.NPIs like 'any' are grammatical only if they appear in a licensing environment like negation ('Sue doesn't have any cats' vs. '*Sue has any cats').This phenomenon is challenging because of the variety of NPI licensing environments that exist.We introduce an artificially generated dataset that manipulates key features of NPI licensing for the experiments.We find that BERT has significant knowledge of these features, but its success varies widely across different experimental methods.We conclude that a variety of methods is necessary to reveal all relevant aspects of a model's grammatical knowledge in a given domain.

Verschiedene Methoden zur Analyse des BERT führen zu unterschiedlichen (aber kompatiblen) Schlussfolgerungen in einer Fallstudie über NPIs.


The primate visual system builds robust, multi-purpose representations of the external world in order to support several diverse downstream cortical processes.Such representations are required to be invariant to the sensory inconsistencies caused by dynamically varying lighting, local texture distortion, etc.A key architectural feature combating such environmental irregularities is â€˜long-range horizontal connectionsâ€™ that aid the perception of the global form of objects.In this work, we explore the introduction of such horizontal connections into standard deep convolutional networks; we present V1Net -- a novel convolutional-recurrent unit that models linear and nonlinear horizontal inhibitory and excitatory connections inspired by primate visual cortical connectivity.We introduce the Texturized Challenge -- a new benchmark to evaluate object recognition performance under perceptual noise -- which we use to evaluate V1Net against an array of carefully selected control models with/without recurrent processing.Additionally, we present results from an ablation study of V1Net demonstrating the utility of diverse neurally inspired horizontal connections for state-of-the-art AI systems on the task of object boundary detection from natural images.We also present the emergence of several biologically plausible horizontal connectivity patterns, namely center-on surround-off, association fields and border-ownership connectivity patterns in a V1Net model trained to perform boundary detection on natural images from the Berkeley Segmentation Dataset 500 (BSDS500).Our findings suggest an increased representational similarity between V1Net and biological visual systems, and highlight the importance of neurally inspired recurrent contextual processing principles for learning visual representations that are robust to perceptual noise and furthering the state-of-the-art in computer vision.

In dieser Arbeit stellen wir V1Net vor - ein neuartiges rekurrentes neuronales Netzwerk, das kortikale horizontale Verbindungen modelliert, die zu robusten visuellen Repräsentationen durch Wahrnehmungsgruppierung führen.

Die Autoren schlagen vor, eine Convolutional Variante des LSTM zu modifizieren, um horizontale Verbindungen einzubeziehen, die von bekannten Interaktionen im visuellen Kortex inspiriert sind.


Humans understand novel sentences by composing meanings and roles of core language components.In contrast, neural network models for natural language modeling fail when such compositional generalization is required.The main contribution of this paper is to hypothesize that language compositionality is a form of group-equivariance.Based on this hypothesis, we propose a set of tools for constructing equivariant sequence-to-sequence models.Throughout a variety of experiments on the SCAN tasks, we analyze the behavior of existing models under the lens of equivariance, and demonstrate that our equivariant architecture is able to achieve the type compositional generalization required in human language understanding.

Wir schlagen eine Verbindung zwischen Permutationsäquivarianz und kompositioneller Verallgemeinerung vor und bieten äquivariante Sprachmodelle

Diese Arbeit konzentriert sich auf das Lernen von lokal äquivarianten Repräsentationen und Funktionen über Eingabe-/Ausgabewörter für die Zwecke der SCAN-Aufgabe.


Variational inference (VI) is a popular approach for approximate Bayesian inference that is particularly promising for highly parameterized models such as deep neural networks.  A key challenge of variational inference is to approximate the posterior over model parameters with a distribution that is simpler and tractable yet sufficiently expressive.In this work, we propose a method for training highly flexible variational distributions by starting with a coarse approximation and iteratively refining it.Each refinement step makes cheap, local adjustments and only requires optimization of simple variational families.We demonstrate theoretically that our method always improves a bound on the approximation (the Evidence Lower BOund) and observe this empirically across a variety of benchmark tasks.  In experiments, our method consistently outperforms recent variational inference methods for deep learning in terms of log-likelihood and the ELBO.  We see that the gains are further amplified on larger scale models, significantly outperforming standard VI and deep ensembles on residual networks on CIFAR10.

Das Papier schlägt einen Algorithmus vor, um die Flexibilität des Variationsposteriores in Bayesschen Neuronalen Netzen durch iterative Optimierung zu erhöhen.

Eine Methode zum Trainieren flexibler Variationsposteriori-Verteilungen, angewandt auf Bayes'sche neuronale Netze, um Variationsinferenz (VI) über die Gewichte durchzuführen.


In this paper, we propose a residual non-local attention network for high-quality image restoration.Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial- and channel-wise features.To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts.Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention block.The trunk branch is used to extract hierarchical features.Local and non-local mask branches aim to adaptively rescale these hierarchical features with mixed attentions.The local mask branch concentrates on more local structures with convolutional operations, while non-local attention considers more about long-range dependencies in the whole feature map.Furthermore, we propose residual local and non-local attention learning to train the very deep network, which further enhance the representation ability of the network.Our proposed method can be generalized for various image restoration applications, such as image denoising, demosaicing, compression artifacts reduction, and super-resolution.Experiments demonstrate that our method obtains comparable or better results compared with recently leading methods quantitatively and visually.

Neuer hochmoderner Rahmen für die Bildwiederherstellung.

In dem Artikel wird eine Architektur für ein Convolutional Neural Network vorgeschlagen, das Blöcke für lokale und nicht-lokale Aufmerksamkeitsmechanismen enthält, die angeblich für die Erzielung hervorragender Ergebnisse in vier Bildwiederherstellungsanwendungen verantwortlich sind.

In diesem Beitrag wird ein nicht-lokales Aufmerksamkeitsnetz für die Bildwiederherstellung vorgeschlagen.


Most approaches to learning action planning models heavily rely on a significantly large volume of training samples or plan observations.In this paper, we adopt a different approach based on deductive learning from domain-specific knowledge, specifically from logic formulae that specify constraints about the possible states of a given domain.The minimal input observability required by our approach is a single example composed of a full initial state and a partial goal state.We will show that exploiting specific domain knowledge enables to constrain the space of possible action models as well as to complete partial observations, both of which turn out helpful to learn good-quality action models.

Hybrider Ansatz zur Modellerstellung, der einen Mangel an verfügbaren Daten durch bereichsspezifisches Wissen von Experten ausgleicht.

Ein Domänenerfassungsansatz, der eine andere Darstellung des partiellen Domänenmodells in Betracht zieht, indem er schematische Mutex-Relationen anstelle von Vor- und Nachbedingungen verwendet.


We release the largest public ECG dataset of continuous raw signals for representation learning containing over 11k patients and 2 billion labelled beats.Our goal is to enable semi-supervised ECG models to be made as well as to discover unknown subtypes of arrhythmia and anomalous ECG signal events.To this end, we propose an unsupervised representation learning task, evaluated in a semi-supervised fashion.  We provide a set of baselines for different feature extractors that can be built upon.  Additionally, we perform qualitative evaluations on results from PCA embeddings, where we identify some clustering of known subtypes indicating the potential for representation learning in arrhythmia sub-type discovery.

Wir veröffentlichen einen Datensatz, der aus Einleitungs-EKG-Daten von 11.000 Patienten besteht, denen das {DEVICENAME}(TM)-Gerät verschrieben wurde.

Diese Arbeit beschreibt einen großen EKG-Datensatz, den die Autoren zu veröffentlichen beabsichtigen, und bietet eine unüberwachte Analyse und Visualisierung des Datensatzes.


As the basic building block of Convolutional Neural Networks (CNNs), the convolutional layer is designed to extract local patterns and lacks the ability to model global context in its nature.Many efforts have been recently made to complement CNNs with the global modeling ability, especially by a family of works on global feature interaction.In these works, the global context information is incorporated into local features before they are fed into convolutional layers.However, research on neuroscience reveals that, besides influences changing the inputs to our neurons, the neurons' ability of modifying their functions dynamically according to context is essential for perceptual tasks, which has been overlooked in most of CNNs.Motivated by this, we propose one novel Context-Gated Convolution (CGC) to explicitly modify the weights of convolutional layers adaptively under the guidance of global context.As such, being aware of the global context, the modulated convolution kernel of our proposed CGC can better extract representative local patterns and compose discriminative features.Moreover, our proposed CGC is lightweight, amenable to modern CNN architectures, and consistently improves the performance of CNNs according to extensive experiments on image classification, action recognition, and machine translation.

Eine neuartige Context-Gated Convolution, die durch explizite Modulation von Convolutional Kernels globale Kontextinformationen in CNNs einbezieht und so repräsentativere lokale Muster erfasst und diskriminierende Merkmale extrahiert.

In dieser Arbeit wird globaler Kontext verwendet, um die Gewichte von Convolutional Layers zu modulieren und CNNs dabei zu helfen, diskriminantere Merkmale mit hoher Leistung und weniger Parametern zu erfassen als bei der Modulation von Merkmalskarten.


We analyze the trade-off between quantization noise and clipping distortion in low precision networks.We identify the statistics of various tensors, and derive exact expressions for the mean-square-error degradation due to clipping.By optimizing these expressions, we show marked improvements over standard quantization schemes that normally avoid clipping.For example, just by choosing the accurate clipping values, more than 40\% accuracy improvement is obtained for the quantization of VGG-16 to 4-bits of precision.Our results have many applications for the quantization of neural networks at both training and inference time.

Wir analysieren den Kompromiss zwischen Quantisierungsstörungen und Clipping-Verzerrung in Netzwerken mit geringer Präzision und zeigen deutliche Verbesserungen gegenüber Standard-Quantisierungsschemata, die normalerweise Clipping vermeiden.

Leitet eine Formel zur Ermittlung der minimalen und maximalen Clipping-Werte für eine gleichmäßige Quantisierung ab, die den quadratischen Fehler, der sich aus der Quantisierung ergibt, entweder für eine Laplace- oder Gauß-Verteilung über den vorquantisierten Wert minimiert.


Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field.But its performance can awfully degrade with insufficient batch size.This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption.Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption.In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before.The extra batch statistics associated with gradients also can severely affect the training of deep neural network.Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN).MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure.We prove the benefits of MABN by both theoretical analysis and experiments.Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO.The code has been released in https://github.com/megvii-model/MABN.

Wir schlagen eine neuartige Normalisierungsmethode vor, um Fälle mit kleinen Batch Größen zu behandeln.

Eine Methode zur Bewältigung des Problems der kleinen Batch Größe von BN, die den gleitenden Mittelwert ohne allzu großen Aufwand anwendet und die Anzahl der Statistiken von BN für eine bessere Stabilität reduziert.


We present a simple proof for the benefit of depth in multi-layer feedforward network with rectifed activation (``"depth separation").Specifically we present a sequence of classification problems f_i such that(a) for any fixed depth rectified network we can find an index m such that problems with index > m require exponential network width to fully represent the function f_m; and(b) for any problem f_m in the family, we present a concrete neural network with linear depth and bounded width that fully represents it.While there are several previous work showing similar results, our proof uses substantially simpler tools and techniques, and should be accessible to undergraduate students in computer science and people with similar backgrounds.

ReLU MLP Tiefenseparationsbeweis mit gemoterischen Argumenten.

Ein Beweis dafür, dass tiefere Netze weniger Einheiten benötigen als flachere für eine Familie von Problemen.


The rich and accessible labeled data fuel the revolutionary success of deep learning.Nonetheless, massive supervision remains a luxury for many real applications, boosting great interest in label-scarce techniques such as few-shot learning (FSL).An intuitively feasible approach to FSL is to conduct data augmentation via synthesizing additional training samples.The key to this approach is how to guarantee both discriminability and diversity of the synthesized samples.In this paper, we propose a novel FSL model, called $\textrm{D}^2$GAN, which synthesizes Diverse and Discriminative features based on Generative Adversarial Networks (GAN).$\textrm{D}^2$GAN secures discriminability of the synthesized features by constraining them to have high correlation with real features of the same classes while low correlation with those of different classes.  Based on the observation that noise vectors that are closer in the latent code space are more likely to be collapsed into the same mode when mapped to feature space, $\textrm{D}^2$GAN incorporates a novel anti-collapse regularization term, which encourages feature diversity by penalizing the ratio of the logarithmic similarity of two synthesized features and the logarithmic similarity of the latent codes generating them.Experiments on three common benchmark datasets verify the effectiveness of $\textrm{D}^2$GAN by comparing with the state-of-the-art.

Ein neuer GAN-basierter Few-Shot Lernalgorithmus durch Synthese verschiedener und diskriminierender Merkmale.

Eine Meta-Lernmethode, die ein generatives Modell erlernt, das die Unterstützungsmenge eines Few-Shot Lerners, der eine Kombination von Verlusten optimiert, erweitern kann.


The lack of crisp mathematical models that capture the structure of real-worlddata sets is a major obstacle to the detailed theoretical understanding of deepneural networks.Here, we first demonstrate the effect of structured data setsby experimentally comparing the dynamics and the performance of two-layernetworks trained on two different data sets:(i) an unstructured synthetic dataset containing random i.i.d. inputs, and(ii) a simple canonical data set suchas MNIST images.Our analysis reveals two phenomena related to the dynamics ofthe networks and their ability to generalise that only appear when training onstructured data sets.Second, we introduce a generative model for data sets,where high-dimensional inputs lie on a lower-dimensional manifold and havelabels that depend only on their position within this manifold.We call it the*hidden manifold model* and we experimentally demonstrate that trainingnetworks on data sets drawn from this model reproduces both the phenomena seenduring training on MNIST.

Wir zeigen, wie sich die Struktur von Datensätzen auf neuronale Netze auswirkt, und stellen ein generatives Modell für synthetische Datensätze vor, das diese Auswirkungen reproduziert.

In dem Beitrag wird untersucht, wie sich verschiedene Einstellungen der Datenstruktur auf das Lernen neuronaler Netze auswirken und wie das Verhalten auf realen Datensätzen beim Lernen auf einem synthetischen Datensatz nachgeahmt werden kann.


In this paper, we study deep diagonal circulant neural networks, that is deep neural networks in which weight matrices are the product of diagonal and circulant ones.Besides making a theoretical analysis of their expressivity, we introduced principled techniques for training these models: we devise an initialization scheme and proposed a smart use of non-linearity functions in order to train deep diagonal circulant networks. Furthermore, we show that these networks outperform recently introduced deep networks with other types of structured layers.We conduct a thorough experimental study to compare the performance of deep diagonal circulant networks with state of the art models based on structured matrices and with dense models.We show that our models achieve better accuracy than other structured approaches while required 2x fewer weights as the next best approach.Finally we train deep diagonal circulant networks to build a compact and accurate models on a real world video classification dataset with over 3.8 million training examples.

Wir trainieren tiefe neuronale Netze, die auf diagonalen und zirkulanten Matrizen basieren, und zeigen, dass diese Art von Netzen sowohl kompakt als auch genau in realen Anwendungen sind.

Die Autoren liefern eine theoretische Analyse der Ausdruckskraft von diagonalen zirkulanten neuronalen Netzen (DCNN) und schlagen ein Initialisierungsschema für tiefe DCNNs vor.


Interpretability has largely focused on local explanations, i.e. explaining why a model made a particular prediction for a sample.These explanations are appealing due to their simplicity and local fidelity.However, they do not provide information about the general behavior of the model.We propose to leverage model distillation to learn global additive explanations that describe the relationship between input features and model predictions.These global explanations take the form of feature shapes, which are more expressive than feature attributions.Through careful experimentation, we show qualitatively and quantitatively that global additive explanations are able to describe model behavior and yield insights about models such as neural nets.A visualization of our approach applied to a neural net as it is trained is available at https://youtu.be/ErQYwNqzEdc

Wir schlagen vor, die Modelldestillation zu nutzen, um globale additive Erklärungen in Form von Merkmalsformen (die aussagekräftiger sind als Merkmalszuweisungen) für Modelle wie neuronale Netze zu lernen, die auf tabellarischen Daten trainiert werden.

In diesem Beitrag werden verallgemeinerte additive Modelle (GAMs) mit Modelldestillation eingesetzt, um globale Erklärungen für neuronale Netze zu liefern.


A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner.These representations are typically used as general purpose features for words across a range of NLP problems.However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem.Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations.In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences.Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods.We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

Ein groß angelegter Multi-Task Lernrahmen mit verschiedenen Trainingszielen zum Erlernen von Satzrepräsentationen fester Länge.

In dieser Arbeit geht es um das Erlernen von Satzeinbettungen durch die Kombination verschiedener Trainingssignale: Überspringen von Gedanken, Vorhersage von Übersetzungen, Klassifizierung von Entailment-Beziehungen und Vorhersage der Konstituentenparse.


In a time where neural networks are increasingly adopted in sensitive applications, algorithmic bias has emerged as an issue with moral implications.While there are myriad ways that a system may be compromised by bias, systematically isolating and evaluating existing systems on such scenarios is non-trivial, i.e., bias may be subtle, natural and inherently difficult to quantify.To this end, this paper proposes the first systematic study of benchmarking state-of-the-art neural models against biased scenarios.More concretely, we postulate that the bias annotator problem can be approximated with neural models, i.e., we propose generative models of latent bias to deliberately and unfairly associate latent features to a specific class.All in all, our framework provides a new way for principled quantification and evaluation of models against biased datasets.Consequently, we find that state-of-the-art NLP models (e.g., BERT, RoBERTa, XLNET) are readily compromised by biased data.

Wir schlagen einen neuronalen Bias Annotator vor, um Modelle auf ihre Robustheit gegenüber verzerrten Textdatensätzen zu testen.

Eine Methode zur Erzeugung voreingenommener Datensätze für NLP, die auf einem bedingten, adversarial regularisierten Autoencoder (CARA) beruht.


We consider the problem of topic modeling in a weakly semi-supervised setting.In this scenario, we assume that the user knows a priori a subset of the topics she wants the model to learn and is able to provide a few exemplar documents for those topics.In addition, while each document may typically consist of multiple topics, we do not assume that the user will identify all its topics exhaustively.      Recent state-of-the-art topic models such as NVDM, referred to herein as Neural Topic Models (NTMs), fall under the variational autoencoder framework.We extend NTMs to the weakly semi-supervised setting by using informative priors in the training objective.After analyzing the effect of informative priors, we propose a simple modification of the NVDM model using a logit-normal posterior that we show achieves better alignment to user-desired topics versus other NTM models.

Wir schlagen vor, Themenmodelle im VAE-Stil zu überwachen, indem wir den Prior auf intelligenter Basis pro Dokument anpassen. Wir finden, dass ein Logit-Normal-Posterior die beste Leistung liefert.

Eine flexible Methode zur schwachen Überwachung eines Themenmodells, um eine bessere Anpassung an die Intuition der Benutzer zu erreichen.


Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability.However, it is by no means obvious how to estimate mutual information (MI) between each hidden layer and the input/desired output, to construct the IP.For instance, hidden layers with many neurons require MI estimators with robustness towards the high dimensionality associated with such layers.MI estimators should also be able to naturally handle convolutional layers, while at the same time being computationally tractable to scale to large networks.None of the existing IP methods to date have been able to study truly deep Convolutional Neural Networks (CNNs), such as the e.g.\ VGG-16.In this paper, we propose an IP analysis using the new matrix--based R\'enyi's entropy coupled with tensor kernels over convolutional layers, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data.The obtained results shed new light on the previous literature concerning small-scale DNNs, however using a completely new approach.Importantly, the new framework enables us to provide the first comprehensive IP analysis of contemporary large-scale DNNs and CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.

Erste umfassende Analyse der Informationsebene von großen tiefen neuronalen Netzen unter Verwendung von matrixbasierter Entropie und Tensorkernels.

Die Autoren schlagen einen Tensor-Kernel-basierten Schätzer für die Schätzung der gegenseitigen Information zwischen hochdimensionalen Schichten in einem neuronalen Netz vor.


Developing agents that can learn to follow natural language instructions has been an emerging research direction.While being accessible and flexible, natural language instructions can sometimes be ambiguous even to humans.To address this, we propose to utilize programs, structured in a formal language, as a precise and expressive way to specify tasks.We then devise a modular framework that learns to perform a task specified by a program â€“ as different circumstances give rise to diverse ways to accomplish the task, our framework can perceive which circumstance it is currently under, and instruct a multitask policy accordingly to fulfill each subtask of the overall task.Experimental results on a 2D Minecraft environment not only demonstrate that the proposed framework learns to reliably accomplish program instructions and achieves zero-shot generalization to more complex instructions but also verify the efficiency of the proposed modulation mechanism for learning the multitask policy.We also conduct an analysis comparing various models which learn from programs and natural language instructions in an end-to-end fashion.

Wir schlagen einen modularen Rahmen vor, der durch Programme spezifizierte Aufgaben bewältigen kann und eine Zero-Shot Verallgemeinerung auf komplexere Aufgaben ermöglicht.

Diese Arbeit untersucht das Training von RL-Agenten mit Anweisungen und Aufgabendekompositionen, die als Programme formalisiert sind, und schlägt ein Modell für einen programmgesteuerten Agenten vor, der ein Programm interpretiert und Teilziele für ein Aktionsmodul vorschlägt.


We analyze the convergence of (stochastic) gradient descent algorithm for learning a convolutional filter with Rectified Linear Unit (ReLU) activation function.Our analysis does not rely on any specific form of the input distribution and our proofs only use the definition of ReLU, in contrast with previous works that are restricted to standard Gaussian input.We show that (stochastic) gradient descent with random initialization can learn the convolutional filter in polynomial time and the convergence rate depends on the smoothness of the input distribution and the closeness of patches.To the best of our knowledge, this is the first recovery guarantee of gradient-based algorithms for convolutional filter on non-Gaussian input distributions.Our theory also justifies the two-stage learning rate strategy in deep neural networks.While our focus is theoretical, we also present experiments that justify our theoretical findings.

Wir beweisen, dass ein zufällig initialisierter (stochastischer) Gradientenabstieg einen Convolutional Filter in polynomieller Zeit lernt.

Untersucht das Problem des Lernens eines einzelnen Convolutional Filter mit SGD und zeigt, dass SGD unter bestimmten Bedingungen einen einzelnen Convolutional Filter lernt.

In dieser Arbeit wird die Annahme der Gaußschen Verteilung auf eine allgemeinere Annahme der Winkelglätte erweitert, die eine größere Familie von Eingangsverteilungen abdeckt.


Deep neural networks (DNNs) are widely adopted in real-world cognitive applications because of their high accuracy.The robustness of DNN models, however, has been recently challenged by adversarial attacks where small disturbance on input samples may result in misclassification.State-of-the-art defending algorithms, such as adversarial training or robust optimization, improve DNNs' resilience to adversarial attacks by paying high computational costs.Moreover, these approaches are usually designed to defend one or a few known attacking techniques only.The effectiveness to defend other types of attacking methods, especially those that have not yet been discovered or explored, cannot be guaranteed.This work aims for a general approach of enhancing the robustness of DNN models under adversarial attacks.In particular, we propose Bamboo -- the first data augmentation method designed for improving the general robustness of DNN without any hypothesis on the attacking algorithms.Bamboo augments the training data set with a small amount of data uniformly sampled on a fixed radius ball around each training data and hence, effectively increase the distance between natural data points and decision boundary.Our experiments show that Bamboo substantially improve the general robustness against arbitrary types of attacks and noises, achieving better results comparing to previous adversarial training methods, robust optimization methods and other data augmentation methods with the same amount of data points.

Die erste Datenerweiterungsmethode, die speziell zur Verbesserung der allgemeinen Robustheit von DNN entwickelt wurde, ohne Hypothesen über die angreifenden Algorithmen aufzustellen.

Es wird eine Trainingsmethode zur Datenerweiterung vorgeschlagen, um die Robustheit des Modells gegenüber Störungen durch den Gegner zu erhöhen, indem gleichmäßige Zufallsstichproben aus einer Kugel mit festem Radius, die auf die Trainingsdaten zentriert ist, erweitert werden.


The ability to synthesize realistic patterns of neural activity is crucial for studying neural information processing.Here we used the Generative Adversarial Networks (GANs) framework to simulate the concerted activity of a population of neurons.We adapted the Wasserstein-GAN variant to facilitate the generation of unconstrained neural population activity patterns while still benefiting from parameter sharing in the temporal domain.We demonstrate that our proposed GAN, which we termed Spike-GAN, generates spike trains that match accurately the first- and second-order statistics of datasets of tens of neurons and also approximates well their higher-order statistics.We applied Spike-GAN to a real dataset recorded from salamander retina and showed that it performs as well as state-of-the-art approaches based on the maximum entropy and the dichotomized Gaussian frameworks.Importantly, Spike-GAN does not require to specify a priori the statistics to be matched by the model, and so constitutes a more flexible method than these alternative approaches.Finally, we show how to exploit a trained Spike-GAN  to construct 'importance maps' to detect the most relevant statistical structures present in a spike train. Spike-GAN provides a powerful, easy-to-use technique for generating realistic spiking neural activity and for describing the most relevant features of the large-scale neural population recordings studied in modern systems neuroscience.

Verwendung von Wasserstein-GANs zur Erzeugung realistischer neuronaler Aktivität und zur Erkennung der wichtigsten Merkmale in neuronalen Populationsmustern.

Eine Methode zur Simulation von Spike Trains von Neuronenpopulationen, die mit empirischen Daten übereinstimmen, unter Verwendung eines semi-convolutional GAN.

In der Arbeit wird vorgeschlagen, GANs für die Synthese realistischer neuronaler Aktivitätsmuster zu verwenden.


Deep latent variable models have become a popular model choice due to the scalable learning algorithms introduced by (Kingma & Welling 2013, Rezende et al. 2014).These approaches maximize a variational lower bound on the intractable log likelihood of the observed data.Burda et al. (2015) introduced a multi-sample variational bound, IWAE, that is at least as tight as the standard variational lower bound and becomes increasingly tight as the number of samples increases.Counterintuitively, the typical inference network gradient estimator for the IWAE bound performs poorly as the number of samples increases (Rainforth et al. 2018, Le et al. 2018).Roeder et a.(2017) propose an improved gradient estimator, however, are unable to show it is unbiased.We show that it is in fact biased and that the bias can be estimated efficiently with a second application of the reparameterization trick.The doubly reparameterized gradient (DReG) estimator does not suffer as the number of samples increases, resolving the previously raised issues.The same idea can be used to improve many recently introduced training techniques for latent variable models.In particular, we show that this estimator reduces the variance of the IWAE gradient, the reweighted wake-sleep update (RWS) (Bornschein & Bengio 2014), and the jackknife variational inference (JVI) gradient (Nowozin 2018).Finally, we show that this computationally efficient, drop-in estimator translates to improved performance for all three objectives on several modeling tasks.

Doppelt reparametrisierte Gradientenschätzer bieten eine unvoreingenommene Varianzreduktion, die zu einer verbesserten Leistung führt.

Der Autor fand experimentell heraus, dass der Schätzer der bestehenden Arbeit (STL) voreingenommen ist und schlägt vor, die Voreingenommenheit zu reduzieren, um den Gradientenschätzer des ELBO zu verbessern.


Zeroth-order optimization is the process of minimizing an objective $f(x)$, given oracle access to evaluations at adaptively chosen inputs $x$.In this paper, we present two simple yet powerful GradientLess Descent (GLD) algorithms that do not rely on an underlying gradient estimate and are numerically stable.We analyze our algorithm from a novel geometric perspective and we show that for {\it any monotone transform} of a smooth and strongly convex objective with latent dimension $k \ge n$, we present a novel analysis that shows convergence within an $\epsilon$-ball of the optimum in $O(kQ\log(n)\log(R/\epsilon))$ evaluations, where the input dimension is $n$, $R$ is the diameter of the input space and $Q$ is the condition number.Our rates are the first of its kind to be both1) poly-logarithmically dependent on dimensionality and2) invariant under monotone transformations.We further leverage our geometric perspective to show that our analysis is optimal.Both monotone invariance and its ability to utilize a low latent dimensionality are key to the empirical success of our algorithms, as demonstrated on synthetic and MuJoCo benchmarks.

Gradientless Descent ist ein nachweislich effizienter gradientenfreier Algorithmus, der monoton-invariant und schnell für hochdimensionale Optimierung nullter Ordnung ist.

Diese Arbeit schlägt stabile GradientLess Descent (GLD) Algorithmen vor, die sich nicht auf eine Gradientenschätzung verlassen.


Many processes can be concisely represented as a sequence of events leading from a starting state to an end state.Given raw ingredients, and a finished cake, an experienced chef can surmise the recipe.Building upon this intuition,  we propose a new class of visual generative models: goal-conditioned predictors (GCP).Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video.GCP instead treats videos as start-goal transformations, making video generation easier by conditioning on the more informative context provided by the first and final frames. Not only do existing forward prediction approaches synthesize better and longer videos when modified to become goal-conditioned,  but GCP models can also utilize structures that are not linear in time, to accomplish hierarchical prediction. To this end, we study both auto-regressive GCP models and novel tree-structured GCP models that generate frames recursively, splitting the video iteratively into finer and finer segments delineated by subgoals. In experiments across simulated and real datasets, our GCP methods generate high-quality sequences over long horizons.  Tree-structured GCPs are also substantially easier to parallelize than auto-regressive GCPs, making training  and  inference  very  efficient, and allowing the model to train on sequences that are thousands of frames in length.Finally, we demonstrate the utility of GCP approaches for imitation learning in the setting without access to expert actions. Videos are on the supplementary website: https://sites.google.com/view/video-gcp

Wir schlagen eine neue Klasse von visuellen generativen Modellen vor: zielkonditionierte Prädiktoren. Wir zeigen experimentell, dass die Konditionierung auf das Ziel es ermöglicht, die Unsicherheit zu reduzieren und Vorhersagen über viel längere Zeiträume zu erstellen.

In dieser Arbeit wird das Problem der Videovorhersage als Interpolation statt als Extrapolation neu formuliert, indem die Vorhersage auf das Start- und Endbild (Zielbild) konditioniert wird, was zu einer höheren Qualität der Vorhersagen führt.


Recent advances in computing technology and sensor design have made it easier to collect longitudinal or time series data from patients, resulting in a gigantic amount of available medical data.Most of the medical time series lack annotations or even when the annotations are available they could be subjective and prone to human errors.Earlier works have developed natural language processing techniques to extract concept annotations and/or clinical narratives from doctor notes.However, these approaches are slow and do not use the accompanying medical time series data.To address this issue, we introduce the problem of concept annotation for the medical time series data, i.e., the task of predicting and localizing medical concepts by using the time series data as input.We propose Relational Multi-Instance Learning (RMIL) - a deep Multi Instance Learning framework based on recurrent neural networks, which uses pooling functions and attention mechanisms for the concept annotation tasks.Empirical results on medical datasets show that our proposed models outperform various multi-instance learning models.

Wir schlagen ein tiefes Multiinstanz-Lernsystem vor, das auf rekurrenten neuronalen Netzen basiert und Pooling-Funktionen und Aufmerksamkeitsmechanismen für die Annotation von Konzepten verwendet.

Das Papier befasst sich mit der Klassifizierung medizinischer Zeitreihendaten und schlägt vor, die zeitliche Beziehung zwischen den Instanzen in jeder Reihe mithilfe einer rekurrenten neuronalen Netzwerkarchitektur zu modellieren.

Vorschlagen einer neuartigen Multiple Instance Learning (MIL) Formulierung, die Relation MIL (RMIL) genannt wird, und erörtern eine Reihe ihrer Varianten mit LSTM, Bi-LSTM, S2S usw. und untersuchen der Integration von RMIL mit verschiedenen Aufmerksamkeitsmechanismen und demonstrieren ihrer Verwendung bei der Vorhersage medizinischer Konzepte aus Zeitreihendaten.


The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing.However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting.We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance.  We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

Die Einbettungsschichten werden mit der Tensor-Train-Zerlegung faktorisiert, um ihren Speicherbedarf zu reduzieren.

Diese Arbeit schlägt ein Low-Rank-Tensor-Zerlegungsmodell zur Parametrisierung der Einbettungsmatrix in der natürlichen Sprachverarbeitung (NLP) vor, das das Netzwerk komprimiert und manchmal die Testgenauigkeit erhöht.


We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor. We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function.We provide empirical evidence that our proposed modification(i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and(ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter).We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence.Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32. Our source code will become available after the review process.

Regulierung des Gewichtsabfalls in adaptiven Gradientenmethoden wie Adam.

Schlägt vor, den Gewichtsabfall von der Anzahl der Optimierungsschritte zu entkoppeln.

In der Arbeit wird eine alternative Methode zur Umsetzung des Gewichtsverfalls in Adam vorgestellt, für die empirische Ergebnisse vorliegen.

Untersucht die Probleme des Gewichtsverfalls in den SGD-Varianten und schlägt eine Entkopplungsmethode zwischen dem Gewichtsverfall und der gradientenbasierten Aktualisierung vor.


Lifelong learning is the problem of learning multiple consecutive tasks in a sequential manner where knowledge gained from previous tasks is retained and used for future learning.It is essential towards the development of intelligent machines that can adapt to their surroundings.In this work we focus on a lifelong learning approach to generative modeling where we continuously incorporate newly observed streaming distributions into our learnt model.We do so through a student-teacher architecture which allows us to learn and preserve all the distributions seen so far without the need to retain the past data nor the past models.Through the introduction of a novel cross-model regularizer, the student model leverages the information learnt by the teacher, which acts as a summary of everything seen till now.The regularizer has the additional benefit of reducing the effect of catastrophic interference that appears when we learn over streaming data.We demonstrate its efficacy on streaming distributions as well as its ability to learn a common latent representation across a complex transfer learning scenario.

Lebenslanges verteilungsbasiertes Lernen durch eine Schüler-Lehrer Architektur in Verbindung mit einem modellübergreifenden Posterior-Regulierer.


Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling.In this paper, we look at geometric data represented as point clouds.We introduce a deep autoencoder (AE) network with excellent reconstruction quality and generalization ability.The learned representations outperform the state of the art in 3D recognition tasks and enable basic shape editing applications via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation.We also perform a thorough study of different generative models including GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space our AEs and, Gaussian mixture models (GMM).Interestingly, GMMs trained in the latent space of our AEs produce samples of the best fidelity and diversity.To perform our quantitative evaluation of generative models, we propose simple measures of fidelity and diversity based on optimally matching between sets point clouds.

Deep Autoencoders zum Erlernen einer guten Darstellung für geometrische 3D-Punktwolkendaten; Generative Modelle für Punktwolken.

Ansätze zum Erlernen generativer Modelle vom Typ GAN unter Verwendung der PointNet-Architektur und des Latent-Space-GAN.


Despite the remarkable performance of deep neural networks (DNNs) on various tasks, they are susceptible to adversarial perturbations which makes it difficult to deploy them in real-world safety-critical applications.In this paper, we aim to obtain robust networks by sparsifying DNN's latent features sensitive to adversarial perturbation.Specifically, we define vulnerability at the latent feature space and then propose a Bayesian framework to prioritize/prune features based on their contribution to both the original and adversarial loss.We also suggest regularizing the features' vulnerability during training to improve robustness further.While such network sparsification has been primarily studied in the literature for computational efficiency and regularization effect of DNNs, we confirm that it is also useful to design a defense mechanism through quantitative evaluation and qualitative analysis.We validate our method, \emph{Adversarial Neural Pruning (ANP)} on multiple benchmark datasets, which results in an improvement in test accuracy and leads to state-of-the-art robustness.ANP also tackles the practical problem of obtaining sparse and robust networks at the same time, which could be crucial to ensure adversarial robustness on lightweight networks deployed to computation and memory-limited devices.

Wir schlagen eine neue Methode zur Unterdrückung der Anfälligkeit des latenten Merkmalsraums vor, um robuste und kompakte Netzwerke zu erhalten.

In dieser Arbeit wird die Methode des "adversen neuronalen Pruning" vorgeschlagen, bei der eine Pruning-Maske und ein neuer Verlust zur Unterdrückung von Schwachstellen trainiert wird, um die Genauigkeit und die Robustheit gegenüber Angriffen zu verbessern.


In anomaly detection (AD), one seeks to identify whether a test sample is abnormal,  given a data set of normal samples.   A recent and promising approach to AD relies on deep generative models, such as variational autoencoders (VAEs),for unsupervised learning of the normal data distribution.In semi-supervised AD (SSAD), the data also includes a small sample of labeled anomalies.In this work,we propose two variational methods for training VAEs for SSAD.The intuitive idea in both methods is to train the encoder to â€˜separateâ€™ between latent vectors for normal and outlier data.We show that this idea can be derived from principled probabilistic formulations of the problem, and propose simple and effective algorithms.  Our methods can be applied to various data types, as we demonstrate on SSAD datasets ranging from natural images to astronomy and medicine, and can be combined with any VAE model architecture.When comparing to state-of-the-art SSAD methods that are not specific to particular data types, we obtain marked improvement in outlier detection.

Wir haben zwei VAE-Modifikationen vorgeschlagen, die negative Datenbeispiele berücksichtigen, und sie für die halbüberwachte Erkennung von Anomalien verwendet.

In den Beiträgen werden zwei VAE-ähnliche Methoden für die halbüberwachte Neuheitserkennung vorgeschlagen, MML-VAE und DP-VAE.


We introduce dynamic instance hardness (DIH) to facilitate the training of machine learning models.DIH is a property of each training sample and is computed as the running mean of the sample's instantaneous hardness as measured over the training history.We use DIH to evaluate how well a model retains knowledge about each training sample over time.We find that for deep neural nets (DNNs), the DIH of a sample in relatively early training stages reflects its DIH in later stages and as a result, DIH can be effectively used to reduce the set of training samples in future epochs.Specifically, during each epoch, only samples with high DIH are trained (since they are historically hard) while samples with low DIH can be safely ignored.DIH is updated each epoch only for the selected samples, so it does not require additional computation.Hence, using DIH during training leads to an appreciable speedup.Also, since the model is focused on the historically more challenging samples, resultant models are more accurate.The above, when formulated as an algorithm, can be seen as a form of curriculum learning, so we call our framework DIH curriculum learning (or DIHCL).The advantages of DIHCL, compared to other curriculum learning approaches, are: (1) DIHCL does not require additional inference steps over the data not selected by DIHCL in each epoch, (2) the dynamic instance hardness, compared to static instance hardness (e.g., instantaneous loss), is more stable as it integrates information over the entire training history up to the present time.Making certain mathematical assumptions, we formulate the problem of DIHCL as finding a curriculum that maximizes a multi-set function $f(\cdot)$, and derive an approximation bound for a DIH-produced curriculum relative to the optimal curriculum.Empirically, DIHCL-trained DNNs significantly outperform random mini-batch SGD and other recently developed curriculum learning methods in terms of efficiency, early-stage convergence, and final performance, and this is shown in training several state-of-the-art DNNs on 11 modern datasets.

Ein neues Verständnis der Trainingsdynamik und Metriken der Auswendiglernhärte führen zu effizientem und nachweisbarem Lernen von Curriculums.

Diese Arbeit formuliert DIH als ein Curriclum Learning Problem, das die Daten zum Trainieren von DNNs effektiver nutzen kann, und leitet die Theorie der Approximationsgrenze ab.


This paper explores many immediate connections between adaptive control and machine learning, both through common update laws as well as common concepts.Adaptive control as a field has focused on mathematical rigor and guaranteed convergence.The rapid advances in machine learning on the other hand have brought about a plethora of new techniques and problems for learning.This paper elucidates many of the numerous common connections between both fields such that results from both may be leveraged together to solve new problems.In particular, a specific problem related to higher order learning is solved through insights obtained from these intersections.

Geschichte der parallelen Entwicklungen von Aktualisierungsgesetzen und Konzepten zwischen adaptiver Steuerung und Optimierung beim maschinellen Lernen.


Recurrent convolution (RC) shares the same convolutional kernels and unrolls them multiple times, which is originally proposed to model time-space signals.We suggest that RC can be viewed as a model compression strategy for deep convolutional neural networks.RC reduces the redundancy across layers and is complementary to most existing model compression approaches.However, the performance of an RC network can't match the performance of its corresponding standard one, i.e. with the same depth but independent convolutional kernels.  This reduces the value of RC for model compression.In this paper, we propose a simple variant which improves RC networks: The batch normalization layers of an RC module are learned independently (not shared) for different unrolling steps.We provide insights on why this works.Experiments on CIFAR show that unrolling a convolutional layer several steps can improve the performance, thus indirectly plays a role in model compression.

Rekurrente Convoltion für die Modellkompression und ein Trick für das Training, d.h. das Lernen unabhängiger BN-Schichten über Schritte.

Der Autor modifiziert das Recurrent Convoltional Neural Network (RCNN) mit unabhängiger Batch Normalisierung, wobei die experimentellen Ergebnisse des RCNN mit der Architektur des neuronalen Netzwerks ResNet kompatibel sind, wenn es die gleiche Anzahl von Schichten enthält.


The visual world is vast and varied, but its variations divide into structured and unstructured factors.Structured factors, such as scale and orientation, admit clear theories and efficient representation design.Unstructured factors, such as what it is that makes a cat look like a cat, are too complicated to model analytically, and so require free-form representation learning.We compose structured Gaussian filters and free-form filters, optimized end-to-end, to factorize the representation for efficient yet general learning.Our experiments on dynamic structure, in which the structured filters vary with the input, equal the accuracy of dynamic inference with more degrees of freedom while improving efficiency.(Please see https://arxiv.org/abs/1904.11487 for the full edition.)

Dynamische rezeptive Felder mit räumlicher Gaußstruktur sind genau und effizient.

In diesem Beitrag wird ein strukturierter Faltungsoperator zur Modellierung von Deformationen lokaler Bildregionen vorgeschlagen, der die Anzahl der Parameter erheblich reduziert.


It is widely known that well-designed perturbations can cause state-of-the-art machine learning classifiers to mis-label an image, with sufficiently small perturbations that are imperceptible to the human eyes.However, by detecting the inconsistency between the image and wrong label, the human observer would be alerted of the attack.In this paper, we aim to design attacks that not only make classifiers generate wrong labels, but also make the wrong labels imperceptible to human observers.To achieve this, we propose an algorithm called LabelFool which identifies a target label similar to the ground truth label and finds a perturbation of the image for this target label.We first find the target label for an input image by a probability model, then move the input in the feature space towards the target label.Subjective studies on ImageNet show that in the label space, our attack is much less recognizable by human observers, while objective experimental results on ImageNet show that we maintain similar performance in the image space as well as attack rates to state-of-the-art attack algorithms.

Ein Trick für adversarial Beispiele, damit die falsch klassifizierten Labels im Bezeichnungsraum für menschliche Beobachter nicht wahrnehmbar sind.

Eine Methode zur Konstruktion von Angriffen, die von Menschen weniger leicht erkannt werden können, indem die Zielklasse so verändert wird, dass sie der Originalklasse des Bildes ähnlich ist.


This paper presents noise type/position classification of various impact noises generated in a building which is a serious conflict issue in apartment complexes.For this study, a collection of floor impact noise dataset is recorded with a single microphone.Noise types/positions are selected based on a report by the Floor Management Center under Korea Environmental Corporation.Using a convolutional neural networks based classifier, the impact noise signals converted to log-scaled Mel-spectrograms are classified into noise types or positions.Also, our model is evaluated on a standard environmental sound dataset ESC-50 to show extensibility on environmental sound classification.

In diesem Beitrag wird eine Klassifizierung der Lärmarten und -positionen verschiedener Trittschallgeräusche vorgestellt, die in einem Gebäude erzeugt werden und ein ernsthaftes Konfliktproblem in Wohnkomplexen darstellen.

Diese Arbeit beschreibt den Einsatz von Convolutional Neural Networks in einem neuartigen Anwendungsbereich der Klassifizierung von Gebäudelärmarten und Lärmpositionen.


Recordings of neural circuits in the brain reveal extraordinary dynamical richness and high variability.At the same time, dimensionality reduction techniques generally uncover low-dimensional structures underlying these dynamics.What determines the dimensionality of activity in neural circuits?What is the functional role of dimensionality in behavior and task learning?In this work we address these questions using recurrent neural network (RNN) models.We find that, depending on the dynamics of the initial network, RNNs learn to increase and reduce dimensionality in a way that matches task demands.These findings shed light on fundamental dynamical mechanisms by which neural networks solve tasks with robust representations that generalize to new cases.

Rekurrente neuronale Netze lernen, um die Dimensionalität ihrer internen Repräsentation in Abhängigkeit von der Dynamik des Ausgangsnetzes aufgabengerecht zu erhöhen und zu verringern.


Domain adaptation addresses the common problem when the target distribution generating our test data drifts from the source (training) distribution.While absent assumptions, domain adaptation is impossible, strict conditions, e.g. covariate or label shift, enable principled algorithms.Recently-proposed domain-adversarial approaches consist of aligning source and target encodings, often motivating this approach as minimizing two (of three) terms in a theoretical bound on target error.Unfortunately, this minimization can cause arbitrary increases in the third term, e.g. they can break down under shifting label distributions.We propose asymmetrically-relaxed distribution alignment, a new approach that overcomes some limitations of standard domain-adversarial algorithms.Moreover, we characterize precise assumptions under which our algorithm is theoretically principled and demonstrate empirical benefits on both synthetic and real datasets.

Anstelle von strikten Verteilungsanpassungen in traditionellen tiefen Domänenanpassungszielen, die versagen, wenn sich die Verteilung der Zielbeschriftungen ändert, schlagen wir vor, ein entspanntes Ziel mit neuen Analysen, neuen Algorithmen und experimenteller Validierung zu optimieren.

In dieser Arbeit werden entspannte Metriken für die Bereichsanpassung vorgeschlagen, die neue theoretische Grenzen für den Zielfehler setzen.