In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs.There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal.In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora.Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation.Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French-to-English and German-to-English translation.The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively.Our implementation is released as an open source project.

Wir stellen die erste erfolgreiche Methode vor, um neuronale maschinelle Übersetzung auf unüberwachte Weise zu trainieren, indem wir nichts anderes als einsprachige Korpora verwenden

Die Autoren stellen ein Modell für unüberwachte NMT vor, das keine parallelen Korpora zwischen den beiden interessierenden Sprachen erfordert.

Dies ist eine Arbeit über unüberwachte MT, die eine Standardarchitektur mit Worteinbettungen in einem gemeinsamen Einbettungsraum nur mit zweisprachigen Wortpapieren und einem Encoder-Decoder trainiert, der mit einsprachigen Daten trainiert wird.


We describe a new training methodology for generative adversarial networks.The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses.This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2.We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10.Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator.Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation.As an additional contribution, we construct a higher-quality version of the CelebA dataset.

Wir trainieren generative adversarial Netze auf progressive Weise und können so hochauflösende Bilder mit hoher Qualität erzeugen.

Einführung von progressivem Wachstum und einer einfachen parameterfreien statistischen Minibatch-Zusammenfassung zur Verwendung beim GAN-Training, um die Synthese von hochauflösenden Bildern zu ermöglichen.


Designing a convolution for a spherical neural network requires a delicate tradeoff between efficiency and rotation equivariance.DeepSphere, a method based on a graph representation of the discretized sphere, strikes a controllable balance between these two desiderata.This contribution is twofold.First, we study both theoretically and empirically how equivariance is affected by the underlying graph with respect to the number of pixels and neighbors.Second, we evaluate DeepSphere on relevant problems.Experiments show state-of-the-art performance and demonstrates the efficiency and flexibility of this formulation.Perhaps surprisingly, comparison with previous work suggests that anisotropic filters might be an unnecessary price to pay.

Ein graphbasiertes sphärisches CNN, das ein interessantes Gleichgewicht von Kompromissen für eine Vielzahl von Anwendungen bietet.

Kombiniert bestehende CNN-Frameworks, die auf der Diskretisierung einer Kugel als Graph basieren, um ein Konvergenzergebnis zu zeigen, das mit der Rotationsäquivalenz auf einer Kugel zusammenhängt.

Die Autoren verwenden die bestehende Graph-CNN-Formulierung und eine Pooling-Strategie, die hierarchische Pixelierungen der Kugel ausnutzt, um aus der diskretisierten Kugel zu lernen.


The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics.In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity.Here, we derive stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm.These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule.We can further use the relations to efficiently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity.Our claims are empirically verified.

Wir beweisen Fluktuations-Dissipations-Beziehungen für SGD, die verwendet werden können, um (i) Lernraten adaptiv festzulegen und (ii) Verlustflächen zu untersuchen.

Die Konzepte des Artikels arbeiten im zeitdiskreten Formalismus, verwenden die Master-Gleichung und sind nicht auf eine lokal quadratische Annäherung der Verlustfunktion oder auf Gaußsche Annahmen für SGD-Störungen angewiesen.

Die Autoren leiten die stationären Fluktuations-Dissipations-Relationen ab, die messbare Größen und Hyperparameter in SGD miteinander verknüpfen, und verwenden die Relationen, um den Trainingsplan adaptiv festzulegen und die Verlustfunktionslandschaft zu analysieren.


Recurrent neural networks (RNNs) are difficult to train on sequence processing tasks, not only because input noise may be amplified through feedback, but also because any inaccuracy in the weights has similar consequences as input noise.We describe a method for denoising the hidden state during training to achieve more robust representations thereby improving generalization performance.Attractor dynamics are incorporated into the hidden state to `clean up' representations at each step of a sequence.The attractor dynamics are trained through an auxillary denoising loss to recover previously experienced hidden states from noisy versions of those states.This state-denoised recurrent neural network (SDRNN) performs multiple steps of internal processing for each external sequence step.On a range of tasks, we show that the SDRNN outperforms a generic RNN as well as a variant of the SDRNN with attractor dynamics on the hidden state but without the auxillary loss.We argue that attractor dynamics---and corresponding connectivity constraints---are an essential component of the deep learning arsenal and should be invoked not only for recurrent networks but also for improving deep feedforward nets and intertask transfer.

Wir schlagen einen Mechanismus zum Entfernen von Störfaktoren des internen Zustands eines RNN vor, um die Generalisierungsleistung zu verbessern.


We consider reinforcement learning in input-driven environments, where an exogenous, stochastic input process affects the dynamics of the system.Input processes arise in many applications, including queuing systems, robotics control with disturbances, and object tracking.Since the state dynamics and rewards depend on the input process, the state alone provides limited information for the expected future returns.Therefore, policy gradient methods with standard state-dependent baselines suffer high variance during training.We derive a bias-free, input-dependent baseline to reduce this variance, and analytically show its benefits over state-dependent baselines.We then propose a meta-learning approach to overcome the complexity of learning a baseline that depends on a long sequence of inputs.Our experimental results show that across environments from queuing systems, computer networks, and MuJoCo robotic locomotion, input-dependent baselines consistently improve training stability and result in better eventual policies.

Für Umgebungen, die teilweise durch externe Eingabeprozesse diktiert werden, leiten wir eine eingabeabhängige Basislinie ab, die nachweislich die Varianz für Policy-Gradienten Methoden reduziert und die Strategie-Leistung in einem breiten Spektrum von RL Aufgaben verbessert.

Die Autoren betrachten das Problem des Lernens in eingabegesteuerten Umgebungen, zeigen, wie das PG-Theorem immer noch für einen eingabebewussten Kritiker gilt, und zeigen, dass eingabeabhängige Basislinien am besten für Vermutungen mit diesem Kritiker zu verwenden sind.

In diesem Beitrag wird der Begriff der eingabeabhängigen Basislinien in Policy-Gradienten Methoden in RL eingeführt und es werden verschiedene Methoden zum Trainieren der eingabeabhängigen Basislinienfunktion vorgeschlagen, um die Varianz von Störungen durch externe Faktoren zu verringern.


Deep networks have shown great performance in classification tasks.However, the parameters learned by the classifier networks usually discard stylistic information of the input, in favour of information strictly relevant to classification.We introduce a network that has the capacity to do both classification and reconstruction by adding a "style memory" to the output layer of the network.We also show how to train such a neural network as a deep multi-layer autoencoder, jointly minimizing both classification and reconstruction losses.The generative capacity of our network demonstrates that the combination of style-memory neurons with the classifier neurons yield good reconstructions of the inputs when the classification is correct.We further investigate the nature of the style memory, and how it relates to composing digits and letters.

Durch die Erweiterung der obersten Schicht eines Klassifizierungsnetzes mit einem Stylespeicher kann es generativ arbeiten.

In dieser Arbeit wird vorgeschlagen, ein neuronales Klassifizierungsnetz nicht nur für die Klassifizierung zu trainieren, sondern auch für die Rekonstruktion einer Repräsentation seiner Eingabe, um die Klasseninformation aus dem Erscheinungsbild zu faktorisieren.

Die Arbeit schlägt vor, einen Autoencoder so zu trainieren, dass die Repräsentation der mittleren Schicht aus dem Klassenlabel des Inputs und einer versteckten Vektordarstellung besteht.


Routing models, a form of conditional computation where examples are routed through a subset of components in a larger network, have shown promising results in recent works.Surprisingly, routing models to date have lacked important properties, such as architectural diversity and large numbers of routing decisions.Both architectural diversity and routing depth can increase the representational power of a routing network.In this work, we address both of these deficiencies.We discuss the significance of architectural diversity in routing models, and explain the tradeoffs between capacity and optimization when increasing routing depth.In our experiments, we find that adding architectural diversity to routing models significantly improves performance, cutting the error rates of a strong baseline by 35% on an Omniglot setup.However, when scaling up routing depth, we find that modern routing techniques struggle with optimization.We conclude by discussing both the positive and negative results, and suggest directions for future research.

Pro-Beispiel Routing Modelle profitieren von der architektonischen Vielfalt, haben aber immer noch Probleme bei der Skalierung auf eine große Anzahl von Routing Entscheidungen.

Erweitert die Art der architektonischen Einheit, die dem Router bei jeder Entscheidung zur Verfügung steht, und skaliert auf tiefere Netze, um die modernste Leistung auf Omniglot zu erreichen.

Diese Arbeit erweitert Routing Netzwerke, um verschiedene Architekturen über geroutete Module hinweg zu nutzen.


Across numerous applications, forecasting relies on numerical solvers for partial differential equations (PDEs).Although the use of deep-learning techniques has been proposed, the uses have been restricted by the fact the training data are obtained using PDE solvers.Thereby, the uses were limited to domains, where the PDE solver was applicable, but no further. We present methods for training on small domains, while applying the trained models on larger domains, with consistency constraints ensuring the solutions are physically meaningful even at the boundary of the small domains.We demonstrate the results on an air-pollution forecasting model for Dublin, Ireland.

Wir stellen RNNs für das Training von Ersatzmodellen von PDEs vor, bei denen Konsistenzbeschränkungen sicherstellen, dass die Lösungen physikalisch sinnvoll sind, selbst wenn beim Training viel kleinere Bereiche verwendet werden als das trainierte Modell angewendet wird.


We address the issue of limit cycling behavior in training Generative Adversarial Networks and propose the use of Optimistic Mirror Decent (OMD) for training Wasserstein GANs.Recent theoretical results have shown that optimistic mirror decent (OMD) can enjoy faster regret rates in the context of zero-sum games.WGANs is exactly a context of solving a zero-sum game with simultaneous no-regret dynamics.  Moreover, we show that optimistic mirror decent addresses the limit cycling problem in training WGANs.We formally show that in the case of bi-linear zero-sum games the last iterate of OMD dynamics converges to an equilibrium, in contrast to GD dynamics which are bound to cycle.We also portray the huge qualitative difference between GD and OMD dynamics with toy examples, even when GD is modified with many adaptations proposed in the recent literature, such as gradient penalty or momentum.We apply OMD WGAN training to a bioinformatics problem of generating DNA sequences.We observe that models trained with OMD achieve consistently smaller KL divergence with respect to the true underlying distribution, than models trained with GD variants.Finally, we introduce a new algorithm, Optimistic Adam, which is an optimistic variant of Adam.We apply it to WGAN training on CIFAR10 and observe improved performance in terms of inception score as compared to Adam.

Wir schlagen die Verwendung von optimistischen Spiegeln vor, um Zyklusprobleme beim Training von GANs anzugehen. Außerdem stellen wir den Optimistischen Adam-Algorithmus vor

In dieser Arbeit wird die Verwendung von optimistischem Mirror Descent zum Training von WGANs vorgeschlagen.

Der Artikel schlägt vor, optimistischen Gradientenabstieg für das GAN-Training zu verwenden, der das bei SGD und seinen Varianten beobachtete Zyklusverhalten vermeidet und vielversprechende Ergebnisse beim GAN-Training liefert.

In diesem Beitrag wird eine einfache Modifikation des Standard-Gradientenabstiegs vorgeschlagen, die die Konvergenz von GANs und anderen Minimax-Optimierungsproblemen verbessern soll.


Learning good representations of users and items is crucially important to recommendation with implicit feedback.Matrix factorization is the basic idea to derive the representations of users and items by decomposing the given interaction matrix.However, existing matrix factorization based approaches share the limitation in that the interaction between user embedding and item embedding is only weakly enforced by fitting the given individual rating value, which may lose potentially useful information.In this paper, we propose a novel Augmented Generalized Matrix Factorization (AGMF) approach that is able to incorporate the historical interaction information of users and items for learning effective representations of users and items.Despite the simplicity of our proposed approach, extensive experiments on four public implicit feedback datasets demonstrate that our approach outperforms state-of-the-art counterparts.Furthermore, the ablation study demonstrates that by using multi-hot encoding to enrich user embedding and item embedding for Generalized Matrix Factorization, better performance, faster convergence, and lower training loss can be achieved.

Eine einfache Erweiterung der verallgemeinerten Matrixfaktorisierung kann den Stand der Technik bei Empfehlungen übertreffen.

Die Arbeit stellt ein Matrixfaktorisierungs Framework vor, um den Effekt historischer Daten beim Lernen von Benutzerpräferenzen in kollaborativen Filtereinstellungen zu verstärken.


We propose an unsupervised method for building dynamic representations of sequential data, particularly of observed interactions.The method simultaneously acquires representations of input data and its dynamics.It is based on a hierarchical generative model composed of two levels.In the first level, a model learns representations to generate observed data.In the second level, representational states encode the dynamics of the lower one.The model is designed as a Bayesian network with switching variables represented in the higher level, and which generates transition models.The method actively explores the latent space guided by its knowledge and the uncertainty about it.That is achieved by updating the latent variables from prediction error signals backpropagated to the latent space.So, no encoder or inference models are used since the generators also serve as their inverse transformations.The method is evaluated in two scenarios, with static images and with videos.The results show that the adaptation over time leads to better performance than with similar architectures without temporal dependencies, e.g., variational autoencoders.With videos, it is shown that the system extracts the dynamics of the data in states that highly correlate with the ground truth of the actions observed.

Eine Methode, die Darstellungen von sequentiellen Daten und ihrer Dynamik durch generative Modelle mit einem aktiven Prozess erstellt.

Kombiniert neuronale Netze und Gaußsche Verteilungen, um eine Architektur und ein generatives Modell für Bilder und Videos zu schaffen, das den Fehler zwischen erzeugten und gelieferten Bildern minimiert.

Die Arbeit schlägt ein Bayes'sches Netzmodell vor, das als neuronales Netz realisiert ist und verschiedene Daten in Form eines linearen dynamischen Systems erlernt.


Activation is a nonlinearity function that plays a predominant role in the convergence and performance of deep neural networks.While Rectified Linear Unit (ReLU) is the most successful activation function, its derivatives have shown superior performance on benchmark datasets.In this work, we explore the polynomials as activation functions (order â‰¥ 2) that can approximate continuous real valued function within a given interval.Leveraging this property, the main idea is to learn the nonlinearity, accepting that the ensuing function may not be monotonic.While having the ability to learn more suitable nonlinearity, we cannot ignore the fact that it is a challenge to achieve stable performance due to exploding gradients - which is prominent with the increase in order.To handle this issue, we introduce dynamic input scaling, output scaling, and lower learning rate for the polynomial weights.Moreover, lower learning rate will control the abrupt fluctuations of the polynomials between weight updates.In experiments on three public datasets, our proposed method matches the performance of prior activation functions, thus providing insight into a networkâ€™s nonlinearity preference.

Wir schlagen Polynome als Aktivierungsfunktionen vor.

Die Autoren führen lernfähige Aktivierungsfunktionen ein, die durch Polynomfunktionen parametrisiert sind, und zeigen Ergebnisse, die etwas besser sind als ReLU.


We introduce CBF, an exploration method that works in the absence of rewards or end of episode signal.CBF is based on intrinsic reward derived from the error of a dynamics model operating in feature space.It was inspired by (Pathak et al., 2017), is easy to implement, and can achieve results such as passing four levels of Super Mario Bros, navigating VizDoom mazes and passing two levels of SpaceInvaders.We investigated the effect of combining the method with several auxiliary tasks, but find inconsistent improvements over the CBF baseline.

Eine einfache intrinsische Motivationsmethode, die die Forward Dynamik nutzt, modelliert Fehler im Merkmalsraum des Regelwerks.


This paper is concerned with the robustness of VAEs to adversarial attacks.We highlight that conventional VAEs are brittle under attack but that methods recently introduced for disentanglement such as Î²-TCVAE (Chen et al., 2018) improve robustness, as demonstrated through a variety of previously proposed adversarial attacks (Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al.(2018)).This motivated us to develop Seatbelt-VAE, a new hierarchical disentangled VAE that is designed to be significantly more robust to adversarial attacks than existing approaches, while retaining high quality reconstructions.

Wir zeigen, dass entwirrte VAEs robuster als einfache VAEs gegenüber Angriffen sind, die darauf abzielen, sie zur Dekodierung der adversarial Eingabe für ein ausgewähltes Ziel zu verleiten. Anschließend entwickeln wir ein noch robusteres hierarchisches entschlüsseltes VAE, Seatbelt-VAE.

Die Autoren schlagen ein neues VAE-Modell namens seatbelt-VAE vor, das sich gegenüber latenten Angriffen als robuster erweist als Benchmarks.


The backpropagation algorithm is the de-facto standard for credit assignment in artificial neural networks due to its empirical results.Since its conception, variants of the backpropagation algorithm have emerged.More specifically, variants that leverage function changes in the backpropagation equations to satisfy their specific requirements.Feedback Alignment is one such example, which replaces the weight transpose matrix in the backpropagation equations with a random matrix in search of a more biologically plausible credit assignment algorithm.In this work, we show that function changes in the  backpropagation procedure is equivalent to adding an implicit learning rate to an artificial neural network.Furthermore, we learn activation function derivatives in the backpropagation equations to demonstrate early convergence in these artificial neural networks.Our work reports competitive performances with early convergence on MNIST and CIFAR10 on sufficiently large deep neural network architectures.

Wir zeigen, dass Funktionsänderungen in der Backpropagation gleichbedeutend mit einer impliziten Lernrate sind.


Unsupervised text style transfer is the task of re-writing text of a given style into a target style without using a parallel corpus of source style and target style sentences for training.Style transfer systems are evaluated on their ability to generate sentences that1) possess the target style,2) are fluent and natural sounding, and3) preserve the non-stylistic parts (content) of the source sentence.We train a reinforcement learning (RL) based unsupervised style transfer system that incorporates rewards for the above measures, and describe novel rewards shaping methods for the same.Our approach does not attempt to disentangle style and content, and leverages the power of massively pre-trained language models as well as the Transformer.Our system significantly outperforms existing state-of-art systems based on human as well as automatic evaluations on target style, fluency and content preservation as well as on overall success of style transfer, on a variety of datasets.

Ein Ansatz des Reinforcement Learrning zur Übertragung von Textstilen.

Stellt eine RL-basierte Methode vor, die ein vortrainiertes Sprachmodell nutzt, um den Textstil zu übertragen, ohne das Ziel der Entflechtung zu verfolgen, und dabei generierte Stilübertragungen eines anderen Modells verwendet.

Die Autoren schlagen eine Kombinationsbelohnung vor, die sich aus Sprachgewandtheit, Inhalt und Stil für die Übertragung von Textstil zusammensetzt.


Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what networks have learned inside the deep generative representations and how photo-realistic images are able to be composed from random noises.In this work, we show that highly-structured semantic hierarchy emerges from the generative representations as the variation factors for synthesizing scenes.By probing the layer-wise representations with a broad set of visual concepts at different abstraction levels, we are able to quantify the causality between the activations and the semantics occurring in the output image.Such a quantification identifies the human-understandable variation factors learned by GANs to compose scenes.The qualitative and quantitative results suggest that the generative representations learned by GAN are specialized to synthesize different hierarchical semantics: the early layers tend to determine the spatial layout and configuration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme.Identifying such a set of manipulatable latent semantics facilitates semantic scene manipulation.

Wir zeigen, dass in den tiefen generativen Repräsentationen eine hochstrukturierte semantische Hierarchie als Ergebnis für die Synthese von Szenen entsteht.

Die Arbeit untersucht die Aspekte, die durch die latenten Variablen kodiert werden, die in die verschiedenen Schichten von StyleGAN eingegeben werden.

Der Artikel präsentiert eine visuell geführte Interpretation der Aktivierungen der Convolution Layers im Generator von StyleGAN auf Layout, Szenenkategorie, Szeneneigenschaften und Farbe.


Variational autoencoders (VAEs) defined over SMILES string and graph-based representations of molecules promise to improve the optimization of molecular properties, thereby revolutionizing the pharmaceuticals and materials industries.However, these VAEs are hindered by the non-unique nature of SMILES strings and the computational cost of graph convolutions.To efficiently pass messages along all paths through the molecular graph, we encode multiple SMILES strings of a single molecule using a set of stacked recurrent neural networks, harmonizing hidden representations of each atom between SMILES representations, and use attentional pooling to build a final fixed-length latent representation.By then decoding to a disjoint set of SMILES strings of the molecule, our All SMILES VAE learns an almost bijective mapping between molecules and latent representations near the high-probability-mass subspace of the prior.Our SMILES-derived but molecule-based latent representations significantly surpass the state-of-the-art in a variety of fully- and semi-supervised property regression and molecular property optimization tasks.

Wir fassen Nachrichten zwischen mehreren SMILES-Zeichenfolgen desselben Moleküls zusammen, um Informationen entlang aller Pfade durch den molekularen Graphen zu übermitteln und so latente Darstellungen zu erzeugen, die den Stand der Technik bei einer Vielzahl von Aufgaben deutlich übertreffen.

Die Methode verwendet mehrere Eingaben von SMILES-Zeichenfolgen, zeichenweise Merkmalsfusion über diese Zeichenfolgen und Netzwerktraining durch mehrere Ausgabeziele von SMILES-Zeichenfolgen, wodurch eine robuste latente Repräsentation mit fester Länge unabhängig von SMILES-Variationen geschaffen wird.

Die Autoren beschreiben ein neuartiges Variations Autoencoder ähnliches Verfahren für Moleküle, das Moleküle als Zeichenketten kodiert, um die für den Informationsaustausch zwischen den Atomen im Molekül erforderlichen Operationen zu reduzieren.


We propose a simple yet highly effective method that addresses the mode-collapse problem in the Conditional Generative  Adversarial  Network (cGAN).Although conditional distributions are multi-modal (i.e., having many modes) in practice, most cGAN approaches tend to learn an overly simplified distribution where an input is always mapped to a single output regardless of variations in latent code.To address such issue, we propose to explicitly regularize the generator to produce diverse outputs depending on latent codes.The proposed regularization is simple, general, and can be easily integrated into most conditional GAN objectives.Additionally, explicit regularization on generator allows our method to control a balance between visual quality and diversity.We demonstrate the effectiveness of our method on three conditional generation tasks: image-to-image translation, image inpainting, and future video prediction.We show that simple addition of our regularization to existing models leads to surprisingly diverse generations, substantially outperforming the previous approaches for multi-modal conditional generation specifically designed in each individual task.

Wir schlagen einen einfachen und allgemeinen Ansatz vor, der das Problem des Moduszusammenbruchs in verschiedenen bedingten GANs vermeidet.

Die Arbeit schlägt einen Regularisierungsterm für das bedingte GAN-Ziel vor, um eine vielfältige multimodale Generierung zu fördern und einen Moduskollaps zu verhindern.

Die Arbeit schlägt eine Methode zur Erzeugung verschiedener Ausgaben für verschiedene bedingte GAN-Frameworks vor, einschließlich Bild-zu-Bild-Übersetzung, Bild-Inpainting und Video-Vorhersage, die auf verschiedene bedingte Synthese-Frameworks für verschiedene Aufgaben angewendet werden können.


The transformer is a state-of-the-art neural translation model that uses attention to iteratively refine lexical representations with information drawn from the surrounding context.Lexical features are fed into the first layer and propagated through a deep network of hidden layers.We argue that the need to represent and propagate lexical features in each layer limits the modelâ€™s capacity for learning and representing other information relevant to the task.To alleviate this bottleneck, we introduce gated shortcut connections between the embedding layer and each subsequent layer within the encoder and decoder.This enables the model to access relevant lexical content dynamically, without expending limited resources on storing it within intermediate states.We show that the proposed modification yields consistent improvements on standard WMT translation tasks and reduces the amount of lexical information passed along the hidden layers.We furthermore evaluate different ways to integrate lexical connections into the transformer architecture and present ablation experiments exploring the effect of proposed shortcuts on model behavior.

Durch die Ausstattung des Transformermodells mit Abkürzungen zur Einbettungsschicht wird die Modellkapazität für das Lernen neuer Informationen frei.


Probability density estimation is a classical and well studied problem, but standard density estimation methods have historically lacked the power to model complex and high-dimensional image distributions.  More recent generative models leverage the power of neural networks to implicitly learn and represent probability models over complex images.  We describe methods to extract explicit probability density estimates from GANs, and explore the properties of these image density functions.  We perform sanity check experiments to provide evidence that these probabilities are reasonable.  However, we also show that density functions of natural images are difficult to interpret and thus limited in use.  We study reasons for this lack of interpretability, and suggest that we can get better interpretability by doing density estimation on latent representations of images.

Wir untersuchen die Beziehung zwischen Wahrscheinlichkeitsdichtewerten und Bildinhalten in nicht-invertierbaren GANs.

Die Autoren versuchen, die Wahrscheinlichkeitsverteilung des Bildes mit Hilfe von GAN zu schätzen und entwickeln eine geeignete Approximation der PDFs im latenten Raum.


Convolutional Neural Networks (CNNs) are composed of multiple convolution layers and show elegant performance in vision tasks.The design of the regular convolution is based on the Receptive Field (RF) where the information within a specific region is processed.In the view of the regular convolution's RF, the outputs of neurons in lower layers with smaller RF are bundled to create neurons in higher layers with larger RF. As a result, the neurons in high layers are able to capture the global context even though the neurons in low layers only see the local information.However, in lower layers of the biological brain, the information outside of the RF changes the properties of neurons.In this work, we extend the regular convolution and propose spatially shuffled convolution (ss convolution).In ss convolution, the regular convolution is able to use the information outside of its RF by spatial shuffling which is a simple and lightweight operation.We perform experiments on CIFAR-10 and ImageNet-1k dataset, and show that ss convolution improves the classification performance across various CNNs.

Wir schlagen eine räumlich gemischte Convolution vor, bei der die reguläre Convolution die Informationen von außerhalb ihres rezeptiven Feldes einbezieht.

Er schlägt eine SS-Konvulation vor, die Informationen außerhalb ihres RF verwendet und bei Tests mit mehreren CNN-Modellen bessere Ergebnisse erzielt.

Die Autoren schlugen eine Shuffle-Strategie für Convolution Layers in Convolution Layers in Convolutional Neural Networks vor.


We propose a framework to model the distribution of sequential data coming froma set of entities connected in a graph with a known topology.The method isbased on a mixture of shared hidden Markov models (HMMs), which are trainedin order to exploit the knowledge of the graph structure and in such a way that theobtained mixtures tend to be sparse.Experiments in different application domainsdemonstrate the effectiveness and versatility of the method.

Eine Methode zur Modellierung der generativen Verteilung von Sequenzen, die aus graphisch verbundenen Einheiten stammen.

Die Autoren schlagen eine Methode zur Modellierung sequentieller Daten aus mehreren miteinander verbundenen Quellen unter Verwendung einer Mischung aus einem gemeinsamen Pool von HMMs vor.


To gain high rewards in muti-agent scenes, it is sometimes necessary to understand other agents and make corresponding optimal decisions.We can solve these tasks by first building models for other agents and then finding the optimal policy with these models.To get an accurate model, many observations are needed and this can be sample-inefficient.What's more, the learned model and policy can overfit to current agents and cannot generalize if the other agents are replaced by new agents.In many practical situations, each agent we face can be considered as a sample from a population with a fixed but unknown distribution.Thus we can treat the task against some specific agents as a task sampled from a task distribution.We apply meta-learning method to build models and learn policies.Therefore when new agents come, we can adapt to them efficiently.Experiments on grid games show that our method can quickly get high rewards.

Unsere Arbeit wendet Meta-Lernen auf Multi-Agenten Reinforcement Learning an, um unserem Agenten zu helfen, sich effizient an neue Gegner anzupassen.

Dieser Beitrag konzentriert sich auf die schnelle Anpassung an neue Verhaltensweisen der anderen Agenten in der Umgebung mit Hilfe einer auf MAML basierenden Methode.

Die Arbeit stellt einen Ansatz für Multi-Agenten-Lernen vor, der auf dem Rahmen des modell-agnostischen Meta-Lernens für die Aufgabe der Gegner-Modellierung für Multi-Agenten RL basiert.


We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation.  This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball.We show that this is an effective regularizer;  for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2% to 5.3%.

Wir charakterisieren die Singulärwerte der linearen Transformation, die mit einer standardmäßigen 2D Mehrkanal Convolutional Layer verbunden sind, und ermöglichen so deren effiziente Berechnung.

Der Beitrag widmet sich der Berechnung der Singulärwerte von Convolutional Layers.

Leitet exakte Formeln für die Berechnung der Singulärwerte von Convolutional Layers tiefer neuronaler Netze ab und zeigt, dass die Berechnung der Singulärwerte viel schneller erfolgen kann als die Berechnung der vollständigen SVD der Convolution Matrix, indem man auf schnelle FFT-Transformationen zurückgreift.


Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning.A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment.Computing a Bayes-optimal policy is however intractable for all but the smallest tasks.In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection.In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty.We also evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher return during training than existing methods.

VariBAD eröffnet einen Weg zu überschaubarer approximativer Bayes-optimaler Exploration für Deep RL unter Verwendung von Ideen aus Meta-Learning, Bayesian RL und approximativer Variationsinferenz.

In diesem Beitrag wird eine neue Methode des tiefen Reinforcement Learnings vorgestellt, die einen effizienten Kompromiss zwischen Erkundung und Ausbeutung ermöglicht und Meta-Lernen, variationale Inferenz und bayesianisches RL kombiniert.


In a continual learning setting, new categories may be introduced over time, and an ideal learning system should perform well on both the original categories and the new categories.While deep neural nets have achieved resounding success in the classical setting, they are known to forget about knowledge acquired in prior episodes of learning if the examples encountered in the current episode of learning are drastically different from those encountered in prior episodes.This makes deep neural nets ill-suited to continual learning.In this paper, we propose a new model that can both leverage the expressive power of deep neural nets and is resilient to forgetting when new categories are introduced.We demonstrate an improvement in terms of accuracy on original classes compared to a vanilla deep neural net.

Wir zeigen, dass metrisches Lernen dazu beitragen kann, katastrophales Vergessen zu reduzieren.

In dieser Arbeit wird metrisches Lernen eingesetzt, um das katastrophale Vergessen in neuronalen Netzen zu reduzieren, indem die Ausdruckskraft der letzten Schicht verbessert wird, was zu besseren Ergebnissen beim kontinuierlichen Lernen führt.


Biomedical knowledge bases are crucial in modern data-driven biomedical sciences, but auto-mated biomedical knowledge base construction remains challenging.In this paper, we consider the problem of disease entity normalization, an essential task in constructing a biomedical knowledge base.  We present NormCo, a deep coherence model which considers the semantics of an entity mention, as well as the topical coherence of the mentions within a single document.NormCo mod-els entity mentions using a simple semantic model which composes phrase representations from word embeddings, and treats coherence as a disease concept co-mention sequence using an RNN rather than modeling the joint probability of all concepts in a document, which requires NP-hard inference.  To overcome the issue of data sparsity, we used distantly supervised data and synthetic data generated from priors derived from the BioASQ dataset.  Our experimental results show thatNormCo outperforms state-of-the-art baseline methods on two disease normalization corpora in terms of (1) prediction quality and (2) efficiency, and is at least as performant in terms of accuracy and F1 score on tagged documents.

Wir stellen NormCo vor, ein tiefes Kohärenzmodell, das sowohl die Semantik einer Entitätserwähnung als auch die thematische Kohärenz der Erwähnungen innerhalb eines einzelnen Dokuments berücksichtigt, um eine Krankheitsentitätsnormalisierung durchzuführen.

Verwendet einen GRU Autoencoder zur Darstellung des "Kontexts" (verwandte Eigenschaften einer bestimmten Krankheit innerhalb eines Satzes) und löst die BioNLP-Aufgabe mit erheblichen Verbesserungen gegenüber den bekanntesten Methoden.


We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others.Multiplicative interaction layers as primitive operations have a long-established presence in the literature, though this often not emphasized and thus under-appreciated.We begin by showing that such layers strictly enrich the representable function classes of neural networks.We conjecture that multiplicative interactions offer a particularly powerful inductive bias when fusing multiple streams of information or when conditional computation is required.We therefore argue that they should be considered in many situation where multiple compute or information paths need to be combined, in place of the simple and oft-used concatenation operation.Finally, we back up our claims and demonstrate the potential of multiplicative interactions by applying them in large-scale complex RL and sequence modelling tasks, where their use allows us to deliver state-of-the-art results, and thereby provides new evidence in support of multiplicative interactions playing a more prominent role when designing new neural network architectures.

Wir untersuchen die Rolle der multiplikativen Interaktion als vereinheitlichendes Framework, um eine Reihe klassischer und moderner neuronaler Netzwerkarchitekturen zu beschreiben, wie z.B. Gating, Aufmerksamkeitsschichten, Hypernetze und dynamische Convolutions.

Stellt die multiplikative Interaktion als einheitliche Charakterisierung für die Darstellung häufig verwendeter Komponenten der Modellarchitektur vor und zeigt empirische Beweise für die überlegene Leistung bei Aufgaben wie RL und Sequenzmodellierung.

Die Arbeit untersucht verschiedene Arten von multiplikativen Interaktionen und stellt fest, dass MI-Modelle in der Lage sind, bei Sprachmodellierungs- und Reinforcement Learning Problemen eine Spitzenleistung zu erzielen.


Developing conditional generative models for text-to-video synthesis is an extremely challenging yet an important topic of research in machine learning.In this work, we address this problem by introducing Text-Filter conditioning Generative Adversarial Network (TFGAN), a GAN model with novel conditioning scheme that aids improving the text-video associations.With a combination of this conditioning scheme and a deep GAN architecture, TFGAN generates photo-realistic videos from text on very challenging real-world video datasets.In addition, we construct a benchmark synthetic dataset of moving shapes to systematically evaluate our conditioning scheme.Extensive experiments demonstrate that TFGAN significantly outperforms the existing approaches, and can also generate videos of novel categories not seen during training.

Ein effektives textkonditionierendes GAN-Framework zur Erzeugung von Videos aus Text.

In dieser Arbeit wird eine GAN-basierte Methode zur Videogenerierung auf der Grundlage von Textbeschreibungen vorgestellt, mit einer neuen Konditionierungsmethode, die Convolution Filter aus dem kodierten Text erzeugt und diese für eine Convolution im Diskriminator verwendet.

Diese Arbeit schlägt bedingte GAN-Modelle für die Text-Video-Synthese vor: die Entwicklung von CNN-Filtern mit Textmerkmalen und die Erstellung eines Datensatzes für bewegte Formen mit verbesserter Leistung bei der Video-/Bilderzeugung.


Over-parameterization is ubiquitous nowadays in training neural networks to benefit both optimization in seeking global optima and generalization in reducing prediction error.However, compressive networks are desired in many real world applications and direct training of small networks may be trapped in local optima.In this paper, instead of pruning or distilling over-parameterized models to compressive ones, we propose a new approach based on \emph{differential inclusions of inverse scale spaces}, that generates a family of models from simple to complex ones by coupling gradient descent and mirror descent to explore model structural sparsity.It has a simple discretization, called the Split Linearized Bregman Iteration (SplitLBI), whose global convergence analysis in deep learning is established that from any initializations, algorithmic iterations converge to a critical point of empirical risks.Experimental evidence shows that\ SplitLBI may achieve state-of-the-art performance in large scale training on ImageNet-2012 dataset etc., while with \emph{early stopping} it unveils effective subnet architecture with comparable test accuracies to dense models after retraining instead of pruning well-trained ones.

SplitLBI wird auf Deep Learning angewandt, um die strukturelle Spärlichkeit von Modellen zu erforschen. Dabei wird eine Spitzenleistung in ImageNet-2012 erzielt und eine effektive Subnetzarchitektur enthüllt.

Es wird ein optimierungsbasierter Algorithmus vorgeschlagen, um wichtige dünn besetzte Strukturen großer neuronaler Netze zu finden, indem das Lernen der Gewichtsmatrix und die Einschränkung der dünn besetzten Strukturen gekoppelt werden, was garantierte Konvergenz bei nicht-konvexen Optimierungsproblemen bietet.


In this paper, we study the learned iterative shrinkage thresholding algorithm (LISTA) for solving sparse coding problems.  Following assumptions made by prior works, we first discover that the code components in its estimations may be lower than expected, i.e., require gains, and to address this problem, a gated mechanism amenable to theoretical analysis is then introduced.Specific design of the gates is inspired by convergence analyses of the mechanism and hence its effectiveness can be formally guaranteed.In addition to the gain gates, we further introduce overshoot gates for compensating insufficient step size in LISTA.Extensive empirical results confirm our theoretical findings and verify the effectiveness of our method.

Wir schlagen Gated-Mechanismen zur Verbesserung der erlernten ISTA für Sparse Coding vor, mit theoretischen Garantien für die Überlegenheit der Methode.

Es werden Erweiterungen von LISTA vorgeschlagen, die die Unterschätzung durch die Einführung von "Gain Gates" und die Einbeziehung von Impulsen mit "Overshoot Gates" angehen und verbesserte Konvergenzraten zeigen.

Diese Arbeit konzentriert sich auf die Lösung von Sparse-Coding Problemen unter Verwendung von Netzwerken des LISTA-Typs, indem es eine "Gain Gating Funktion" vorschlägt, um die Schwäche der "no false positive" Annahme zu mildern.


The learning of hierarchical representations for image classification has experienced an impressive series of successes due in part to the availability of large-scale labeled data for training.On the other hand, the trained classifiers have traditionally been evaluated on a handful of test images, which are deemed to be extremely sparsely distributed in the space of all natural images.It is thus questionable whether recent performance improvements on the excessively re-used test sets generalize to real-world natural images with much richer content variations.In addition, studies on adversarial learning show that it is effortless to construct adversarial examples that fool nearly all image classifiers, adding more complications to relative performance comparison of existing models.This work presents an efficient framework for comparing image classifiers, which we name the MAximum Discrepancy (MAD) competition.Rather than comparing image classifiers on fixed test sets, we adaptively sample a test set from an arbitrarily large corpus of unlabeled images so as to maximize the discrepancies between the classifiers, measured by the distance over WordNet hierarchy.Human labeling on the resulting small and model-dependent image sets reveals the relative performance of the competing classifiers and provides useful insights on potential ways to improve them.We report the MAD competition results of eleven ImageNet classifiers while noting that the framework is readily extensible and cost-effective to add future classifiers into the competition.

Wir stellen einen effizienten und adaptiven Rahmen für den Vergleich von Bildklassifizierern vor, um die Diskrepanzen zwischen den Klassifizierern zu maximieren, anstelle eines Vergleichs auf festen Testsätzen.

Mechanismus zur Fehlersuche, der Bildklassifikatoren vergleicht, indem er ihre "am meisten nicht übereinstimmende" Testmenge stichprobenartig prüft und die Unstimmigkeit durch einen von der WordNet-Ontologie abgeleiteten semantischen Abstand misst.


Robustness of neural networks has recently been highlighted by the adversarial examples, i.e., inputs added with well-designed  perturbations which are imperceptible to humans but can cause the network to give incorrect outputs.In this paper, we design a new CNN architecture that by itself has good robustness.We introduce a simple but powerful technique, Random Mask, to modify existing CNN structures.We show that CNN with Random Mask achieves state-of-the-art performance against black-box adversarial attacks without applying any adversarial training.We next investigate the adversarial examples which â€œfoolâ€ a CNN with Random Mask.Surprisingly, we find that these adversarial examples often â€œfoolâ€ humans as well.This raises fundamental questions on how to define adversarial examples and robustness properly.

Wir schlagen eine Technik vor, die CNN-Strukturen modifiziert, um die Robustheit zu verbessern und gleichzeitig eine hohe Testgenauigkeit zu erhalten. Wir stellen in Frage, ob die derzeitige Definition von adversarial Beispielen angemessen ist, indem wir Gegenbeispiele erzeugen, die Menschen täuschen können.

In dieser Arbeit wird eine einfache Technik zur Verbesserung der Robustheit neuronaler Netze gegen Blackbox-Angriffe vorgeschlagen.

Die Autoren schlagen eine einfache Methode vor, um die Robustheit von Convolutional Neural Networks gegenüber adversarial Beispielen zu erhöhen, mit überraschend guten Ergebnissen.


Supervised deep learning methods require cleanly labeled large-scale datasets, but collecting such data is difficult and sometimes impossible.There exist two popular frameworks to alleviate this problem: semi-supervised learning and robust learning to label noise.Although these frameworks relax the restriction of supervised learning, they are studied independently.Hence, the training scheme that is suitable when only small cleanly-labeled data are available remains unknown.In this study, we consider learning from bi-quality data as a generalization of these studies, in which a small portion of data is cleanly labeled, and the rest is corrupt.Under this framework, we compare recent algorithms for semi-supervised and robust learning.The results suggest that semi-supervised learning outperforms robust learning with noisy labels.We also propose a training strategy for mixing mixup techniques to learn from such bi-quality data effectively.

Wir schlagen vor, halb-überwachtes und robustes Lernen auf störhaften Etiketten in einer gemeinsamen Umgebung zu vergleichen.

Die Autoren schlagen eine Strategie für das Training eines Modells in einer formalen Umgebung vor, die auf einer Verwechslung basiert und die semi-supervised und robuste Lernaufgaben als Spezialfälle einschließt.


Hierarchical Sparse Coding (HSC) is a powerful model to efficiently represent multi-dimensional, structured data such as images.The simplest solution to solve this computationally hard problem is to decompose it into independent layerwise subproblems.However, neuroscientific evidence would suggest inter-connecting these subproblems as in the Predictive Coding (PC) theory, which adds top-down connections between consecutive layers.In this study, a new model called Sparse Deep Predictive Coding (SDPC) is introduced to assess the impact of this inter-layer feedback connection.In particular, the SDPC is compared with a Hierarchical Lasso (Hi-La) network made out of a sequence of Lasso layers.A 2-layered SDPC and a Hi-La networks are trained on 3 different databases and with different sparsity parameters on each layer.First, we show that the overall prediction error generated by SDPC is lower thanks to the feedback mechanism as it transfers prediction error between layers.Second, we demonstrate that the inference stage of the SDPC is faster to converge than for the Hi-La model.Third, we show that the SDPC also accelerates the learning process.Finally, the qualitative analysis of both models dictionaries, supported by their activation probability, show that the SDPC features are more generic and informative.

In dieser Arbeit wird der positive Effekt von Top-Down-Verbindungen im Hierarchical Sparse Coding Algorithmus experimentell nachgewiesen.

In diesem Beitrag wird eine Studie vorgestellt, in der Techniken für die hierarchische spärliche Kodierung verglichen werden. Es wird gezeigt, dass der Top-Down-Term bei der Reduzierung von Vorhersagefehlern von Vorteil ist und schneller lernen kann.


Explaining a deep learning model can help users understand its behavior and allow researchers to discern its shortcomings.Recent work has primarily focused on explaining models for tasks like image classification or visual question answering.  In this paper, we introduce an explanation approach for image similarity models, where a model's output is a score measuring the similarity of two inputs rather than a classification.  In this task, an explanation depends on both of the input images, so standard methods do not apply.We propose an explanation method that pairs a saliency map identifying important image regions with an attribute that best explains the match.  We find that our explanations provide additional information not typically captured by saliency maps alone, and can also improve performance on the classic task of attribute recognition.Our approach's ability to generalize is demonstrated on two datasets from diverse domains, Polyvore Outfits and Animals with Attributes 2.

Ein Black-Box-Ansatz zur Erklärung der Vorhersagen eines Bildähnlichkeitsmodells.

Stellt eine Methode zur Erklärung von Bildähnlichkeitsmodellen vor, die Attribute identifiziert, die positiv zur Ähnlichkeitsbewertung beitragen, und sie mit einer generierten Auffälligkeitskarte verbindet.

Die Arbeit schlägt einen Erklärungsmechanismus vor, der die typischen Regionen von Auffälligkeitskarten mit Attributen für die Ähnlichkeitsanpassung tiefer neuronaler Netze verbindet.


Adversarial examples have been shown to be an effective way of assessing the robustness of neural sequence-to-sequence (seq2seq) models, by applying perturbations to the input of a model leading to large degradation in performance.However, these perturbations are only indicative of a weakness in the model if they do not change the semantics of the input in a way that would change the expected output.Using the example of machine translation (MT), we propose a new evaluation framework for adversarial attacks on seq2seq models taking meaning preservation into account and demonstrate that existing methods may not preserve meaning in general.Based on these findings, we propose new constraints for attacks on word-based MT systems and show, via human and automatic evaluation, that they produce more semantically similar adversarial inputs.Furthermore, we show that performing adversarial training with meaning-preserving attacks is beneficial to the model in terms of adversarial robustness without hurting test performance.

Wie Sie gegnerische Angriffe auf seq2seq bewerten sollten.

Die Autoren untersuchen Möglichkeiten zur Generierung von Gegenbeispielen und zeigen, dass das adversarial Training mit dem Angriff, der am besten mit den eingeführten Kriterien für den Bedeutungserhalt übereinstimmt, zu einer verbesserten Robustheit gegenüber dieser Art von Angriff führt, ohne dass es zu einer Verschlechterung in der nicht-adversarischen Umgebung kommt.

Das Papier handelt von bedeutungserhaltenden adversen Störungen im Kontext von Seq2Seq-Modellen


We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs.The proposed technique keeps the contribution of positive and negative weights to the layer output in equilibrium.We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.

Eine alternative Normalisierungstechnik zur Batch-Normalisierung

Führt eine Normalisierungstechnik ein, die die Gewichte von Convolutional Layers normalisiert.

In diesem Manuskript wird eine neue schichtweise Transformation, EquiNorm, zur Verbesserung der Batch Normalisierung eingeführt, die nicht die Eingaben in die Schichten, sondern die Schichtgewichte verändert.


We present a framework for building unsupervised representations of entities and their compositions, where each entity is viewed as a probability distribution rather than a fixed length vector.In particular, this distribution is supported over the contexts which co-occur with the entity and are embedded in a suitable low-dimensional space.This enables us to consider the problem of representation learning with a perspective from Optimal Transport and take advantage of its numerous tools such as Wasserstein distance and Wasserstein barycenters.We elaborate how the method can be applied for obtaining unsupervised representations of text and illustrate the performance quantitatively as well as qualitatively on tasks such as measuring sentence similarity and word entailment, where we empirically observe significant gains (e.g., 4.1% relative improvement over Sent2vec and GenSen).The key benefits of the proposed approach include:(a) capturing uncertainty and polysemy via modeling the entities as distributions,(b) utilizing the underlying geometry of the particular task (with the ground cost),(c) simultaneously providing interpretability with the notion of optimal transport between contexts and(d) easy applicability on top of existing point embedding methods.In essence, the framework can be useful for any unsupervised or supervised problem (on text or other modalities); and only requires a co-occurrence structure inherent to many problems.The code, as well as pre-built histograms, are available under https://github.com/context-mover.

Stellen Sie jede Entität als eine Wahrscheinlichkeitsverteilung über Kontexte dar, die in einen Grundraum eingebettet sind.

Es wird vorgeschlagen, Worteinbettungen aus einem Histogramm über Kontextwörter zu konstruieren, anstatt als Punktvektoren, was die Messung von Entfernungen zwischen zwei Wörtern im Sinne eines optimalen Transports zwischen den Histogrammen durch eine Methode ermöglicht, die die Darstellung einer Entität vom Standard "Punkt in einem Vektorraum" zu einem Histogramm mit Bins an einigen Punkten in diesem Vektorraum erweitert.


Over the last few years, the phenomenon of adversarial examples --- maliciously constructed inputs that fool trained machine learning models --- has captured the attention of the research community, especially when the adversary is restricted to making small modifications of a correctly handled input.At the same time, less surprisingly, image classifiers lack human-level performance on randomly corrupted images, such as images with additive Gaussian noise.In this work, we show that these are two manifestations of the same underlying phenomenon.We establish this connection in several ways.First, we find that adversarial examples exist at the same distance scales we would expect from a linear model with the same performance on corrupted images.Next, we show that Gaussian data augmentation during training improves robustness to small adversarial perturbations and that adversarial training improves robustness to several types of image corruptions.Finally, we present a model-independent upper bound on the distance from a corrupted image to its nearest error given test performance and show that in practice we already come close to achieving the bound, so that improving robustness further for the corrupted image distribution requires significantly reducing test error.All of this suggests that improving adversarial robustness should go hand in hand with improving performance in the presence of more general and realistic image corruptions.This yields a computationally tractable evaluation metric for defenses to consider: test error in noisy image distributions.

In Anbetracht der beobachteten Fehlerquoten von Modellen außerhalb der natürlichen Datenverteilung sollte man mit kleinen negativen Störungen rechnen.

In diesem Beitrag wird eine alternative Betrachtungsweise für adversarial Beispiele in hochdimensionalen Räumen vorgeschlagen, indem die "Fehlerrate" in einer Gauß-Verteilung betrachtet wird, die an jedem Testpunkt zentriert ist.


Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training.Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019).However, surprisingly,  the simple baseline of just pre-training and fine-tuning compact models has been overlooked.In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work.Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation.The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements.Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data.One surprising observation is that they have a compound effect even when sequentially applied on the same data.To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.

Untersucht, wie selbstüberwachtes Lernen und Wissensdestillation im Zusammenhang mit der Erstellung kompakter Modelle zusammenwirken.

Untersucht das Training von kompakten, vortrainierten Sprachmodellen durch Destillation und zeigt, dass die Verwendung eines Lehrers zur Destillation eines kompakten Schülermodells besser funktioniert als das direkte Vortraining des Modells.

Dieser Beitrag zeigt, dass das Vortraining eines Schülers direkt auf die maskierte Sprachmodellierung besser ist als die Destillation, und dass es am besten ist, beides zu kombinieren und von diesem vortrainierten Schülermodell zu destillieren.


In this paper, we investigate lossy compression of deep neural networks (DNNs) by weight quantization and lossless source coding for memory-efficient deployment.Whereas the previous work addressed non-universal scalar quantization and entropy coding of DNN weights, we for the first time introduce universal DNN compression by universal vector quantization and universal source coding.In particular, we examine universal randomized lattice quantization of DNNs, which randomizes DNN weights by uniform random dithering before lattice quantization and can perform near-optimally on any source without relying on knowledge of its probability distribution.Moreover, we present a method of fine-tuning vector quantized DNNs to recover the performance loss after quantization.Our experimental results show that the proposed universal DNN compression scheme compresses the 32-layer ResNet (trained on CIFAR-10) and the AlexNet (trained on ImageNet) with compression ratios of $47.1$ and $42.5$, respectively.

Wir stellen das universelle Komprimierungsschema für tiefe neuronale Netze vor, das universell für die Komprimierung beliebiger Modelle anwendbar ist und unabhängig von deren Gewichtsverteilung nahezu optimal funktioniert.

Es wird eine Pipeline für die Netzwerkkomprimierung eingeführt, die der tiefen Komprimierung ähnelt und randomisierte Gitterquantisierung anstelle der klassischen Vektorquantisierung sowie universelle Quellcodierung (bzip2) anstelle der Huffman-Codierung verwendet.


What would be learned by variational autoencoder(VAE) and what influence the disentanglement of VAE?This paper tries to preliminarily address VAE's intrinsic dimension, real factor, disentanglement and indicator issues theoretically in the idealistic situation and implementation issue practically through noise modeling perspective in the realistic case.  On intrinsic dimension issue, due to information conservation, the idealistic VAE learns and only learns intrinsic factor dimension.Besides, suggested by mutual information separation property, the constraint induced by Gaussian prior to the VAE objective encourages the information sparsity in dimension.On disentanglement issue,   subsequently, inspired by information conservation theorem the clarification on disentanglement in this paper is made.On real factor issue, due to factor equivalence, the idealistic VAE possibly learns any factor set in the equivalence class.  On indicator issue, the behavior of current disentanglement metric is discussed, and several performance indicators regarding the disentanglement and generating influence are subsequently raised to evaluate the performance of VAE model and to supervise the used factors.On implementation issue, the experiments under noise modeling and constraints empirically testify the theoretical analysis and also show their own characteristic in pursuing disentanglement.

In diesem Beitrag wird versucht, die Entflechtung theoretisch in einer idealistischen Situation und praktisch durch die Modellierung von Noise in einem realistischen Fall zu untersuchen.

Untersucht die Bedeutung der Modellierung mit Störungen in der Gaußschen VAE und schlägt vor, die Störungen mit Hilfe der Empirical-Bayes-Methode zu trainieren.

Änderung der Behandlung von Störfaktoren bei der Entwicklung von VAE-Modellen


Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization.Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures.We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.

Wir untersuchen die Gewichtsabnahme-Regularisierung für verschiedene Optimierer und identifizieren drei verschiedene Mechanismen, durch die die Gewichtsabnahme die Generalisierung verbessert.

Diskutiert die Auswirkung der Gewichtsabnahme auf das Training von Deep-Network-Modellen mit und ohne Batch Normalisierung und bei Verwendung von Optimierungsmethoden erster/zweiter Ordnung und stellt die Hypothese auf, dass eine größere Lernrate einen Regularisierungseffekt hat.


In this paper we present the first freely available dataset for the development and evaluation of domain adaptation methods, for the sound event detection task.The dataset contains 40 log mel-band energies extracted from $100$ different synthetic sound event tracks, with additive noise from nine different acoustic scenes (from indoor, outdoor, and vehicle environments), mixed at six different sound-to-noise ratios, SNRs, (from -12 to -27 dB with a step of -3 dB), and totaling to 5400 (9 * 100 * 6) sound files and a total length of 30 564 minutes.We provide the dataset as is, the code to re-create the dataset and remix the sound event tracks and the acoustic scenes with different SNRs, and a baseline method that tests the adaptation performance with the proposed dataset and establishes some first results.

Der erste frei verfügbare Domänenanpassungs Datensatz für die Erkennung von Schallereignissen.


This paper aims to address the limitations of mutual information estimators based on variational optimization.By redefining the cost using generalized functions from nonextensive statistical mechanics we raise the upper bound of previous estimators and enable the control of the bias variance trade off.Variational based estimators outperform previous methods especially in high dependence high dimensional scenarios found in machine learning setups.Despite their performance, these estimators either exhibit a high variance or are upper bounded by log(batch size).Our approach inspired by nonextensive statistical mechanics uses different generalizations for the logarithm and the exponential in the partition function.This enables the estimator to capture changes in mutual information over a wider range of dimensions and correlations of the input variables whereas previous estimators saturate them.

Schätzer der gegenseitigen Information auf der Grundlage der nicht-extensiven statistischen Mechanik.

In diesem Beitrag wird versucht, neuartige Variationsschranken für die gegenseitige Information aufzustellen, indem der Parameter q eingeführt und die q-Algebra definiert wird. Es wird gezeigt, dass die Schranken eine geringere Varianz haben und hohe Werte erreichen.


Generative adversarial networks (GANs) are a widely used framework for learning generative models.Wasserstein GANs (WGANs), one of the most successful variants of GANs, require solving a minmax problem to global optimality, but in practice, are successfully trained with stochastic gradient descent-ascent.In this paper, we show that, when the generator is a one-layer network, stochastic gradient descent-ascent converges to a global solution in polynomial time and sample complexity.

Wir zeigen, dass der stochastische Gradientenabstieg zu einem globalen Optimum für WGAN mit einem einschichtigen Generatoren-Netzwerk konvergiert.

Es wird versucht zu beweisen, dass der Stochastic Gradient Descent-Ascent zu einer globalen Lösung für das Min-Max Problem des WGAN konvergieren kann.


Classifiers such as deep neural networks have been shown to be vulnerable against adversarial perturbations on problems with high-dimensional input space.While adversarial training improves the robustness of classifiers against such adversarial perturbations, it leaves classifiers sensitive to them on a non-negligible fraction of the inputs.We argue that there are two different kinds of adversarial perturbations: shared perturbations which fool a classifier on many inputs and singular perturbations which only fool the classifier on a small fraction of the data.We find that adversarial training increases the robustness of classifiers against shared perturbations.Moreover, it is particularly effective in removing universal perturbations, which can be seen as an extreme form of shared perturbations.Unfortunately, adversarial training does not consistently increase the robustness against singular perturbations on unseen inputs.However, we find that adversarial training decreases robustness of the remaining perturbations against image transformations such as changes to contrast and brightness or  Gaussian blurring.It thus makes successful attacks on the classifier in the physical world less likely.Finally, we show that even singular perturbations can be easily detected and must thus exhibit generalizable patterns even though the perturbations are specific for certain inputs.

Wir zeigen empirisch, dass das adversarische Training universelle Störungen wirksam beseitigt, die adversarischen Beispiele weniger robust gegenüber Bildtransformationen macht und sie für einen Erkennungsansatz erkennbar bleiben.

Analysiert adversariales Training und seine Auswirkung auf universelle adversarische Beispiele sowie standardmäßige (Basisiteration) adversarische Beispiele und wie adversariales Training die Erkennung beeinflusst.

Die Autoren zeigen, dass adversarial Training wirksam gegen "gemeinsame" adversarial Störungen schützt, insbesondere gegen universelle Störungen, aber weniger wirksam gegen singuläre Störungen.


We address the challenging problem of efficient deep learning model deployment, where the goal is to design neural network architectures that can fit different hardware platform constraints.Most of the traditional approaches either manually design or use Neural Architecture Search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally expensive and unscalable.Our key idea is to decouple model training from architecture search to save the cost.To this end, we propose to train a once-for-all network (OFA) that supports diverse architectural settings (depth, width, kernel size, and resolution).Given a deployment scenario, we can then quickly get a specialized sub-network by selecting from the OFA network without additional training.To prevent interference between many sub-networks during training, we also propose a novel progressive shrinking algorithm, which can train a surprisingly large number of sub-networks ($> 10^{19}$) simultaneously.Extensive experiments on various hardware platforms (CPU, GPU, mCPU, mGPU, FPGA accelerator) show that OFA consistently outperforms SOTA NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3) while reducing orders of magnitude GPU hours and $CO_2$ emission.In particular, OFA achieves a new SOTA 80.0% ImageNet top1 accuracy under the mobile setting ($<$600M FLOPs).Code and pre-trained models are released at https://github.com/mit-han-lab/once-for-all.

Wir stellen Techniken vor, um ein einziges Netzwerk zu trainieren, das für viele Hardware-Plattformen geeignet ist.

Die Methode führt zu einem Netz, aus dem man Teilnetze für verschiedene Ressourcenbeschränkungen (Latenz, Speicher) extrahieren kann, die gute Leistungen erbringen, ohne dass ein erneutes Training erforderlich ist.

In diesem Beitrag wird versucht, das Problem der Suche nach den besten Architekturen für spezielle ressourcenbeschränkte Einsatzszenarien mit einer auf Vorhersagen basierenden NAS-Methode anzugehen.


A deep generative model is a powerful method of learning a data distribution, which has achieved tremendous success in numerous scenarios.However, it is nontrivial for a single generative model to faithfully capture the distributions of the complex data such as images with complicate structures.In this paper, we propose a novel approach of cascaded boosting for boosting generative models, where meta-models (i.e., weak learners) are cascaded together to produce a stronger model.Any hidden variable meta-model can be leveraged as long as it can support the likelihood evaluation.We derive a decomposable variational lower bound of the boosted model, which allows each meta-model to be trained separately and greedily.We can further improve the learning power of the generative models by combing our cascaded boosting framework with the multiplicative boosting framework.

Vorschlag für einen Ansatz zur Verstärkung generativer Modelle durch Kaskadierung von Modellen mit verborgenen Variablen

In diesem Papier wird ein neuartiger Ansatz des kaskadierten Boostings für generative Modelle vorgeschlagen, bei dem jedes Meta-Modell separat und gierig trainiert werden kann.


Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks.Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline.We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena.We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

Wir untersuchen die Satzstruktur in ELMo und verwandten kontextuellen Einbettungsmodellen. Wir stellen fest, dass bestehende Modelle die Syntax effizient kodieren und Hinweise auf weitreichende Abhängigkeiten zeigen, aber nur geringe Verbesserungen bei semantischen Aufgaben bieten.

Die von den Autoren vorgeschlagene "Edge Probing"-Methode konzentriert sich auf die Beziehungen zwischen den Bereichen und nicht auf die einzelnen Wörter, was es den Autoren ermöglicht, syntaktische Konstituenten, Abhängigkeiten, Entitätskennzeichnungen und semantische Rollenbezeichnungen zu untersuchen.

Bietet neue Einblicke in die Erfassung kontextualisierter Worteinbettungen durch die Zusammenstellung einer Reihe von Edge Probing Aufgaben.


Deep reinforcement learning has succeeded in sophisticated games such as Atari, Go, etc.Real-world decision making, however, often requires reasoning with partial information extracted from complex visual observations.This paper presents  Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for partial and complex observations.DPFRL encodes a differentiable particle filter with learned transition and observation models in a neural network, which allows for reasoning with partial observations over multiple time steps.While a standard particle filter relies on a generative observation model, DPFRL learns a discriminatively parameterized model that is training directly for decision making.We show that the discriminative parameterization results in significantly improved performance, especially for tasks with complex visual observations, because it circumvents the difficulty of modelling observations explicitly.In most cases, DPFRL outperforms state-of-the-art POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark that we introduce.We further show that DPFRL performs well for visual navigation with real-world data.

Wir stellen DPFRL vor, ein Rahmenwerk für Reinforcement Learning unter partiellen und komplexen Beobachtungen mit einem voll differenzierbaren diskriminativen Partikelfilter.

Es werden Ideen für das Training von DLR-Agenten mit latenten Zustandsvariablen vorgestellt, die als Glaubensverteilung modelliert sind, so dass sie mit teilweise beobachteten Umgebungen umgehen können.

Diese Arbeit stellt eine prinzipielle Methode für POMDP RL vor: Diskriminatives Partikelfilter Reinforcement Learning, das es erlaubt, mit partiellen Beobachtungen über mehrere Zeitschritte zu argumentieren und dabei den Stand der Technik in Benchmarks zu erreichen.


Extending models with auxiliary latent variables is a well-known technique to in-crease model expressivity.Bachman & Precup (2015); Naesseth et al. (2018); Cremer et al. (2017); Domke & Sheldon (2018) show that Importance Weighted Autoencoders (IWAE) (Burda et al., 2015) can be viewed as extending the variational family with auxiliary latent variables.Similarly, we show that this view encompasses many of the recent developments in variational bounds (Maddisonet al., 2017; Naesseth et al., 2018; Le et al., 2017; Yin & Zhou, 2018; Molchanovet al., 2018; Sobolev & Vetrov, 2018).The success of enriching the variational family with auxiliary latent variables motivates applying the same techniques to the generative model.We develop a generative model analogous to the IWAE bound and empirically show that it outperforms the recently proposed Learned Accept/Reject Sampling algorithm (Bauer & Mnih, 2018), while being substantially easier to implement.Furthermore, we show that this generative process provides new insights on ranking Noise Contrastive Estimation (Jozefowicz et al.,2016; Ma & Collins, 2018) and Contrastive Predictive Coding (Oord et al., 2018).

Monte Carlo Ziele werden mit Hilfe der Variationsinferenz für Hilfsvariablen analysiert, was zu einer neuen Analyse von CPC und NCE sowie zu einem neuen generativen Modell führt.

Vorschlage einer anderen Sichtweise zur Verbesserung der Variationsschranken mit latenten Hilfsvariablenmodellen und untersuchen der Verwendung dieser Modelle im generativen Modell.


Stochastic Gradient Descent or SGD is the most popular optimization algorithm for large-scale problems.SGD estimates the gradient by uniform sampling with sample size one.There have been several other works that suggest faster epoch wise convergence by using weighted non-uniform sampling for better gradient estimates.Unfortunately, the per-iteration cost of maintaining this adaptive distribution for gradient estimation is more than calculating the full gradient.As a result, the false impression of faster convergence in iterations leads to slower convergence in time, which we call a chicken-and-egg loop.In this paper, we break this barrier by providing the first demonstration of a sampling scheme, which leads to superior gradient estimation, while keeping the sampling cost per iteration similar to that of the uniform sampling.Such an algorithm is possible due to the sampling view of Locality Sensitive Hashing (LSH), which came to light recently.As a consequence of superior and fast estimation, we reduce the running time of all existing gradient descent algorithms.We demonstrate the benefits of our proposal on both SGD and AdaGrad.

Wir verbessern den Ablauf aller bestehenden Gradientenabstiegsalgorithmen.

Die Autoren schlagen vor, stochastische Gradienten aus einer monotonen Funktion, die proportional zu den Gradientenstärken ist, mit Hilfe von LSH abzutasten.

Betrachtet SGD über ein Ziel in der Form einer Summe über Beispiele eines quadratischen Verlustes.


In recent years we have made significant progress identifying computational principles that underlie neural function.While not yet complete, we have sufficient evidence that a synthesis of these ideas could result in an understanding of how neural computation emerges from a combination of innate dynamics and plasticity, and which could potentially be used to construct new AI technologies with unique capabilities.I discuss the relevant principles, the advantages they have for computation, and how they can benefit AI.Limitations of current AI are generally recognized, but fewer people are aware that we understand enough about the brain to immediately offer novel AI formulations.

Die Grenzen der gegenwärtigen KI sind allgemein anerkannt, aber weniger Menschen sind sich bewusst, dass wir genug über das Gehirn wissen, um sofort neue KI-Formulierungen anzubieten.


Recent work has demonstrated how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments.We propose question-answering as a general paradigm to decode and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling â€“ action-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019).After training agents with these predictive objectives in a visually-rich, 3D environment with an assortment of objects, colors, shapes, and spatial configurations, we probe their internal state representations with a host of synthetic (English) questions, without backpropagating gradients from the question-answering decoder into the agent.The performance of different agents when probed in this way reveals that they learn to encode detailed, and seemingly compositional, information about objects, properties and spatial relations from their physical environment.Our approach is intuitive, i.e. humans can easily interpret the responses of the model as opposed to inspecting continuous vectors, and model-agnostic, i.e. applicable to any modeling approach.By revealing the implicit knowledge of objects, quantities, properties and relations acquired by agents as they learn, question-conditional agent probing can stimulate the design and development of stronger predictive learning objectives.

Wir verwenden die Beantwortung von Fragen, um zu bewerten, wie viel Wissen über die Umwelt Agenten durch selbstüberwachte Vorhersage lernen können.

Er schlägt QA als ein Werkzeug vor, um zu untersuchen, was Agenten in der Welt lernen, und argumentiert, dass dies eine intuitive Methode für Menschen ist, die beliebige Komplexität zulässt.

Die Autoren schlagen ein Framework vor, um die von prädiktiven Modellen erstellten Repräsentationen zu bewerten, die genügend Informationen enthalten, um Fragen über die Umgebung zu beantworten, auf die sie trainiert wurden. Sie zeigen, dass die von SimCore erstellten Repräsentationen genügend Informationen enthielten, damit das LSTM die Fragen genau beantworten konnte.


In most real-world scenarios, training datasets are highly class-imbalanced, where deep neural networks suffer from generalizing to a balanced testing criterion.In this paper, we explore a novel yet simple way to alleviate this issue via synthesizing less-frequent classes with adversarial examples of other classes.Surprisingly, we found this counter-intuitive method can effectively learn generalizable features of minority classes by transferring and leveraging the diversity of the majority information.Our experimental results on various types of class-imbalanced datasets in image classification and natural language processing show that the proposed method not only improves the generalization of minority classes significantly compared to other re-sampling or re-weighting methods, but also surpasses other methods of state-of-art level for the class-imbalanced classification.

Wir entwickeln eine neue Methode zur unausgewogenen Klassifizierung unter Verwendung von adversarial Beispielen.

Schlägt ein neues Optimierungsziel vor, das synthetische Stichproben erzeugt, indem es die Mehrheitsklassen anstelle der Minderheitsklassen überabtasten lässt und so das Problem der Überanpassung der Minderheitsklassen löst.

Die Autoren schlagen vor, die Ungleichgewichtsklassifizierung mit Hilfe von Re-Sampling Methoden anzugehen. Sie zeigen, dass Gegenbeispiele in der Minderheitenklasse helfen würden, ein neues Modell zu trainieren, das besser verallgemeinert.


Active matter consists of active agents which transform energy extracted from surroundings into momentum, producing a variety of collective phenomena.A model, synthetic active system composed of microtubule polymers driven by protein motors spontaneously forms a liquid-crystalline nematic phase.Extensile stress created by the protein motors precipitates continuous buckling and folding of the microtubules creating motile topological defects and turbulent fluid flows.Defect motion is determined by the rheological properties of the material; however, these remain largely unquantified.Measuring defects dynamics can yield fundamental insights into active nematics, a class of materials that include bacterial films and animal cells.Current methods for defect detection lack robustness and precision, and require fine-tuning for datasets with different visual quality.  In this study, we applied Deep Learning to train a defect detector to automatically analyze microscopy videos of the microtubule active nematic.  Experimental results indicate that our method is robust and accurate.It is expected to significantly increase the amount of video data that can be processed.

Eine interessante Anwendung von CNN in Experimenten zur Physik der weichen kondensierten Materie.

Die Autoren zeigen, dass ein Deep-Learning Ansatz sowohl die Erkennungsgenauigkeit als auch die Erkennungsrate von Defekten in nematischen Flüssigkristallen verbessern kann.

Anwendung eines bekannten neuronalen Modells (YOLO) zur Erkennung von Bounding Boxes von Objekten in Bildern.


In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed.The results of our experiment show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.

Eine Analyse der Auswirkungen von Kompositionalität und Lokalität auf das Repräsentationslernen beim Zero-Shot Lernen.

Schlägt einen Evaluierungsrahmen für ZSL vor, bei dem das Modell nicht vortrainiert werden darf und stattdessen die Modellparameter zufällig initialisiert werden, um besser zu verstehen, was in ZSL passiert.


It is becoming increasingly clear that many machine learning classifiers are vulnerable to adversarial examples.In attempting to explain the origin of adversarial examples, previous studies have typically focused on the fact that neural networks operate on high dimensional data, they overfit, or they are too linear.Here we show that distributions of logit differences have a universal functional form.This functional form is independent of architecture, dataset, and training protocol; nor does it change during training.This leads to adversarial error having a universal scaling, as a power-law, with respect to the size of the adversarial perturbation.We show that this universality holds for a broad range of datasets (MNIST, CIFAR10, ImageNet, and random data), models (including state-of-the-art deep networks, linear models, adversarially trained networks, and networks trained on randomly shuffled labels), and attacks (FGSM, step l.l., PGD).Motivated by these results, we study the effects of reducing prediction entropy on adversarial robustness.Finally, we study the effect of network architectures on adversarial sensitivity.To do this, we use neural architecture search with reinforcement learning to find adversarially robust architectures on CIFAR10.Our resulting architecture is more robust to white \emph{and} black box attacks compared to previous attempts.

Bei allen untersuchten Datensätzen und Modellen weisen die Fehler des Gegners eine ähnliche Potenzgesetzform auf, und die Architektur spielt eine Rolle.


Reinforcement learning (RL) has led to increasingly complex looking behavior in recent years.However, such complexity can be misleading and hides over-fitting.We find that visual representations may be a useful metric of complexity, and both correlates well objective optimization and causally effects reward optimization.We then propose curious representation learning (CRL) which allows us to use better visual representation learning algorithms to correspondingly increase visual representation in policy through an intrinsic objective on both simulated environments and transfer to real images.Finally, we show better visual representations induced by CRL allows us to obtain better performance on Atari without any reward than other curiosity objectives.

Wir stellen eine Formulierung von Neugier als ein visuelles Repräsentationslernproblem vor und zeigen, dass sie gute visuelle Repräsentationen in Agenten ermöglicht.

In diesem Beitrag wird neugierbasiertes RL-Training als Lernen eines visuellen Repräsentationsmodells formuliert, wobei argumentiert wird, dass die Konzentration auf bessere LR und die Maximierung des Modellverlusts für neuartige Szenen eine bessere Gesamtleistung ergibt.


This paper introduces the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances comprising the scene and infer their complete object geometry.This enables a semantically meaningful decomposition of a scanned scene into individual, complete 3D objects, including hidden and unobserved object parts.This will open up new possibilities for interactions with object in a scene, for instance for virtual or robotic agents.To address this task, we propose 3D-SIC, a new data-driven approach that jointly detects object instances and predicts their completed geometry.The core idea of 3D-SIC is a novel end-to-end 3D neural network architecture that leverages joint color and geometry feature learning.The fully-convolutional nature of our 3D network enables efficient inference of semantic instance completion for 3D scans at scale of large indoor environments in a single forward pass.In a series evaluation, we evaluate on both real and synthetic scan benchmark data, where we outperform state-of-the-art approaches by over 15 in mAP@0.5 on ScanNet, and over 18 in mAP@0.5 on SUNCG.

Aus einem unvollständigen RGB-D-Scan einer Szene wollen wir die einzelnen Objektinstanzen, aus denen die Szene besteht, erkennen und ihre vollständige Objektgeometrie ableiten.

Vorschlagen einer durchgängigen 3D CNN-Struktur, die Farbmerkmale und 3D-Merkmale kombiniert, um die fehlende 3D-Struktur einer Szene aus RGB-D-Scans vorherzusagen.

Die Autoren schlagen ein neuartiges durchgängiges 3D Convolutional Network vor, das die semantische 3D Instanzvervollständigung in Form von Objektbegrenzungsrahmen, Klassenbezeichnungen und vollständiger Objektgeometrie vorhersagt.


Style transfer usually refers to the task of applying color and texture information from a specific style image to a given content image while preserving the structure of the latter.Here we tackle the more generic problem of semantic style transfer: given two unpaired collections of images, we aim to learn a mapping between the corpus-level style of each collection, while preserving semantic content shared across the two domains.We introduce XGAN ("Cross-GAN"), a dual adversarial autoencoder, which captures a shared representation of the common domain semantic content in an unsupervised way, while jointly learning the domain-to-domain image translations in both directions.  We exploit ideas from the domain adaptation literature and define a semantic consistency loss which encourages the model to preserve semantics in the learned embedding space.We report promising qualitative results for the task of face-to-cartoon translation.The cartoon dataset we collected for this purpose will also be released as a new benchmark for semantic style transfer.

XGAN ist ein unüberwachtes Modell für die Bild-zu-Bild-Übersetzung auf Merkmalsebene, das auf semantische Stilübertragungsprobleme wie die Gesicht-zu-Karikatur-Aufgabe angewendet wird, für die wir einen neuen Datensatz vorstellen.

In diesem Beitrag wird ein neues GAN-basiertes Modell für die ungepaarte Bild-zu-Bild-Übersetzung vorgeschlagen, das dem DTN ähnelt.


Training neural networks on large datasets can be accelerated by distributing the workload over a network of machines.As datasets grow ever larger, networks of hundreds or thousands of machines become economically viable.The time cost of communicating gradients limits the effectiveness of using such large machine counts, as may the increased chance of network faults.We explore a particularly simple algorithm for robust, communication-efficient learning---signSGD.Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote.This algorithm uses 32x less communication per iteration than full-precision, distributed SGD.Under natural conditions verified by experiment, we prove that signSGD converges in the large and mini-batch settings, establishing convergence for a parameter regime of Adam as a byproduct.Aggregating sign gradients by majority vote means that no individual worker has too much power.We prove that unlike SGD, majority vote is robust when up to 50% of workers behave adversarially.The class of adversaries we consider includes as special cases those that invert or randomise their gradient estimate.On the practical side, we built our distributed training system in Pytorch.Benchmarking against the state of the art collective communications library (NCCL), our framework---with the parameter server housed entirely on one machine---led to a 25% reduction in time for training resnet50 on Imagenet when using 15 AWS p3.2xlarge machines.

Die Arbeiter senden Gradientenzeichen an den Server, und die Aktualisierung wird durch Mehrheitsabstimmung entschieden. Wir zeigen, dass dieser Algorithmus konvergent, kommunikationseffizient und fehlertolerant ist, sowohl in der Theorie als auch in der Praxis.

Stellt eine verteilte Implementierung von signSGD mit Mehrheitsentscheidung als Aggregation vor.