Profiling cellular phenotypes from microscopic imaging can provide meaningful biological information resulting from various factors affecting the cells.One motivating application is drug development: morphological cell features can be captured from images, from which similarities between different drugs applied at different dosages can be quantified.The general approach is to find a function mapping the images to an embedding space of manageable dimensionality whose geometry captures relevant features of the input images.An important known issue for such methods is separating relevant biological signal from nuisance variation.For example, the embedding vectors tend to be more correlated for cells that were cultured and imaged during the same week than for cells from a different week, despite having identical drug compounds applied in both cases.In this case, the particular batch a set of experiments were conducted in constitutes the domain of the data; an ideal set of image embeddings should contain only the relevant biological information (e.g. drug effects).We develop a general framework for adjusting the image embeddings in order to `forget' domain-specific information while preserving relevant biological information.To do this, we minimize a loss function based on distances between marginal distributions (such as the Wasserstein distance) of embeddings across domains for each replicated treatment.For the dataset presented, the replicated treatment is the negative control.We find that for our transformed embeddings (1) the underlying geometric structure is not only preserved but the embeddings also carry improved biological signal (2) less domain-specific information is present.

Wir korrigieren unerwünschte Abweichungen bei der Einbettung von Bildern in verschiedenen Bereichen, wobei nur relevante Informationen erhalten bleiben.

Erörtert eine Methode zur Anpassung von Bildeinbettungen, um technische Variationen von biologischen Signalen zu trennen.

Die Autoren stellen eine Methode vor, um domänenspezifische Informationen zu entfernen und gleichzeitig die relevanten biologischen Informationen zu erhalten, indem sie ein Netzwerk trainieren, das die Wasserstein-Distanz zwischen den Distrbutionen minimiert.


This paper presents a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size.MINE is  back-propable and we prove that it is strongly consistent.We illustrate a handful of applications in which MINE is succesfully applied  to enhance the property of generative models in both unsupervised and supervised settings.We apply our framework to estimate the information bottleneck, and apply it in tasks related to supervised classification problems.Our results  demonstrate substantial added flexibility and improvement in these settings.

Ein in Stichprobengröße und Dimensionen skalierbarer Schätzer der gegenseitigen Information.


Reinforcement learning methods have recently achieved impressive results on a wide range of control problems.However, especially with complex inputs, they still require an extensive amount of training data in order to converge to a meaningful solution.This limitation largely prohibits  their usage for complex input spaces such as video signals, and it is still impossible to use it for a number of complex problems in a real world environments, including many of those for video based control.Supervised learning, on the contrary, is capable of learning on a relatively small number of samples, however it does not take into account reward-based control policies and is not capable to provide independent control policies.  In this article we propose a model-free control method, which uses a combination of reinforcement and supervised learning for autonomous control and paves the way towards policy based control in real world environments.We use SpeedDreams/TORCS video game to demonstrate that our approach requires much less samples (hundreds of thousands against millions or tens of millions) comparing to the state-of-the-art reinforcement learning techniques on similar data, and at the same time overcomes both supervised and reinforcement learning approaches in terms of quality.Additionally, we demonstrate the applicability of the method to MuJoCo control problems.

Die neue Kombination aus verstärktem und überwachtem Lernen, die die Anzahl der erforderlichen Proben für das Training auf Videos drastisch reduziert.

In dieser Arbeit wird vorgeschlagen, gelabelte kontrollierte Daten zu nutzen, um das verstärkungsbasierte Lernen einer Kontrollpolitik zu beschleunigen.


A typical experiment to study cognitive function is to train animals to perform tasks, while the researcher records the electrical activity of the animals neurons.The main obstacle faced, when using this type of electrophysiological experiment to uncover the circuit mechanisms underlying complex behaviors, is our incomplete access to relevant circuits in the brain.One promising approach is to model neural circuits using an artificial neural network (ANN), which can provide complete access to the â€œneural circuitsâ€ responsible for a behavior.More recently, reinforcement learning models have been adopted to understand the functions of cortico-basal ganglia circuits as reward-based learning has been found in mammalian brain.In this paper, we propose a Biologically-plausible Actor-Critic with Episodic Memory (B-ACEM) framework to model a prefrontal cortex-basal ganglia-hippocampus (PFC-BG) circuit, which is verified to capture the behavioral findings from a well-known perceptual decision-making task, i.e., random dots motion discrimination.This B-ACEM framework links neural computation to behaviors, on which we can explore how episodic memory should be considered to govern future decision.Experiments are conducted using different settings of the episodic memory and results show that all patterns of episodic memories can speed up learning.In particular, salient events are prioritized to propagate reward information and guide decisions.Our B-ACEM framework and the built-on experiments give inspirations to both designs for more standard decision-making models in biological system and a more biologically-plausible ANN.

Schnelles Lernen über das episodische Gedächtnis, verifiziert durch einen biologisch plausiblen Rahmen für den präfrontalen Kortex-Basalganglien-Hippocampus-Schaltkreis (PFC-BG).


Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in deep learning and approximation theory.In a seminal paper, Telgarsky high- lighted the benefits of depth by presenting a family of functions (based on sim- ple triangular waves) for which DNNs achieve zero classification error, whereas shallow networks with fewer than exponentially many nodes incur constant error.Even though Telgarskyâ€™s work reveals the limitations of shallow neural networks, it doesnâ€™t inform us on why these functions are difficult to represent and in fact he states it as a tantalizing open question to characterize those functions that cannot be well-approximated by smaller depths.In this work, we point to a new connection between DNNs expressivity and Sharkovskyâ€™s Theorem from dynamical systems, that enables us to characterize the depth-width trade-offs of ReLU networks for representing functions based on the presence of a generalized notion of fixed points, called periodic points (a fixed point is a point of period 1).Motivated by our observation that the triangle waves used in Telgarskyâ€™s work contain points of period 3 â€“ a period that is special in that it implies chaotic behaviour based on the celebrated result by Li-Yorke â€“ we proceed to give general lower bounds for the width needed to represent periodic functions as a function of the depth.Technically, the crux of our approach is based on an eigenvalue analysis of the dynamical systems associated with such functions.

In dieser Arbeit zeigen wir eine neue Verbindung zwischen der Ausdruckskraft von DNNs und Sharkovskys Theorem aus dynamischen Systemen auf, die es uns ermöglicht, den Tiefen-Breiten Kompromiss von ReLU-Netzen zu charakterisieren .

Zeigt, wie die Ausdruckskraft von NN von ihrer Tiefe und Breite abhängt, und fördert das Verständnis für den Nutzen tiefer Netze zur Darstellung bestimmter Funktionsklassen.

Die Autoren leiten mit Hilfe der Analyse dynamischer Systeme Bedingungen für einen Tiefen-Breiten Kompromiss ab, wenn ReLu-Netze in der Lage sind, periodische Funktionen darzustellen.


We investigate low-bit quantization to reduce computational cost of deep neural network (DNN) based keyword spotting (KWS).We propose approaches to further reduce quantization bits via integrating quantization into keyword spotting model training, which we refer to as quantization-aware training.Our experimental results on large dataset indicate that quantization-aware training can recover performance models quantized to lower bits representations.By combining quantization-aware training and weight matrix factorization, we are able to significantly reduce model size and computation for small-footprint keyword spotting, while maintaining performance.

Wir untersuchen quantisierungsbewusstes Training in sehr niedrig quantisierten Keyword-Spottern, um die Kosten für das On-Device Keyword-Spotting zu reduzieren.

In diesem Beitrag wird eine Kombination aus Low-Rank-Decomposition und Quanitization-Ansatz zur Komprimierung von DNN-Modellen für das Keyword-Spotting vorgeschlagen.


Single-cell RNA-sequencing (scRNA-seq) is a powerful tool for analyzing biological systems.However, due to biological and technical noise, quantifying the effects of multiple experimental conditions presents an analytical challenge.To overcome this challenge, we developed MELD: Manifold Enhancement of Latent Dimensions.MELD leverages tools from graph signal processing to learn a latent dimension within the data scoring the prototypicality of each datapoint with respect to experimental or control conditions.We call this dimension the Enhanced Experimental Signal (EES).MELD learns the EES by filtering the noisy categorical experimental label in the graph frequency domain to recover a smooth signal with continuous values.This method can be used to identify signature genes that vary between conditions and identify which cell types are most affected by a given perturbation.We demonstrate the advantages of MELD analysis in two biological datasets, including T-cell activation in response to antibody-coated beads and treatment of human pancreatic islet cells with interferon gamma.

Ein neuartiger Rahmen für die Graphsignalverarbeitung zur Quantifizierung der Auswirkungen von experimentellen Störungen in biomedizinischen Einzelzelldaten.

In diesem Beitrag werden mehrere Methoden zur Verarbeitung von Versuchsergebnissen zu biologischen Zellen vorgestellt und ein MELD-Algorithmus vorgeschlagen, der harte Gruppenzuordnungen auf weiche Zuordnungen abbildet, so dass relevante Gruppen von Zellen geclustert werden können.


Models of user behavior are critical inputs in many prescriptive settings and can be viewed as decision rules that transform state information available to the user into actions.Gaussian processes (GPs), as well as nonlinear extensions thereof, provide a flexible framework to learn user models in conjunction with approximate Bayesian inference.However, the resulting models may not be interpretable in general.We propose decision-rule GPs (DRGPs) that apply GPs in a transformed space defined by decision rules that have immediate interpretability to practitioners.We illustrate this modeling tool on a real application and show that structural variational inference techniques can be used with DRGPs.We find that DRGPs outperform the direct use of GPs in terms of out-of-sample performance.

Wir schlagen eine Klasse von Benutzermodellen vor, die auf der Anwendung von Gaußschen Prozessen auf einen durch Entscheidungsregeln definierten transformierten Raum basieren.


While Bayesian optimization (BO) has achieved great success in optimizing expensive-to-evaluate black-box functions, especially tuning hyperparameters of neural networks, methods such as random search (Li et al., 2016) and multi-fidelity BO (e.g. Klein et al. (2017)) that exploit cheap approximations, e.g. training on a smaller training data or with fewer iterations, can outperform standard BO approaches that use only full-fidelity observations.In this paper, we propose a novel Bayesian optimization algorithm, the continuous-fidelity knowledge gradient (cfKG) method, that can be used when fidelity is controlled by one or more continuous settings such as training data size and the number of training iterations.cfKG characterizes the value of the information gained by sampling a point at a given fidelity, choosing to sample at the point and fidelity with the largest value per unit cost.Furthermore, cfKG can be generalized, following Wu et al. (2017), to settings where derivatives are available in the optimization process, e.g. large-scale kernel learning, and where more than one point can be evaluated simultaneously.Numerical experiments show that cfKG outperforms state-of-art algorithms when optimizing synthetic functions, tuning convolutional neural networks (CNNs) on CIFAR-10 and SVHN, and in large-scale kernel learning.

Wir schlagen einen Bayes-optimalen Bayes'schen Optimierungsalgorithmus für das Tuning von Hyperparametern vor, der billige Approximationen ausnutzt.

Untersucht die Optimierung von Hyperparametern durch Bayes'sche Optimierung unter Verwendung des Knowledge Gradient Frameworks und ermöglicht dem Bayes'schen Optimierer die Abstimmung von Treue und Kosten.


Neural networks trained only to optimize for training accuracy can often be fooled by adversarial examples --- slightly perturbed inputs misclassified with high confidence.Verification of networks enables us to gauge their vulnerability to such adversarial examples.We formulate verification of piecewise-linear neural networks as a mixed integer program.On a representative task of finding minimum adversarial distortions, our verifier is two to three orders of magnitude quicker than the state-of-the-art.We achieve this computational speedup via tight formulations for non-linearities, as well as a novel presolve algorithm that makes full use of all information available.The computational speedup allows us to verify properties on convolutional and residual networks with over 100,000 ReLUs --- several orders of magnitude more than networks previously verified by any complete verifier.In particular, we determine for the first time the exact adversarial accuracy of an MNIST classifier to perturbations with bounded l-âˆž norm Îµ=0.1: for this classifier, we find an adversarial example for 4.38% of samples, and a certificate of robustness to norm-bounded perturbations for the remainder.Across all robust training procedures and network architectures considered, and for both the MNIST and CIFAR-10 datasets, we are able to certify more samples than the state-of-the-art and find more adversarial examples than a strong first-order attack.

Wir verifizieren die Robustheit von tiefen neuronalen Modellen mit über 100.000 ReLUs, wobei wir mehr Beispiele als der Stand der Technik zertifizieren und mehr negative Beispiele finden als ein starker Angriff erster Ordnung.

Führt eine sorgfältige Studie über gemischt-ganzzahlige lineare Programmieransätze zur Überprüfung der Robustheit neuronaler Netze gegenüber Störungen durch Gegner durch und schlägt drei Verbesserungen der MILP-Formulierungen zur Überprüfung neuronaler Netze vor.


Uncertainty estimation is an essential step in the evaluation of the robustness for deep learning models in computer vision, especially when applied in risk-sensitive areas.However, most state-of-the-art deep learning models either fail to obtain uncertainty estimation or need significant modification (e.g., formulating a proper Bayesian treatment) to obtain it.None of the previous methods are able to take an arbitrary model off the shelf and generate uncertainty estimation without retraining or redesigning it.To address this gap, we perform the first systematic exploration into training-free uncertainty estimation. We propose three simple and scalable methods to analyze the variance of output from a trained network under tolerable perturbations: infer-transformation, infer-noise, and infer-dropout.They operate solely during inference, without the need to re-train, re-design, or fine-tune the model, as typically required by other state-of-the-art uncertainty estimation methods.Surprisingly, even without involving such perturbations in training, our methods produce comparable or even better uncertainty estimation when compared to other training-required state-of-the-art methods.Last but not least, we demonstrate that the uncertainty from our proposed methods can be used to improve the neural network training.

Eine Reihe von Methoden, um eine Unsicherheitsabschätzung eines beliebigen Modells zu erhalten, ohne es neu zu entwerfen, neu zu trainieren oder zu fine-tunen.

Beschreibt mehrere Ansätze zur Messung der Unsicherheit in beliebigen neuronalen Netzen, wenn während des Trainings keine Verzerrung auftritt.


Capturing spatiotemporal dynamics is an essential topic in video recognition.In this paper, we present learnable higher-order operation as a generic family of building blocks for capturing higher-order correlations from high dimensional input video space.We prove that several successful architectures for visual classification tasks are in the family of higher-order neural networks, theoretical and experimental analysis demonstrates their underlying mechanism is higher-order.  On the task of video recognition, even using RGB only without fine-tuning with other video datasets, our higher-order models can achieve results on par with or better than the existing state-of-the-art methods on both Something-Something (V1 and V2) and Charades datasets.

Vorgeschlagene Operation höherer Ordnung für kontextbezogenes Lernen.

Es wird ein neuer 3D Convolutional Block vorgeschlagen, der den Video-Input mit seinem Kontext verarbeitet, basierend auf der Annahme, dass relevanter Kontext um das Objekt des Bildes herum vorhanden ist.


Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters.To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedures.The consistency loss dramatically improves generalization performance over supervised-only training; however, we show that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data.Motivated by these observations, we propose to train consistency-based methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modified learning rate schedule.We also propose fast-SWA, which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule.With weight averaging, we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100, over many different quantities of labeled training data.For example, we achieve 5.0% error on CIFAR-10 with only 4000 labels, compared to the previous best result in the literature of 6.3%.

Konsistenzbasierte Modelle für halbüberwachtes Lernen konvergieren nicht zu einem einzigen Punkt, sondern erkunden weiterhin eine Reihe plausibler Lösungen am Rande eines flachen Bereichs. Die Mittelwertbildung trägt zur Verbesserung der Generalisierungsleistung bei.

Der Artikel schlägt vor, Stochastic Weight Averaging auf den Kontext des halbüberwachten Lernens anzuwenden. Es wird argumentiert, dass die halbüberwachten MT/Pi-Modelle besonders gut für SWA geeignet sind und schlägt schnelles SWA vor, um das Training zu beschleunigen.


In this paper, we find that by designing a novel loss function entitled, ''tracking loss'', Convolutional Neural Network (CNN) based object detectors can be successfully converted to well-performed visual trackers without any extra computational cost.This property is preferable to visual tracking where annotated video sequences for training are always absent, because rich features learned by detectors from still images could be utilized by dynamic trackers.It also avoids extra machinery such as feature engineering and feature aggregation proposed in previous studies.Tracking loss achieves this property by exploiting the internal structure of feature maps within the detection network and treating different feature points discriminatively.Such structure allows us to simultaneously consider discrimination quality and bounding box accuracy which is found to be crucial to the success.We also propose a network compression method to accelerate tracking speed without performance reduction.That also verifies tracking loss will remain highly effective even if the network is drastically compressed.Furthermore, if we employ a carefully designed tracking loss ensemble, the tracker would be much more robust and accurate.Evaluation results show that our trackers (including the ensemble tracker and two baseline trackers), outperform all state-of-the-art methods on VOT 2016 Challenge in terms of Expected Average Overlap (EAO) and robustness.We will make the code publicly available.

Wir konvertieren erfolgreich einen populären Detektor RPN in einen gut funktionierenden Tracker aus der Sicht der Verlustfunktion.


We study the problem of semantic code repair, which can be broadly defined as automatically fixing non-syntactic bugs in source code.The majority of past work in semantic code repair assumed access to unit tests against which candidate repairs could be validated.In contrast, the goal here is to develop a strong statistical model to accurately predict both bug locations and exact fixes without access to information about the intended correct behavior of the program.Achieving such a goal requires a robust contextual repair model, which we train on a large corpus of real-world source code that has been augmented with synthetically injected bugs.Our framework adopts a two-stage approach where first a large set of repair candidates are generated by rule-based processors, and then these candidates are scored by a statistical model using a novel neural network architecture which we refer to as Share, Specialize, and Compete.Specifically, the architecture (1) generates a  shared encoding of the source code using an RNN over the abstract syntax tree, (2) scores each candidate repair using specialized network modules, and (3) then normalizes these scores together so they can compete against one another in comparable probability space.We evaluate our model on a real-world test set gathered from GitHub containing four common categories of bugs.Our model is able to predict the exact correct repair 41% of the time with a single guess, compared to 13% accuracy for an attentional sequence-to-sequence model.

Eine neuronale Architektur zur Bewertung und Einstufung von Programmreparaturkandidaten, um semantische Programmreparaturen statisch ohne Zugriff auf Unit-Tests durchzuführen.

Stellt eine neuronale Netzarchitektur vor, die aus den Teilen share, specialize und compete besteht, um Code in vier Fällen zu reparieren.


Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) (Tsipras et al., 2019).Such a dilemma is shown to be rooted in the inherently higher sample complexity (Schmidt et al., 2018) and/or model capacity (Nakkiran, 2019), for learning a high-accuracy and robust classifier.In view of that, give a classification task, growing the model capacity appears to help draw a win-win between accuracy and robustness, yet at the expense of model size and latency, therefore posing challenges for resource-constrained applications.Is it possible to co-design model accuracy, robustness and efficiency to achieve their triple wins?This paper studies multi-exit networks associated with input-adaptive efficient inference, showing their strong promise in achieving a â€œsweet point" in co-optimizing model accuracy, robustness, and efficiency.Our proposed solution, dubbed Robust Dynamic Inference Networks (RDI-Nets), allows for each input (either clean or adversarial) to adaptively choose one of the multiple output layers (early branches or the final one) to output its prediction.That multi-loss adaptivity adds new variations and flexibility to adversarial attacks and defenses, on which we present a systematical investigation.We show experimentally that by equipping existing backbones with such robust adaptive inference, the resulting RDI-Nets can achieve better accuracy and robustness, yet with over 30% computational savings, compared to the defended original models.

Ist es möglich, die Genauigkeit, Robustheit und Effizienz von Modellen mitzugestalten, um ihre dreifachen Ziele zu erreichen? Ja!

Nutzt input-adaptive Mehrfach-Frühausgänge für den Bereich des adversarial Angriffs und der Verteidigung, indem es die durchschnittliche Inferenzkomplexität reduziert, ohne der Annahme einer größeren Kapazität zu widersprechen.


Although deep convolutional networks have achieved improved performance in many natural language tasks, they have been treated as black boxes because they are difficult to interpret.Especially, little is known about how they represent language in their intermediate layers.In an attempt to understand the representations of deep convolutional networks trained on language tasks, we show that individual units are selectively responsive to specific morphemes, words, and phrases, rather than responding to arbitrary and uninterpretable patterns.In order to quantitatively analyze such intriguing phenomenon, we propose a concept alignment method based on how units respond to replicated text.We conduct analyses with different architectures on multiple datasets for classification and translation tasks and provide new insights into how deep models understand natural language.

Wir zeigen, dass einzelne Einheiten in CNN Repräsentationen, die in NLP Aufgaben erlernt werden, selektiv auf bestimmte natürliche Sprachkonzepte reagieren.

Verwendet grammatikalische Einheiten natürlicher Sprache, die Bedeutungen bewahren, um zu zeigen, dass die Einheiten von tiefen CNNs, die in NLP Aufgaben gelernt wurden, als Konzeptdetektor für natürliche Sprache fungieren können.


We study the problem of building models that disentangle independent factors of variation.Such models encode features that can efficiently be used for classification and to transfer attributes between different images in image synthesis.As data we use a weakly labeled training set, where labels indicate what single factor has changed between two data samples, although the relative value of the change is unknown.This labeling is of particular interest as it may be readily available without annotation costs.We introduce an autoencoder model and train it through constraints on image pairs and triplets.We show the role of feature dimensionality and adversarial training theoretically and experimentally.We formally prove the existence of the reference ambiguity, which is inherently present in the disentangling task when weakly labeled data is used.The numerical value of a factor has different meaning in different reference frames.When the reference depends on other factors, transferring that factor becomes ambiguous.We demonstrate experimentally that the proposed model can successfully transfer attributes on several datasets, but show also cases when the reference ambiguity occurs.

Es handelt sich dabei um eine überwiegend theoretische Arbeit, die die Herausforderungen bei der Entflechtung von Variationsfaktoren unter Verwendung von Autoencodern und GAN beschreibt.

Dieser Beitrag befasst sich mit der Entflechtung von Variationsfaktoren in Bildern, zeigt, dass man im Allgemeinen ohne weitere Annahmen nicht zwischen zwei verschiedenen Variationsfaktoren unterscheiden kann, und schlägt eine neuartige AE+GAN-Architektur vor, um zu versuchen, die Variationsfaktoren zu entflechten.

Diese Arbeit untersucht die Herausforderungen bei der Entflechtung unabhängiger Variationsfaktoren bei schwach markierten Daten und führt den Begriff der Referenzmehrdeutigkeit für die Zuordnung von Datenpunkten ein.


In information retrieval, learning to rank constructs a machine-based ranking model which given a query, sorts the search results by their degree of relevance or importance to the query.Neural networks have been successfully applied to this problem, and in this paper, we propose an attention-based deep neural network which better incorporates different embeddings of the queries and search results with an attention-based mechanism.This model also applies a decoder mechanism to learn the ranks of the search results in a listwise fashion.The embeddings are trained with convolutional neural networks or the word2vec model.We demonstrate the performance of this model with image retrieval and text querying data sets.

Lernen, mit mehreren Einbettungen und Aufmerksamkeiten zu rangieren.

Es wird vorgeschlagen, die Aufmerksamkeit zu nutzen, um mehrere Eingabedarstellungen sowohl für die Suchanfrage als auch für die Suchergebnisse in der Aufgabe "Lernen zu ranken" zu kombinieren.


Computational neuroscience aims to fit reliable models of in vivo neural activity and interpret them as abstract computations.Recent work has shown that functional diversity of neurons may be limited to that of relatively few cell types; other work has shown that incorporating constraints into artificial neural networks (ANNs) can improve their ability to mimic neural data.Here we develop an algorithm that takes as input recordings of neural activity and returns clusters of neurons by cell type and models of neural activity constrained by these clusters.The resulting models are both more predictive and more interpretable, revealing the contributions of functional cell types to neural computation and ultimately informing the design of future ANNs.

Wir haben einen Algorithmus entwickelt, der als Eingabe Aufzeichnungen neuronaler Aktivität nimmt und Cluster von Neuronen nach Zelltyp und Modelle neuronaler Aktivität liefert, die durch diese Cluster eingeschränkt werden.


Graph Neural Networks (GNNs) are a powerful representational tool for solving problems on graph-structured inputs.In almost all cases so far, however, they have been applied to directly recovering a final solution from raw inputs, without explicit guidance on how to structure their problem-solving.Here, instead, we focus on learning in the space of algorithms: we train several state-of-the-art GNN architectures to imitate individual steps of classical graph algorithms, parallel (breadth-first search, Bellman-Ford) as well as sequential (Prim's algorithm).As graph algorithms usually rely on making discrete decisions within neighbourhoods, we hypothesise that maximisation-based message passing neural networks are best-suited for such objectives, and validate this claim empirically.We also demonstrate how learning in the space of algorithms can yield new opportunities for positive transfer between tasks---showing how learning a shortest-path algorithm can be substantially improved when simultaneously learning a reachability algorithm.

Wir überwachen neuronale Graphen-Netzwerke, um Zwischenergebnisse und schrittweise Ausgaben klassischer Graphen-Algorithmen zu imitieren, und gewinnen dabei sehr günstige Erkenntnisse.

Schlägt vor, neuronale Netze so zu trainieren, dass sie Graphenalgorithmen imitieren, indem sie Primitive und Unterroutinen lernen und nicht die endgültige Ausgabe.


Prospection is an important part of how humans come up with new task plans, but has not been explored in depth in robotics.Predicting multiple task-level is a challenging problem that involves capturing both task semantics and continuous variability over the state of the world.Ideally, we would combine the ability of machine learning to leverage big data for learning the semantics of a task, while using techniques from task planning to reliably generalize to new environment.In this work, we propose a method for learning a model encoding just such a representation for task planning.We learn a neural net that encodes the k most likely outcomes from high level actions from a given world.Our approach creates comprehensible task plans that allow us to predict changes to the environment many time steps into the future.We demonstrate this approach via application to a stacking task in a cluttered environment, where the robot must select between different colored blocks while avoiding obstacles, in order to perform a task.We also show results on a simple navigation task.Our algorithm generates realistic image and pose predictions at multiple points in a given task.

Wir beschreiben eine Architektur zur Generierung verschiedener Hypothesen für Zwischenziele bei Robotermanipulationsaufgaben.

Bewertet die Qualität eines vorgeschlagenen generativen Vorhersagemodells zur Erstellung von Plänen für die Roboterausführung.

In dieser Arbeit wird eine Methode zum Erlernen einer hochrangigen Übergangsfunktion vorgeschlagen, die für die Aufgabenplanung nützlich ist.


Adaptive gradient algorithms perform gradient-based updates using the history of gradients and are ubiquitous in training deep neural networks.While adaptive gradient methods theory is well understood for minimization problems, the underlying factors driving their empirical success in min-max problems such as GANs remain unclear.In this paper, we aim at bridging  this gap from both theoretical and empirical perspectives.First, we analyze a variant of Optimistic Stochastic Gradient (OSG) proposed in~\citep{daskalakis2017training} for solving a class of non-convex non-concave min-max problem and establish $O(\epsilon^{-4})$ complexity for finding $\epsilon$-first-order stationary point, in which the algorithm only requires invoking one stochastic first-order oracle while enjoying state-of-the-art iteration complexity achieved by stochastic extragradient method by~\citep{iusem2017extragradient}.Then we propose an adaptive variant of OSG named Optimistic Adagrad (OAdagrad) and reveal an \emph{improved} adaptive complexity $\widetilde{O}\left(\epsilon^{-\frac{2}{1-\alpha}}\right)$~\footnote{Here $\widetilde{O}(\cdot)$ compresses a logarithmic factor of $\epsilon$.}, where $\alpha$ characterizes the growth rate of the cumulative stochastic gradient and $0\leq \alpha\leq 1/2$.To the best of our knowledge, this is the first work for establishing adaptive complexity in non-convex non-concave min-max optimization.Empirically, our experiments show that indeed adaptive gradient algorithms outperform their non-adaptive counterparts in GAN training.Moreover, this observation can be explained by the slow growth rate of the cumulative stochastic gradient, as observed empirically.

Dieser Artikel bietet eine neuartige Analyse von adaptiven Gradientenalgorithmen zur Lösung von nicht-konvexen, nicht-konkaven Min-Max-Problemen als GANs und erklärt den Grund, warum adaptive Gradientenmethoden ihre nicht-adaptiven Gegenstücke durch empirische Studien übertreffen.

Entwickelt Algorithmen für die Lösung von Variationsungleichungen im stochastischen Umfeld und schlägt eine Variante der Extragradientenmethode vor.


We consider the problem of unsupervised learning of a low dimensional, interpretable, latent state of a video containing a moving object.The problem of distilling dynamics from pixels has been extensively considered through the lens of graphical/state space models that exploit Markov structure for cheap computation and structured graphical model priors for enforcing interpretability on latent representations.We take a step towards extending these approaches by discarding the Markov structure; instead, repurposing the recently proposed Gaussian Process Prior Variational Autoencoder for learning sophisticated latent trajectories.We describe the model and perform experiments on a synthetic dataset and see that the model reliably reconstructs smooth dynamics exhibiting U-turns and loops.We also observe that this model may be trained without any beta-annealing or freeze-thaw of training parameters.Training is performed purely end-to-end on the unmodified evidence lower bound objective.This is in contrast to previous works, albeit for slightly different use cases, where application specific training tricks are often required.

Wir lernen sohpisticated Trajektorien eines Objekts rein aus Pixeln mit einem Spielzeug Videodatensatz durch die Verwendung einer VAE-Struktur mit einem Gauß Prozess Prior.


Dreams and our ability to recall them are among the most puzzling questions in sleep research.Specifically, putative differences in brain network dynamics between individuals with high versus low dream recall rates, are still poorly understood.In this study, we addressed this question as a classification problem where we applied deep convolutional networks (CNN) to sleep EEG recordings to predict whether subjects belonged to the high or low dream recall group (HDR and LDR resp.).Our model achieves significant accuracy levels across all the sleep stages, thereby indicating subtle signatures of dream recall in the sleep microstructure.We also visualized the feature space to inspect the subject-specificity of the learned features, thus ensuring that the network captured population level differences.Beyond being the first study to apply deep learning to sleep EEG in order to classify HDR and LDR, guided backpropagation allowed us to visualize the most discriminant features in each sleep stage.The significance of these findings and future directions are discussed.

Wir untersuchen die neuronalen Grundlagen der Traumerinnerung mit Hilfe von Convolutional Neural Networks und Techniken zur Visualisierung von Merkmalen, wie tSNE und Guided Backpropagation.


This paper considers multi-agent reinforcement learning (MARL) in networked system control.Specifically, each agent learns a decentralized control policy based on local observations and messages from connected neighbors.We formulate such a networked MARL (NMARL) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent.Further, we propose a new differentiable communication protocol, called NeurComm, to reduce information loss and non-stationarity in NMARL.Based on experiments in realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control, an appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms, while NeurComm outperforms existing communication protocols in both learning efficiency and control performance.

In diesem Papier werden eine neue Formulierung und ein neues Kommunikationsprotokoll für vernetzte Multi-Agenten Kontrollprobleme vorgeschlagen

Befasst sich mit N-MARLs, bei denen die Agenten ihre Politik nur auf der Grundlage von Nachrichten von benachbarten Knoten aktualisieren, und zeigt, dass die Einführung eines räumlichen Diskontierungsfaktors das Lernen stabilisiert.


Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights.Recent work developing this class of methods has explored ever richer parameterizations of the approximate posterior in the hope of improving performance.In contrast, here we share a curious experimental finding that suggests instead restricting the variational distribution to a more compact parameterization.For a variety of deep Bayesian neural networks trained using Gaussian mean-field variational inference, we find that the posterior standard deviations consistently exhibit strong low-rank structure after convergence.This means that by decomposing these variational parameters into a low-rank factorization, we can make our variational approximation more compact without decreasing the models' performance.Furthermore, we find that such factorized parameterizations improve the signal-to-noise ratio of stochastic gradient estimates of the variational lower bound, resulting in faster convergence.

Die VB mit mittlerem Feld verwendet doppelt so viele Parameter; wir binden die Varianzparameter in der VB mit mittlerem Feld ohne Verlust an ELBO, wodurch wir an Geschwindigkeit und geringeren Varianzgradienten gewinnen.


Aspect extraction in online product reviews is a key task in sentiment analysis and opinion mining.Training supervised neural networks for aspect extraction is not possible when ground truth aspect labels are not available, while the unsupervised neural topic models fail to capture the particular aspects of interest.In this work, we propose a weakly supervised approach for training neural networks for aspect extraction in cases where only a small set of seed words, i.e., keywords that describe an aspect, are available.Our main contributions are as follows.First, we show that current weakly supervised networks fail to leverage the predictive power of the available seed words by comparing them to a simple bag-of-words classifier.  Second, we propose a distillation approach for aspect extraction where the seed words are considered by the bag-of-words classifier (teacher) and distilled to the parameters of a neural network (student).Third, we show that regularization encourages the student to consider non-seed words for classification and, as a result, the student outperforms the teacher, which only considers the seed words.Finally, we empirically show that our proposed distillation approach outperforms (by up to 34.4% in F1 score) previous weakly supervised approaches for aspect extraction in six domains of Amazon product reviews.

Wir nutzen einige wenige Schlüsselwörter als schwache Überwachung für das Training neuronaler Netze zur Extraktion von Aspekten.

Erörtert eine Variante der Wissensdestillation, bei der ein "Lehrer" auf der Grundlage eines Bag-of-Words-Klassifikators mit Startwörtern und ein "Schüler", der ein auf Einbettung basierendes neuronales Netz ist, verwendet werden.


Forming perceptual groups and individuating objects in visual scenes is an essential step towards visual intelligence.This ability is thought to arise in the brain from computations implemented by bottom-up, horizontal, and top-down connections between neurons.However, the relative contributions of these connections to perceptual grouping are poorly understood.We address this question by systematically evaluating neural network architectures featuring combinations of these connections on two synthetic visual tasks, which stress low-level "Gestalt" vs. high-level object cues for perceptual grouping.We show that increasing the difficulty of either task strains learning for networks that rely solely on bottom-up processing.Horizontal connections resolve this limitation on tasks with Gestalt cues by supporting incremental spatial propagation of activities, whereas top-down connections rescue learning on tasks with high-level object cues by modifying coarse predictions about the position of the target object.Our findings dissociate the computational roles of bottom-up, horizontal and top-down connectivity, and demonstrate how a model featuring all of these interactions can more flexibly learn to form perceptual groups.

Horizontale und von oben nach unten gerichtete Rückkopplungsverbindungen sind für komplementäre Wahrnehmungsgruppierungsstrategien in biologischen und rekurrenten Sehsystemen verantwortlich.

Unter Verwendung neuronaler Netze als Computermodell des Gehirns wird die Effizienz verschiedener Strategien zur Lösung von zwei visuellen Aufgaben untersucht.


Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images.However, their application in the audio domain has received limited attention,and autoregressive models, such as WaveNet, remain the state of the art in generative modelling of audio signals such as human speech.To address this paucity, we introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech.Our architecture is composed of a conditional feed-forward generator producing raw speech audio, and an ensemble of discriminators which operate on random windows of different sizes.The discriminators analyse the audio both in terms of general realism, as well as how well the audio corresponds to the utterance that should be pronounced.  To measure the performance of GAN-TTS, we employ both subjective human evaluation (MOS - Mean Opinion Score), as well as novel quantitative metrics (FrÃ©chet DeepSpeech Distance and Kernel DeepSpeech Distance), which we find to be well correlated with MOS.We show that GAN-TTS is capable of generating high-fidelity speech with naturalness comparable to the state-of-the-art models, and unlike autoregressive models, it is highly parallelisable thanks to an efficient feed-forward generator.Listen to GAN-TTS reading this abstract at http://tiny.cc/gantts.

Wir stellen GAN-TTS vor, ein Generatives Adversariales Netzwerk für Text-to-Speech, das einen Mean Opinion Score (MOS) von 4,2 erreicht.

Löst die GAN-Herausforderung bei der Synthese von Rohwellenformen und beginnt, die bestehende Leistungslücke zwischen autoregressiven Modellen und GANs für Rohaudios zu schließen.


This paper proposes a Pruning in Training (PiT) framework of learning to reduce the parameter size of networks.Different from existing works, our PiT framework employs the sparse penalties to train networks and thus help rank the importance of weights and filters.Our PiT algorithms can directly prune the network without any fine-tuning.The pruned networks can still achieve comparable performance to the original networks.In particular, we introduce the (Group) Lasso-type Penalty (L-P /GL-P), and (Group) Split LBI Penalty (S-P / GS-P) to regularize the networks, and a pruning strategy proposed  is used in help prune the network.We conduct the extensive experiments on MNIST, Cifar-10, and miniImageNet.The results validate the efficacy of our proposed methods.Remarkably, on MNIST dataset, our PiT framework can save 17.5% parameter size of LeNet-5, which achieves the 98.47% recognition accuracy.

Wir schlagen einen Lernalgorithmus für Netzwerk Pruning vor, indem wir Strukturspärlichkeitsstrafen durchsetzen.

In diesem Beitrag wird ein Ansatz für das Pruning beim Training eines Netzes unter Verwendung von Lasso- und Split-LBI-Penalties vorgestellt.


We first pose the Unsupervised Continual Learning (UCL) problem: learning salient representations from a non-stationary stream of unlabeled data in which the number of object classes varies with time.Given limited labeled data just before inference, those representations can also be associated with specific object types to perform classification.To solve the UCL problem, we propose an architecture that involves a single module, called Self-Taught Associative Memory (STAM), which loosely models the function of a cortical column in the mammalian brain.Hierarchies of STAM modules learn based on a combination of Hebbian learning, online clustering, detection of novel patterns and forgetting outliers, and top-down predictions.We illustrate the operation of STAMs in the context of learning handwritten digits in a continual manner with only 3-12 labeled examples per class.STAMs suggest a promising direction to solve the UCL problem without catastrophic forgetting.

Wir stellen unüberwachtes kontinuierliches Lernen (UCL) und eine neuroinspirierte Architektur vor, die das UCL-Problem löst.

Vorschlag Hierarchien von STAM-Modulen zu verwenden, um das UCL-Problem zu lösen, und erbringen des Nachweis, dass die von den Modulen erlernten Repräsentationen für die Few-Shot Klassifizierung gut geeignet sind.


Recent advances have made it possible to create deep complex-valued neural networks.Despite this progress, the potential power of fully complex intermediate computations and representations has not yet been explored for many challenging learning problems.Building on recent advances, we propose a novel mechanism for extracting signals in the frequency domain.As a case study, we perform audio source separation in the Fourier domain.Our extraction mechanism could be regarded as a local ensembling method that combines a complex-valued convolutional version of Feature-Wise Linear Modulation (FiLM) and a signal averaging operation.We also introduce a new explicit amplitude and phase-aware loss, which is scale and time invariant, taking into account the complex-valued components of the spectrogram.Using the Wall Street Journal Dataset, we compare our phase-aware loss to several others that operate both in the time and frequency domains and demonstrate the effectiveness of our proposed signal extraction method and proposed loss.When operating in the complex-valued frequency domain, our deep complex-valued network substantially outperforms its real-valued counterparts even with half the depth and a third of the parameters.Our proposed mechanism improves significantly deep complex-valued networks' performance and we demonstrate the usefulness of its regularizing effect.

Neue Methode zur Signalextraktion im Fourier-Bereich.

Trägt eine komplexwertige Convolutional Version der merkmalsweisen linearen Modulation bei, die eine Parameteroptimierung ermöglicht und einen Verlust entwirft, bei dem Betrag und Phase berücksichtigt werden.


It is challenging to disentangle an object into two orthogonal spaces of content and style since each can influence the visual observation in a different and unpredictable way.It is rare for one to have access to a large number of data to help separate the influences.In this paper, we present a novel framework to learn this disentangled representation in a completely unsupervised manner.We address this problem in a two-branch Autoencoder framework.For the structural content branch, we project the latent factor into a soft structured point tensor and constrain it with losses derived from prior knowledge.This encourages the branch to distill geometry information.Another branch learns the complementary style information.The two branches form an effective framework that can disentangle object's content-style representation without any human annotation.We evaluate our approach on four image datasets, on which we demonstrate the superior disentanglement and visual analogy quality both in synthesized and real-world data.We are able to generate photo-realistic images with 256x256 resolution that are clearly disentangled in content and style.

Wir stellen ein neuartiges Framework vor, mit dem sich die getrennte Darstellung von Inhalt und Stil auf völlig unüberwachte Weise erlernen lässt.

Vorschlag eines Modells auf der Grundlage eines Autoencoders zur Entflechtung der Darstellung eines Objekts; die Ergebnisse zeigen, dass das Modell Darstellungen erzeugen kann, die Inhalt und Stil erfassen.


We develop the Y-learner for estimating heterogeneous treatment effects in experimental and observational studies.The Y-learner is designed to leverage the abilities of neural networks to optimize multiple objectives and continually update, which allows for better pooling of underlying feature information between treatment and control groups.We evaluate the Y-learner on three test problems: (1) A set of six simulated data benchmarks from the literature.(2) A real-world large-scale experiment on voter persuasion.(3) A task from the literature that estimates artificially generated treatment effects on MNIST didgits.The Y-learner achieves state of the art results on two of the three tasks.On the MNIST task, it gets the second best results.

Wir entwickeln eine CATE-Schätzungsstrategie, die sich einige der faszinierenden Eigenschaften neuronaler Netze zunutze macht.

Zeigt Verbesserungen von X-learner durch die Modellierung der Behandlungsreaktionsfunktion, der Kontrollreaktionsfunktion und der Abbildung des unterstellten Behandlungseffekts auf den bedingten durchschnittlichen Behandlungseffekt als neuronale Netze.

Die Autoren schlagen den Y-Learner zur Schätzung des bedingten durchschnittlichen Behandlungseffekts (CATE) vor, der gleichzeitig die Parameter der Ergebnisfunktionen und den CATE-Schätzer aktualisiert.


With the rapid proliferation of IoT devices, our cyberspace is nowadays dominated by billions of low-cost computing nodes, which expose an unprecedented heterogeneity to our computing systems.Dynamic analysis, one of the most effective approaches to finding software bugs, has become paralyzed due to the lack of a generic emulator capable of running diverse previously-unseen firmware.In recent years, we have witnessed devastating security breaches targeting IoT devices.These security concerns have significantly hamstrung further evolution of IoT technology.In this work, we present Laelaps, a device emulator specifically designed to run diverse software on low-cost IoT devices.We do not encode into our emulator any specific information about a device.Instead, Laelaps infers the expected behavior of firmware via symbolic-execution-assisted peripheral emulation and generates proper inputs to steer concrete execution on the fly.This unique design feature makes Laelaps the first generic device emulator capable of running diverse firmware with no a priori knowledge about the target device.To demonstrate the capabilities of Laelaps, we deployed two popular dynamic analysis techniques---fuzzing testing and dynamic symbolic execution---on top of our emulator.We successfully identified both self-injected and real-world vulnerabilities.


Deep neural models, such as convolutional and recurrent networks, achieve phenomenal results over spatial data such as images and text.However, when considering tabular data, gradient boosting of decision trees (GBDT) remains the method of choice.Aiming to bridge this gap, we propose \emph{deep neural forests} (DNF)--  a novel architecture that combines elements from decision trees as well as dense residual connections. We present the results of extensive empirical study in which we examine the performance of GBDTs, DNFs and (deep) fully-connected networks. These results indicate that DNFs achieve comparable results to GBDTs on tabular data, and open the door to end-to-end neural modeling of multi-modal data.To this end, we present a successful application of DNFs as part of a hybrid architecture for a multi-modal driving scene understanding classification task.

Eine Architektur für tabellarische Daten, die Verzweigungen von Entscheidungsbäumen nachbildet und eine dichte Restkonnektivität verwendet.

In dieser Arbeit wird ein tiefer neuronaler Wald vorgeschlagen, ein Algorithmus, der auf tabellarische Daten abzielt und die Stärken des Gradient Boosting von Entscheidungsbäumen integriert.

Eine neuartige neuronale Netzwerkarchitektur, die die Funktionsweise von Entscheidungswäldern nachahmt, um das allgemeine Problem des Trainings von tiefen Modellen für tabellarische Daten zu lösen, und die eine mit GBDT vergleichbare Effektivität aufweist.


Hyperparameter tuning is one of the most time-consuming workloads in deep learning.State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable.Recently researchers have shown renewed interest in simpler methods like momentum SGD as they may yield better results.Motivated by this trend, we ask: can simple adaptive methods, based on SGD perform as well or better?We revisit the momentum SGD algorithm and show that hand-tuning a single learning rate and momentum makes it competitive with Adam.We then analyze its robustness to learning rate misspecification and objective curvature variation.Based on these insights, we design YellowFin, an automatic tuner for momentum and learning rate in SGD.YellowFin optionally uses a negative-feedback loop to compensate for the momentum dynamics in asynchronous settings on the fly.We empirically show YellowFin can converge in fewer iterations than Adam on ResNets and LSTMs for image recognition, language modeling and constituency parsing, with a speedup of up to $3.28$x in synchronous and up to $2.69$x in asynchronous settings.

YellowFin ist ein SGD-basierter Optimierer, der sowohl die Dynamik als auch die Lernrate anpassen kann.

Schlägt eine Methode zur automatischen Abstimmung des Momentum-Parameters in Momentum-SGD-Methoden vor, die bessere Ergebnisse und eine schnellere Konvergenzgeschwindigkeit als der moderne Adam-Algorithmus erzielt.


Robustness and security of machine learning (ML) systems are intertwined, wherein a non-robust ML system (classifiers, regressors, etc.) can be subject to attacks using a wide variety of exploits.With the advent of scalable deep learning methodologies, a lot of emphasis has been put on the robustness of supervised, unsupervised and reinforcement learning algorithms.Here, we study the robustness of the latent space of a deep variational autoencoder (dVAE), an unsupervised generative framework, to show that it is indeed possible to perturb the latent space, flip the class predictions and keep the classification probability approximately equal before and after an attack.This means that an agent that looks at the outputs of a decoder would remain oblivious to an attack.

Angriffe auf den latenten Raum von variationalen Autoencodern zur Veränderung der semantischen Bedeutung von Eingaben.

Dieses Papier befasst sich mit Sicherheit und maschinellem Lernen und schlägt einen Man-in-Middle-Angriff vor, der die VAE-Kodierung der Eingabedaten so verändert, dass die dekodierte Ausgabe falsch klassifiziert wird.


Graph-based dependency parsing consists of two steps: first, an encoder produces a feature representation for each parsing substructure of the input sentence, which is then used to compute a score for the substructure; and second, a decoder} finds the parse tree whose substructures have the largest total score.Over the past few years, powerful neural techniques have been introduced into the encoding step which substantially increases parsing accuracies.However, advanced decoding techniques, in particular high-order decoding, have seen a decline in usage.It is widely believed that contextualized features produced by neural encoders can help capture high-order decoding information and hence diminish the need for a high-order decoder.In this paper, we empirically evaluate the combinations of different neural and non-neural encoders with first- and second-order decoders and provide a comprehensive analysis about the effectiveness of these combinations with varied training data sizes.We find that: first, when there is large training data, a strong neural encoder with first-order decoding is sufficient to achieve high parsing accuracy and only slightly lags behind the combination of neural encoding and second-order decoding; second, with small training data, a non-neural encoder with a second-order decoder outperforms the other combinations in most cases.

Eine empirische Studie, die die Wirksamkeit verschiedener Encoder-Decoder-Kombinationen für die Aufgabe des Dependency Parsing untersucht.

Empirische Analyse verschiedener Kodierer, Dekodierer und deren Abhängigkeiten für graphbasiertes Dependency Parsing.


Meta-learning will be crucial to creating lifelong, generalizable AI.In practice, however, it is hard to define the meta-training task distribution that is used to train meta-learners.If made too small, tasks are too similar for a model to meaningfully generalize.If made too large, generalization becomes incredibly difficult.We argue that both problems can be alleviated by introducing a teacher model that controls the sequence of tasks that a meta-learner is trained on.This teacher model is incentivized to start the student meta-learner on simple tasks then adaptively increase task difficulty in response to student progress.While this approach has been previously studied in curriculum generation, our main contribution is in extending it to meta-learning.

Lehrer, der Meta-Lernende wie Menschen ausbildet.


Using higher order knowledge to reduce training data has become a popular research topic.However, the ability for available methods to draw effective decision boundaries is still limited: when training set is small, neural networks will be biased to certain labels.Based on this observation, we consider constraining output probability distribution as higher order domain knowledge.We design a novel algorithm that jointly optimizes output probability distribution on a clustered embedding space to make neural networks draw effective decision boundaries.  While directly applying probability constraint is not effective, users need to provide additional very weak supervisions: mark some batches that have output distribution greatly differ from target probability distribution.We use experiments to empirically prove that our model can converge to an accuracy higher than other state-of-art semi-supervised learning models with less high quality labeled training examples.

Wir führen einen Ansatz zur Einbettung des Raums ein, um die Wahrscheinlichkeitsverteilung der Ausgabe eines neuronalen Netzes einzuschränken.

In diesem Beitrag wird eine Methode zum halbüberwachten Lernen mit tiefen neuronalen Netzen vorgestellt, und das Modell erreicht bei einem geringen Trainingsumfang eine relativ hohe Genauigkeit.

In dieser Arbeit wird die Label-Verteilung in das Modelllernen integriert, wenn nur eine begrenzte Anzahl von Trainingsinstanzen zur Verfügung steht, und es werden zwei Techniken vorgeschlagen, um das Problem der falsch verzerrten Output-Label Verteilung zu lösen.


We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy).  Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pretrained on a large text corpus.We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis.  We also present an analysis showing that exposing the deep internals of the pretrained network is crucial, allowing downstream models to mix different types of semi-supervision signals.

Wir stellen eine neue Art von tiefer kontextualisierter Wortrepräsentation vor, die den Stand der Technik für eine Reihe von anspruchsvollen NLP-Aufgaben deutlich verbessert.


This work addresses the long-standing problem of robust event localization in the presence of temporally of misaligned labels in the training data.We propose a novel versatile loss function that generalizes a number of training regimes from standard fully-supervised cross-entropy to count-based weakly-supervised learning.Unlike classical models which are constrained to strictly fit the annotations during training, our soft localization learning approach relaxes the reliance on the exact position of labels instead.Training with this new loss function exhibits strong robustness to temporal misalignment of labels, thus alleviating the burden of precise annotation of temporal sequences.We demonstrate state-of-the-art performance against standard benchmarks in a number of challenging experiments and further show that robustness to label noise is not achieved at the expense of raw performance.

In dieser Arbeit wird eine neuartige Verlustfunktion für das robuste Training von DNN zur zeitlichen Lokalisierung in Gegenwart von falsch ausgerichteten Etiketten eingeführt.

Ein neuer Verlust für Trainingsmodelle, die vorhersagen, wo Ereignisse in einer Trainingssequenz mit störhaften Beschriftungen auftreten, indem sie geglättete Beschriftung und Vorhersagesequenz vergleichen.


The driving force behind deep networks is their ability to compactly represent rich classes of functions.The primary notion for formally reasoning about this phenomenon is expressive efficiency, which refers to a situation where one network must grow unfeasibly large in order to replicate functions of another.To date, expressive efficiency analyses focused on the architectural feature of depth, showing that deep networks are representationally superior to shallow ones.In this paper we study the expressive efficiency brought forth by connectivity, motivated by the observation that modern networks interconnect their layers in elaborate ways.We focus on dilated convolutional networks, a family of deep models delivering state of the art performance in sequence processing tasks.By introducing and analyzing the concept of mixed tensor decompositions, we prove that interconnecting dilated convolutional networks can lead to expressive efficiency.In particular, we show that even a single connection between intermediate layers can already lead to an almost quadratic gap, which in large-scale settings typically makes the difference between a model that is practical and one that is not.Empirical evaluation demonstrates how the expressive efficiency of connectivity, similarly to that of depth, translates into gains in accuracy.This leads us to believe that expressive efficiency may serve a key role in developing new tools for deep network design.

Wir führen den Begriff der gemischten Tensorzerlegungen ein und beweisen damit, dass die Verbindung von erweiterten Convolutional Networks deren Ausdruckskraft erhöht.

In diesem Beitrag wird theoretisch bestätigt, dass die Verbindung von Netzen mit unterschiedlichen Dilatationen zu einer ausdrucksstarken Effizienz bei der gemischten Tensorzerlegung führen kann.

Die Autoren untersuchen erweiterte Convolutional Networks und zeigen, dass die Verflechtung von zwei erweiterten Convolutional Networks A und B in verschiedenen Stadien ausdrucksstärker ist als eine Nichtverflechtung.

Es zeigt sich, dass die strukturelle Annahme eines einzigen perfekten Binärbaums im WaveNet seine Leistung beeinträchtigt und dass WaveNet-ähnliche Architekturen mit komplexeren gemischten Baumstrukturen besser abschneiden.


We apply multi-task learning to image classification tasks on MNIST-like datasets.MNIST dataset has been referred to as the {\em drosophila} of machine learning and has been the testbed of many learning theories.The NotMNIST dataset and the FashionMNIST dataset have been created with the MNIST dataset as reference.In this work, we exploit these MNIST-like datasets for multi-task learning.The datasets are pooled together for learning the parameters of joint classification networks.Then the learned parameters are used as the initial parameters to retrain disjoint classification networks.The baseline recognition model are all-convolution neural networks.Without multi-task learning, the recognition accuracies for MNIST, NotMNIST and FashionMNIST are 99.56\%, 97.22\% and 94.32\% respectively.With multi-task learning to pre-train the networks, the recognition accuracies are respectively 99.70\%, 97.46\% and 95.25\%.The results re-affirm that multi-task learning framework, even with data with different genres, does lead to significant improvement.

In diesem Beitrag wird ein neuronales Multitasking-Netzwerk zur Klassifizierung von MNIST-ähnlichen Datensätzen vorgestellt.


Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network.To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization.This approach provides us with a broad and unifying view on much prior work on this topic.Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal.In particular, they specify a concrete security guarantee that would protect against a well-defined class of adversaries.These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks.They also suggest robustness against a first-order adversary as a natural security guarantee.We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models.

Wir bieten eine prinzipielle, optimierungsbasierte Neubetrachtung des Begriffs der adversarial Beispiele und entwickeln Methoden, die Modelle hervorbringen, die gegen eine Vielzahl von Gegnern robust sind.

Untersucht eine Minimax Formulierung für das Lernen von tiefen Netzwerken, um deren Robustheit zu erhöhen, wobei der projizierte Gradientenabstieg als Hauptgegner verwendet wird.

In dieser Arbeit wird vorgeschlagen, neuronale Netze durch den Rahmen von Sattelpunktproblemen widerstandsfähig gegen gegnerische Verluste zu machen.


In recent years there has been a rapid increase in classification methods on graph structured data.Both in graph kernels and graph neural networks, one of the implicit assumptions of successful state-of-the-art models was that incorporating graph isomorphism features into the architecture leads to better empirical performance.However, as we discover in this work, commonly used data sets for graph classification have repeating instances which cause the problem of isomorphism bias, i.e. artificially increasing the accuracy of the models by memorizing target information from the training set.This prevents fair competition of the algorithms and raises a question of the validity of the obtained results.We analyze 54 data sets, previously extensively used for graph-related tasks, on the existence of isomorphism bias, give a set of recommendations to machine learning practitioners to properly set up their models, and open source new data sets for the future experiments.

Viele Graphklassifizierungsdatensätze weisen Duplikate auf, was Fragen hinsichtlich der Generalisierungsfähigkeiten und des fairen Vergleichs der Modelle aufwirft.

Die Autoren diskutieren den Isomorphismus-Bias in Graphen-Datensätzen, den Overfitting-Effekt beim Lernen von Netzwerken, wenn Graphen-Isomorphismus-Merkmale in das Modell aufgenommen werden, theoretisch analog zu Datenleckeffekten.


Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently.However, learning from demonstrations often suffers from the covariate shift problem, which resultsin cascading errors of the learned policy.We introduce a notion of conservatively extrapolated value functions, which provably lead to policies with self-correction.We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation.We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks.We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm, which is shown to outperform prior works in sample efficiency.

Wir führen einen Begriff von konservativ-extrapolierten Wertfunktionen ein, die nachweislich zu Strategien führen, die sich selbst korrigieren können, um nahe an den Demonstrationszuständen zu bleiben, und lernen sie mit einer neuartigen negativen Sampling-Technik.

Ein Algorithmus namens Wertiteration mit negativem Sampling, um das Problem der Kovariatenverschiebung beim Imitationslernen zu lösen.


A structured understanding of our world in terms of objects, relations, and hierarchies is an important component of human cognition.Learning such a structured world model from raw sensory data remains a challenge.As a step towards this goal, we introduce Contrastively-trained Structured World Models (C-SWMs).C-SWMs utilize a contrastive approach for representation learning in environments with compositional structure.We structure each state embedding as a set of object representations and their relations, modeled by a graph neural network.This allows objects to be discovered from raw pixel observations without direct supervision as part of the learning process.We evaluate C-SWMs on compositional environments involving multiple interacting objects that can be manipulated independently by an agent, simple Atari games, and a multi-object physics simulation.Our experiments demonstrate that C-SWMs can overcome limitations of models based on pixel reconstruction and outperform typical representatives of this model class in highly structured environments, while learning interpretable object-based representations.

Kontrastiv trainierte strukturierte Weltmodelle (C-SWMs) lernen objektorientierte Zustandsdarstellungen und ein relationales Modell einer Umgebung aus rohen Pixeleingaben.

Die Autoren überwinden das Problem der Verwendung von pixelbasierten Verlusten bei der Konstruktion und dem Lernen von strukturierten Weltmodellen, indem sie einen kontrastiven latenten Raum verwenden.


Neural machine translation (NMT) models learn representations containing substantial linguistic information.However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons.We develop unsupervised methods for discovering important neurons in NMT models.Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision.We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena.Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.

Unüberwachte Methoden zum Auffinden, Analysieren und Kontrollieren wichtiger Neuronen in der NMT

In dieser Arbeit wird vorgeschlagen, "sinnvolle" Neuronen in neuronalen maschinellen Übersetzungsmodellen zu finden, indem ein Ranking auf der Grundlage der Korrelation zwischen Modellpaaren, verschiedenen Epochen oder verschiedenen Datensätzen erstellt wird, und es wird ein Kontrollmechanismus für die Modelle vorgeschlagen.


Computations for the softmax function in neural network models are expensive when the number of output classes is large.This can become a significant issue in both training and inference for such models.In this paper, we present Doubly Sparse Softmax (DS-Softmax), Sparse Mixture of Sparse of Sparse Experts, to improve the efficiency for softmax inference.During training, our method learns a two-level class hierarchy by dividing entire output class space into several partially overlapping experts.Each expert is responsible for a learned subset of the output class space and each output class only belongs to a small number of those experts.During inference, our method quickly locates the most probable expert to compute small-scale softmax.Our method is learning-based and requires no knowledge of the output class partition space a priori.We empirically evaluate our method on several real-world tasks and demonstrate that we can achieve significant computation reductions without loss of

Wir präsentieren doppelt spärlichen Softmax, die spärliche Mischung aus spärlichen Experten, um die Effizienz der Softmax-Inferenz durch Ausnutzung der zweistufigen überlappenden Hierarchie zu verbessern.

Die Arbeit schlägt die neue Softmax-Algorithmus-Implementierung mit zwei hierarchischen Ebenen der Spärlichkeit, die den Betrieb in der Sprach Modellierung beschleunigt.


Our work presents empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance.Compared to previously studied indicators of generalization, we show that layer rotation has the additional benefit of being easily monitored and controlled, as well as having a network-independent optimum: the training procedures during which all layers' weights reach a cosine distance of 1 from their initialization consistently outperform other configurations -by up to 20% test accuracy.Finally, our results also suggest that the study of layer rotation can provide a unified framework to explain the impact of weight decay and adaptive gradient methods on generalization.

In diesem Beitrag werden empirische Belege für die Entdeckung eines Generalisierungsindikators vorgestellt: die Entwicklung des Kosinusabstands zwischen dem Gewichtsvektor jeder Schicht und ihrer Initialisierung während des Trainings.


Models of code can learn distributed representations of a program's syntax and semantics to predict many non-trivial properties of a program.Recent state-of-the-art models leverage highly structured representations of programs, such as trees, graphs and paths therein (e.g. data-flow relations), which are precise and abundantly available for code.This provides a strong inductive bias towards semantically meaningful relations, yielding more generalizable representations than classical sequence-based models.Unfortunately, these models primarily rely on graph-based message passing to represent relations in code, which makes them de facto local due to the high cost of message-passing steps, quite in contrast to modern, global sequence-based models, such as the Transformer.In this work, we bridge this divide between global and structured models by introducing two new hybrid model families that are both global and incorporate structural bias: Graph Sandwiches, which wrap traditional (gated) graph message-passing layers in sequential message-passing layers; and Graph Relational Embedding Attention Transformers (GREAT for short), which bias traditional Transformers with relational information from graph edge types.By studying a popular, non-trivial program repair task, variable-misuse identification, we explore the relative merits of traditional and hybrid model families for code representation.Starting with a  graph-based model that already improves upon the prior state-of-the-art for this task by 20%, we show that our proposed hybrid models improve an additional 10-15%, while training both faster and using fewer parameters.

Modelle von Quellcode, die globale und strukturelle Merkmale kombinieren, lernen leistungsfähigere Darstellungen von Programmen.

Eine neue Methode zur Modellierung des Quellcodes für die Fehlerbehebung unter Verwendung eines Sandwich-Modells wie [RNN GNN RNN], das die Lokalisierungs- und Reparaturgenauigkeit erheblich verbessert.


Recurrent neural networks (RNNs) are particularly well-suited for modeling long-term dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate.While a number of works attempt to mitigate this effect through gated recurrent units, skip-connections, parametric constraints and design choices, we propose a novel incremental RNN (iRNN), where hidden state vectors keep track of incremental changes, and as such approximate state-vector increments of Rosenblatt's (1962) continuous-time RNNs.iRNN exhibits identity gradients and is able to account for long-term dependencies (LTD).We show that our method is computationally efficient overcoming overheads of many existing methods that attempt to improve RNN training, while suffering no performance degradation.We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks.

Inkrementelle RNNs lösen das Problem der Explosion/des verschwindenden Gradienten, indem sie die Zustandsvektoren auf der Grundlage der Differenz zwischen dem vorherigen Zustand und dem durch eine ODE vorhergesagten Zustand aktualisieren.

Die Autoren befassen sich mit dem Problem der Signalausbreitung in rekurrenten neuronalen Netzen, indem sie ein Attraktorsystem für den Signalübergang aufbauen und prüfen, ob es zu einem Gleichgewicht konvergiert.


Recent empirical results on over-parameterized deep networks are marked by a striking absence of the classic U-shaped test error curve: test error keeps decreasing in wider networks.Researchers are actively working on bridging this discrepancy by proposing better complexity measures.Instead, we directly measure prediction bias and variance for four classification and regression tasks on modern deep networks.We find that both bias and variance can decrease as the number of parameters grows.Qualitatively, the phenomenon persists over a number of gradient-based optimizers.To better understand the role of optimization, we decompose the total variance into variance due to training set sampling and variance due to initialization.Variance due to initialization is significant in the under-parameterized regime.In the over-parameterized regime, total variance is much lower and dominated by variance due to sampling.We provide theoretical analysis in a simplified setting that is consistent with our empirical findings.

Wir liefern Beweise gegen klassische Behauptungen über den Kompromiss zwischen Verzerrung und Varianz und schlagen eine neuartige Zerlegung der Varianz vor.


Real world images often contain large amounts of private / sensitive information that should be carefully protected without reducing their utilities.In this paper, we propose a privacy-preserving deep learning framework with a learnable ob- fuscator for the image classification task.Our framework consists of three mod- els: learnable obfuscator, classifier and reconstructor.The learnable obfuscator is used to remove the sensitive information in the images and extract the feature maps from them.The reconstructor plays the role as an attacker, which tries to recover the image from the feature maps extracted by the obfuscator.In order to best protect usersâ€™ privacy in images, we design an adversarial training methodol- ogy for our framework to optimize the obfuscator.Through extensive evaluations on real world datasets, both the numerical metrics and the visualization results demonstrate that our framework is qualified to protect usersâ€™ privacy and achieve a relatively high accuracy on the image classification task.

Wir haben ein neuartiges Deep Learning System zur Bildklassifizierung vorgeschlagen, das sowohl Bilder genau klassifizieren als auch die Privatsphäre der Nutzer schützen kann.

In diesem Beitrag wird ein Verfahren vorgeschlagen, das die privaten Informationen im Bild bewahrt und die Nutzbarkeit des Bildes nicht beeinträchtigt.

In dieser Arbeit wird vorgeschlagen, adversarische Netzwerke zu verwenden, um Bilder zu verschleiern und sie so ohne Bedenken hinsichtlich des Datenschutzes zu sammeln, um sie für das Training von maschinellen Lernmodellen zu verwenden.


Bitcoin is a virtual coinage system that enables users to trade virtually free of a central trusted authority.All transactions on the Bitcoin blockchain are publicly available for viewing, yet as Bitcoin is built mainly for security itâ€™s original structure does not allow for direct analysis of address transactions. Existing analysis methods of the Bitcoin blockchain can be complicated, computationally expensive or inaccurate.We propose a computationally efficient model to analyze bitcoin blockchain addresses and allow for their use with existing machine learning algorithms.We compare our approach against Multi Level Sequence Learners (MLSLs), one of the best performing models on bitcoin address data.

Ein 2vec-Modell für Transaktionsgraphen von Kryptowährungen.

Die Arbeit schlägt vor, einen Autoencoder, networkX, und node2Vec zu verwenden, um vorherzusagen, ob eine Bitcoin-Adresse nach einem Jahr leer sein wird, aber die Ergebnisse sind schlechter als eine bestehende Basislinie.


Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood.In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations.First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable.Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients.Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.

Konvergenzbeweis der stochastischen Subgradientenmethode und Variationen bei konvex-konkaven Minimax-Problemen

Eine Analyse des simultanen stochastischen Subgradienten, des simultanen Gradienten mit Optimismus und des simultanen Gradienten mit Verankerung im Kontext von konvex-konkaven Minmax Spielen.

Dieses Papier analysiert die Dynamik des stochastischen Gradientenabstiegs, wenn er auf konvex-konkave Spiele angewendet wird, sowie GD mit Optimismus und einen neuen verankerten GD-Algorithmus, der unter schwächeren Annahmen konvergiert als SGD oder SGD mit Optimismus.


Small spacecraft now have precise attitude control systems available commercially, allowing them to slew in 3 degrees of freedom, and capture images within short notice.When combined with appropriate software, this agility can significantly increase response rate, revisit time and coverage.In prior work, we have demonstrated an algorithmic framework that combines orbital mechanics, attitude control and scheduling optimization to plan the time-varying, full-body orientation of agile, small spacecraft in a constellation.The proposed schedule optimization would run at the ground station autonomously, and the resultant schedules uplinked to the spacecraft for execution.The algorithm is generalizable over small steerable spacecraft, control capability, sensor specs, imaging requirements, and regions of interest.In this article, we modify the algorithm to run onboard small spacecraft, such that the constellation can make time-sensitive decisions to slew and capture images autonomously, without ground control.We have developed a communication module based on Delay/Disruption Tolerant Networking (DTN) for onboard data management and routing among the satellites, which will work in conjunction with the other modules to optimize the schedule of agile communication and steering.We then apply this preliminary framework on representative constellations to simulate targeted measurements of episodic precipitation events and subsequent urban floods.The command and control efficiency of our agile algorithm is compared to non-agile (11.3x improvement) and non-DTN (21% improvement) constellations.

Wir schlagen einen algorithmischen Rahmen vor, um Konstellationen von kleinen Raumfahrzeugen mit 3-DOF-Neuorientierungsfähigkeiten zu planen, die mit Inter-Sat-Verbindungen vernetzt sind.

In diesem Beitrag wird ein Kommunikationsmodul zur Optimierung des Kommunikationsplans für das Problem von Raumfahrzeugkonstellationen vorgeschlagen und der Algorithmus in verteilten und zentralisierten Einstellungen verglichen.


Importance sampling (IS) is a standard Monte Carlo (MC) tool to compute information about random variables such as moments or quantiles with unknown distributions.  IS is asymptotically consistent as the number of MC samples, and hence deltas (particles) that parameterize the density estimate, go to infinity.However, retaining infinitely many particles is intractable.We propose a scheme for only keeping a \emph{finite representative subset} of particles and their augmented importance weights that is \emph{nearly consistent}. To do so in {an online manner}, we approximate importance sampling in two ways.  First, we replace the deltas by kernels, yielding kernel density estimates (KDEs).  Second, we sequentially project KDEs onto nearby lower-dimensional subspaces.We characterize the asymptotic bias of this scheme as determined by a compression parameter and kernel bandwidth, which yields a tunable tradeoff between consistency and memory.In experiments,  we observe a favorable tradeoff between memory and accuracy, providing for the first time near-consistent compressions of arbitrary posterior distributions.

Wir haben einen neuartigen komprimierten Kernelized Importance Sampling Algorithmus vorgeschlagen.


We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator?(2) how to correctly use cross-validation to choose the regularization parameter?and (3) how to accelerate computation without losing too much accuracy?We consider the three problems in a unified large-data linear model.We give a precise representation of ridge regression as a covariance matrix-dependent linear combination of the true parameter and the noise. We study the bias of $K$-fold cross-validation for choosing the regularization parameter, and propose a simple bias-correction.We analyze the accuracy of primal and dual sketching for ridge regression, showing they are surprisingly accurate.Our results are illustrated by simulations and by analyzing empirical data.

Wir untersuchen die Struktur der Ridge-Regression in einem hochdimensionalen asymptotischen Framework und gewinnen Erkenntnisse über Cross-Validation und Sketching.

Eine theoretische Untersuchung der Ridge-Regression unter Ausnutzung einer neuen asymptotischen Charakterisierung des Ridge-Regressionsschätzers.


Attention mechanisms have advanced the state of the art in several machine learning tasks.Despite significant empirical gains, there is a lack of theoretical analyses on understanding their effectiveness.In this paper, we address this problem by studying the landscape of population and empirical loss functions of attention-based neural networks.Our results show that, under mild assumptions, every local minimum of a two-layer global attention model has low prediction error, and attention models require lower sample complexity than models not employing attention.We then extend our analyses to the popular self-attention model, proving that they deliver consistent predictions with a more expressive class of functions.Additionally, our theoretical results provide several guidelines for designing attention mechanisms.Our findings are validated with satisfactory experimental results on MNIST and IMDB reviews dataset.

Wir analysieren die Verlustlandschaft von neuronalen Netzen mit Aufmerksamkeit und erklären, warum Aufmerksamkeit beim Training neuronaler Netze hilfreich ist, um eine gute Leistung zu erzielen.

Diese Arbeit beweist aus theoretischer Sicht, dass Aufmerksamkeitsnetzwerke besser generalisieren können als Nicht-Aufmerksamkeits-Baselines für feste Aufmerksamkeit (ein- und mehrschichtig) und Selbstaufmerksamkeit in der einschichtigen Umgebung.


Recent advances in deep learning techniques has shown the usefulness of the deep neural networks in extracting features required to perform the task at hand.However, these features learnt are in particular helpful only for the initial task.This is due to the fact that the features learnt are very task specific and does not capture the most general and task agnostic features of the input.In fact the way humans are seen to learn is by disentangling features which task agnostic.This indicates that leaning task agnostic features by disentangling only the most informative features from the input data.Recently Variational Auto-Encoders (VAEs) have shown to be the de-facto models to capture the latent variables in a generative sense.As these latent features can be represented as continuous and/or discrete variables, this indicates us to use VAE with a mixture of continuous and discrete variables for the latent space.We achieve this by performing our experiments using a modified version of joint-vae to learn the disentangled features.

Mischungsmodell für neuronale Entflechtung.


To improve how neural networks function it is crucial to understand their learning process.The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task.However, empirical evidence for this theory is conflicting, as compression was only observed when networks used saturating activation functions.In contrast, networks with non-saturating activation functions achieved comparable levels of task performance but did not show compression.In this paper we developed more robust mutual information estimation techniques, that adapt to hidden activity of neural networks and produce more sensitive measurements of activations from all functions, especially unbounded functions.Using these adaptive estimation techniques, we explored compression in networks with a range of different activation functions.With two improved methods of estimation, firstly, we show that saturation of the activation function is not required for compression, and the amount of compression varies between different activation functions.We also find that there is a large amount of variation in compression between different network initializations.Secondary, we see that L2 regularization leads to significantly increased compression, while preventing overfitting.Finally, we show that only compression of the last layer is positively correlated with generalization.

Wir haben robuste Schätzungen der gegenseitigen Information für DNNs entwickelt und sie verwendet, um die Kompression in Netzwerken mit nicht sättigenden Aktivierungsfunktionen zu beobachten.

In dieser Arbeit wurde die weit verbreitete Annahme untersucht, dass tiefe neuronale Netze bei überwachten Aufgaben eine Informationskompression durchführen.

In diesem Papier wird eine Methode zur Schätzung der gegenseitigen Information für Netze mit unbeschränkten Aktivierungsfunktionen und die Verwendung der L2-Regularisierung vorgeschlagen, um eine stärkere Kompression zu erreichen.


In this work, we address the problem of musical timbre transfer, where the goal is to manipulate the timbre of a sound sample from one instrument to match another instrument while preserving other musical content, such as pitch, rhythm, and loudness.In principle, one could apply image-based style transfer techniques to a time-frequency representation of an audio signal, but this depends on having a representation that allows independent manipulation of timbre as well as high-quality waveform generation.We introduce TimbreTron, a method for musical timbre transfer which applies â€œimageâ€ domain style transfer to a time-frequency representation of the audio signal, and then produces a high-quality waveform using a conditional WaveNet synthesizer.We show that the Constant Q Transform (CQT) representation is particularly well-suited to convolutional architectures due to its approximate pitch equivariance.Based on human perceptual evaluations, we confirmed that TimbreTron recognizably transferred the timbre while otherwise preserving the musical content, for both monophonic and polyphonic samples.We made an accompanying demo video here: https://www.cs.toronto.edu/~huang/TimbreTron/index.html which we strongly encourage you to watch before reading the paper.

Wir stellen das TimbreTron vor, eine Pipeline zur Durchführung von qualitativ hochwertigem Timbre-Transfer auf musikalische Wellenformen unter Verwendung von CQT-Domain-Transfer.

Eine Methode zur Konvertierung von Aufnahmen eines bestimmten Musikinstruments in ein anderes durch Anwendung von CycleGAN, das für die Übertragung von Bildern entwickelt wurde, um Spektrogramme zu übertragen.

Die Autoren verwenden mehrere Techniken/Werkzeuge, um eine neuronale Klangfarbenübertragung (Umwandlung von Musik von einem Instrument in ein anderes) ohne gepaarte Trainingsbeispiele zu ermöglichen.

Beschreibt ein Modell für die Übertragung von musikalischen Klangfarben. Die Ergebnisse zeigen, dass das vorgeschlagene System sowohl für die Übertragung von Tonhöhe und Tempo als auch für die Anpassung der Klangfarbe effektiv ist.


Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them.But also generic hardware and software implementations of deep learning run more efficiently for sparse networks.Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints.We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network.DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded.We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance.DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.

In diesem Beitrag wird Deep Rewiring vorgestellt, ein Algorithmus, mit dem tiefe neuronale Netze trainiert werden können, wenn die Konnektivität des Netzes während des Trainings stark eingeschränkt ist.

Ein Ansatz zur Implementierung von Deep Learning direkt auf spärlich verbundenen Graphen, der ein effizientes Online-Training von Netzwerken und schnelles und flexibles Lernen ermöglicht.

Die Autoren stellen einen einfachen Algorithmus vor, der mit begrenztem Speicherplatz trainieren kann