Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs.MPT is typically used in combination with a technique called loss scaling, that works by scaling up the loss value up before the start of backpropagation in order to minimize the impact of numerical underflow on training.Unfortunately, existing methods make this loss scale value a hyperparameter that needs to be tuned per-model, and a single scale cannot be adapted to different layers at different training stages.We introduce a loss scaling-based training method called adaptive loss scaling that makes MPT easier and more practical to use, by removing the need to tune a model-specific loss scale hyperparameter.We achieve this by introducing layer-wise loss scale values which are automatically computed during training to deal with underflow more effectively than existing methods.We present experimental results on a variety of networks and tasks that show our approach can shorten the time to convergence and improve accuracy, compared with using the existing state-of-the-art MPT and single-precision floating point.

Wir entwickeln eine adaptive Verlustskalierung, um das Training mit gemischter Präzision zu verbessern, das die Ergebnisse des Stands der Technik übertrifft.

Vorschlag für eine adaptive Verlustskalierungsmethode während der Backpropagation für das Training mit gemischter Präzision, bei der die Skalierungsrate automatisch festgelegt wird, um die Rückflussrate zu reduzieren.

Die Autoren schlagen eine Methode zum Trainieren von Modellen mit FP16-Präzision vor, die eine ausgefeiltere Methode zur gleichzeitigen und automatischen Minimierung der Rückflussrate in jeder Schicht verwendet.


Many real-world problems, e.g. object detection, have outputs that are naturally expressed as sets of entities.This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors.We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks.Specifically, in our formulation we incorporate the permutation as unobservable variable and estimate its distribution during the learning process using alternating optimization.We demonstrate the validity of this new formulation on two relevant vision problems: object detection, for which our formulation outperforms state-of-the-art detectors such as Faster R-CNN and YOLO, and a complex CAPTCHA test, where we observe that, surprisingly, our set based network acquired the ability of mimicking arithmetics without any rules being coded.

Wir präsentieren einen neuartigen Ansatz für das Erlernen der Vorhersage von Mengen mit unbekannter Permutation und Kardinalität unter Verwendung von Feed-Forward Deep Neural Networks.

Eine Formulierung zum Erlernen der Verteilung über unbeobachtbare Permutationsvariablen auf der Grundlage von tiefen Netzwerken für das Problem der Mengenvorhersage.


Foveation is an important part of human vision, and a number of deep networks have also used foveation.However, there have been few systematic comparisons between foveating and non-foveating deep networks, and between different variable-resolution downsampling methods.Here we define several such methods, and compare their performance on ImageNet recognition with a Densenet-121 network.The best variable-resolution method slightly outperforms uniform downsampling.Thus in our experiments, foveation does not substantially help or hinder object recognition in deep networks.

Wir vergleichen die Leistung der Objekterkennung bei Bildern, die einheitlich und mit drei verschiedenen Foveationsschemen abgetastet wurden.


We explore the concept of co-design in the context of neural network verification.Specifically, we aim to train deep neural networks that not only are robust to adversarial perturbations but also whose robustness can be verified more easily.To this end, we identify two properties of network models - weight sparsity and so-called ReLU stability - that turn out to significantly impact the complexity of the corresponding verification task.We demonstrate that improving weight sparsity alone already enables us to turn computationally intractable verification problems into tractable ones.Then, improving ReLU stability leads to an additional 4-13x speedup in verification times.An important feature of our methodology is its "universality," in the sense that it can be used with a broad range of training procedures and verification approaches.

Wir entwickeln Methoden, um tiefe neuronale Modelle zu trainieren, die sowohl robust gegenüber negativen Störungen sind als auch deren Robustheit wesentlich einfacher zu verifizieren ist.

Diese Arbeit stellt verschiedene Möglichkeiten vor, einfache ReLU-Netze zu regularisieren, um die Robustheit gegenüber Angreifern, die nachweisbare Robustheit gegenüber Angreifern und die Verifikationsgeschwindigkeit zu optimieren.

In diesem Beitrag werden Methoden zur Ausbildung robuster neuronaler Netze vorgeschlagen, die schneller überprüft werden können. Dabei werden Pruning Methoden zur Förderung der Gewichtsdistanz und Regularisierungen zur Förderung der ReLU-Stabilität eingesetzt.


Batch Normalization (BatchNorm) has shown to be effective for improving and accelerating the training of deep neural networks.However, recently it has been shown that it is also vulnerable to adversarial perturbations.In this work, we aim to investigate the cause of adversarial vulnerability of the BatchNorm.We hypothesize that the use of different normalization statistics during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulnerability in the BatchNorm layer.We empirically proved this by experiments on various neural network architectures and datasets.Furthermore, we introduce Robust Normalization (RobustNorm) and experimentally show that it is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm.

Untersuchung, wie BatchNorm eine Sicherheitslücke verursacht und wie man sie vermeiden kann.

Dieses Werk befasst sich mit der Anfälligkeit von BatchNorm für unerwünschte Störungen und schlägt eine Alternative namens RobustNorm vor, die anstelle der Normalisierung eine Min-Max-Reskalierung verwendet.

Dieses Papier untersucht den Grund für die Anfälligkeit von BatchNorm und schlägt Robust Normalization vor, eine Normalisierungsmethode, die unter einer Vielzahl von Angriffsmethoden deutlich bessere Ergebnisse erzielt.


Electronic Health Records (EHR) comprise of longitudinal clinical observations portrayed with sparsity, irregularity, and high-dimensionality which become the major obstacles in drawing reliable downstream outcome.Despite greatly numbers of imputation methods are being proposed to tackle these issues, most of the existing methods ignore correlated features or temporal dynamics and entirely put aside the uncertainty.In particular, since the missing values estimates have the risk of being imprecise, it motivates us to pay attention to reliable and less certain information differently.In this work, we propose a novel variational-recurrent imputation network (V-RIN), which unified imputation and prediction network, by taking into account the correlated features, temporal dynamics, and further utilizing the uncertainty to alleviate the risk of biased missing values estimates.Specifically, we leverage the deep generative model to estimate the missing values based on the distribution among variables and a recurrent imputation network to exploit the temporal relations in conjunction with utilization of the uncertainty.We validated the effectiveness of our proposed model with publicly available real-world EHR dataset, PhysioNet Challenge 2012, and compared the results with other state-of-the-art competing methods in the literature.

Unser variational-rekurrentes Imputationsnetzwerk (V-RIN) berücksichtigt die korrelierten Merkmale, die zeitliche Dynamik und nutzt die Unsicherheit, um das Risiko einer verzerrten Schätzung fehlender Werte zu verringern.

Ein Netzwerk zur Imputation fehlender Daten, das Korrelationen, zeitliche Beziehungen und Datenunsicherheiten für das Problem der spärlichen Daten in EHRs berücksichtigt und eine höhere AUC bei der Klassifizierung von Mortalitätsraten erzielt.

In dem Papier wird eine Methode vorgestellt, die VAE und GRU mit Unsicherheitsfaktoren für die sequentielle Imputation fehlender Daten und die Ergebnisvorhersage kombiniert.


Despite the state-of-the-art accuracy of Deep Neural Networks (DNN) in various classification problems, their deployment onto resource constrained edge computing devices remains challenging due to their large size and complexity.Several recent studies have reported remarkable results in reducing this complexity through quantization of DNN models.However, these studies usually do not consider the changes in the loss function when performing quantization, nor do they take the different importances of DNN model parameters to the accuracy into account.We address these issues in this paper by proposing a new method, called adaptive quantization, which simplifies a trained DNN model by finding a unique, optimal precision for each network parameter such that the increase in loss is minimized.The optimization problem at the core of this method iteratively uses the loss function gradient to determine an error margin for each parameter and assigns it a precision accordingly.Since this problem uses linear functions, it is computationally cheap and, as we will show, has a closed-form approximate solution.Experiments on MNIST, CIFAR, and SVHN datasets showed that the proposed method can achieve near or better than state-of-the-art reduction in model size with similar error rates.Furthermore, it can achieve compressions close to floating-point model compression methods without loss of accuracy.

Eine adaptive Methode für die Festkomma-Quantisierung neuronaler Netze, die eher auf theoretischer Analyse als auf Heuristiken beruht.

Schlägt eine Methode zur Quantisierung neuronaler Netze vor, die es ermöglicht, Gewichte je nach ihrer Wichtigkeit mit unterschiedlicher Genauigkeit zu quantisieren, wobei der Verlust berücksichtigt wird.

In dieser Arbeit wird eine Technik zur Quantisierung der Gewichte eines neuronalen Netzes vorgeschlagen, bei der die Bittiefe/Präzision für jeden Parameter unterschiedlich ist.


We study the problem of learning permutation invariant representations that can capture containment relations.We propose training a model on a novel task: predicting the size of the symmetric difference between pairs of multisets, sets which may contain multiple copies of the same object.With motivation from fuzzy set theory, we formulate both multiset representations and how to predict symmetric difference sizes given these representations.We model multiset elements as vectors on the standard simplex and multisets as the summations of such vectors, and we predict symmetric difference as the l1-distance between multiset representations.We demonstrate that our representations more effectively predict the sizes of symmetric differences than DeepSets-based approaches with unconstrained object representations.Furthermore, we demonstrate that the model learns meaningful representations, mapping objects of different classes to different standard basis vectors.

Auf der Grundlage der Fuzzy-Mengen-Theorie schlagen wir ein Modell vor, das nur die Größen der symmetrischen Unterschiede zwischen Paaren von Multisets angibt und Repräsentationen solcher Multisets und ihrer Elemente lernt.

Dieses Papier schlägt eine neue Aufgabe des Mengenlernens vor, die Vorhersage der Größe der symmetrischen Differenz zwischen mehreren Mengen, und gibt eine Methode zur Lösung der Aufgabe auf der Grundlage der Fuzzy-Mengentheorie.


It is important to collect credible training samples $(x,y)$ for building data-intensive learning systems (e.g., a deep learning system).In the literature, there is a line of studies on eliciting distributional information from self-interested agents who hold a relevant information.  Asking people to report complex distribution $p(x)$, though theoretically viable, is challenging in practice.This is primarily due to the heavy cognitive loads required for human agents to reason and report this high dimensional information.Consider the example where we are interested in building an image classifier via first collecting a certain category of  high-dimensional image data.While classical elicitation results apply to eliciting a complex and generative (and continuous) distribution $p(x)$ for this image data, we are interested in eliciting samples $x_i \sim p(x)$ from agents.This paper introduces a deep learning aided method to incentivize credible sample contributions from selfish and rational agents.The challenge to do so is to design an incentive-compatible score function to score each reported sample to induce truthful reports, instead of an arbitrary or even adversarial one.We show that with accurate estimation of a certain $f$-divergence function we are able to achieve approximate incentive compatibility in eliciting truthful samples.We then present an efficient estimator with theoretical guarantee via studying the variational forms of $f$-divergence function.Our work complements the literature of information elicitation via introducing the problem of \emph{sample elicitation}.  We also show a connection between this sample elicitation problem and $f$-GAN, and how this connection can help reconstruct an estimator of the distribution based on collected samples.

In diesem Werk wird eine Deep-Learning-gestützte Methode vorgeschlagen, um glaubwürdige Proben von eigennützigen Agenten zu erhalten.

Die Autoren schlagen einen Rahmen für die Auswahl von Stichproben für das Problem der Auswahl von glaubwürdigen Stichproben von Agenten für komplexe Verteilungen vor, schlagen vor, dass tiefe neuronale Rahmen in diesem Rahmen angewendet werden können, und verbinden Stichprobenauswahl und f-GAN.

In diesem Papier wird das Problem der Stichprobenauswahl untersucht und ein Ansatz des Deep Learning vorgeschlagen, der sich auf den dualen Ausdruck der f-Divergenz stützt, die sich als Maximum über eine Menge von Funktionen t schreibt.


The celebrated Sequence to Sequence learning (Seq2Seq) technique and its numerous variants achieve excellent performance on many tasks.However, many machine learning tasks have inputs naturally represented as graphs; existing Seq2Seq models face a significant challenge in achieving accurate conversion from graph form to the appropriate sequence.To address this challenge, we introduce a general end-to-end graph-to-sequence neural encoder-decoder architecture that maps an input graph to a sequence of vectors and uses an attention-based LSTM method to decode the target sequence from these vectors.Our method first generates the node and graph embeddings using an improved graph-based neural network with a novel aggregation strategy to incorporate edge direction information in the node embeddings.We further introduce an attention mechanism that aligns node embeddings and the decoding sequence to better cope with large graphs.Experimental results on bAbI, Shortest Path, and Natural Language Generation tasks demonstrate that our model achieves state-of-the-art performance and significantly outperforms existing graph neural networks, Seq2Seq, and Tree2Seq models; using the proposed bi-directional node embedding aggregation strategy, the model can converge rapidly to the optimal performance.

Graph to Sequence Learning mit aufmerksamkeitsbasierten neuronalen Netzen

Eine graph2seq-Architektur, die einen Graphen-Encoder, der GGNN- und GCN-Komponenten mit einem Aufmerksamkeits-Sequenz-Encoder kombiniert, und die Verbesserungen gegenüber den Basislinien zeigt.

In dieser Arbeit wird ein Ende-zu-Ende Graph Encoder zu Sequenz Decoder Modell mit einem dazwischen liegenden Aufmerksamkeitsmechanismus vorgeschlagen.


We address the problem of learning to discover 3D parts for objects in unseen categories.Being able to learn the geometry prior of parts and transfer this prior to unseen categories pose fundamental challenges on data-driven shape segmentation approaches.Formulated as a contextual bandit problem, we propose a learning-based iterative grouping framework which learns a grouping policy to progressively merge small part proposals into bigger ones in a bottom-up fashion.At the core of our approach is to restrict the local context for extracting part-level features, which encourages the generalizability to novel categories.On a recently proposed large-scale fine-grained 3D part dataset, PartNet, we demonstrate that our method can transfer knowledge of parts learned from 3 training categories to 21 unseen testing categories without seeing any annotated samples.Quantitative comparisons against four strong shape segmentation baselines show that we achieve the state-of-the-art performance.

Ein Zero-Shot-Segmentierungsrahmen für die Segmentierung von 3D-Objektteilen. Modellierung der Segmentierung als Entscheidungsprozess und Lösung als kontextuelles Bandit-Problem.

Eine Methode zur Segmentierung von 3D-Punktansammlungen von Objekten in Einzelteilen, die sich auf die Verallgemeinerung von Teilgruppierungen auf neue Objektkategorien konzentriert, die während des Trainings nicht gesehen wurden, und die eine starke Leistung im Vergleich zu den Basislinien zeigt.

In dieser Arbeit wird eine Methode zur Segmentierung von Teilen in Objektpunktansammlungen vorgeschlagen.


This paper presents the ballistic graph neural network.Ballistic graph neural network tackles the weight distribution from a transportation perspective and has many different properties comparing to the traditional graph neural network pipeline.The ballistic graph neural network does not require to calculate any eigenvalue.The filters propagate exponentially faster($\sigma^2 \sim T^2$) comparing to traditional graph neural network($\sigma^2 \sim T$).We use a perturbed coin operator to perturb and optimize the diffusion rate.Our results show that by selecting the diffusion speed, the network can reach a similar accuracy with fewer parameters.We also show the perturbed filters act as better representations comparing to pure ballistic ones.We provide a new perspective of training graph neural network, by adjusting the diffusion rate, the neural network's performance can be improved.

Eine neue Perspektive für die Erfassung der Korrelation zwischen Knoten auf der Grundlage von Diffusionseigenschaften.

Eine neue Diffusionsoperation für neuronale Graphennetze, die keine Eigenwertberechnung erfordert und sich im Vergleich zu herkömmlichen neuronalen Graphennetzen exponentiell schneller ausbreiten kann.

In dieser Arbeit wird vorgeschlagen, das Problem der Diffusionsgeschwindigkeit durch die Einführung eines ballistischen Laufs zu lösen.


In this paper, we propose a \textit{weak supervision} framework for neural ranking tasks based on the data programming paradigm \citep{Ratner2016}, which enables us to leverage multiple weak supervision signals from different sources.Empirically, we consider two sources of weak supervision signals, unsupervised ranking functions and semantic feature similarities.We train a BERT-based passage-ranking model (which achieves new state-of-the-art performances on two benchmark datasets with full supervision) in our weak supervision framework.Without using ground-truth training labels, BERT-PR models outperform BM25 baseline by a large margin on all three datasets and even beat the previous state-of-the-art results with full supervision on two of datasets.

Wir schlagen eine Trainings-Pipeline mit schwacher Überwachung vor, die auf dem Datenprogrammierungs-Framework für Ranking-Aufgaben basiert, in dem wir ein BERT-basiertes Ranking-Modell trainieren und neue SOTA-Ergebnisse erzielen.

Die Autoren schlagen eine Kombination aus BERT und dem Framework für schwache Überwachung vor, um das Problem des Passagen-Rankings anzugehen, und erzielen damit bessere Ergebnisse als der vollständig überwachte Stand der Technik.


We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective.We demonstrate a very universal Frequency Principle (F-Principle) --- DNNs often fit target functions from low to high frequencies --- on high-dimensional benchmark datasets, such as MNIST/CIFAR10, and deep networks, such as VGG16.This F-Principle of DNNs is opposite to the learning behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibits faster convergence for higher frequencies, for various scientific computing problems.With a naive theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions.The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function.This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.

Bei realen Problemen haben wir festgestellt, dass DNNs während des Trainingsprozesses häufig Zielfunktionen von niedrigen zu hohen Frequenzen anpassen.

In dieser Arbeit wird der Verlust von neuronalen Netzen im Fourier-Bereich analysiert und festgestellt, dass DNNs dazu neigen, niederfrequente Komponenten vor hochfrequenten zu lernen.

Das Werk untersucht den Trainingsprozess von NNs durch Fourier-Analyse und kommt zu dem Schluss, dass NNs niederfrequente Komponenten vor hochfrequenten Komponenten lernen.


The problem of accelerating drug discovery relies heavily on automatic tools to optimize precursor molecules to afford them with better biochemical properties.Our work in this paper substantially extends prior state-of-the-art on graph-to-graph translation methods for molecular optimization.In particular, we realize coherent multi-resolution representations by interweaving the encoding of substructure components with the atom-level encoding of the original molecular graph.Moreover, our graph decoder is fully autoregressive, and interleaves each step of adding a new substructure with the process of resolving its attachment to the emerging molecule.We evaluate our model on multiple molecular optimization tasks and show that our model significantly outperforms previous state-of-the-art baselines.

Wir schlagen einen hierarchisch gekoppelten Encoder-Decoder mit mehreren Auflösungen für die Übersetzung von Graphen in Graphen vor.

Ein hierarchisches Graph-zu-Graph-Übersetzungsmodell zur Erzeugung molekularer Graphen unter Verwendung chemischer Substrukturen als Bausteine, das vollständig autoregressiv ist und kohärente Mehrfachauflösungsrepräsentationen erlernt, wodurch es frühere Modelle übertrifft.

Die Autoren stellen eine hierarchische Graph-zu-Graph-Übersetzungsmethode zur Erzeugung neuartiger organischer Moleküle vor.


Equivariance is a nice property to have as it produces much more parameter efficient neural architectures and preserves the structure of the input through the feature mapping.Even though some combinations of transformations might never appear (e.g. an upright face with a horizontal nose), current equivariant architectures consider the set of all possible transformations in a transformation group when learning feature representations.Contrarily, the human visual system is able to attend to the set of relevant transformations occurring in the environment and utilizes this information to assist and improve object recognition.Based on this observation, we modify conventional equivariant feature mappings such that they are able to attend to the set of co-occurring transformations in data and generalize this notion to act on groups consisting of multiple symmetries.We show that our proposed co-attentive equivariant neural networks consistently outperform conventional rotation equivariant and rotation & reflection equivariant neural networks on rotated MNIST and CIFAR-10.

Wir nutzen die Aufmerksamkeit, um äquivariante neuronale Netze auf die Menge oder die gemeinsam auftretenden Transformationen in den Daten zu beschränken.

In dieser Arbeit wird Aufmerksamkeit mit Gruppenäquivarianz kombiniert, wobei insbesondere die p4m-Gruppe der Rotationen, Translationen und Spiegelungen betrachtet wird, und es wird eine Form der Selbstaufmerksamkeit abgeleitet, die die Äquivarianzeigenschaft nicht zerstört.

Die Autoren schlagen einen Selbstbeobachtungsmechanismus für rotationsäquivariante neuronale Netze vor, der die Klassifizierungsleistung gegenüber regulären rotationsäquivarianten Netzen verbessert.


The fast generation and refinement of protein backbones would constitute a major advancement to current methodology for the design and development of de novo proteins.In this study, we train Generative Adversarial Networks (GANs) to generate fixed-length full-atom protein backbones, with the goal of sampling from the distribution of realistic 3-D backbone fragments.We represent protein structures by pairwise distances between all backbone atoms, and present a method for directly recovering and refining the corresponding backbone coordinates in a differentiable manner.We show that interpolations in the latent space of the generator correspond to smooth deformations of the output backbones, and that test set structures not seen by the generator during training exist in its image.Finally, we perform sequence design, relaxation, and ab initio folding of a subset of generated structures, and show that in some cases we can recover the generated folds after forward-folding.Together, these results suggest a mechanism for fast protein structure refinement and folding using external energy functions.

Wir trainieren ein GAN zur Generierung und Wiederherstellung von Protein-Rückgraten mit vollständigen Atomen und zeigen, dass wir in ausgewählten Fällen die generierten Proteine nach Sequenzdesign und in von Anfang an Vorwärtsfaltung wiederherstellen können.

Ein generatives Modell für das Proteinrückgrat, das ein GAN, ein Autoencoder-ähnliches Netzwerk und einen Verfeinerungsprozess verwendet, sowie eine Reihe von qualitativen Bewertungen, die auf positive Ergebnisse hindeuten.

In diesem Beitrag wird ein Ende-zu-Ende Ansatz für die Generierung von Protein-Rückgraten unter Verwendung generativer adversarialer Netzwerke vorgestellt.


Few-Shot Learning (learning with limited labeled data) aims to overcome the limitations of traditional machine learning approaches which require thousands of labeled examples to train an effective model.Considered as a hallmark of human intelligence, the community has recently witnessed several contributions on this topic, in particular through meta-learning, where a model learns how to learn an effective model for few-shot learning.The main idea is to acquire prior knowledge from a set of training tasks, which is then used to perform (few-shot) test tasks.Most existing work assumes that both training and test tasks are drawn from the same distribution, and a large amount of labeled data is available in the training tasks.This is a very strong assumption which restricts the usage of meta-learning strategies in the real world where ample training tasks following the same distribution as test tasks may not be available.In this paper, we propose a novel meta-learning paradigm wherein a few-shot learning model is learnt, which simultaneously overcomes domain shift between the train and test tasks via adversarial domain adaptation.We demonstrate the efficacy the proposed method through extensive experiments.

Beim Meta Learning für Few Shot Learning wird davon ausgegangen, dass Trainings- und Testaufgaben aus der gleichen Verteilung gezogen werden. Was macht man, wenn das nicht der Fall ist? Meta-Learning mit Bereichsanpassung auf Aufgabenebene.

Diese Arbeit schlägt ein Modell vor, das unüberwachte adversarische Domänenanpassung mit prototypischen Netzwerken kombiniert, die besser abschneiden als die Basissysteme für Few-Shot Learning Aufgaben mit Domänenverschiebung.

Die Autoren schlugen eine Meta-Domänenanpassung vor, um das Szenario der Domänenverschiebung in einer Meta-Lernumgebung anzugehen, und konnten in mehreren Experimenten Leistungsverbesserungen nachweisen.


Universal probabilistic programming systems (PPSs) provide a powerful framework for specifying rich and complex probabilistic models.However, this expressiveness comes at the cost of substantially complicating the process of drawing inferences from the model.In particular, inference can become challenging when the support of the model varies between executions.Though general-purpose inference engines have been designed to operate in such settings, they are typically inefficient, often relying on proposing from the prior to make transitions.To address this, we introduce a new inference framework: Divide, Conquer, and Combine (DCC).DCC divides the program into separate straight-line sub-programs, each of which has a fixed support allowing more powerful inference algorithms to be run locally, before recombining their outputs in a principled fashion.We show how DCC can be implemented as an automated and general-purpose PPS inference engine, and empirically confirm that it can provide substantial performance improvements over previous approaches.

Divide, Conquer, and Combine ist ein neues Inferenzschema, das auf probabilistischen Programmen mit stochastischer Unterstützung angewendet werden kann, d.h. die Existenz der Variablen selbst ist stochastisch.


Detecting communities or the modular structure of real-life networks (e.g. a socialnetwork or a product purchase network) is an important task because the way anetwork functions is often determined by its communities.The traditional approaches to community detection involve modularity-based approaches,which generally speaking, construct partitions based on heuristics thatseek to maximize the ratio of the edges within the partitions to those betweenthem.Node embedding approaches, which represent each node in a graph as areal-valued vector, transform the problem of community detection in a graph tothat of clustering a set of vectors.Existing node embedding approaches are primarilybased on first initiating uniform random walks from each node to constructa context of a node and then seeks to make the vector representation ofthe node close to its context.However, standard node embedding approaches donot directly take into account the community structure of a network while constructingthe context around each node.To alleviate this, we explore two differentthreads of work.First, we investigate the use of biased random walks (specifically,maximum entropy based walks) to obtain more centrality preserving embeddingof nodes, which we hypothesize may lead to more effective clusters in the embeddedspace.Second, we propose a community structure aware node embeddingapproach where we incorporate modularity-based partitioning heuristics intothe objective function of node embedding.We demonstrate that our proposed approachfor community detection outperforms a number of modularity-based baselinesas well as K-means on a standard node-embedded vector space (specifically,node2vec) on a wide range of real-life networks of different sizes and densities.

Ein gemeinschaftserhaltender Knoteneinbettungsalgorithmus, der zu einer effektiveren Erkennung von Gemeinschaften mit einer Clusterbildung im eingebetteten Raum führt.


A point cloud is an agile 3D representation, efficiently modeling an object's surface geometry.However, these surface-centric properties also pose challenges on designing tools to recognize and synthesize point clouds.This work presents a novel autoregressive model, PointGrow, which generates realistic point cloud samples from scratch or conditioned from given semantic contexts.Our model operates recurrently, with each point sampled according to a conditional distribution given its previously-generated points.Since point cloud object shapes are typically encoded by long-range interpoint dependencies, we augment our model with dedicated self-attention modules to capture these relations.Extensive evaluation demonstrates that PointGrow achieves satisfying performance on both unconditional and conditional point cloud generation tasks, with respect to fidelity, diversity and semantic preservation.Further, conditional PointGrow learns a smooth manifold of given images where 3D shape interpolation and arithmetic calculation can be performed inside.

Ein autoregressives Deep-Learning-Modell zur Erzeugung verschiedener Punkt Clouds.

Ein Ansatz zur Erzeugung von 3D-Formen als Punkt Clouds, der die lexikografische Ordnung von Punkten nach Koordinaten berücksichtigt und ein Modell zur Vorhersage von Punkten in dieser Reihenfolge trainiert.

In diesem Beitrag wird ein generatives Modell für Punkt Clouds vorgestellt, das ein Pixel-RNN-ähnliches autoregressives Modell und ein Aufmerksamkeitsmodell zur Behandlung von Interaktionen über größere Entfernungen verwendet.


Reinforcement learning and evolutionary algorithms can be used to create sophisticated control solutions.Unfortunately explaining how these solutions work can be difficult to due to their "black box" nature.In addition, the time-extended nature of control algorithms often prevent direct applications of explainability techniques used for standard supervised learning algorithms.This paper attempts to address explainability of blackbox control algorithms through six different techniques:1) Bayesian rule lists,2) Function analysis,3) Single time step integrated gradients,4) Grammar-based decision trees,5) Sensitivity analysis combined with temporal modeling with LSTMs, and6) Explanation templates.These techniques are tested on a simple 2d domain, where a simulated rover attempts to navigate through obstacles to reach a goal.For control, this rover uses an evolved multi-layer perception that maps an 8d field of obstacle and goal sensors to an action determining where it should go in the next time step.Results show that some simple insights in explaining the neural network are possible, but that good explanations are difficult.

Beschreibt eine Reihe von Erklärungsmethoden, die auf einen einfachen neuronalen Netzcontroller für die Navigation angewendet werden.

Dieser Beitrag bietet Einblicke und Erklärungen für das Problem, Erklärungen für ein mehrschichtiges Perzeptron zu liefern, das als inverser Controller für die Rover-Bewegung verwendet wird, sowie Ideen, wie man ein Black-Box-Modell erklären kann.


The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments.This challenging task demands that the agent be aware of which instruction was completed, which instruction is needed next, which way to go, and its navigation progress towards the goal.In this paper, we introduce a self-monitoring agent with two complementary components: (1) visual-textual co-grounding module to locate the instruction completed in the past, the instruction required for the next action, and the next moving direction from surrounding images and (2) progress monitor to ensure the grounded instruction correctly reflects the navigation progress.We test our self-monitoring agent on a standard benchmark and analyze our proposed approach through a series of ablation studies that elucidate the contributions of the primary components.Using our proposed method, we set the new state of the art by a significant margin (8% absolute increase in success rate on the unseen test set).Code is available at https://github.com/chihyaoma/selfmonitoring-agent.

Wir schlagen einen selbstüberwachenden Agenten für die Seh- und Sprachnavigationsaufgabe vor.

Eine Methode für die Navigation in Bild und Sprache, die den Fortschritt im Unterricht mit Hilfe eines Fortschrittsmonitors und eines visuell-textuellen Co-Grounding-Moduls verfolgt und bei Standard-Benchmarks gut abschneidet.

In diesem Beitrag wird ein Modell für die Navigation durch Sehen und Sprache mit panoramischer visueller Aufmerksamkeit und einem zusätzlichen Verlust bei der Fortschrittskontrolle beschrieben, das Ergebnisse auf dem neuesten Stand der Technik liefert.


Environments in Reinforcement Learning (RL) are usually only partially observable.To address this problem, a possible solution is to provide the agent with information about past  observations.While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alternative representation which is based on the record of the past events observed in a given episode.Inspired by the human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using self-supervision. We evaluate our history representation method using two challenging RL benchmarks: some games of the Atari-57 suite and the 3D environment Obstacle Tower.Using these benchmarks we show the advantage of our solution with respect to common RNN-based approaches.

Ereigniserkennung zur Darstellung der Historie für den Agenten in RL

Die Autoren untersuchen das RL-Problem unter teilweise beobachteten Bedingungen und schlagen eine Lösung vor, die ein FFNN verwendet, aber eine Geschichtsdarstellung bietet und PPO übertrifft.

In diesem Beitrag wird eine neue Methode zur Darstellung der Vergangenheit als Eingabe für einen RL-Agenten vorgeschlagen, die sich als besser erweist als PPO und eine RNN-Variante von PPO.


The unconditional generation of high fidelity images is a longstanding benchmarkfor testing the performance of image decoders.Autoregressive image modelshave been able to generate small images unconditionally, but the extension ofthese methods to large images where fidelity can be more readily assessed hasremained an open problem.Among the major challenges are the capacity to encodethe vast previous context and the sheer difficulty of learning a distribution thatpreserves both global semantic coherence and exactness of detail.To address theformer challenge, we propose the Subscale Pixel Network (SPN), a conditionaldecoder architecture that generates an image as a sequence of image slices of equalsize.The SPN compactly captures image-wide spatial dependencies and requires afraction of the memory and the computation.To address the latter challenge, wepropose to use multidimensional upscaling to grow an image in both size and depthvia intermediate stages corresponding to distinct SPNs.We evaluate SPNs on theunconditional generation of CelebAHQ of size 256 and of ImageNet from size 32to 128.We achieve state-of-the-art likelihood results in multiple settings, set upnew benchmark results in previously unexplored settings and are able to generatevery high fidelity large scale samples on the basis of both datasets.

Wir zeigen, dass autoregressive Modelle sehr realitätsnahe Bilder erzeugen können.

Eine Architektur, die Decoder einsetzt, Decoder zur Größenanpassung und Decoder zur Tiefenanpassung, um das Problem des Erlernens von weitreichenden Abhängigkeiten in Bildern zu lösen, um Bilder mit hoher Wiedergabetreue zu erhalten.

Dieser Beitrag befasst sich mit dem Problem der Generierung von Bildern mit hoher Genauigkeit und zeigt erfolgreich überzeugende Imagenet-Beispiele mit einer Auflösung von 128x128 für ein Likelihood-Density-Modell.


Real-world dynamical systems often consist of multiple stochastic subsystems that interact with each other.Modeling and forecasting the behavior of such dynamics are generally not easy, due to the inherent hardness in understanding the complicated interactions and evolutions of their constituents.This paper introduces the relational state-space model (R-SSM), a sequential hierarchical latent variable model that makes use of graph neural networks (GNNs) to simulate the joint state transitions of multiple correlated objects.By letting GNNs cooperate with SSM, R-SSM provides a flexible way to incorporate relational information into the modeling of multi-object dynamics.We further suggest augmenting the model with normalizing flows instantiated for vertex-indexed random variables and propose two auxiliary contrastive objectives to facilitate the learning.The utility of R-SSM is empirically evaluated on synthetic and real time series datasets.

Ein tiefes hierarchisches Zustandsraummodell, in dem die Zustandsübergänge korrelierter Objekte durch graphische neuronale Netze koordiniert werden.

Ein hierarchisches latentes Variablenmodell für sequenzielle dynamische Prozesse mehrerer Objekte, wenn jedes Objekt eine signifikante Stochastizität aufweist.

Diese Arbeit stellt ein relationales Zustandsraummodell vor, das die gemeinsamen Zustandsübergänge korrelierter Objekte simuliert, die in einer Graphenstruktur hierarchisch koordiniert sind.


Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses).When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed.While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents.This paper proposes to add such inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated.Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

Wir führen eine neue induktive Verzerrung ein, die Baumstrukturen in rekurrente neuronale Netze integriert.

In diesem Papier wird ON-LSTM vorgeschlagen, eine neue RNN-Einheit, die die latente Baumstruktur in rekurrente Modelle integriert und gute Ergebnisse bei der Sprachmodellierung, dem unbeaufsichtigten Parsing, der gezielten syntaktischen Auswertung und der logischen Inferenz erzielt.


Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures.A completely satisfactory explanation for their success remains elusive.Here, we present a novel explanation for the benefits of skip connections in training very deep networks.The difficulty of training deep networks is partly due to the singularities caused by the non-identifiability of the model.Several such singularities have been identified in previous works:(i) overlap singularities caused by the permutation symmetry of nodes in a given layer,(ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes,(iii) singularities generated by the linear dependence of the nodes.These singularities cause degenerate manifolds in the loss landscape that slow down learning.We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent.Moreover, for typical initializations, skip connections move the network away from the "ghosts" of these singularities and sculpt the landscape around them to alleviate the learning slow-down.These hypotheses are supported by evidence from simplified models, as well as from experiments with deep networks trained on real-world datasets.

Degenerierte Mannigfaltigkeiten, die sich aus der Nicht-Identifizierbarkeit des Modells ergeben, verlangsamen das Lernen in tiefen Netzen; Skip-Verbindungen helfen, indem sie Degenerationen aufbrechen.

Die Autoren zeigen, dass Eliminations- und Überlappungssingularitäten das Lernen in tiefen neuronalen Netzen behindern, und demonstrieren, dass Skip-Verbindungen die Häufigkeit dieser Singularitäten reduzieren und das Lernen beschleunigen können.

Das Werk untersucht die Verwendung von Skip-Verbindungen in tiefen Netzwerken als eine Möglichkeit, Singularitäten in der Hessian-Matrix während des Trainings zu mildern.


Representation learning is a central challenge across a range of machine learning areas.In reinforcement learning, effective and functional representations have the potential to tremendously accelerate learning progress and solve more challenging problems.Most prior work on representation learning has focused on generative approaches, learning representations that capture all the underlying factors of variation in the observation space in a more disentangled or well-ordered manner.In this paper, we instead aim to learn functionally salient representations: representations that are not necessarily complete in terms of capturing all factors of variation in the observation space, but rather aim to capture those factors of variation that are important for decision making -- that are "actionable".These representations are aware of the dynamics of the environment, and capture only the elements of the observation that are necessary for decision making rather than all factors of variation, eliminating the need for explicit reconstruction.We show how these learned representations can be useful to improve exploration for sparse reward problems, to enable long horizon hierarchical reinforcement learning, and as a state representation for learning policies for downstream tasks.We evaluate our method on a number of simulated environments, and compare it to prior methods for representation learning, exploration, and hierarchical reinforcement learning.

Lernen von Zustandsdarstellungen, die die für die Kontrolle notwendigen Faktoren erfassen.

Ein Ansatz für das Repräsentationslernen im Rahmen des Reinforcement Learnings, der zwei Stufen funktional in Bezug auf die Aktionen unterscheidet, die erforderlich sind, um sie zu erreichen.

In dem Werk wird eine Methode zum Erlernen von Darstellungen vorgestellt, bei denen die Nähe im euklidischen Abstand Zustände darstellt, die durch ähnliche Strategien erreicht werden.


We explore the behavior of a standard convolutional neural net in a setting that introduces classification tasks sequentially and requires the net to master new tasks while preserving mastery of previously learned tasks.  This setting corresponds to that which human learners face as they acquire domain expertise, for example, as an individual reads a textbook chapter-by-chapter.Through simulations involving sequences of 10 related tasks, we find reason for optimism that nets will scale well as they advance from having a single skill to becoming domain experts.We observed two key phenomena.First, forward facilitation---the accelerated learning of task n+1 having learned n previous tasks---grows with n. Second, backward interference---the forgetting of the n previous tasks when learning task n+1---diminishes with n.  Forward facilitation is the goal of research on metalearning, and reduced backward interference is the goal of research on ameliorating catastrophic forgetting.We find that both of these goals are attained simply through broader exposure to a domain.

Wir untersuchen das Verhalten eines CNN bei der Bewältigung neuer Aufgaben unter Beibehaltung der Beherrschung bereits gelernter Aufgaben


We demonstrate a low effort method that unsupervisedly constructs task-optimized embeddings from existing word embeddings to gain performance on a supervised end-task.This avoids additional labeling or building more complex model architectures by instead providing specialized embeddings better fit for the end-task(s).Furthermore, the method can be used to roughly estimate whether a specific kind of end-task(s) can be learned form, or is represented in, a given unlabeled dataset, e.g. using publicly available probing tasks.We evaluate our method for diverse word embedding probing tasks and by size of embedding training corpus -- i.e. to explore its use in reduced (pretraining-resource) settings.

Morty überarbeitet vordefinierte Worteinbettungen, um entweder: (a) Verbesserung der gesamten Einbettungsleistung (für Multi-Task-Einstellungen) oder Verbesserung der Single-Task-Leistung, wobei nur minimaler Aufwand erforderlich ist.


Data augmentation is commonly used to encode invariances in learning methods.However, this process is often performed in an inefficient manner, as artificial examples are created by applying a number of transformations to all points in the training set.The resulting explosion of the dataset size can be an issue in terms of storage and training costs, as well as in selecting and tuning the optimal set of transformations to apply.In this work, we demonstrate that it is possible to significantly reduce the number of data points included in data augmentation while realizing the same accuracy and invariance benefits of augmenting the entire dataset.We propose a novel set of subsampling policies, based on model influence and loss, that can achieve a 90% reduction in augmentation set size while maintaining the accuracy gains of standard data augmentation.

Das selektive Erweitern von schwer zu klassifizierenden Punkten führt zu einem effizienten Training.

Die Autoren untersuchen das Problem der Identifizierung von Subsampling-Strategien für die Datenerweiterung und schlagen Strategien vor, die auf Modelleinfluss und -verlust sowie auf einem empirischen Benchmarking der vorgeschlagenen Methoden basieren.

Die Autoren schlagen vor, Einfluss- oder verlustbasierte Methoden zu verwenden, um eine Teilmenge von Punkten auszuwählen, die zur Erweiterung von Datensätzen für Trainingsmodelle verwendet werden, wobei der Verlust additiv über die Datenpunkte ist.


Over the last few years exciting work in deep generative models has produced models able to suggest new organic molecules by generating strings, trees, and graphs representing their structure.While such models are able to generate molecules with desirable properties, their utility in practice is limited due to the difficulty in knowing how to synthesize these molecules.We therefore propose a new molecule generation model, mirroring a more realistic real-world process, where reactants are selected and combined to form more complex molecules.More specifically, our generative model proposes a bag of initial reactants (selected from a pool of commercially-available molecules) and uses a reaction model to predict how they react together to generate new molecules.Modeling the entire process of constructing a molecule during generation offers a number of advantages.First, we show that such a model has the ability to generate a wide, diverse set of valid and unique molecules due to the useful inductive biases of modeling reactions.Second, modeling synthesis routes rather than final molecules offers practical advantages to chemists who are not only interested in new molecules but also suggestions on stable and safe synthetic routes.Third, we demonstrate the capabilities of our model to also solve one-step retrosynthesis problems, predicting a set of reactants that can produce a target product.

Ein tiefes generatives Modell für organische Moleküle, das zunächst Reaktionsbausteine erzeugt und diese dann mithilfe eines Reaktionsvorhersageprogramms kombiniert.

Ein molekulares generatives Modell, das Moleküle in einem zweistufigen Prozess erzeugt und Synthesewege für die erzeugten Moleküle bereitstellt, so dass die Benutzer die synthetische Zugänglichkeit der erzeugten Verbindungen untersuchen können.


Deep neural networks are complex non-linear models used as predictive analytics tool and have demonstrated state-of-the-art performance on many classification tasks.  However, they have no inherent capability to recognize when their predictions might go wrong.There have been several efforts in the recent past to detect natural errors i.e.  misclassified inputs but these mechanisms pose additional energy requirements.  To address this issue, we present a novel post-hoc framework to detect natural errors in an energy efficient way.  We achieve this by appending relevant features based linear classifiers per class referred as Relevant features based Auxiliary Cells (RACs).   The proposed technique makes use of the consensus between RACs appended at few selected hidden layers to distinguish the correctly classified inputs from misclassified inputs.The combined confidence of RACs is utilized to determine if classification should terminate at an early stage.We demonstrate the effectiveness of our technique on various image classification datasets such as CIFAR10, CIFAR100 and Tiny-ImageNet.Our results show that for CIFAR100 dataset trained on VGG16 network, RACs can detect 46% of the misclassified examples along with 12% reduction in energy compared to the baseline network while 69% of the examples are correctly classified.

Verbesserung der Robustheit und Energieeffizienz eines tiefen neuronalen Netzes unter Verwendung der versteckten Darstellungen.

Dieser Artikel zielt darauf ab, die Fehlklassifikationen von tiefen neuronalen Netzen auf energieeffiziente Weise zu reduzieren, indem relevante merkmalsbasierte Hilfszellen nach einer oder mehreren versteckten Schichten hinzugefügt werden, um zu entscheiden, ob die Klassifizierung vorzeitig beendet werden soll.


Many methods have been developed to represent knowledge graph data, which implicitly exploit low-rank latent structure in the data to encode known information and enable unknown facts to be inferred.To predict whether a relationship holds between entities, their embeddings are typically compared in the latent space following a relation-specific mapping.Whilst link prediction has steadily improved, the latent structure, and hence why such models capture semantic information, remains unexplained.We build on recent theoretical interpretation of word embeddings as a basis to consider an explicit structure for representations of relations between entities.For identifiable relation types, we are able to predict properties and justify the relative performance of leading knowledge graph representation methods, including their often overlooked ability to make independent predictions.

Verständnis der Struktur von Wissensgraphen unter Verwendung von Erkenntnissen aus Worteinbettungen.

In diesem Beitrag wird versucht, die latente Struktur zu verstehen, die den Methoden zur Einbettung von Wissensgraphen zugrunde liegt, und es wird gezeigt, dass die Fähigkeit eines Modells, einen Beziehungstyp darzustellen, von den Beschränkungen der Modellarchitektur in Bezug auf die Beziehungsbedingungen abhängt.

Diese Arbeit schlägt eine detaillierte Studie über die Erklärbarkeit von Link-Prädiktionsmodellen (LP) vor, indem es eine neue Interpretation von Worteinbettungen verwendet, um ein besseres Verständnis der Modellleistung von LPs zu ermöglichen.


Many real-world applications involve multivariate, geo-tagged time series data: at each location, multiple sensors record corresponding measurements.For example, air quality monitoring system records PM2.5, CO, etc.The resulting time-series data often has missing values due to device outages or communication errors.In order to impute the missing values, state-of-the-art methods are built on Recurrent Neural Networks (RNN), which process each time stamp sequentially, prohibiting the direct modeling of the relationship between distant time stamps.Recently, the self-attention mechanism has been proposed for sequence modeling tasks such as machine translation, significantly outperforming RNN because the relationship between each two time stamps can be modeled explicitly.In this paper, we are the first to adapt the self-attention mechanism for multivariate, geo-tagged time series data.In order to jointly capture the self-attention across different dimensions (i.e. time, location and sensor measurements) while keep the size of attention maps reasonable, we propose a novel approach called Cross-Dimensional Self-Attention (CDSA) to process each dimension sequentially, yet in an order-independent manner.On three real-world datasets, including one our newly collected NYC-traffic dataset, extensive experiments demonstrate the superiority of our approach compared to state-of-the-art methods for both imputation and forecasting tasks.

Ein neuartiger Mechanismus zur Selbstbeobachtung für die Imputation multivariater, geogetaggter Zeitreihen.

In diesem Beitrag wird das Problem der Anwendung des Transformatorennetzwerks auf räumlich-zeitliche Daten in einer rechnerisch effizienten Art und Weise behandelt und es werden Möglichkeiten zur Implementierung von 3D-Attention untersucht.

In diesem Beitrag wird die Wirksamkeit von Transformationsmodellen für die Imputation von Zeitreihendaten über verschiedene Dimensionen der Eingabe empirisch untersucht.


The conversion of scanned documents to digital forms is performed using an Optical Character Recognition (OCR) software.This work focuses on improving the quality of scanned documents in order to improve the OCR output.We create an end-to-end document enhancement pipeline which takes in a set of noisy documents and produces clean ones.Deep neural network based denoising auto-encoders are trained to improve the OCR quality.We train a blind model that works on different noise levels of scanned text documents.Results are shown for blurring and watermark noise removal from noisy scanned documents.

Wir haben ein REDNET (ResNet Encoder-Decoder) mit 8 Skip-Verbindungen entwickelt und getestet, um Störungen aus Dokumenten zu entfernen, einschließlich Unschärfe und Wasserzeichen, was zu einem leistungsstarken tiefen Netzwerk für die Bereinigung von Dokumentenbildern führt.


The existence of adversarial examples, or intentional mis-predictions constructed from small changes to correctly predicted examples, is one of the most significant challenges in neural network research today.Ironically, many new defenses are based on a simple observation - the adversarial inputs themselves are not robust and small perturbations to the attacking input often recover the desired prediction.While the intuition is somewhat clear, a detailed understanding of this phenomenon is missing from the research literature.This paper presents a comprehensive experimental analysis of when and why perturbation defenses work and potential mechanisms that could explain their effectiveness (or ineffectiveness) in different settings.

Wir identifizieren eine Familie von Schutztechniken und zeigen, dass sowohl eine deterministische verlustbehaftete Kompression als auch zufällige Störungen der Eingabe zu einer ähnlichen Verbesserung der Robustheit führen.

In diesem Beitrag wird erörtert, wie ein bestimmter gegnerischer Angriff destabilisiert werden kann, was gegnerische Bilder nicht robust macht und ob es für Angreifer möglich ist, ein universelles Modell von Störungen zu verwenden, um ihre gegnerischen Beispiele robust gegen solche Störungen zu machen.

Das Papier untersucht die Robustheit von gegnerischen Angriffen gegenüber Transformationen ihrer Eingaben.


There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD.In this work, we fill in the important, yet ambiguous concept of â€˜ease-of-useâ€™ by defining an optimizerâ€™s tunability:  How easy is it to find good hyperparameter configurations using automatic random hyperparameter search?We propose a practical and universal quantitative measure for optimizer tunability that can form the basis for a fair optimizer benchmark.  Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find  that Adam is the most tunable for the majority of problems, especially with a low budget for hyperparameter tuning.

Wir bieten eine Methode zum Benchmarking von Optimierern an, die den Prozess der Hyperparameterabstimmung berücksichtigt.

Einführung einer neuartigen Metrik zur Erfassung der Abstimmbarkeit eines Optimierers und ein umfassender empirischer Vergleich von Deep-Learning-Optimierern bei unterschiedlichem Umfang der Hyperparameter-Abstimmung.

In diesem Beitrag wird ein einfaches Maß für die Abstimmbarkeit eingeführt, das den Vergleich von Optimierern unter Ressourcenbeschränkungen ermöglicht. Es zeigt sich, dass die Abstimmung der Lernrate von Adam-Optimierern am einfachsten ist, um gut funktionierende Hyperparameter-Konfigurationen zu finden.


The phase problem in diffraction physics is one of the oldest inverse problems in all of science.The central difficulty that any approach to solving this inverse problem must overcome is that half of the information, namely the phase of the diffracted beam, is always missing.In the context of electron microscopy, the phase problem is generally non-linear and solutions provided by phase-retrieval techniques are known to be poor approximations to the physics of electrons interacting with matter.Here, we show that a semi-supervised learning approach can effectively solve the phase problem in electron microscopy/scattering.In particular, we introduce a new Deep Neural Network (DNN), Y-net, which simultaneously learns a reconstruction algorithm via supervised training in addition to learning a physics-based regularization via unsupervised training.We demonstrate that this constrained, semi-supervised approach is an order of magnitude more data-efficient and accurate than the same model trained in a purely supervised fashion.In addition, the architecture of the Y-net model provides for a straightforward evaluation of the consistency of the model's prediction during inference and is generally applicable to the phase problem in other settings.

Wir führen ein halbüberwachtes tiefes neuronales Netzwerk ein, um die Lösung des Phasenproblems in der Elektronenmikroskopie zu approximieren


Word embeddings extract semantic features of words from large datasets of text.Most embedding methods rely on a log-bilinear model to predict the occurrenceof a word in a context of other words.Here we propose word2net, a method thatreplaces their linear parametrization with neural networks.For each term in thevocabulary, word2net posits a neural network that takes the context as input andoutputs a probability of occurrence.Further, word2net can use the hierarchicalorganization of its word networks to incorporate additional meta-data, such assyntactic features, into the embedding model.For example, we show how to shareparameters across word networks to develop an embedding model that includespart-of-speech information.We study word2net with two datasets, a collectionof Wikipedia articles and a corpus of U.S. Senate speeches.Quantitatively, wefound that word2net outperforms popular embedding methods on predicting held-out words and that sharing parameters based on part of speech further boostsperformance.Qualitatively, word2net learns interpretable semantic representationsand, compared to vector-based methods, better incorporates syntactic information.

Word2net ist eine neuartige Methode zum Erlernen neuronaler Netzwerkrepräsentationen von Wörtern, die syntaktische Informationen nutzen kann, um bessere semantische Merkmale zu lernen.

Dieser Artikel erweitert SGNS mit einem architektonischen Wechsel von einem Bag-of-Words-Modell zu einem Feedforward-Modell und trägt eine neue Form der Regularisierung bei, indem es eine Teilmenge von Schichten zwischen verschiedenen assoziierten Netzwerken bindet.

Eine Methode zur Verwendung nichtlinearer Kombination von Kontextvektoren zum Erlernen der Vektordarstellung von Wörtern, wobei die Hauptidee darin besteht, jede Worteinbettung durch ein neuronales Netz zu ersetzen.


A key goal in neuroscience is to understand brain mechanisms of cognitive functions.An emerging approach is to study â€œbrain statesâ€ dynamics using functional magnetic resonance imaging (fMRI).So far in the literature, brain states have typically been studied using 30 seconds of fMRI data or more, and it is unclear to which extent brain states can be reliably identified from very short time series.In this project, we applied graph convolutional networks (GCN) to decode brain activity over short time windows in a task fMRI dataset, i.e. associate a given window of fMRI time series with the task used.Starting with a populational brain graph with nodes defined by a parcellation of cerebral cortex and the adjacent matrix extracted from functional connectome, GCN takes a short series of fMRI volumes as input, generates high-level domain-specific graph representations, and then predicts the corresponding cognitive state.We investigated the performance of this GCN "cognitive state annotation" in the Human Connectome Project (HCP) database, which features 21 different experimental conditions spanning seven major cognitive domains, and high temporal resolution task fMRI data.Using a 10-second window, the 21 cognitive states were identified with an excellent average test accuracy of 89% (chance level 4.8%).As the HCP task battery was designed to selectively activate a wide range of specialized functional networks, we anticipate the GCN annotation to be applicable as a base model for other transfer learning applications, for instance, adapting to new task domains.

Unter Verwendung eines 10-Sekunden-Fensters von fMRI-Signalen identifizierte unser GCN-Modell 21 verschiedene Aufgabenbedingungen aus dem HCP-Datensatz mit einer Testgenauigkeit von 89 %.


Modern deep neural networks (DNNs) require high memory consumption and large computational loads.  In order to deploy DNN algorithms efficiently on edge or mobile devices, a series of DNN compression algorithms have been explored, including the line of works on factorization methods.Factorization methods approximate the weight matrix of a DNN layer with multiplication of two or multiple low-rank matrices.However, it is hard to measure the ranks of DNN layers during the training process.Previous works mainly induce low-rank through implicit approximations or via costly singular value decomposition (SVD) process on every training step.The former approach usually induces a high accuracy loss while the latter prevents DNN factorization from efficiently reaching a high compression rate.In this work, we propose SVD training, which first applies SVD to decompose DNN's layers and then performs training on the full-rank decomposed weights.To improve the training quality and convergence, we add orthogonality regularization to the singular vectors, which ensure the valid form of SVD and avoid gradient vanishing/exploding.Low-rank is encouraged by applying sparsity-inducing regularizers on the singular values of each layer.Singular value pruning is applied at the end to reach a low-rank model.We empirically show that SVD training can significantly reduce the rank of DNN layers and achieve higher reduction on computation load under the same accuracy, comparing to not only previous factorization methods but also state-of-the-art filter pruning methods.

Effiziente Induktion von tiefen neuronalen Netzen mit niedrigem Rang durch SVD-Training mit spärlichen Singulärwerten und orthogonalen Singulärvektoren.

In diesem Artikel wird ein Ansatz zur Netzwerkkompression vorgestellt, bei dem die Gewichtsmatrix in jeder Schicht einen niedrigen Rang hat und die Gewichtsmatrizen explizit in eine SVD-ähnliche Faktorisierung zur Behandlung als neue Parameter faktorisiert werden.

Vorschlag, jede Schicht eines tiefen neuronalen Netzes vor dem Training mit einer Low-Rank-Matrixzerlegung zu parametrisieren, die Convolutions durch zwei aufeinanderfolgende Convolutions zu ersetzen und dann die zerlegte Methode zu trainieren.


The recent rise in popularity of few-shot learning algorithms has enabled models to quickly adapt to new tasks based on only a few training samples.Previous few-shot learning works have mainly focused on classification and reinforcement learning. In this paper, we propose a few-shot meta-learning system that focuses exclusively on regression tasks.Our model is based on the idea that the degree of freedom of the unknown function can be significantly reduced if it is represented as a linear combination of a set of appropriate basis functions.This enables a few labelled samples to approximate the function.We design a Feature Extractor network to encode basis functions for a task distribution, and a  Weights Generator to generate the weight vector for a novel task.We show that our model outperforms the current state of the art meta-learning methods in various regression tasks.

Wir schlagen ein Few-Shot Learning Modell vor, das speziell auf Regressionsaufgaben zugeschnitten ist.

In dieser Arbeit wird ein neues Shot-Learning-Verfahren für Regressionsprobleme mit kleinen Stichproben vorgeschlagen.

Eine Methode, die mit wenigen Beispielen ein Regressionsmodell erlernt und andere Methoden übertrifft.


Most classification and segmentation datasets assume a closed-world scenario in which predictions are expressed as distribution over a predetermined set of visual classes.However, such assumption implies unavoidable and often unnoticeable failures in presence of out-of-distribution (OOD) input.These failures are bound to happen in most real-life applications since current visual ontologies are far from being comprehensive.We propose to address this issue by discriminative detection of OOD pixels in input data.Different from recent approaches, we avoid to bring any decisions by only observing the training dataset of the primary model trained to solve the desired computer vision task.Instead, we train a dedicated OOD modelwhich discriminates the primary training set from a much larger "background" dataset which approximates the variety of the visual world.We perform our experiments on high resolution natural images in a dense prediction setup.We use several road driving datasets as our training distribution, while we approximate the background distribution with the ILSVRC dataset.We evaluate our approach on WildDash test, which is currently the only public test dataset with out-of-distribution images.The obtained results show that the proposed approach succeeds to identify out-of-distribution pixels while outperforming previous work by a wide margin.

Wir präsentieren einen neuen Ansatz zur Erkennung von Pixeln, die nicht in der Verteilung liegen, bei der semantischen Segmentierung.

Diese Arbeit befasst sich mit der Erkennung von Verteilungsfehlern, um den Segmentierungsprozess zu unterstützen, und schlägt einen Ansatz für das Training eines binären Klassifizierers vor, der Bildfelder aus einer bekannten Gruppe von Klassen von denen einer unbekannten unterscheidet.

Diese Arbeit zielt auf die Erkennung von Pixeln außerhalb der Verteilung für die semantische Segmentierung ab, und diese Arbeit nutzt Daten aus anderen Bereichen, um unbestimmte Klassen zu erkennen und die Unsicherheit besser zu modellieren.


Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices.Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy.The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead.To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs.However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large.The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results.It is difficult for even deep reinforcement learning (DRL) DDPG-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy.In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer.Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy.

Genaue, schnelle und automatisierte Quantisierung neuronaler Netze nach dem Kernelprinzip mit gemischter Präzision unter Verwendung von hierarchischem Deep Reinforcement Learning

Eine Methode zur Quantisierung von Gewichten und Aktivierungen neuronaler Netze, die tiefes Reinforcement Learning verwendet, um die Bitbreite für einzelne Kernel in einer Schicht auszuwählen, und die eine bessere Leistung bzw. Latenz als frühere Ansätze erzielt.

In diesem Artikel wird vorgeschlagen, automatisch nach Quantisierungsschemata für jeden Kernel im neuronalen Netz zu suchen, wobei hierarchisches RL zur Steuerung der Suche verwendet wird.


Recent visual analytics systems make use of multiple machine learning models to better fit the data as opposed to traditional single, pre-defined model systems.However, while multi-model visual analytic systems can be effective, their added complexity poses usability concerns, as users are required to interact with the parameters of multiple models.Further, the advent of various model algorithms and associated hyperparameters creates an exhaustive model space to sample models from.This poses complexity to navigate this model space to find the right model for the data and the task.In this paper, we present Gaggle, a multi-model visual analytic system that enables users to interactively navigate the model space.Further translating user interactions into inferences, Gaggle simplifies working with multiple models by automatically finding the best model from the high-dimensional model space to support various user tasks.Through a qualitative user study, we show how our approach helps users to find a best model for a classification and ranking task.The study results confirm that Gaggle is intuitive and easy to use, supporting interactive model space navigation and automated model selection without requiring any technical expertise from users.

Gaggle, ein interaktives visuelles Analysesystem, das den Benutzern hilft, sich interaktiv im Modellraum für Klassifizierungs- und Rankingaufgaben zu bewegen.

Ein neues visuelles Analysesystem, das es auch unerfahrenen Nutzern ermöglichen soll, interaktiv in einem Modellraum zu navigieren, indem es einen demonstrationsbasierten Ansatz verwendet.

Ein visuelles Analysesystem, das unerfahrenen Analysten hilft, sich im Modellraum zurechtzufinden, um Klassifizierungs- und Rangordnungsaufgaben durchzuführen.


Chinese text classification has received more and more attention today.However, the problem of Chinese text representation still hinders the improvement of Chinese text classification, especially the polyphone and the homophone in social media.To cope with it effectively, we propose a new structure, the Extractor, based on attention mechanisms and design novel attention networks named Extractor-attention network (EAN).Unlike most of previous works, EAN uses a combination of a word encoder and a Pinyin character encoder instead of a single encoder.It improves the capability of Chinese text representation.Moreover, compared with the hybrid encoder methods, EAN has more complex combination architecture and more reducing parameters structures.Thus, EAN can take advantage of a large amount of information that comes from multi-inputs and alleviates efficiency issues.The proposed model achieves the state of the art results on 5 large datasets for Chinese text classification.

Wir schlagen ein neuartiges Aufmerksamkeitsnetz mit dem Hybird-Encoder vor, um das Problem der Textdarstellung bei der Klassifizierung chinesischer Texte zu lösen, insbesondere die sprachlichen Phänomene der Aussprache wie Polyphone und Homophone.

Dieses Papier schlägt ein aufmerksamkeitsbasiertes Modell vor, das aus einem Wort-Encoder und einem Pinyin-Encoder für die Klassifizierung chinesischer Texte besteht, und erweitert die Architektur um den Pinyin-Zeichen-Encoder.

Vorschlag für ein Aufmerksamkeitsnetz, bei dem sowohl Wort als auch Pinyin für die chinesische Repräsentation berücksichtigt werden, mit verbesserten Ergebnissen, die in mehreren Datensätzen für die Textklassifizierung gezeigt wurden.


Recent advances in learning from demonstrations (LfD) with deep neural networks have enabled learning complex robot skills that involve high dimensional perception such as raw image inputs. LfD algorithms generally assume learning from single task demonstrations.In practice, however, it is more efficient for a teacher to demonstrate a multitude of tasks without careful task set up, labeling, and engineering.Unfortunately in such cases, traditional imitation learning techniques fail to represent the multi-modal nature of the data, and often result in sub-optimal behavior.In this paper we present an LfD approach for learning multiple modes of behavior from visual data.Our approach is based on a stochastic deep neural network (SNN), which represents the underlying intention in the demonstration as a stochastic activation in the network.We present an efficient algorithm for training SNNs, and for learning with vision inputs, we also propose an architecture that associates the intention with a stochastic attention module.We demonstrate our method on real robot visual object reaching tasks, and show thatit can reliably learn the multiple behavior modes in the demonstration data.Video results are available at https://vimeo.com/240212286/fd401241b9.

Multimodales Imitation Learning aus unstrukturierten Demonstrationen unter Verwendung stochastischer neuronaler Netzmodelle.

Ein neuer, auf Stichproben basierender Ansatz für die Inferenz in latenten Variablenmodellen, der auf multimodales Imitation Learning anwendbar ist und besser funktioniert als deterministische neuronale Netze und stochastische neuronale Netze für eine reale visuelle Robotikaufgabe.

Dieser Artikel zeigt, wie man mehrere Modalitäten durch Imitation Learning aus visuellen Daten mit Hilfe stochastischer neuronaler Netze erlernen kann, und eine Methode zum Lernen aus Demonstrationen, bei denen mehrere Modalitäten derselben Aufgabe gegeben sind.


The interpretability of neural networks has become crucial for their applications in real world with respect to the reliability and trustworthiness.Existing explanation generation methods usually provide important features by scoring their individual contributions to the model prediction and ignore the interactions between features, which eventually provide a bag-of-words representation as explanation.In natural language processing, this type of explanations is challenging for human user to understand the meaning of an explanation and draw the connection between explanation and model prediction, especially for long texts.In this work, we focus on detecting the interactions between features, and propose a novel approach to build a hierarchy of explanations based on feature interactions.The proposed method is evaluated with three neural classifiers, LSTM, CNN, and BERT, on two benchmark text classification datasets.The generated explanations are assessed by both automatic evaluation measurements and human evaluators.Experiments show the effectiveness of the proposed method in providing explanations that are both faithful to models, and understandable to humans.

Ein neuartiger Ansatz zur Konstruktion hierarchischer Erklärungen für die Textklassifizierung durch Erkennung von Merkmalsinteraktionen.

Eine neuartige Methode zur Bereitstellung von Erklärungen für Vorhersagen, die von Textklassifizierern getroffen wurden, die bei der Bewertung der Wichtigkeit auf Wortebene besser abschneidet, sowie eine neue Metrik, der Kohäsionsverlust, zur Bewertung der Wichtigkeit auf Bereichsebene.

Eine Interpretationsmethode auf der Grundlage von Merkmalsinteraktionen und Merkmalswichtigkeitspunkten im Vergleich zu unabhängigen Merkmalsbeiträgen.


Making deep convolutional neural networks more accurate typically comes at the cost of increased computational and memory resources.In this paper, we reduce this cost by exploiting the fact that the importance of features computed by convolutional layers is highly input-dependent, and propose feature boosting and suppression (FBS), a new method to predictively amplify salient convolutional channels and skip unimportant ones at run-time.FBS introduces small auxiliary connections to existing convolutional layers.In contrast to channel pruning methods which permanently remove channels, it preserves the full network structures and accelerates convolution by dynamically skipping unimportant input and output channels.FBS-augmented networks are trained with conventional stochastic gradient descent, making it readily available for many state-of-the-art CNNs.We compare FBS to a range of existing channel pruning and dynamic execution schemes and demonstrate large improvements on ImageNet classification.Experiments show that FBS can respectively provide 5Ã— and 2Ã— savings in compute on VGG-16 and ResNet-18, both with less than 0.6% top-5 accuracy loss.

Wir sorgen dafür, dass Convolutional Layers schneller laufen, indem wir Kanäle bei der Merkmalsberechnung dynamisch verstärken und unterdrücken.

Eine Methode zur Verstärkung und Unterdrückung von Merkmalen für dynamisches Channel Pruning, die die Bedeutung jedes Kanals vorhersagt und dann eine affine Funktion zur Verstärkung/Unterdrückung der Kanalbedeutung verwendet.

Vorschlag für eine Channel Pruning Methode zur dynamischen Auswahl von Kanälen während der Prüfung.


We propose a novel way of reducing the number of parameters in the storage-hungry fully connected layers of a neural network by using pre-defined sparsity, where the majority of connections are absent prior to starting training.Our results indicate that convolutional neural networks can operate without any loss of accuracy at less than 0.5% classification layer connection density, or less than 5% overall network connection density.We also investigate the effects of pre-defining the sparsity of networks with only fully connected layers.Based on our sparsifying technique, we introduce the `scatter' metric to characterize the quality of a particular connection pattern.As proof of concept, we show results on CIFAR, MNIST and a new dataset on classifying Morse code symbols, which highlights some interesting trends and limits of sparse connection patterns.

Neuronale Netze können so vordefiniert werden, dass sie ohne Leistungseinbußen eine spärliche Konnektivität aufweisen.

Diese Arbeit untersucht spärliche Verbindungsmuster in den oberen Schichten von Convolutional Bildklassifikations Netzwerken und stellt Heuristiken zur Verteilung von Verbindungen zwischen Fenstern/Gruppen und ein Maß namens Streuung zur Konstruktion von Konnektivitätsmasken vor.

Vorschlag zur Verringerung der Anzahl von Parametern, die von einem tiefen Netz gelernt werden, durch die Einrichtung von spärlichen Verbindungsgewichten in Klassifizierungsschichten und der Einführung eines Konzepts der "Streuung".


Deep neural networks are vulnerable to adversarial examples, which becomes one of the most important problems in the development of deep learning.While a lot of efforts have been made in recent years, it is of great significance to perform correct and complete evaluations of the adversarial attack and defense algorithms.In this paper, we establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness on image classification tasks.After briefly reviewing plenty of representative attack and defense methods, we perform large-scale experiments with two robustness curves as the fair-minded evaluation criteria to fully understand the performance of these methods.Based on the evaluation results, we draw several important findings and provide insights for future research.

Wir bieten ein umfassendes, strenges und kohärentes Benchmarking zur Bewertung der adversarial Robustheit von Deep-Learning-Modellen.

In diesem Artikel werden verschiedene Arten von Klassifizierungsmodellen unter verschiedenen Angriffsmethoden bewertet.

Eine groß angelegte empirische Studie, in der verschiedene Angriffs- und Verteidigungstechniken miteinander verglichen werden, sowie die Verwendung von Kurven für die Genauigkeit im Vergleich zum Störungsbudget und die Genauigkeit im Vergleich zur Angriffsstärke zur Bewertung von Angriffen und Verteidigungen.


We propose a modification to traditional Artificial Neural Networks (ANNs), which provides the ANNs with new aptitudes motivated by biological neurons.  Biological neurons work far beyond linearly summing up synaptic inputs and then transforming the integrated information.  A biological neuron change firing modes accordingly to peripheral factors (e.g., neuromodulators) as well as intrinsic ones.  Our modification connects a new type of ANN nodes, which mimic the function of biological neuromodulators and are termed modulators, to enable other traditional ANN nodes to adjust their activation sensitivities in run-time based on their input patterns.  In this manner, we enable the slope of the activation function to be context dependent.  This modification produces statistically significant improvements in comparison with traditional ANN nodes in the context of Convolutional Neural Networks and Long Short-Term Memory networks.

Wir schlagen eine Modifikation traditioneller Künstlicher Neuronaler Netze vor, die durch die Biologie der Neuronen motiviert ist und es ermöglicht, die Form der Aktivierungsfunktion kontextabhängig zu gestalten.

Eine Methode zur Skalierung der Aktivierungen einer Schicht von Neuronen in einem ANN in Abhängigkeit von den Eingaben in diese Schicht, die Verbesserungen gegenüber den Grundlinien aufweist.

Einführung einer architektonischen Änderung für Basisneuronen in einem neuronalen Netz und die Idee, die Ausgabe der Linearkombinationen der Neuronen mit einem Modulator zu multiplizieren, bevor sie in die Aktivierungsfunktion eingespeist wird.


In this work, we study how the large-scale pretrain-finetune framework changes the behavior of a neural language generator.We focus on the transformer encoder-decoder model for the open-domain dialogue response generation task.We find that after standard fine-tuning, the model forgets important language generation skills acquired during large-scale pre-training.We demonstrate the forgetting phenomenon through a detailed behavior analysis from the perspectives of context sensitivity and knowledge transfer.Adopting the concept of data mixing, we propose an intuitive fine-tuning strategy named "mix-review''.We find that mix-review effectively regularize the fine-tuning process, and the forgetting problem is largely alleviated.Finally, we discuss interesting behavior of the resulting dialogue model and its implications.

Wir identifizieren das Problem des Vergessens beim Fine-Tuning von vortrainierten NLG-Modellen und schlagen eine Mix-Review-Strategie vor, um dieses Problem zu lösen.

In dieser Arbeit wird das Problem des Vergessens im Rahmen des Pretraining Fine-Tunings aus der Perspektive der Kontextsensitivität und des Wissenstransfers analysiert und eine Fine-Tuningstrategie vorgeschlagen, die die Methode des Gewichtsabfalls übertrifft.

Untersuchung des Vergessensproblems im Pretrain-Finetuning Framework, insbesondere bei Aufgaben zur Erzeugung von Dialogantworten, und Vorschlag einer Mix-Review-Strategie, um das Vergessensproblem zu mildern.


Combining domain knowledge models with neural models has been challenging.  End-to-end trained neural models often perform better (lower Mean Square Error) than domain knowledge models or domain/neural combinations, and the combination is inefficient to train.  In this paper, we demonstrate that by composing domain models with machine learning models, by using extrapolative testing sets, and invoking decorrelation objective functions, we create models which can predict more complex systems.The models are interpretable, extrapolative, data-efficient, and capture predictable but complex non-stochastic behavior such as unmodeled degrees of freedom and systemic measurement noise.  We apply this improved modeling paradigm to several simulated systems and an actual physical system in the context of system identification.   Several ways of composing domain models with neural models are examined for time series, boosting, bagging, and auto-encoding on various systems of varying complexity and non-linearity.  Although this work is preliminary, we show that the ability to combine models is a very promising direction for neural modeling.

Verbesserte Modellierung komplexer Systeme unter Verwendung hybrider neuronaler/domänenbezogener Modellkompositionen, neuer Dekorrelationsverlustfunktionen und extrapolativer Testsätze.

In diesem Beitrag werden Experimente durchgeführt, um die extrapolativen Vorhersagen verschiedener hybrider Modelle, die physikalische Modelle, neuronale Netze und stochastische Modelle umfassen, zu vergleichen und die Herausforderung der nicht modellierten Dynamik als Engpass zu bewältigen.

In diesem Beitrag werden Ansätze zur Kombination von neuronalen Netzen mit Nicht-NN-Modellen vorgestellt, um das Verhalten komplexer physikalischer Systeme vorherzusagen.


Humans can learn task-agnostic priors from interactive experience and utilize the priors for novel tasks without any finetuning.In this paper, we propose Scoring-Aggregating-Planning (SAP), a framework that can learn task-agnostic semantics and dynamics priors from arbitrary quality interactions as well as the corresponding sparse rewards and then plan on unseen tasks in zero-shot condition.The framework finds a neural score function for local regional state and action pairs that can be aggregated to approximate the quality of a full trajectory; moreover, a dynamics model that is learned with self-supervision can be incorporated for planning.Many of previous works that leverage interactive data for policy learning either need massive on-policy environmental interactions or assume access to expert data while we can achieve a similar goal with pure off-policy imperfect data.Instantiating our framework results in a generalizable policy to unseen tasks.Experiments demonstrate that the proposed method can outperform baseline methods on a wide range of applications including gridworld, robotics tasks and video games.

Wir lernen dichte Werte und ein dynamisches Modell als Priors aus Explorationsdaten und verwenden sie, um eine gute Strategie für neue Aufgaben in Zero-Shot Bedingungen zu entwickeln.

In diesem Beitrag wird die Verallgemeinerung von Zero Shots auf neue Umgebungen diskutiert und ein Ansatz mit Ergebnissen zu Grid-World, Super Mario Bros und 3D Robotics vorgeschlagen.

Eine Methode, die darauf abzielt, aufgabenagnostische Prioritäten für die Zero-Shot-Generalisierung zu erlernen, mit der Idee, einen Modellierungsansatz zusätzlich zum modellbasierten RL-Framework einzusetzen.


Particle-based inference algorithm is a promising method to efficiently generate samples for an intractable target distribution by iteratively updating a set of particles.As a noticeable example, Stein variational gradient descent (SVGD) provides a deterministic and computationally efficient update, but it is known to underestimate the variance in high dimensions, the mechanism of which is poorly understood.In this work we explore a connection between SVGD and MMD-based inference algorithm via Stein's lemma.By comparing the two update rules, we identify the source of bias in SVGD as a combination of high variance and deterministic bias, and empirically demonstrate that the removal of either factors leads to accurate estimation of the variance.In addition, for learning high-dimensional Gaussian target, we analytically derive the converged variance for both algorithms, and confirm that only SVGD suffers from the "curse of dimensionality".

Analyse der zugrundeliegenden Mechanismen des Varianzkollapses von SVGD in hohen Dimensionen.


We describe an approach to understand the peculiar and counterintuitive generalization properties of deep neural networks.  The approach involves going beyond worst-case theoretical capacity control frameworks that have been popular in machine learning in recent years to revisit old ideas in the statistical mechanics of neural networks.  Within this approach, we present a prototypical Very Simple Deep Learning (VSDL) model, whose behavior is controlled by two control parameters, one describing an effective amount of data, or load, on the network (that decreases when noise is added to the input), and one with an effective temperature interpretation (that increases when algorithms are early stopped).  Using this model, we describe how a very simple application of ideas from the statistical mechanics theory of generalization provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc.

Neudenken der Verallgemeinerung erfordert ein Überdenken alter Ideen: Ansätze der statistischen Mechanik und komplexes Lernverhalten

Die Autoren schlagen vor, dass Ideen aus der statistischen Mechanik zum Verständnis der Generalisierungseigenschaften von tiefen neuronalen Netzen beitragen, und geben einen Ansatz vor, der starke qualitative Beschreibungen empirischer Ergebnisse in Bezug auf tiefe neuronale Netze und Lernalgorithmen liefert.

Eine Reihe von Ideen zum theoretischen Verständnis der Verallgemeinerungseigenschaften von mehrschichtigen neuronalen Netzen und eine qualitative Analogie zwischen dem Verhalten beim Deep Learning und den Ergebnissen der quantitativen statistischen Physikanalyse von ein- und zweischichtigen neuronalen Netzen.


Computations for the softmax function in neural network models are expensive when the number of output classes is large.This can become a significant issue in both training and inference for such models.In this paper, we present Doubly Sparse Softmax (DS-Softmax), Sparse Mixture of Sparse of Sparse Experts, to improve the efficiency for softmax inference.During training, our method learns a two-level class hierarchy by dividing entire output class space into several partially overlapping experts.Each expert is responsible for a learned subset of the output class space and each output class only belongs to a small number of those experts.During inference, our method quickly locates the most probable expert to compute small-scale softmax.Our method is learning-based and requires no knowledge of the output class partition space a priori.We empirically evaluate our method on several real-world tasks and demonstrate that we can achieve significant computation reductions without loss of performance.

Wir präsentieren doppelt spärlichen Softmax, die spärliche Mischung aus spärlichen Experten, um die Effizienz der Softmax-Inferenz durch Ausnutzung der zweistufigen überlappenden Hierarchie zu verbessern.

In diesem Beitrag wird eine schnelle Annäherung an die Softmax-Berechnung vorgeschlagen, wenn die Anzahl der Klassen sehr groß ist.

In diesem Artikel wird eine spärliche Mischung aus spärlichen Experten vorgeschlagen, die eine zweistufige Klassenhierarchie für eine effiziente Softmax-Inferenz erlernt.


Supervised machine learning models for high-value computer vision applications such as medical image classification often require large datasets labeled by domain experts, which are slow to collect, expensive to maintain, and static with respect to changes in the data distribution.In this context, we assess the utility of observational supervision, where we take advantage of passively-collected signals such as eye tracking or â€œgazeâ€ data, to reduce the amount of hand-labeled data needed for model training.Specifically, we leverage gaze information to directly supervise a visual attention layer by penalizing disagreement between the spatial regions the human labeler looked at the longest and those that most heavily influence model output.We present evidence that constraining the model in this way can reduce the number of labeled examples required to achieve a given performance level by as much as 50%, and that gaze information is most helpful on more difficult tasks.

Wir untersuchen die Verwendung von passiv gesammelten Eye-Tracking-Daten, um die Menge an markierten Daten, die während des Trainings benötigt werden, zu reduzieren.

Eine Methode zur Nutzung von Blickinformationen, um die Komplexität der Stichprobe eines Modells und den erforderlichen Beschriftungsaufwand zu reduzieren, um eine Zielleistung zu erreichen, mit verbesserten Ergebnissen bei mittelgroßen Stichproben und schwierigeren Aufgaben.

Eine Methode zur Einbeziehung von Blicksignalen in Standard-CNNs für die Bildklassifizierung durch Hinzufügen eines Verlustfunktionsterms, der auf der Differenz zwischen der Klassenaktivierungskarte des Modells und der aus Blickverfolgungsinformationen konstruierten Zuordnungen basiert.


We study the robustness to symmetric label noise of GNNs training procedures.By combining the nonlinear neural message-passing models (e.g. Graph Isomorphism Networks, GraphSAGE, etc.) with loss correction methods, we present a noise-tolerant approach for the graph classification task.Our experiments show that test accuracy can be improved under the artificial symmetric noisy setting.

Wir wenden eine Verlustkorrektur auf graphische neuronale Netze an, um ein Modell zu trainieren, das robuster gegen Rauschen ist.

Diese Arbeit führt eine Verlustkorrektur für Graph Neural Networks ein, um mit symmetrischem Graph Label Störungen umzugehen, und konzentriert sich auf eine Graph-Klassifikationsaufgabe.

In diesem Beitrag wird die Verwendung eines Rauschkorrekturverlustes im Zusammenhang mit neuronalen Graphennetzen vorgeschlagen, um mit verrauschten Etiketten umzugehen.