Deep Neural Networks (DNNs) are increasingly deployed in cloud servers and autonomous agents due to their superior performance.The deployed DNN is either leveraged in a white-box setting (model internals are publicly known) or a black-box setting (only model outputs are known) depending on the application.A practical concern in the rush to adopt DNNs is protecting the models against Intellectual Property (IP) infringement.We propose BlackMarks, the first end-to-end multi-bit watermarking framework that is applicable in the black-box scenario.BlackMarks takes the pre-trained unmarked model and the ownerâ€™s binary signature as inputs.The output is the corresponding marked model with specific keys that can be later used to trigger the embedded watermark.To do so, BlackMarks first designs a model-dependent encoding scheme that maps all possible classes in the task to bit â€˜0â€™ and bit â€˜1â€™.Given the ownerâ€™s watermark signature (a binary string), a set of key image and label pairs is designed using targeted adversarial attacks.The watermark (WM) is then encoded in the distribution of output activations of the DNN by fine-tuning the model with a WM-specific regularized loss.To extract the WM, BlackMarks queries the model with the WM key images and decodes the ownerâ€™s signature from the corresponding predictions using the designed encoding scheme.We perform a comprehensive evaluation of BlackMarksâ€™ performance on MNIST, CIFAR-10, ImageNet datasets and corroborate its effectiveness and robustness.BlackMarks preserves the functionality of the original DNN and incurs negligible WM embedding overhead as low as 2.054%.

Vorschlag des ersten Wasserzeichen Frameworks für die Einbettung und Extraktion von Multibit-Signaturen unter Verwendung der DNN-Ausgänge.

Es wird eine Methode für das Multi-Bit-Wasserzeichen von neuronalen Netzen in einer Black-Box Umgebung vorgeschlagen und gezeigt, dass die Vorhersagen bestehender Modelle eine Multi-Bit-Zeichenkette tragen können, die später zur Überprüfung des Eigentums verwendet werden kann.

Die Arbeit schlägt einen Ansatz für Modell-Wasserzeichen vor, bei dem das Wasserzeichen eine in das Modell eingebettete Bitfolge ist, die Teil eines Feinabstimmungsprozesses ist.


Adversarial training provides a principled approach for training robust neural networks.From an optimization perspective, the adversarial training is essentially solving a minmax robust optimization problem.The outer minimization is trying to learn a robust classifier, while the inner maximization is trying to generate adversarial samples.Unfortunately, such a minmax problem is very difficult to solve due to the lack of convex-concave structure.This work proposes a new adversarial training method based on a general learning-to-learn framework.Specifically, instead of applying the existing hand-design algorithms for the inner problem, we learn an optimizer, which is parametrized as a convolutional neural network.At the same time, a robust classifier is learned to defense the adversarial attack generated by the learned optimizer.From the perspective of generative learning, our proposed method can be viewed as learning a deep generative model for generating adversarial samples, which is adaptive to the robust classification.Our experiments demonstrate that our proposed method significantly outperforms existing adversarial training methods on CIFAR-10 and CIFAR-100 datasets.

Sie wissen nicht, wie man optimiert? Dann lernen Sie einfach zu optimieren!

In diesem Artikel wird ein Weg vorgeschlagen, Bildklassifizierungsmodelle so zu trainieren, dass sie resistent gegen L-infinity perturbation Attacken sind.

In diesem Beitrag wird vorgeschlagen, einen Angreifer mit Hilfe des Learning-to-Learn-Konzepts zu erlernen.


In this work we introduce a new framework for performing temporal predictionsin the presence of uncertainty.It is based on a simple idea of disentangling com-ponents of the future state which are predictable from those which are inherentlyunpredictable, and encoding the unpredictable components into a low-dimensionallatent variable which is fed into the forward model.Our method uses a simple su-pervised training objective which is fast and easy to train.We evaluate it in thecontext of video prediction on multiple datasets and show that it is able to consi-tently generate diverse predictions without the need for alternating minimizationover a latent space or adversarial training.

Eine einfache und leicht zu trainierende Methode für multimodale Vorhersagen in Zeitreihen.

In diesem Beitrag wird ein Modell zur Vorhersage von Zeitserien vorgestellt, das eine deterministische Zuordnung erlernt und ein weiteres Netz trainiert, um künftige Bilder anhand der Eingabe und des residual Fehlers des ersten Netzes vorherzusagen.

In dem Beitrag wird ein Modell für die Vorhersage unter Unsicherheit vorgeschlagen, bei dem zwischen deterministischer Komponentenvorhersage und unsicherer Komponentenvorhersage unterschieden wird.


Conducting reinforcement-learning experiments can be a complex and timely process.A full experimental pipeline will typically consist of a simulation of an environment, an implementation of one or many learning algorithms, a variety of additional components designed to facilitate the agent-environment interplay, and any requisite analysis, plotting, and logging thereof.In light of this complexity, this paper introduces simple rl, a new open source library for carrying out reinforcement learning experiments in Python 2 and 3 with a focus on simplicity.The goal of simple_rl is to support seamless, reproducible methods for running reinforcement learning experiments.This paper gives an overview  of the core design philosophy of the package, how it differs from existing libraries, and showcases its central features.

Dieser Artikel stellt simple_rl vor, eine neue Open-Source-Bibliothek zur Durchführung von Reinforcement Learning Experimenten in Python 2 und 3 mit dem Schwerpunkt auf Einfachheit.


Wasserstein GAN(WGAN) is a model that minimizes the Wasserstein distance between a data distribution and sample distribution.Recent studies have proposed stabilizing the training process for the WGAN and implementing the Lipschitz constraint.In this study, we prove the local stability of optimizing the simple gradient penalty $\mu$-WGAN(SGP $\mu$-WGAN) under suitable assumptions regarding the equilibrium and penalty measure $\mu$.The measure valued differentiation concept is employed to deal with the derivative of the penalty terms, which is helpful for handling abstract singular measures with lower dimensional support.Based on this analysis, we claim that penalizing the data manifold or sample manifold is the key to regularizing the original WGAN with a gradient penalty.Experimental results obtained with unintuitive penalty measures that satisfy our assumptions are also provided to support our theoretical results.

Diese Arbeit beschäftigt sich mit der Stabilität einer einfachen Gradientenbestrafung,  $\mu$-WGAN-Optimierung genannt, durch Einführung eines Konzepts der maßstäblichen Differenzierung.

WGAN mit einem quadratischen, nullpunktzentrierten Gradientenbestrafungs Term für ein allgemeines Maß wird untersucht.

Charakterisiert die Konvergenz des gradientenbestraften Wasserstein-GAN.


We present Random Partition Relaxation (RPR), a method for strong quantization of the parameters of convolutional neural networks to binary (+1/-1) and ternary (+1/0/-1) values.Starting from a pretrained model, we first quantize the weights and then relax random partitions of them to their continuous values for retraining before quantizing them again and switching to another weight partition for further adaptation.  We empirically evaluate the performance of RPR with ResNet-18, ResNet-50 and GoogLeNet on the ImageNet classification task for binary and ternary weight networks.We show accuracies beyond the state-of-the-art for binary- and ternary-weight GoogLeNet and competitive performance for ResNet-18 and ResNet-50 using a SGD-based training method that can easily be integrated into existing frameworks.

Modernste Trainingsmethode für binäre und ternäre Gewichtsnetze auf der Grundlage der alternierenden Optimierung von zufällig entspannten Gewichtspartitionen.

In der Arbeit wird ein neues Trainingsschema zur Optimierung eines ternären neuronalen Netzes vorgeschlagen.

Die Autoren schlagen RPR vor, eine Methode zur zufälligen Partitionierung und Quantisierung von Gewichten und zum Trainieren der verbleibenden Parameter, gefolgt von Entspannung in abwechselnden Zyklen, um quantisierte Modelle zu trainieren.


Learning long-term dependencies is a key long-standing challenge of recurrent neural networks (RNNs).Hierarchical recurrent neural networks (HRNNs) have been considered a promising approach as long-term dependencies are resolved through shortcuts up and down the hierarchy.Yet, the memory requirements of Truncated Backpropagation Through Time (TBPTT) still prevent training them on very long sequences.In this paper, we empirically show that in (deep) HRNNs, propagating gradients back from higher to lower levels can be replaced by locally computable losses, without harming the learning capability of the network, over a wide range of tasks.This decoupling by local losses reduces the memory requirements of training by a factor exponential in the depth of the hierarchy in comparison to standard TBPTT.

Wir ersetzen einige Gradientenpfade in hierarchischen RNNs durch einen Hilfsverlust. Wir zeigen, dass dies die Speicherkosten reduzieren kann, während die Leistung erhalten bleibt.

In dem Artikel wird eine hierarchische RNN-Architektur vorgestellt, die speichereffizienter trainiert werden kann.

Die vorgeschlagene Arbeit schlägt vor, die verschiedenen Schichten der Hierarchie in RNN mit Hilfe von Hilfsverlusten zu entkoppeln.


In a typical deep learning approach to a computer vision task, Convolutional Neural Networks (CNNs) are used to extract features at varying levels of abstraction from an image and compress a high dimensional input into a lower dimensional decision space through a series of transformations.In this paper, we investigate how a class of input images is eventually compressed over the course of these transformations.In particular, we use singular value decomposition to analyze the relevant variations in feature space.These variations are formalized as the effective dimension of the embedding.We consider how the effective dimension varies across layers within class.We show that across datasets and architectures, the effective dimension of a class increases before decreasing further into the network, suggesting some sort of initial whitening transformation.Further, the decrease rate of the effective dimension deeper in the network corresponds with training performance of the model.

Neuronale Netze, die eine gute Klassifizierung vornehmen, projizieren Punkte in sphärische Formen, bevor sie in geringere Dimensionen komprimiert werden.


Deep learning methods have achieved high performance in sound recognition tasks.Deciding how to feed the training data is important for further performance improvement.We propose a novel learning method for deep sound recognition: Between-Class learning (BC learning).Our strategy is to learn a discriminative feature space by recognizing the between-class sounds as between-class sounds.We generate between-class sounds by mixing two sounds belonging to different classes with a random ratio.We then input the mixed sound to the model and train the model to output the mixing ratio.The advantages of BC learning are not limited only to the increase in variation of the training data; BC learning leads to an enlargement of Fisherâ€™s criterion in the feature space and a regularization of the positional relationship among the feature distributions of the classes.The experimental results show that BC learning improves the performance on various sound recognition networks, datasets, and data augmentation schemes, in which BC learning proves to be always beneficial.Furthermore, we construct a new deep sound recognition network (EnvNet-v2) and train it with BC learning.As a result, we achieved a performance surpasses the human level.

Wir schlagen eine neuartige Lernmethode für tiefe Tonerkennung vor, das sogenannte BC-Lernen.

Die Autoren definierten eine neue Lernaufgabe, bei der ein DNN das Mischungsverhältnis zwischen Geräuschen aus zwei verschiedenen Klassen vorhersagen muss, um die Unterscheidungskraft des schließlich gelernten Netzwerks zu erhöhen.

Es wird eine Methode zur Verbesserung der Leistung einer generischen Lernmethode durch die Erzeugung von Trainingsmustern "zwischen den Klassen" vorgeschlagen und die grundlegende Intuition und Notwendigkeit der vorgeschlagenen Technik vorgestellt.


Spatiotemporal forecasting has become an increasingly important prediction task in machine learning and statistics due to its vast applications, such as climate modeling, traffic prediction, video caching predictions, and so on.While numerous studies have been conducted, most existing works assume that the data from different sources or across different locations are equally reliable.Due to cost, accessibility, or other factors, it is inevitable that the data quality could vary, which introduces significant biases into the model and leads to unreliable prediction results.The problem could be exacerbated in black-box prediction models, such as deep neural networks.In this paper, we propose a novel solution that can automatically infer data quality levels of different sources through local variations of spatiotemporal signals without explicit labels.Furthermore, we integrate the estimate of data quality level with graph convolutional networks to exploit their efficient structures.We evaluate our proposed method on forecasting temperatures in Los Angeles.

Wir schlagen eine Methode vor, die das zeitlich variierende Qualitätsniveau der Daten für räumlich-zeitliche Prognosen ohne explizit zugewiesene Labels ableitet.

Einführung einer neuen Definition von Datenqualität, die sich auf den Begriff der lokalen Variation stützt, der in (Zhou und Scholkopf) definiert wurde, und ihn auf mehrere heterogene Datenquellen ausweitet.

In dieser Arbeit wurde eine neue Methode zur Bewertung der Qualität verschiedener Datenquellen mit dem zeitvariablen Graphenmodell vorgeschlagen, wobei das Qualitätsniveau als Regularisierungsterm in der Zielfunktion verwendet wird


Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higher-level shape structure such as the repetition and reflective symmetry of object parts.In contrast, recent advances in 3D shape sensing focus more on low-level geometry but less on these higher-level relationships.In this paper, we propose 3D shape programs, integrating bottom-up recognition systems with top-down, symbolic program structure to capture both low-level geometry and high-level structural priors for 3D shapes.Because there are no annotations of shape programs for real shapes, we develop neural modules that not only learn to infer 3D shape programs from raw, unannotated shapes, but also to execute these programs for shape reconstruction.After initial bootstrapping, our end-to-end differentiable model learns 3D shape programs by reconstructing shapes in a self-supervised manner.Experiments demonstrate that our model accurately infers and executes 3D shape programs for highly complex shapes from various categories.It can also be integrated with an image-to-shape module to infer 3D shape programs directly from an RGB image, leading to 3D shape reconstructions that are both more accurate and more physically plausible.

Wir schlagen 3D-Form Programme vor, eine strukturierte, kompositorische Formdarstellung. Unser Modell lernt, Form Programme abzuleiten und auszuführen, um 3D Formen zu erklären.

Ein Ansatz zur Ableitung von Formprogrammen aus 3D Modellen. Die Architektur besteht aus einem rekurrenten Netzwerk, das eine 3D Form kodiert und Anweisungen ausgibt, und einem zweiten Modul, das das Programm in 3D rendert.

In diesem Beitrag wird eine semantische Beschreibung auf hoher Ebene für 3D Formen vorgestellt, die durch das ShapeProgram gegeben ist.


Deep Reinforcement Learning (Deep RL) has been receiving increasingly more attention  thanks to its encouraging performance on a variety of control tasks.Yet, conventional regularization techniques in training neural networks (e.g., $L_2$ regularization, dropout) have been largely ignored in RL methods, possibly because agents are typically trained and evaluated in the same environment.In this work, we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks.Interestingly, we find conventional regularization techniques on the policy networks can often bring large improvement on the task performance, and the improvement is typically more significant when the task is more difficult.We also compare with the widely used entropy regularization and find $L_2$ regularization is generally better.Our findings are further confirmed to be robust against the choice of training hyperparameters.We also study the effects of regularizing different components and find that only regularizing the policy network is typically enough.We hope our study provides guidance for future practices in regularizing policy optimization algorithms.

Wir zeigen, dass konventionelle Regularisierungsmethoden (z.B. $L_2$, Dropout), die in RL Methoden weitgehend ignoriert wurden, bei der Optimierung von Richtlinien sehr effektiv sein können.

Die Autoren untersuchen eine Reihe bestehender Methoden zur direkten Optimierung von Strategien im Bereich des Reinforcement Learnings und bieten eine detaillierte Untersuchung der Auswirkungen von Vorschriften auf die Leistung und das Verhalten von Agenten, die diesen Methoden folgen.

Diese Arbeit bietet eine Studie über die Auswirkung der Regularisierung auf die Leistung in Trainingsumgebungen in Regel-Optimierungsmethoden in mehreren kontinuierlichen Steuerungsaufgaben.


We introduce FigureQA, a visual reasoning corpus of over one million question-answer pairs grounded in over 100,000 images.The images are synthetic, scientific-style figures from five classes: line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts.We formulate our reasoning task by generating questions from 15 templates; questions concern various relationships between plot elements and examine characteristics like the maximum, the minimum, area-under-the-curve, smoothness, and intersection.To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure.To facilitate the training of machine learning systems, the corpus also includes side data that can be used to formulate auxiliary objectives.In particular, we provide the numerical data used to generate each figure as well as bounding-box annotations for all plot elements.We study the proposed visual reasoning task by training several models, including the recently proposed Relation Network as strong baseline.Preliminary results indicate that the task poses a significant machine learning challenge.We envision FigureQA as a first step towards developing models that can intuitively recognize patterns from visual representations of data.

Wir präsentieren einen Frage-Antwort-Datensatz, FigureQA, als einen ersten Schritt zur Entwicklung von Modellen, die intuitiv Muster aus visuellen Darstellungen von Daten erkennen können.

In diesem Beitrag wird ein Datensatz zur Beantwortung von Beispiel Fragen zu Abbildungen vorgestellt, der Schlussfolgerungen zu Abbildungselementen enthält.

In diesem Beitrag wird ein neuer Datensatz für visuelles Reasoning namens Figure-QA vorgestellt, der aus 140.000 Abbildungen und 1,55 Mio. QA-Paaren besteht und bei der Entwicklung von Modellen helfen kann, die nützliche Informationen aus visuellen Datendarstellungen extrahieren können.


In this paper, I discuss some varieties of explanation that can arisein intelligent agents.I distinguish between process accounts, whichaddress the detailed decisions made during heuristic search, andpreference accounts, which clarify the ordering of alternativesindependent of how they were generated.I also hypothesize which types of users will appreciate which types of explanation.In addition, I discuss three facets of multi-step decision making-- conceptual inference, plan generation, and plan execution --in which explanations can arise.I also consider alternative waysto present questions to agents and for them provide their answers.

In diesem Positionspapier werden verschiedene Arten von Selbsterklärungen analysiert, die in Planungs- und verwandten Systemen auftreten können.

Erörtert verschiedene Aspekte von Erklärungen, insbesondere im Zusammenhang mit sequenzieller Entscheidungsfindung.


Generative deep learning has sparked a new wave of Super-Resolution (SR) algorithms that enhance single images with impressive aesthetic results, albeit with imaginary details.Multi-frame Super-Resolution (MFSR) offers a more grounded approach to the ill-posed problem, by conditioning on multiple low-resolution views.This is important for satellite monitoring of human impact on the planet -- from deforestation, to human rights violations -- that depend on reliable imagery.To this end, we present HighRes-net, the first deep learning approach to MFSR that learns its sub-tasks in an end-to-end fashion:(i) co-registration,(ii) fusion,(iii) up-sampling, and(iv) registration-at-the-loss.Co-registration of low-res views is learned implicitly through a reference-frame channel, with no explicit registration mechanism.We learn a global fusion operator that is applied recursively on an arbitrary number of low-res pairs.We introduce a registered loss, by learning to align the SR output to a ground-truth through ShiftNet.We show that by learning deep representations of multiple views, we can super-resolve low-resolution signals and enhance Earth observation data at scale.Our approach recently topped the European Space Agency's MFSR competition on real-world satellite imagery.

Der erste Deep-Learning-Ansatz für MFSR, der Registrierung, Fusion und Up-Sampling durchgängig löst.

In diesem Beitrag wird ein durchgängiger Algorithmus für die Superauflösung mehrerer Bilder vorgeschlagen, der auf paarweisen Co-Registrierungen und Fusionsblöcken (Convolutional Residual Blocks) beruht, die in einem Encoder-Decoder Netzwerk "HighRes-Net" eingebettet sind, das das Superauflösungsbild schätzt.

Diese Arbeit schlägt einen Rahmen vor, der eine rekursive Fusion mit Co-Registrierungs Verlusten beinhaltet, um das Problem zu lösen, dass die Ergebnisse der Superauflösung und die hochauflösenden Labels nicht pixelgenau ausgerichtet sind.


Large mini-batch parallel SGD is commonly used for distributed training of deep networks.Approaches that use tightly-coupled exact distributed averaging based on AllReduce are sensitive to slow nodes and high-latency communication.In this work we show the applicability of Stochastic Gradient Push (SGP) for distributed training.SGP uses a gossip algorithm called PushSum for approximate distributed averaging, allowing for much more loosely coupled communications which can be beneficial in high-latency or high-variability scenarios.The tradeoff is that approximate distributed averaging injects additional noise in the gradient which can affect the train and test accuracies.We prove that SGP converges to a stationary point of smooth, non-convex objective functions.Furthermore, we validate empirically the potential of SGP.For example, using 32 nodes with 8 GPUs per node to train ResNet-50 on ImageNet, where nodes communicate over 10Gbps Ethernet, SGP completes 90 epochs in around 1.5 hours while AllReduce SGD takes over 5 hours, and the top-1 validation accuracy of SGP remains within 1.2% of that obtained using AllReduce SGD.

Verwenden von Gossip-basierten approximativen verteilten Durchschnittsberechnungen für verteiltes Training über Netze mit hoher Latenz, anstelle von exakten verteilten Durchschnittsberechnungen wie AllReduce.

Die Autoren schlagen vor, Gossip-Algorithmen als allgemeine Methode zur Berechnung des ungefähren Durchschnitts über eine Gruppe von Arbeitnehmern zu verwenden.

Die Arbeit beweist die Konvergenz der SGP für nicht-konvexe glatte Funktionen und zeigt, dass die SGP eine signifikante Beschleunigung in der Low-Latency-Umgebung erreichen kann, ohne zu viel Vorhersageleistung zu opfern.


In this paper, we extend the persona-based sequence-to-sequence (Seq2Seq) neural network conversation model to a multi-turn dialogue scenario by modifying the state-of-the-art hredGAN architecture to simultaneously capture utterance attributes such as speaker identity, dialogue topic, speaker sentiments and so on.The proposed system, phredGAN has a persona-based HRED generator (PHRED) and a conditional discriminator.We also explore two approaches to accomplish the conditional discriminator: (1) $phredGAN_a$, a system that passes the attribute representation as an additional input into a traditional adversarial discriminator, and (2) $phredGAN_d$, a dual discriminator system which in addition to the adversarial discriminator, collaboratively predicts the attribute(s) that generated the input utterance.To demonstrate the superior performance of phredGAN over the persona SeqSeq model, we experiment with two conversational datasets, the Ubuntu Dialogue Corpus (UDC) and TV series transcripts from the Big Bang Theory and Friends.Performance comparison is made with respect to a variety of quantitative measures as well as crowd-sourced human evaluation.We also explore the trade-offs from using either variant of $phredGAN$ on datasets with many but weak attribute modalities (such as with Big Bang Theory and Friends) and ones with few but strong attribute modalities (customer-agent interactions in Ubuntu dataset).

In diesem Beitrag wird ein adversarial Learning Framework für neuronale Konversationsmodelle mit Personas entwickelt.

Diese Arbeit schlägt eine Erweiterung von hredGAN vor, um gleichzeitig eine Reihe von Attributeinbettungen zu lernen, die die Persona jedes Sprechers repräsentieren und Persona-basierte Antworten generieren kann.


We introduce bio-inspired artificial neural networks consisting of neurons that are additionally characterized by spatial positions.To simulate properties of biological systems we add the costs penalizing long connections and the proximity of neurons in a two-dimensional space.Our experiments show that in the case where the network performs two different tasks, the neurons naturally split into clusters, where each cluster is responsible for processing a different task.This behavior not only corresponds to the biological systems, but also allows for further insight into interpretability or continual learning.

Biologisch inspirierte künstliche neuronale Netze, die aus in einem zweidimensionalen Raum angeordneten Neuronen bestehen, sind in der Lage, unabhängige Gruppen für die Ausführung verschiedener Aufgaben zu bilden.


The transformer has become a central model for many NLP tasks from translation to language modeling to representation learning.Its success demonstrates the effectiveness of stacked attention as a replacement for recurrence for many tasks.In theory attention also offers more insights into the modelâ€™s internal decisions; however, in practice when stacked it quickly becomes nearly as fully-connected as recurrent models.In this work, we propose an alternative transformer architecture, discrete transformer, with the goal of better separating out internal model decisions.The model uses hard attention to ensure that each step only depends on a fixed context.Additionally, the model uses a separate â€œsyntacticâ€ controller to separate out network structure from decision making.Finally we show that this approach can be further sparsified with direct regularization.Empirically, this approach is able to maintain the same level of performance on several datasets, while discretizing reasoning decisions over the data.

Diskreter Transformer, der mit harter Aufmerksamkeit sicherstellt, dass jeder Schritt nur von einem festen Kontext abhängt.

In diesem Beitrag werden Modifikationen an der Standard Transformer Architektur vorgestellt, mit dem Ziel, die Interpretierbarkeit zu verbessern und gleichzeitig die Leistung bei NLP-Aufgaben zu erhalten.

In diesem Artikel werden drei diskrete Transformer vorgeschlagen: ein diskretes und stochastisches Gumbel-Softmax-basiertes Aufmerksamkeitsmodul, ein syntaktischer und semantischer Two-Stream Transformer und eine Sparsity-Regularisierung.


Deep predictive coding networks are neuroscience-inspired unsupervised learning models that learn to predict future sensory states.We build upon the PredNet implementation by Lotter, Kreiman, and Cox (2016) to investigate if predictive coding representations are useful to predict brain activity in the visual cortex.We use representational similarity analysis (RSA) to compare PredNet representations to functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data from the Algonauts Project (Cichy et al., 2019).In contrast to previous findings in the literature (Khaligh-Razavi & Kriegeskorte, 2014), we report empirical data suggesting that unsupervised models trained to predict frames of videos without further fine-tuning may outperform supervised image classification baselines in terms of correlation to spatial (fMRI) and temporal (MEG) data.

Wir zeigen empirische Belege dafür, dass Vorhersehende Coding Modelle Repräsentationen liefern, die stärker mit Gehirndaten korrelieren als überwachte Bilderkennungsmodelle.


The incorporation of prior knowledge into learning is essential in achieving good performance based on small noisy samples.Such knowledge is often incorporated through the availability of related data arising from domains and tasks similar to the one of current interest.Ideally one would like to allow both the data for the current task and for previous related tasks to self-organize the learning system in such a way that commonalities and differences between the tasks are learned in a data-driven fashion.We develop a framework for learning multiple tasks simultaneously, based on sharing features that are common to all tasks, achieved through the use of a modular deep feedforward neural network consisting of shared branches, dealing with the common features of all tasks, and private branches, learning the specific unique aspects of each task.Once an appropriate weight sharing architecture has been established, learning takes place through standard algorithms for feedforward networks, e.g., stochastic gradient descent and its variations.The method deals with meta-learning (such as domain adaptation, transfer and multi-task learning) in a unified fashion, and can easily deal with data arising from different types of sources.Numerical experiments demonstrate the effectiveness of learning in domain adaptation and transfer learning setups, and provide evidence for the flexible and task-oriented representations arising in the network.

Ein allgemeines Framework für die Handhabung von Transfer- und Multi-Task Lernen unter Verwendung von Paaren von Autoencodern mit aufgabenspezifischen und gemeinsamen Gewichten.

Vorschlagen eines generischen Frameworks für Ende-zu-Ende Transfer Lernen / Domänenanpassung mit tiefen neuronalen Netzen.

Diese Arbeit schlägt ein Modell vor, das es Architekturen von tiefen neuronalen Netzen ermöglicht, Parameter über verschiedene Datensätze hinweg gemeinsam zu nutzen, und wendet es auf das Transferlernen an.

Der Artikel konzentriert sich auf das Erlernen gemeinsamer Merkmale aus Daten aus verschiedenen Bereichen und endet mit einer allgemeinen Architektur für Multi-Task-, Semi-Supervised- und Transfer-Lernen.


Deep neural networks and decision trees operate on largely separate paradigms; typically, the former performs representation learning with pre-specified architectures, while the latter is characterised by learning hierarchies over pre-specified features with data-driven architectures.We unite the two via adaptive neural trees (ANTs), a model that incorporates representation learning into edges, routing functions and leaf nodes of a decision tree, along with a backpropagation-based training algorithm that adaptively grows the architecture from primitive modules (e.g., convolutional layers).ANTs allow increased interpretability via hierarchical clustering, e.g., learning meaningful class associations, such as separating natural vs. man-made objects.We demonstrate this on classification and regression tasks, achieving over 99% and 90% accuracy on the MNIST and CIFAR-10 datasets, and outperforming standard neural networks, random forests and gradient boosted trees on the SARCOS dataset.Furthermore, ANT optimisation naturally adapts the architecture to the size and complexity of the training data.

Wir schlagen ein Framework vor, um Entscheidungsbäume und neuronale Netze zu kombinieren, und zeigen anhand von Bildklassifizierungsaufgaben, dass er die komplementären Vorteile der beiden Ansätze nutzt und gleichzeitig die Grenzen früherer Arbeiten überwindet.

Die Autoren schlugen ein neues Modell, Adaptive Neural Trees, vor, indem sie das Repräsentationslernen und die Gradientenoptimierung von neuronalen Netzen mit dem Architekturlernen von Entscheidungsbäumen kombinierten.

In diesem Beitrag wird der Ansatz der adaptiven neuronalen Bäume vorgeschlagen, um die beiden Lernparadigmen der tiefen neuronalen Netze und der Entscheidungsbäume zu kombinieren.


While natural language processing systems often focus on a single language, multilingual transfer learning has the potential to improve performance, especially for low-resource languages. We introduce XLDA, cross-lingual data augmentation, a method that replaces a segment of the input text with its translation in another language.XLDA enhances performance of all 14 tested languages of the cross-lingual natural language inference (XNLI) benchmark.With improvements of up to 4.8, training with XLDA achieves state-of-the-art performance for Greek, Turkish, and Urdu.XLDA is in contrast to, and performs markedly better than, a more naive approach that aggregates examples in various languages in a way that each example is solely in one language.On the SQuAD question answering task, we see that XLDA provides a 1.0 performance increase on the English evaluation set.Comprehensive experiments suggest that most languages are effective as cross-lingual augmentors, that XLDA is robust to a wide range of translation quality, and that XLDA is even more effective for randomly initialized models than for pretrained models.

Die Übersetzung von Teilen der Eingabe während des Trainings kann die sprachübergreifende Leistung verbessern.

In diesem Beitrag wird eine Methode zur sprachübergreifenden Datenerweiterung vorgeschlagen, um die Sprachinferenz und die Beantwortung von Fragen zu verbessern.

In diesem Beitrag wird vorgeschlagen, sprachübergreifende Daten durch heuristische Swaps mit alignierten Übersetzungen zu ergänzen, wie es zweisprachige Menschen beim Code-Switching tun.


Training conditional generative latent-variable models is challenging in scenarios where the conditioning signal is very strong and the decoder is expressive enough to generate a plausible output given only the condition; the generative model tends to ignore the latent variable, suffering from posterior collapse. We find, and empirically show, that one of the major reasons behind posterior collapse is rooted in the way that generative models are conditioned, i.e., through concatenation of the latent variable and the condition. To mitigate this problem, we propose to explicitly make the latent variables depend on the condition by unifying the conditioning and latent variable sampling, thus coupling them so as to prevent the model from discarding the root of variations. To achieve this, we develop a conditional Variational Autoencoder architecture that learns a distribution not only of the latent variables, but also of the condition, the latter acting as prior on the former. Our experiments on the challenging tasks of conditional human motion prediction and image captioning demonstrate the effectiveness of our approach at avoiding posterior collapse. Video results of our approach are anonymously provided in http://bit.ly/iclr2020

Wir schlagen einen bedingten variationalen Autoencoder vor, der den nachgelagerten Kollaps in Szenarien abmildert, in denen das Konditionierungssignal stark genug ist, damit ein ausdrucksstarker Decoder eine plausible Ausgabe daraus erzeugen kann.

In diesem Beitrag werden stark konditionierte generative Modelle betrachtet und eine Zielfunktion sowie eine Parametrisierung der Variationsverteilung vorgeschlagen, so dass die latenten Variablen explizit von den Eingabebedingungen abhängen.

In diesem Artikel wird argumentiert, dass ein nachgelagerter Kollaps wahrscheinlicher ist als bei einer einfachen VAE, wenn der Decoder auf der Verkettung von latenten Variablen und Hilfsinformationen beruht.


We propose a study of the stability of several few-shot learning algorithms subject to variations in the hyper-parameters and optimization schemes while controlling the random seed.  We propose a methodology for testing for statistical differences in model performances under several replications.To study this specific design, we attempt to reproduce results from three prominent papers: Matching Nets, Prototypical Networks, and TADAM.We analyze on the miniImagenet dataset on the standard classification task in the 5-ways, 5-shots learning setting at test time.We find that the selected implementations exhibit stability across random seed, and repeats.

Wir schlagen eine Studie über die Stabilität verschiedener Algorithmen zum Few-Shot Learning vor, die Variationen in den Hyperparametern und Optimierungsschemata unterworfen sind, während wir die zufällige Seed kontrollieren.

In dieser Arbeit wird die Reproduzierbarkeit beim Few-Shot Learning untersucht.


We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning.In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach.Accordingly, the choice of representation -- the mapping of observation space to goal space -- is crucial.To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation.We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice.Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods.

Wir übersetzen eine Beschränkung der Suboptimalität von Repräsentationen in ein praktisches Trainingsziel im Kontext des hierarchischen Reinforcement Learning.

Die Autoren schlagen einen neuartigen Ansatz für das Lernen einer Repräsentation für HRL vor und stellen eine interessante Verbindung zwischen dem Lernen der Repräsentation und der Begrenzung der Suboptimalität her, die zu einem gradientenbasierten Algorithmus führt.

In dieser Arbeit wird ein Weg vorgeschlagen, mit Suboptimalität im Zusammenhang mit Lernrepräsentationen umzugehen, die sich auf die Suboptimalität der hierarchischen Regeln in Bezug auf die Aufgabenbelohnung beziehen.


Heuristic search research often deals with finding algorithms for offline planning which aim to minimize the number of expanded nodes or planning time.In online planning, algorithms for real-time search or deadline-aware search have been considered before.However, in this paper, we are interested in the problem of {\em situated temporal planning} in which an agent's plan can depend on exogenous events in the external world, and thus it becomes important to take the passage of time into account during the planning process.  Previous work on situated temporal planning has proposed simple pruning strategies, as well as complex schemes for a simplified version of the associated metareasoning problem. In this paper, we propose a simple metareasoning technique,  called the crude greedy scheme, which can be applied in a situated temporal planner.Our empirical evaluation shows that the crude greedy scheme outperforms standard heuristic search based on cost-to-go estimates.

Metareasoning in einem situierten temporalen Planer.

Dieser Beitrag befasst sich mit dem Problem der situierten zeitlichen Planung und schlägt eine weitere Vereinfachung der zuvor von Shperberg vorgeschlagenen gierigen Strategien vor.


Neural networks are vulnerable to small adversarial perturbations.Existing literature largely focused on understanding and mitigating the vulnerability of learned models.In this paper, we demonstrate an intriguing phenomenon about the most popular robust training method in the literature, adversarial training: Adversarial robustness, unlike clean accuracy, is sensitive to the input data distribution.Even a semantics-preserving transformations on the input data distribution can cause a significantly different robustness for the adversarial trained model that is both trained and evaluated on the new distribution.Our discovery of such sensitivity on data distribution is based on a study which disentangles the behaviors of clean accuracy and robust accuracy of the Bayes classifier.Empirical investigations further confirm our finding.We construct semantically-identical variants for MNIST and CIFAR10 respectively, and show that standardly trained models achieve comparable clean accuracies on them, but adversarially trained models achieve significantly different robustness accuracies.This counter-intuitive phenomenon indicates that input data distribution alone can affect the adversarial robustness of trained neural networks, not necessarily the tasks themselves.Lastly, we discuss the practical implications on evaluating adversarial robustness, and make initial attempts to understand this complex phenomenon.

Die Robustheit trainierter PGD-Modelle reagiert empfindlich auf semantikerhaltende Transformationen von Bilddatensätzen, was bedeutet, dass die Bewertung robuster Lernalgorithmen in der Praxis heikel ist.

Die Arbeit verdeutlicht den Unterschied zwischen sauberer und robuster Genauigkeit und zeigt, dass eine Änderung der Randverteilung der Eingabedaten P(x) unter Beibehaltung ihrer Semantik P(y|x) die Robustheit des Modells beeinflusst.

In diesem Beitrag wird die Ursache für die mangelnde Robustheit von Klassifikatoren gegenüber Störungen der aadversarial Eingaben bei l-inf begrenzten Störungen untersucht.


Many tasks in natural language processing involve comparing two sentences to compute some notion of relevance, entailment, or similarity.Typically this comparison is done either at the word level or at the sentence level, with no attempt to leverage the inherent structure of the sentence.When sentence structure is used for comparison, it is obtained during a non-differentiable pre-processing step, leading to propagation of errors.We introduce a model of structured alignments between sentences, showing how to compare two sentences by matching their latent structures.Using a structured attention mechanism, our model matches possible spans in the first sentence to possible spans in the second sentence, simultaneously discovering the tree structure of each sentence and performing a comparison, in a model that is fully differentiable and is trained only on the comparison objective.We evaluate this model on two sentence comparison tasks: the Stanford natural language inference dataset and the TREC-QA dataset.We find that comparing spans results in superior performance to comparing words individually, and that the learned trees are consistent with actual linguistic structures.

Übereinstimmung von Sätzen durch Lernen der latenten Konstituentenbaumstrukturen mit einer Variante des Inside-Outside Algorithmus, eingebettet in eine neuronale Netzwerkschicht.

In diesem Beitrag wird ein strukturierter Aufmerksamkeitsmechanismus zur Berechnung von Alignment-Scores unter allen möglichen Abständen in zwei gegebenen Sätzen vorgestellt.

In diesem Artikel wird ein Modell für strukturierte Übereinstimmungen zwischen Sätzen vorgeschlagen, um Sätze durch den Abgleich ihrer latenten Strukturen zu vergleichen.


Learning disentangled representation from any unlabelled data is a non-trivial problem.In this paper we propose Information Maximising Autoencoder (InfoAE) where the encoder learns powerful disentangled representation through maximizing the mutual information between the representation and given information in an unsupervised fashion.We have evaluated our model on MNIST dataset and achieved approximately 98.9 % test accuracy while using complete unsupervised training.

Unüberwachtes Lernen der Entflechtungsdarstellung.

Die Autoren stellen einFramework vor, in dem ein Auto Encoder (E, D) so regularisiert wird, dass seine latente Repräsentation gegenseitige Informationen mit generierten latenten Raumrepräsentation teilt.


Effective training of neural networks requires much data.In the low-data regime,parameters are underdetermined, and learnt networks generalise poorly.DataAugmentation (Krizhevsky et al., 2012) alleviates this by using existing datamore effectively.However standard data augmentation produces only limitedplausible alternative data.Given there is potential to generate a much broader setof augmentations, we design and train a generative model to do data augmentation.The model, based on image conditional Generative Adversarial Networks, takesdata from a source domain and learns to take any data item and generalise itto generate other within-class data items.As this generative process does notdepend on the classes themselves, it can be applied to novel unseen classes of data.We show that a Data Augmentation Generative Adversarial Network (DAGAN)augments standard vanilla classifiers well.We also show a DAGAN can enhancefew-shot learning systems such as Matching Networks.We demonstrate theseapproaches on Omniglot, on EMNIST having learnt the DAGAN on Omniglot, andVGG-Face data.In our experiments we can see over 13% increase in accuracy inthe low-data regime experiments in Omniglot (from 69% to 82%), EMNIST (73.9%to 76%) and VGG-Face (4.5% to 12%); in Matching Networks for Omniglot weobserve an increase of 0.5% (from 96.9% to 97.4%) and an increase of 1.8% inEMNIST (from 59.5% to 61.3%).

Bedingte GANs, die so trainiert werden, dass sie datenerweiterte Beispiele ihrer bedingten Eingaben erzeugen, die zur Verbesserung von Standard Klassifizierungs- und One-Shot-Lernsystemen wie Matching-Netzwerken und Pixel-Distanz verwendet werden.

Die Autoren schlagen eine Methode zur Datenerweiterung vor, bei der die klassenübergreifenden Transformationen mit Hilfe von bedingten GAN auf einen niedrigdimensionalen latenten Raum abgebildet werden.


Answering questions about data can require understanding what parts of an input X influence the response Y. Finding such an understanding can be built by testing relationships between variables through a machine learning model.For example, conditional randomization tests help determine whether a variable relates to the response given the rest of the variables.However, randomization tests require users to specify test statistics.We formalize a class of proper test statistics that are guaranteed to select a feature when it provides information about the response even when the rest of the features are known.We show that f-divergences provide a broad class of proper test statistics.In the class of f-divergences, the KL-divergence yields an easy-to-compute proper test statistic that relates to the AMI.Questions of feature importance can be asked at the level of an individual sample.  We show that estimators from the same AMI test can also be used to find important features in a particular instance.We provide an example to show that perfect predictive models are insufficient for instance-wise feature selection.We evaluate our method on several simulation experiments, on a genomic dataset, a clinical dataset for hospital readmission, and on a subset of classes in ImageNet.Our method outperforms several baselines in various simulated datasets, is able to identify biologically significant genes, can select the most important predictors of a hospital readmission event, and is able to identify distinguishing features in an image-classification task.

Wir entwickeln eine einfache, auf Regression basierende, modellagnostische Methode zur Auswahl von Merkmalen, um datengenerierende Prozesse mit FDR-Kontrolle zu interpretieren, und übertreffen mehrere populäre Grundlinien auf mehreren simulierten, medizinischen und Bilddatensätzen.

Diese Arbeit schlägt eine praktische Verbesserung des bedingten Randomisierungstests und eine neue Teststatistik vor, beweist, dass f-Divergenz eine mögliche Wahl ist, und zeigt, dass KL-Divergenz einige bedingte Verteilungen aufhebt.

Diese Arbeit befasst sich mit dem Problem, nützliche Merkmale in einer Eingabe zu finden, die von einer variablen Antwort abhängig sind, selbst wenn alle anderen Eingabevariablen konditioniert sind.

Eine modellunabhängige Methode zur Interpretation des Einflusses von Eingabemerkmalen auf die Reaktion eines Modells auf Maschinenebene bis hin zur Instanzebene sowie geeignete Teststatistiken für die modellunabhängige Merkmalsauswahl.


Supervised learning depends on annotated examples, which are taken to be the ground truth.But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk.Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem).Given a fixed annotation budget and unlimited unlabeled data, redundant annotation comes at the expense of fewer labeled examples.This raises two fundamental questions: (1) How can we best learn from noisy workers?(2) How should we allocate our labeling budget to maximize the performance of a classifier?We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data.The alternating minimization proceeds in rounds, estimating worker quality from disagreement with the current model and then updating the model by optimizing a loss function that accounts for the current estimate of worker quality.Unlike previous approaches, even with only one annotation per example, our algorithm can estimate worker quality.We establish a generalization error bound for models learned with our algorithm and establish theoretically that it's better to label many examples once (vs less multiply) when worker quality exceeds a threshold.Experiments conducted on both ImageNet (with simulated noisy workers) and MS-COCO (using the real crowdsourced labels) confirm our algorithm's benefits.

Ein neuer Ansatz zum Lernen eines Modells aus verrauschten Crowdsourced Annotations.

In diesem Artikel wird eine Methode zum Lernen aus störhaften Labels vorgeschlagen, die sich auf den Fall konzentriert, dass die Daten nicht redundant beschriftet sind, mit theoretischer und experimenteller Validierung

Diese Arbeit konzentriert sich auf das Problem des Lernens aus der Crowd, bei dem die gemeinsame Aktualisierung der Klassifikatorgewichte und der Konfusionsmatrizen der Arbeiter bei dem Schätzproblem mit seltenen Crowdsourced Labels helfen kann.

Es wird ein überwachter Lernalgorithmus für die Modellierung der Qualität von Labels und Mitarbeitern vorgeschlagen und der Algorithmus wird verwendet, um zu untersuchen, wie viel Redundanz beim Crowdsourcing erforderlich ist und ob eine geringe Redundanz mit reichlich Störbeispielen zu besseren Labels führt.


Neural networks make mistakes.The reason why a mistake is made often remains a mystery.As such neural networks often are considered a black box.It would be useful to have a method that can give an explanation that is intuitive to a user as to why an image is misclassified.In this paper we develop a method for explaining the mistakes of a classifier model by visually showing what must be added to an image such that it is correctly classified.Our work combines the fields of adversarial examples, generative modeling and a correction technique based on difference target propagation to create an technique that creates explanations of why an image is misclassified.In this paper we explain our method and demonstrate it on MNIST and CelebA.This approach could aid in demystifying neural networks for a user.

Neue Methode zur Erklärung, warum ein neuronales Netz ein Bild falsch klassifiziert hat.

In diesem Artikel wird eine Methode zur Erklärung der Klassifizierungsfehler von neuronalen Netzen vorgeschlagen.

Ziel ist es, die Klassifizierung neuronaler Netze besser zu verstehen und den latenten Raum eines variationalen Autoencoders zu erforschen und die Störungen des latenten Raums zu berücksichtigen, um eine korrekte Klassifizierung zu erhalten.


In the context of multi-task learning, neural networks with branched architectures have often been employed to jointly tackle the tasks at hand.Such ramified networks typically start with a number of shared layers, after which different tasks branch out into their own sequence of layers.Understandably, as the number of possible network configurations is combinatorially large, deciding what layers to share and where to branch out becomes cumbersome.Prior works have either relied on ad hoc methods to determine the level of layer sharing, which is suboptimal, or utilized neural architecture search techniques to establish the network design, which is considerably expensive.In this paper, we go beyond these limitations and propose a principled approach to automatically construct branched multi-task networks, by leveraging the employed tasks' affinities.Given a specific budget, i.e. number of learnable parameters, the proposed approach generates architectures, in which shallow layers are task-agnostic, whereas deeper ones gradually grow more task-specific.Extensive experimental analysis across numerous, diverse multi-tasking datasets shows that, for a given budget, our method consistently yields networks with the highest performance, while for a certain performance threshold it requires the least amount of learnable parameters.

Eine Methode zur automatischen Konstruktion von verzweigten Multitasking-Netzwerken mit starker experimenteller Bewertung auf verschiedenen Multitasking-Datensätzen.

In diesem Beitrag wird ein neuartiges Multi-Task Learning Framework mit sanften Parametern vorgeschlagen, das auf einer baumartigen Struktur basiert.

In diesem Artikel wird eine Methode zur Ableitung der Architektur von Multitasking Netzwerken vorgestellt, um zu bestimmen, welcher Teil des Netzwerks von den verschiedenen Aufgaben gemeinsam genutzt werden sollte.


Typical recent neural network designs are primarily convolutional layers, but the tricks enabling structured efficient linear layers (SELLs) have not yet been adapted to the convolutional setting.We present a method to express the weight tensor in a convolutional layer using diagonal matrices, discrete cosine transforms (DCTs) and permutations that can be optimised using standard stochastic gradient methods.A network composed of such structured efficient convolutional layers (SECL) outperforms existing low-rank networks and demonstrates competitive computational efficiency.

Es ist möglich, die Gewichtsmatrix in einem Convolutional Layer zu ersetzen, um sie als strukturierte, effiziente Schicht zu trainieren, die genauso gut funktioniert wie die Low-Rank Decomposition.

Diese Arbeit wendet frühere Structured Efficient Linear Layers auf Convolutional Layers an und schlägt Structured Efficient Convolutional Layers als Ersatz für die ursprünglichen Convolutional Layers vor.


Blind document deblurring is a fundamental task in the field of document processing and restoration, having wide enhancement applications in optical character recognition systems, forensics, etc.Since this problem is highly ill-posed, supervised and unsupervised learning methods are well suited for this application.Using various techniques, extensive work has been done on natural-scene deblurring.However, these extracted features are not suitable for document images.We present SVDocNet, an end-to-end trainable U-Net based spatial recurrent neural network (RNN) for blind document deblurring where the weights of the RNNs are determined by different convolutional neural networks (CNNs).This network achieves state of the art performance in terms of both quantitative measures and qualitative results.

Wir stellen SVDocNet vor, ein durchgängig trainierbares U-Netz auf der Basis eines räumlich rekurrenten neuronalen Netzes (RNN) für die blinde Entschlüsselung von Dokumenten.


In contrast to the monolithic deep architectures used in deep learning today for computer vision, the visual cortex processes retinal images via two functionally distinct but interconnected networks: the ventral pathway for processing object-related information and the dorsal pathway for processing motion and transformations.Inspired by this cortical division of labor and properties of the magno- and parvocellular systems, we explore an unsupervised approach to feature learning that jointly learns object features and their transformations from natural videos.We propose a new convolutional bilinear sparse coding model that (1) allows independent feature transformations and (2) is capable of processing large images.Our learning procedure leverages smooth motion in natural videos.Our results show that our model can learn groups of features and their transformations directly from natural videos in a completely unsupervised manner.The learned "dynamic filters" exhibit certain equivariance properties, resemble cortical spatiotemporal filters, and capture the statistics of transitions between video frames.Our model can be viewed as one of the first approaches to demonstrate unsupervised learning of primary "capsules" (proposed by Hinton and colleagues for supervised learning) and has strong connections to the Lie group approach to visual perception.

Wir erweitern die bilineare Sparse Codierung und nutzen Videosequenzen, um dynamische Filter zu lernen.


Conventional out-of-distribution (OOD) detection schemes based on variational autoencoder or Random Network Distillation (RND) are known to assign lower uncertainty to the OOD data than the target distribution.In this work, we discover that such conventional novelty detection schemes are also vulnerable to the blurred images.Based on the observation, we construct a novel RND-based OOD detector, SVD-RND, that utilizes blurred images during training.Our detector is simple, efficient in test time, and outperforms baseline OOD detectors in various domains.Further results show that SVD-RND learns a better target distribution representation than the baselines.Finally, SVD-RND combined with geometric transform achieves near-perfect detection accuracy in CelebA domain.

Wir schlagen einen neuartigen OOD-Detektor vor, der unscharfe Bilder als Negativbeispiele verwendet. Unser Modell erreicht eine signifikante OOD-Erkennungsleistung in verschiedenen Bereichen.

In diesem Beitrag wird die Idee vorgestellt, unscharfe Bilder als Regularisierungsbeispiele zu verwenden, um die Leistung bei der Erkennung von Verteilungsfehlern auf der Grundlage von Random Network Distillation zu verbessern.

In diesem Artikel wird die Out-of-Data Verteilung angegangen, indem RND auf Datenerweiterungen angewandt wird, indem ein Modell trainiert wird, um die Ausgaben eines Zufallsnetzwerks mit einer Erweiterung als Eingabe abzugleichen.


Training large deep neural networks on massive datasets is Â computationally very challenging.There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue.The most prominent algorithm in this line of research is LARS, which by Â employing layerwise adaptive learning rates trains ResNet on ImageNet in a few minutes.However, LARS performs poorly for attention models like BERT, indicating that its performance gains are not consistent across tasks.In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches.Using this strategy, we develop a new layerwise adaptive large batch optimization technique called LAMB; we then provide convergence analysis of LAMB as well as LARS, showing convergence to a stationary point in general nonconvex settings.Our empirical results demonstrate the superior performance of LAMB across various tasks such as BERT and ResNet-50 training with very little hyperparameter tuning.In particular, for BERT training, our optimizer enables use of very large batch sizes of 32868 without any degradation of performance. Â By increasing the batch size to the memory limit of a TPUv3 Pod, BERT training time can be reduced from 3 days to just 76 minutes (Table 1).

Ein schneller Optimierer für allgemeine Anwendungen und Training in großen Batches.

In dieser Arbeit haben die Autoren eine Studie zum Large-Batch-Training für BERT durchgeführt und erfolgreich ein BERT-Modell in 76 Minuten trainiert.

In diesem Beitrag wird eine schichtweise Anpassungsstrategie entwickelt, die es ermöglicht, BERT-Modelle mit großen 32k-Minibatches im Vergleich zur Basislinie von 512 Batches zu trainieren.


Model-agnostic meta-learning (MAML) is known as a powerful meta-learning method.However, MAML is notorious for being hard to train because of the existence of two learning rates.Therefore, in this paper, we derive the conditions that inner learning rate $\alpha$ and meta-learning rate $\beta$ must satisfy for MAML to converge to minima with some simplifications.We find that the upper bound of $\beta$ depends on $ \alpha$, in contrast to the case of using the normal gradient descent method.Moreover, we show that the threshold of $\beta$ increases as $\alpha$ approaches its own upper bound.This result is verified by experiments on various few-shot tasks and architectures; specifically, we perform sinusoid regression and classification of Omniglot and MiniImagenet datasets with a multilayer perceptron and a convolutional neural network.Based on this outcome, we present a guideline for determining the learning rates: first, search for the largest possible $\alpha$; next, tune $\beta$ based on the chosen value of $\alpha$.

Wir haben die Rolle von zwei Lernraten beim modellagnostischen Meta-Lernen bei der Konvergenz analysiert.

Die Autoren haben das Problem der Instabilität der Optimierung in MAML durch die Untersuchung der beiden Lernraten angegangen.

In dieser Arbeit wird eine Methode untersucht, mit der die beiden im MAML-Trainingsalgorithmus verwendeten Lernraten eingestellt werden können.


We present a neural framework for learning associations between interrelated groups of words such as the ones found in Subject-Verb-Object (SVO) structures.Our model induces a joint function-specific word vector space, where vectors of e.g. plausible SVO compositions lie close together.The model retains information about word group membership even in the joint space, and can thereby effectively be applied to a number of tasks reasoning over the SVO structure.We show the robustness and versatility of the proposed framework by reporting state-of-the-art results on the tasks of estimating selectional preference (i.e., thematic fit) and event similarity.The results indicate that the combinations of representations learned with our task-independent model outperform task-specific architectures from prior work, while reducing the number of parameters by up to 95%.The proposed framework is versatile and holds promise to support learning function-specific representations beyond the SVO structures.

Aufgabenunabhängiges neuronales Modell für das Lernen von Assoziationen zwischen zusammenhängenden Wortgruppen.

In dem Papier wird eine Methode zum Training funktionsspezifischer Wortvektoren vorgeschlagen, bei der jedes Wort mit drei Vektoren in jeweils einer anderen Kategorie (Subjekt-Verb-Objekt) dargestellt wird.

In diesem Beitrag wird ein neuronales Netz zum Erlernen funktionsspezifischer Arbeitsrepräsentationen vorgeschlagen und der Vorteil gegenüber Alternativen aufgezeigt.


The fabrication of semiconductor involves etching process to remove selected areas from wafers.However, the measurement of etched structure in micro-graph heavily relies on time-consuming manual routines.Traditional image processing usually demands on large number of annotated data and the performance is still poor.We treat this challenge as segmentation problem and use deep learning approach to detect masks of objects in etched structure of wafer.Then, we use simple image processing to carry out automatic measurement on the objects.We attempt Generative Adversarial Network (GAN) to generate more data to overcome the problem of very limited dataset.We download 10 SEM (Scanning Electron Microscope) images of 4 types from Internet, based on which we carry out our experiments.Our deep learning based method demonstrates superiority over image processing approach with  mean accuracy reaching over 96% for the measurements, compared with the ground truth.To the best of our knowledge, it is the first time that deep learning has been applied in semiconductor industry for automatic measurement.

Einsatz einer Deep-Learning-Methode zur automatischen Vermessung von SEM-Bildern in der Halbleiterindustrie.


Generating and scheduling activities is particularly challengingwhen considering both consumptive resources andcomplex resource interactions such as time-dependent resourceusage.We present three methods of determining validtemporal placement intervals for an activity in a temporallygrounded plan in the presence of such constraints.We introducethe Max Duration and Probe algorithms which aresound, but incomplete, and the Linear algorithm which issound and complete for linear rate resource consumption.We apply these techniques to the problem of schedulingawakes for a planetary rover where the awake durationsare affected by existing activities.We demonstrate how theProbe algorithm performs competitively with the Linear algorithmgiven an advantageous problem space and well-definedheuristics.We show that the Probe and Linear algorithmsoutperform the Max Duration algorithm empirically.We then empirically present the runtime differences betweenthe three algorithms.The Probe algorithm is currently base-linedfor use in the onboard scheduler for NASAâ€™s next planetaryrover, the Mars 2020 rover.

Dieses Papier beschreibt und analysiert drei Methoden zur Terminierung von Aktivitäten mit nicht fester Dauer bei Vorhandensein von verbrauchenden Ressourcen.

In diesem Beitrag werden drei Ansätze für die Planung von Aktivitäten an Bord eines planetarischen Rovers unter Berücksichtigung von Ressourcenbeschränkungen vorgestellt.


A disentangled representation of a data set should be capable of recovering the underlying factors that generated it.One question that arises is whether using Euclidean space for latent variable models can produce a disentangled representation when the underlying generating factors have a certain geometrical structure.Take for example the images of a car seen from different angles.The angle has a periodic structure but a 1-dimensional representation would fail to capture this topology.How can we address this problem?The submissions presented for the first stage of the  NeurIPS2019 Disentanglement Challenge consist of a Diffusion Variational Autoencoder ($\Delta$VAE) with a hyperspherical latent space which can for example recover periodic true factors.The training of the $\Delta$VAE is enhanced by incorporating a modified version of the Evidence Lower Bound (ELBO) for tailoring the encoding capacity of the posterior approximate.

Beschreibung der Einreichung zur NeurIPS2019 Disentanglement Challenge basierend auf hypersphärischen variationalen Autoencodern.


Anomaly detection, finding patterns that substantially deviate from those seen previously, is one of the fundamental problems of artificial intelligence.Recently, classification-based methods were shown to achieve superior results on this task.In this work, we present a unifying view and propose an open-set method to relax current generalization assumptions.Furthermore, we extend the applicability of transformation-based methods to non-image data using random affine transformations.Our method is shown to obtain state-of-the-art accuracy and is applicable to broad data types.The strong performance of our method is extensively validated on multiple datasets from different domains.

Eine Anomalie-Erkennung, die Zufallstransformationen zur Klassifizierung für die Verallgemeinerung auf Nicht-Bilddaten verwendet.

In diesem Beitrag wird ein tiefes Verfahren zur Erkennung von Anomalien vorgeschlagen, das die jüngsten tiefen One-class Klassifizierungs- und transformationsbasierten Klassifizierungsansätze vereint.

In dieser Arbeit wird ein Ansatz zur klassifikationsbasierten Erkennung von Anomalien für allgemeine Daten unter Verwendung der affinen Transformation y = Wx+b vorgeschlagen.


Recent improvements in large-scale language models have driven progress on automatic generation of syntactically and semantically consistent text for many real-world applications.Many of these advances leverage the availability of large corpora.While training on such corpora encourages the model to understand long-range dependencies in text, it can also result in the models internalizing the social biases present in the corpora.This paper aims to quantify and reduce biases exhibited by language models.Given a conditioning context (e.g. a writing prompt) and a language model, we analyze if (and how) the sentiment of the generated text is affected by changes in values of sensitive attributes (e.g. country names, occupations, genders, etc.) in the conditioning context, a.k.a. counterfactual evaluation.We quantify these biases by adapting individual and group fairness metrics from the fair machine learning literature.Extensive evaluation on two different corpora (news articles and Wikipedia) shows that state-of-the-art Transformer-based language models exhibit biases learned from data.We propose embedding-similarity and sentiment-similarity regularization methods that improve both individual and group fairness metrics without sacrificing perplexity and semantic similarity---a positive step toward development and deployment of fairer language models for real-world applications.

Wir verringern die Verzerrung der Stimmung auf der Grundlage einer kontrafaktischen Bewertung der Texterstellung mithilfe von Sprachmodellen.

In diesem Artikel wird die semantische Verzerrung in Sprachmodellen gemessen, wie sie sich in dem von den Modellen erzeugten Text widerspiegelt, und es werden andere objektive Begriffe zu den üblichen Sprachmodellierungszielen hinzugefügt, um die Verzerrung zu verringern.

In diesem Papier wird vorgeschlagen, die Verzerrung in vortrainierten Sprachmodellen zu bewerten, indem ein festes Sentiment-System verwendet und verschiedene Präfix-Vorlagen getestet werden.

Eine Methode, die auf semantischer Ähnlichkeit basiert, und eine Methode, die auf sentimentaler Ähnlichkeit basiert, um die neuronalen Sprachmodelle, die aus großen Datensätzen trainiert wurden, zu entzerren.


Topic modeling of text documents is one of the most important tasks in representation learning.In this work, we propose iTM-VAE, which is a Bayesian nonparametric (BNP) topic model with variational auto-encoders.On one hand, as a BNP topic model, iTM-VAE potentially has infinite topics and can adapt the topic number to data automatically.On the other hand, different with the other BNP topic models, the inference of iTM-VAE is modeled by neural networks, which has rich representation capacity and can be computed in a simple feed-forward manner.Two variants of iTM-VAE are also proposed in this paper, where iTM-VAE-Prod models the generative process in products-of-experts fashion for better performance and iTM-VAE-G places a prior over the concentration parameter such that the model can adapt a suitable concentration parameter to data automatically.Experimental results on 20News and Reuters RCV1-V2 datasets show that the proposed models outperform the state-of-the-arts in terms of perplexity, topic coherence and document retrieval tasks.Moreover, the ability of adjusting the concentration parameter to data is also confirmed by experiments.

Ein nichtparametrisches Bayes'sches Themenmodell mit variationalen Autoencodern, das bei öffentlichen Benchmarks den Stand der Technik in Bezug auf Komplexität, Themenkohärenz und Abfrageaufgaben erreicht.

In diesem Beitrag wird ein unendliches Themenmodell mit variationalen Autoencodern konstruiert, indem der Stick-breaking variationale Autoencoder von Nalisnick & Smith mit latenter Dirichlet-Zuweisung und mehreren in Miao verwendeten Inferenztechniken kombiniert wird.


Knowledge Distillation (KD) is a widely used technique in recent deep learning research to obtain small and simple models whose performance is on a par with their large and complex counterparts.Standard Knowledge Distillation tends to be time-consuming because of the training time spent to obtain a teacher model that would then provide guidance for the student model.It might be possible to cut short the time by training a teacher model on the fly, but it is not trivial to have such a high-capacity teacher that gives quality guidance to student models this way.To improve this, we present a novel framework of Knowledge Distillation exploiting dark knowledge from the whole training set.In this framework, we propose a simple and effective implementation named Distillation by Utilizing Peer Samples (DUPS) in one generation.We verify our algorithm on numerous experiments.Compared with standard training on modern architectures, DUPS achieves an average improvement of 1%-2% on various tasks with nearly zero extra cost.Considering some typical Knowledge Distillation methods which are much more time-consuming, we also get comparable or even better performance using DUPS.

Wir stellen einen neuen Rahmen für die Wissensdestillation vor, der Peer-Beispiele als Lehrer verwendet.

Vorschlagen einer Methode zur Verbesserung der Effektivität der Wissensdestillation, indem die verwendeten Bezeichnungen abgeschwächt werden und ein Datensatz anstelle eines einzelnen Beispiels verwendet wird.

In dieser Arbeit wird vorgeschlagen, den zusätzlichen Rechenaufwand für das Training durch Wissensdestillation zu bewältigen, indem auf der kürzlich vorgeschlagenen Snapshot-Destillationstechnik aufgebaut wird.


We develop a metalearning approach for learning hierarchically structured poli- cies, improving sample efficiency on unseen tasks through the use of shared primitivesâ€”policies that are executed for large numbers of timesteps.Specifi- cally, a set of primitives are shared within a distribution of tasks, and are switched between by task-specific policies.We provide a concrete metric for measuring the strength of such hierarchies, leading to an optimization problem for quickly reaching high reward on unseen tasks.We then present an algorithm to solve this problem end-to-end through the use of any off-the-shelf reinforcement learning method, by repeatedly sampling new tasks and resetting task-specific policies.We successfully discover meaningful motor primitives for the directional movement of four-legged robots, solely by interacting with distributions of mazes.We also demonstrate the transferability of primitives to solve long-timescale sparse-reward obstacle courses, and we enable 3D humanoid robots to robustly walk and crawl with the same policy.

Erlernen hierarchischer Teilstrategien durch Ende-zu-Ende Training über eine Verteilung von Aufgaben

Die Autoren betrachten das Problem des Lernens einer sinnvollen Menge von Subregeln, die zwischen Aufgaben geteilt werden können, um das Lernen auf neue Aufgaben aus der Aufgabenverteilung zu starten.

In dieser Arbeit wird eine neuartige Methode zur Induzierung einer zeitlichen hierarchischen Struktur in einer spezialisierten Multi-Task-Umgebung vorgeschlagen.


This paper proposes a new model for document embedding.Existing approaches either require complex inference or use recurrent neural networks that are difficult to parallelize.We take a different route and use recent advances in language modeling to develop a convolutional neural network embedding model.This allows us to train deeper architectures that are fully parallelizable.Stacking layers together increases the receptive filed allowing each successive layer to model increasingly longer range semantic dependences within the document.Empirically we demonstrate superior results on two publicly available benchmarks.Full code will be released with the final version of this paper.

Convolutional Neural Network Modell für die unbeaufsichtigte Einbettung von Dokumenten.

Es wird ein neues Modell für die allgemeine Aufgabe der Induktion von Dokumentrepräsentationen (Einbettungen) vorgestellt, das eine CNN-Architektur zur Verbesserung der Recheneffizienz verwendet.

In diesem Artikel wird vorgeschlagen, CNNs mit einem Skip-Gram-ähnlichen Ziel als schnellen Weg zur Ausgabe von Dokumenteneinbettungen zu verwenden.


We prove bounds on the generalization error of convolutional networks.The bounds are in terms of the training loss, the number ofparameters, the Lipschitz constant of the loss and the distance fromthe weights to the initial weights. They are independent of thenumber of pixels in the input, and the height and width of hiddenfeature maps. We present experiments with CIFAR-10, along with varyinghyperparameters of a deep convolutional network, comparing our boundswith practical generalization gaps.

Wir beweisen Verallgemeinerungsgrenzen für Convolutional neuronale Netze, die die Gewichtskopplung berücksichtigen.

Untersucht die Verallgemeinerungsfähigkeit von CNNs und verbessert die oberen Schranken des Verallgemeinerungsfehlers, wobei eine Korrelation zwischen dem Verallgemeinerungsfehler von gelernten CNNs und dem dominanten Term der oberen Schranke gezeigt wird.

In diesem Papier wird eine Verallgemeinerungsschranke für neuronale Convolutional Neural Networks vorgestellt, die auf der Anzahl der Parameter, der Lipschitz-Konstante und dem Abstand der endgültigen Gewichte von der Initialisierung basiert.


MobileNets family of computer vision neural networks have fueled tremendous progress in the design and organization of resource-efficient architectures in recent years.New applications with stringent real-time requirements in highly constrained devices require further compression of MobileNets-like already computeefficient networks.Model quantization is a widely used technique to compress and accelerate neural network inference and prior works have quantized MobileNets to 4 âˆ’ 6 bits albeit with a modest to significant drop in accuracy.While quantization to sub-byte values (i.e. precision â‰¤ 8 bits) has been valuable, even further quantization of MobileNets to binary or ternary values is necessary to realize significant energy savings and possibly runtime speedups on specialized hardware, such as ASICs and FPGAs.Under the key observation that convolutional filters at each layer of a deep neural network may respond differently to ternary quantization, we propose a novel quantization method that generates per-layer hybrid filter banks consisting of full-precision and ternary weight filters for MobileNets.The layer-wise hybrid filter banks essentially combine the strengths of full-precision and ternary weight filters to derive a compact, energy-efficient architecture for MobileNets.Using this proposed quantization method, we quantized a substantial portion of weight filters of MobileNets to ternary values resulting in 27.98% savings in energy, and a 51.07% reduction in the model size, while achieving comparable accuracy and no degradation in throughput on specialized hardware in comparison to the baseline full-precision MobileNets.

Zweifache Einsparungen bei der Modellgröße, 28 % Energieeinsparung für MobileNets auf ImageNet ohne Genauigkeitsverlust durch hybride Schichten, die aus konventionellen Filtern mit voller Genauigkeit und ternären Filtern bestehen.

Der Schwerpunkt liegt auf der Quantisierung der MobileNets-Architektur auf ternäre Werte, wodurch der Platz- und Rechenbedarf gesenkt wird, um neuronale Netze energieeffizienter zu machen.

Das Papier schlägt eine schichtweise hybride Filter Bank vor, die nur einen Teil der Convolutional Filter auf ternäre Werte für die MobileNets Architektur quantisiert.


Performing controlled experiments on noisy data is essential in thoroughly understanding deep learning across a spectrum of noise levels.Due to the lack of suitable datasets, previous research have only examined deep learning on controlled synthetic noise, and real-world noise has never been systematically studied in a controlled setting.To this end, this paper establishes a benchmark of real-world noisy labels at 10 controlled noise levels.As real-world noise possesses unique properties, to understand the difference, we conduct a large-scale study across a variety of noise levels and types, architectures, methods, and training settings.Our study shows that: (1) Deep Neural Networks (DNNs) generalize much better on real-world noise.(2) DNNs may not learn patterns first on real-world noisy data.(3) When networks are fine-tuned, ImageNet architectures generalize well on noisy data.(4) Real-world noise appears to be less harmful, yet it is more difficult for robust DNN methods to improve.(5) Robust learning methods that work well on synthetic noise may not work as well on real-world noise, and vice versa.We hope our benchmark, as well as our findings, will facilitate deep learning research on noisy data.

Wir haben einen Benchmark für kontrolliertes echtes Stören erstellt und einige interessante Erkenntnisse über gestörte Daten in der realen Welt gewonnen.

In diesem Beitrag werden 6 bestehende Methoden zum Erlernen von gestörten Labeln in zwei Trainingseinstellungen verglichen: von Grund auf und mit Fine-Tuning.

Die Autoren erstellen einen großen Datensatz und einen Benchmark für kontrolliertes echte Störungen, um kontrollierte Experimente mit gestörten Daten beim Deep Learning durchzuführen.


Designing RNA molecules has garnered recent interest in medicine, synthetic biology, biotechnology and bioinformatics since many functional RNA molecules were shown to be involved in regulatory processes for transcription, epigenetics and translation.Since an RNA's function depends on its structural properties, the RNA Design problem is to find an RNA sequence which satisfies given structural constraints.Here, we propose a new algorithm for the RNA Design problem, dubbed LEARNA.LEARNA uses deep reinforcement learning to train a policy network to sequentially design an entire RNA sequence given a specified target structure.By meta-learning across 65000 different RNA Design tasks for one hour on 20 CPU cores, our extension Meta-LEARNA constructs an RNA Design policy that can be applied out of the box to solve novel RNA Design tasks.Methodologically, for what we believe to be the first time, we jointly optimize over a rich space of architectures for the policy network, the hyperparameters of the training procedure and the formulation of the decision process.Comprehensive empirical results on two widely-used RNA Design benchmarks, as well as a third one that we introduce, show that our approach achieves new state-of-the-art performance on the former while also being orders of magnitudes faster in reaching the previous state-of-the-art performance.In an ablation study, we analyze the importance of our method's different components.

Wir lernen, das RNA-Design Problem mit Reinforcement Learning unter Verwendung von Meta-Learning und AutoML Ansätzen zu lösen.

Anwendung der Policy-Gradienten-Optimierung zur Generierung von RNA-Sequenzen, die sich in eine Ziel-Sekundärstruktur falten, was zu einer deutlichen Verbesserung der Genauigkeit und der Laufzeit führte.


Pruning is a popular technique for compressing a neural network: a large pre-trained network is fine-tuned while connections are successively removed.However, the value of pruning has largely evaded scrutiny.In this extended abstract, we examine residual networks obtained through Fisher-pruning and make two interesting observations.First, when time-constrained, it is better to train a simple, smaller network from scratch than prune a large network.Second, it is the architectures obtained through the pruning process  --- not the learnt weights --- that prove valuable.Such architectures are powerful when trained from scratch.Furthermore, these architectures are easy to approximate without any further pruning: we can prune once and obtain a family of new, scalable network architectures for different memory requirements.

Das Trainieren kleiner Netze schlägt das Pruning, aber das Pruning findet gute kleine Netze zum Trainieren, die leicht zu kopieren sind.


Supervised learning problems---particularly those involving social data---are often subjective.That is, human readers, looking at the same data, might come to legitimate but completely different conclusions based on their personal experiences.Yet in machine learning settings feedback from multiple human annotators is often reduced to a single ``ground truth'' label, thus hiding the true, potentially rich and diverse interpretations of the data found across the social spectrum.We explore the rewards and challenges of discovering and learning representative distributions of the labeling opinions of a large human population.A major, critical cost to this approach is the number of humans needed to provide enough labels not only to obtain representative samples but also to train a machine to predict representative distributions on unlabeled data.We propose aggregating label distributions over, not just individuals, but also data items, in order to maximize the costs of humans in the loop.We test different aggregation approaches on state-of-the-art deep learning models.Our results suggest that careful label aggregation methods can greatly reduce the number of samples needed to obtain representative distributions.

Wir untersuchen das Problem des Lernens, die zugrundeliegende Vielfalt von Überzeugungen in überwachten Lernbereichen vorherzusagen.


Recent advancements in deep learning techniques such as Convolutional Neural Networks(CNN) and Generative Adversarial Networks(GAN) have achieved breakthroughs in the problem of semantic image inpainting, the task of reconstructing missing pixels in given images.While much more effective than conventional approaches, deep learning models require large datasets and great computational resources for training, and inpainting quality varies considerably when training data vary in size and diversity.To address these problems, we present in this paper a inpainting strategy of \textit{Comparative Sample Augmentation}, which enhances the quality of training set by filtering out irrelevant images and constructing additional images using information about the surrounding regions of the images to be inpainted.Experiments on multiple datasets demonstrate that our method extends the applicability of deep inpainting models to training sets with varying sizes, while maintaining inpainting quality as measured by qualitative and quantitative metrics for a large class of deep models, with little need for model-specific consideration.

Wir haben eine Strategie eingeführt, die das Inpainting von Modellen auf Datensätzen unterschiedlicher Größe ermöglicht.

Hilfe beim Bild Inpainting mit GANs durch Verwendung eines vergleichenden Augmentierungsfilters und Hinzufügen von Zufallsstörungen zu jedem Pixel.


Generative adversarial networks (GANs) are a family of generative models that do not minimize a single training criterion.Unlike other generative models, the data distribution is learned via a game between a generator (the generative model) and a discriminator (a teacher providing training signal) that each minimize their own cost.GANs are designed to reach a Nash equilibrium at which each player cannot reduce their cost without changing the other playersâ€™ parameters.One useful approach for the theory of GANs is to show that a divergence between the training distribution and the model distribution obtains its minimum value at equilibrium.Several recent research directions have been motivated by the idea that this divergence is the primary guide for the learning process and that every step of learning should decrease the divergence.We show that this view is overly restrictive.During GAN training, the discriminator provides learning signal in situations where the gradients of the divergences between distributions would not be useful.We provide empirical counterexamples to the view of GAN training as divergence minimization.Specifically, we demonstrate that GANs are able to learn distributions in situations where the divergence minimization point of view predicts they would fail.We also show that gradient penalties motivated from the divergence minimization perspective are equally helpful when applied in other contexts in which the divergence minimization perspective does not predict they would be helpful.This contributes to a growing body of evidence that GAN training may be more usefully viewed as approaching Nash equilibria via trajectories that do not necessarily minimize a specific divergence at each step.

Wir finden Belege dafür, dass die Minimierung der Divergenz möglicherweise keine genaue Charakterisierung des GAN-Trainings ist.

Der Beitrag zielt darauf ab, empirische Beweise dafür zu liefern, dass die Theorie der Divergenzminimierung eher ein Werkzeug ist, um das Ergebnis des Trainings von GANs zu verstehen, als eine notwendige Bedingung, die während des Trainings selbst durchgesetzt werden muss.

Diese Arbeit untersucht nicht-sättigende GANs und die Auswirkungen von zwei bestraften Gradienten-Ansätzen, unter Berücksichtigung mehrerer Gedankenexperimente, um Beobachtungen zu demonstrieren und sie an realen Datenexperimenten zu validieren.


Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications.Recent works have developed accurate MI estimators through provably low-bias approximations and tight variational lower bounds assuming abundant supply of samples, but require an unrealistic number of samples to guarantee statistical significance of the estimation.In this work, we focus on improving data efficiency and propose a Data-Efficient MINE Estimator (DEMINE) that can provide a tight lower confident interval of MI under limited data, through adding cross-validation to the MINE lower bound (Belghazi et al., 2018).Hyperparameter search is employed and a novel meta-learning approach with task augmentation is developed to increase robustness to hyperparamters, reduce overfitting and improve accuracy.With improved data-efficiency, our DEMINE estimator enables statistical testing of dependency at practical dataset sizes.We demonstrate the effectiveness of DEMINE on synthetic benchmarks and a real world fMRI dataset, with application of inter-subject correlation analysis.

Ein neuer und praktischer statistischer Test der Abhängigkeit unter Verwendung neuronaler Netze, der an synthetischen und realen fMRI-Datensätzen getestet wurde.

Es wird eine auf neuronalen Netzen basierende Schätzung von gegenseitigen Informationen vorgeschlagen, die zuverlässig mit kleinen Datensätzen arbeiten kann, wobei die Komplexität der Stichprobe durch Entkopplung des Netzwerk-Lernproblems und des Schätzproblems reduziert wird.


Language and vision are processed as two different modal in current work for image captioning.However, recent work on Super Characters method shows the effectiveness of two-dimensional word embedding, which converts text classification problem into image classification problem.In this paper, we propose the SuperCaptioning method, which borrows the idea of two-dimensional word embedding from Super Characters method, and processes the information of language and vision together in one single CNN model.The experimental results on Flickr30k data shows the proposed method gives high quality image captions.An interactive demo is ready to show at the workshop.

Bildbeschriftung durch zweidimensionale Worteinbettung.


Determining the optimal order in which data examples are presented to Deep Neural Networks during training is a non-trivial problem.However, choosing a non-trivial scheduling method may drastically improve convergence.In this paper, we propose a Self-Paced Learning (SPL)-fused Deep Metric Learning (DML) framework, which we call Learning Embeddings for Adaptive Pace (LEAP).Our method parameterizes mini-batches dynamically based on the \textit{easiness} and \textit{true diverseness} of the sample within a salient feature representation space.In LEAP, we train an \textit{embedding} Convolutional Neural Network (CNN) to learn an expressive representation space by adaptive density discrimination using the Magnet Loss.The \textit{student} CNN classifier dynamically selects samples to form a mini-batch based on the \textit{easiness} from cross-entropy losses and \textit{true diverseness} of examples from the representation space sculpted by the \textit{embedding} CNN.We evaluate LEAP using deep CNN architectures for the task of supervised image classification on MNIST, FashionMNIST, CIFAR-10, CIFAR-100, and SVHN.We show that the LEAP framework converges faster with respect to the number of mini-batch updates required to achieve a comparable or better test performance on each of the datasets.

LEAP kombiniert die Stärken des adaptiven Sampling mit denen des Mini-Batch-Online-Lernens und des adaptiven Repräsentationslernens, um eine repräsentative, selbstgesteuerte Strategie in einem Ende-zu-Ende DNN Trainingsprotokoll zu formulieren.

Es wird eine Methode zur Erstellung von Minibatches für ein Schülernetz vorgestellt, bei der ein zweiter erlernter Repräsentationsraum zur dynamischen Auswahl von Beispielen nach ihrer "Einfachheit und wahren Vielfalt" verwendet wird.

Experimente die Klassifizierungsgenauigkeit auf MNIST, FashionMNIST und CIFAR-10 Datensätze zu lernen, eine Darstellung mit Lehrplan Lern-Stil Minibatch-Auswahl in einem Ende-zu-Ende Rahmen.


Conventional deep reinforcement learning typically determines an appropriate primitive action at each timestep, which requires enormous amount of time and effort for learning an effective policy, especially in large and complex environments.To deal with the issue fundamentally, we incorporate macro actions, defined as sequences of primitive actions, into the primitive action space to form an augmented action space.The problem lies in how to find an appropriate macro action to augment the primitive action space.  The agent using a proper augmented action space is able to jump to a farther state and thus speed up the exploration process as well as facilitate the learning procedure.In previous researches, macro actions are developed by mining the most frequently used action sequences or repeating previous actions.However, the most frequently used action sequences are extracted from a past policy, which may only reinforce the original behavior of that policy.On the other hand, repeating actions may limit the diversity of behaviors of the agent.Instead, we propose to construct macro actions by a genetic algorithm, which eliminates the dependency of the macro action derivation procedure from the past policies of the agent.  Our approach appends a macro action to the primitive action space once at a time and evaluates whether the augmented action space leads to promising performance or not.   We perform extensive experiments and show that the constructed macro actions are able to speed up the learning process for a variety of deep reinforcement learning methods.Our experimental results also demonstrate that the macro actions suggested by our approach are transferable among deep reinforcement learning methods and similar environments.We further provide a comprehensive set of ablation analysis to validate our methodology.

Wir schlagen vor, Makro-Aktionen mit Hilfe eines genetischen Algorithmus zu konstruieren, der die Abhängigkeit der Makro-Aktionsableitung von den vergangenen Strategien des Agenten eliminiert.

Dieses Papier schlägt einen generischen Algorithmus für die Konstruktion von Makro-Aktionen für Deep Reinforcement Learning vor, indem eine Makro-Aktion an den primitiven Aktionsraum angehängt wird.


A key problem in neuroscience and life sciences more generally is that the data generation process is often best thought of as a hierarchy of dynamic systems.One example of this is in-vivo calcium imaging data, where observed calcium transients are driven by a combination of electro-chemical kinetics where hypothesized trajectories around manifolds determining the frequency of these transients.A recent approach using sequential variational auto-encoders demonstrated it was possible to learn the latent dynamic structure of reaching behaviour from spiking data modelled as a Poisson process.Here we extend this approach using a ladder method to infer the spiking events driving calcium transients along with the deeper latent dynamic system.We show strong performance of this approach on a benchmark synthetic dataset against a number of alternatives.

Wir schlagen eine Erweiterung von LFADS vor, die in der Lage ist, Spike Trains abzuleiten, um Kalzium-Fluoreszenzspuren mit hierarchischen VAEs zu rekonstruieren.