Deep multitask networks, in which one neural network produces multiple predictive outputs, are more scalable and often better regularized than their single-task counterparts.Such advantages can potentially lead to gains in both speed and performance, but multitask networks are also difficult to train without finding the right balance between tasks.We present a novel gradient normalization (GradNorm) technique which automatically balances the multitask loss function by directly tuning the gradients to equalize task training rates.We show that for various network architectures, for both regression and classification tasks, and on both synthetic and real datasets, GradNorm improves accuracy and reduces overfitting over single networks, static baselines, and other adaptive multitask loss balancing techniques.GradNorm also matches or surpasses the performance of exhaustive grid search methods, despite only involving a single asymmetry hyperparameter $\alpha$.Thus, what was once a tedious search process which incurred exponentially more compute for each task added can now be accomplished within a few training runs, irrespective of the number of tasks.Ultimately, we hope to demonstrate that gradient manipulation affords us great control over the training dynamics of multitask networks and may be one of the keys to unlocking the potential of multitask learning.

Wir zeigen, wie man die Leistung eines Multitasking-Netzes steigern kann, indem man eine adaptive Multitasking Verlustfunktion einstellt, die durch direkten Ausgleich der Netzgradienten gelernt wird.

In dieser Arbeit wird ein dynamisches Aktualisierungsschema für die Gewichte vorgeschlagen, das die Gewichte für verschiedene Aufgabenverluste während der Trainingszeit aktualisiert, indem es die Verlustquoten der verschiedenen Aufgaben nutzt.


Image segmentation aims at grouping pixels that belong to the same object or region.At the heart of image segmentation lies the problem of determining whether a pixel is inside or outside a region, which we denote as the "insideness" problem.Many Deep Neural Networks (DNNs) variants excel in segmentation benchmarks, but regarding insideness, they have not been well visualized or understood: What representations do DNNs use to address the long-range relationships of insideness?How do architectural choices affect the learning of these representations?In this paper, we take the reductionist approach by analyzing DNNs solving the insideness problem in isolation, i.e. determining the inside of closed (Jordan) curves.We demonstrate analytically that state-of-the-art feed-forward and recurrent architectures can implement solutions of the insideness problem for any given curve.Yet, only recurrent networks could  learn these general solutions when the training enforced a specific "routine" capable of breaking down the long-range relationships.Our results highlights the need for new training strategies that decompose the learning into appropriate stages, and that lead to the general class of solutions necessary for DNNs to understand insideness.

DNNs für die Bildsegmentierung können Lösungen für das Insideness-Problem implementieren, aber nur einige rekurrente Netze können sie mit einer bestimmten Art von Überwachung lernen.

In diesem Beitrag wird die Innensicht zur Untersuchung der semantischen Segmentierung in der Ära des Deep Learning eingeführt, und die Ergebnisse können dazu beitragen, dass Modelle besser verallgemeinert werden.


We address the challenging problem of deep representation learning--the efficient adaption of a pre-trained deep network to different tasks.Specifically, we propose to explore gradient-based features.These features are gradients of the model parameters with respect to a task-specific loss given an input sample.Our key innovation is the design of a linear model that incorporates both gradient features and the activation of the network.We show that our model provides a local linear approximation to a underlying deep model, and discuss important theoretical insight.Moreover, we present an efficient algorithm for the training and inference of our model without computing the actual gradients.Our method is evaluated across a number of representation learning tasks on several datasets and using different network architectures.We demonstrate strong results in all settings.And our results are well-aligned with our theoretical insight.

Ausgehend von einem trainierten Modell untersuchten wir die Gradienten der Modellparameter pro Stichprobe im Verhältnis zu einem aufgabenspezifischen Verlust und konstruierten ein lineares Modell, das die Gradienten der Modellparameter und die Aktivierung des Modells kombiniert.

In diesem Beitrag wird vorgeschlagen, die Gradienten bestimmter Schichten von Convolutional Networks als Merkmale in einem linearisierten Modell für Transferlernen und schnelle Anpassung zu verwenden.


Recovering 3D geometry shape, albedo and lighting from a single image has wide applications in many areas, which is also a typical ill-posed problem.In order to eliminate the ambiguity, face prior knowledge like linear 3D morphable models (3DMM) learned from limited scan data are often adopted to the reconstruction process.However, methods based on linear parametric models cannot generalize well for facial images in the wild with various ages, ethnicity, expressions, poses, and lightings.Recent methods aim to learn a nonlinear parametric model using convolutional neural networks (CNN) to regress the face shape and texture directly.However, the models were only trained on a dataset that is generated from a linear 3DMM.Moreover, the identity and expression representations are entangled in these models, which hurdles many facial editing applications.In this paper, we train our model with adversarial loss in a semi-supervised manner on hybrid batches of unlabeled and labeled face images to exploit the value of large amounts of unlabeled face images from unconstrained photo collections.A novel center loss is introduced to make sure that different facial images from the same person have the same identity shape and albedo.Besides, our proposed model disentangles identity, expression, pose, and lighting representations, which improves the overall reconstruction performance and facilitates facial editing applications, e.g., expression transfer.Comprehensive experiments demonstrate that our model produces high-quality reconstruction compared to state-of-the-art methods and is robust to various expression, pose, and lighting conditions.

Wir trainieren unser Gesichtsrekonstruktionsmodell mit adversarialem Verlust in halbüberwachter Weise auf hybriden Batches von unbeschrifteten und beschrifteten Gesichtsbildern, um den Wert großer Mengen unbeschrifteter Gesichtsbilder aus unbeschränkten Fotosammlungen zu nutzen.

In dieser Arbeit wird ein halbüberwachtes und kontradiktorisches Trainingsverfahren vorgeschlagen, um nichtlineare entwirrte Repräsentationen von Gesichtsbildern mit Verlustfunktionen genau zu bestimmen und so die beste Leistung bei der Gesichtsrekonstruktion zu erzielen.


Human conversations naturally evolve around related entities and connected concepts, while may also shift from topic to topic.This paper presents ConceptFlow, which leverages commonsense knowledge graphs to explicitly model such conversation flows for better conversation response generation.ConceptFlow grounds the conversation inputs to the latent concept space and represents the potential conversation flow as a concept flow along the commonsense relations.The concept is guided by a graph attention mechanism that models the possibility of the conversation evolving towards different concepts.The conversation response is then decoded using the encodings of both utterance texts and concept flows, integrating the learned conversation structure in the concept space.Our experiments on Reddit conversations demonstrate the advantage of ConceptFlow over previous commonsense aware dialog models and fine-tuned GPT-2 models, while using much fewer parameters but with explicit modeling of conversation structures.

In diesem Beitrag wird ConceptFlow vorgestellt, das den Gesprächsfluss explizit in einem Commonsense-Wissensgraphen modelliert, um eine bessere Gesprächsgenerierung zu ermöglichen.

Die Arbeit schlägt ein System für die Generierung einer Single-Turn-Antwort auf eine gepostete Äußerung in einer offenen Domäne Dialog Einstellung mit der Diffiusion in die Nachbarn der geerdeten Konzepte.


Biological neural networks face homeostatic and resource constraints that restrict the allowed configurations of connection weights.If a constraint is tight it defines a very small solution space, and the size of these constraint spaces determines their potential overlap with the solutions for computational tasks.We study the geometry of the solution spaces for constraints on neurons' total synaptic weight and on individual synaptic weights, characterizing the connection degrees (numbers of partners) that maximize the size of these solution spaces.We then hypothesize that the size of constraints' solution spaces could serve as a cost function governing neural circuit development.We develop analytical approximations and bounds for the model evidence of the maximum entropy degree distributions under these cost functions.We test these on a published electron microscopic connectome of an associative learning center in the fly brain, finding evidence for a developmental progression in circuit structure.

Wir untersuchen die Hypothese, dass die Entropie von Lösungsräumen für Beschränkungen der synaptischen Gewichte (die "Flexibilität" der Beschränkung) als Kostenfunktion für die Entwicklung neuronaler Schaltkreise dienen könnte.


In  this  preliminary  work,  we  study  the  generalization  properties  of  infinite  ensembles  of infinitely-wide neural networks.  Amazingly, this model family admits tractable calculations for many information-theoretic quantities.  We report analytical and empirical investigations in the search for signals that correlate with generalization.

Unendliche Ensembles von unendlich großen neuronalen Netzen sind aus informationstheoretischer Sicht eine interessante Modellfamilie.


Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks.There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns unified multilingual representations using monolingual and cross-lingual objectives jointly.In this paper, we first conduct direct comparisons of representations learned using both of these methods across diverse cross-lingual tasks.Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training is task-dependent.Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches.Extensive experiments on various tasks demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark.We further show that our proposed framework can generalize to contextualized representations and achieves state-of-the-art results on the CoNLL cross-lingual NER benchmark.

Wir führen eine vergleichende Studie über sprachübergreifendes Alignment und gemeinsame Trainingsmethoden durch und vereinen diese beiden bisher exklusiven Paradigmen in einem neuen Rahmen.

Dieser Beitrag vergleicht Ansätze zur zweisprachigen Lexikoninduktion und zeigt, welche Methode bei Lexikon, Induktion, NER- und MT-Aufgaben besser abschneidet.


Large number of weights in deep neural networks make the models difficult to be deployed in low memory environments such as, mobile phones, IOT edge devices as well as "inferencing as a service" environments on the cloud. Prior work has considered reduction in the size of the models, through compression techniques like weight pruning, filter pruning, etc. or through low-rank decomposition of the convolution layers.In this paper, we demonstrate the use of multiple techniques to achieve not only higher model compression but also reduce the compute resources required during inferencing.We do filter pruning followed by low-rank decomposition using Tucker decomposition for model compression.We show that our approach achieves upto 57\% higher model compression when compared to either Tucker Decomposition or Filter pruning alone  at similar accuracy for GoogleNet.Also, it reduces the Flops by upto 48\% thereby making the inferencing faster.

Kombination orthogonaler Modellkomprimierungstechniken, um die Modellgröße und die Anzahl der bei der Inferenzierung erforderlichen Flops erheblich zu verringern.

In dieser Arbeit wird eine Kombination aus Tucker Zerlegung und Filter Pruning vorgeschlagen.


We review the limitations of BLEU and ROUGE -- the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce JAUNE:  a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries against hypothesis summaries.

Einführen von JAUNE, einer Methode die BLEU- und ROUGE-Punkte durch multidimensionale, modellbasierte Evaluatoren zur Bewertung von Zusammenfassungen ersetzt.

In diesem Beitrag wird eine neue JAUNE-Metrik für die Bewertung von maschinellen Übersetzungs- und Textzusammenfassungssystemen vorgeschlagen, die zeigt, dass ihr Modell besser mit den tatsächlichen Ähnlichkeitsbezeichnungen übereinstimmt als BLEU.


This paper presents a new Graph Neural Network (GNN) type using feature-wise linear modulation (FiLM).Many standard GNN variants propagate information along the edges of a graph by computing ``messages'' based only on the representation of the source of each edge.In GNN-FiLM, the representation of the target node of an edge is additionally used to compute a transformation that can be applied to all incoming messages, allowing feature-wise modulation of the passed information.Results of experiments comparing different GNN architectures on three tasks from the literature are presented, based on re-implementations of baseline methods.Hyperparameters for all methods were found using extensive search, yielding somewhat surprising results: differences between baseline models are smaller than reported in the literature.Nonetheless, GNN-FiLM outperforms baseline methods on a regression task on molecular graphs and performs competitively on other tasks.

Neuer GNN-Formalismus mit umfangreichen Experimenten; die Unterschiede zwischen GGNN/GCN/GAT sind geringer als angenommen.

Die Arbeit schlägt eine neue Graph Neural Network Architektur vor, die eine merkmalsweise lineare Modulation verwendet, um die Weiterleitung von Nachrichten von der Quelle zum Zielknoten auf der Grundlage der Darstellung des Zielknotens zu konditionieren.


To deal simultaneously with both, the attributed network embedding and clustering, we propose a new model.It exploits both content and structure information, capitalising on their simultaneous use.The proposed model relies on the approximation of the relaxed continuous embedding solution by the true discrete clustering one.Thereby, we show that incorporating an embedding representation provides simpler and more interpretable solutions.Experiment results demonstrate that the proposed algorithm performs better, in terms of clustering and embedding, than the state-of-art algorithms, including deep learning methods devoted to similar tasks for attributed network datasets with different proprieties.

In diesem Beitrag wird ein neuartiger Matrixzerlegungsrahmen für die gleichzeitige Einbettung und Clustering von Netzwerkdaten vorgeschlagen.

In dieser Arbeit wird ein Algorithmus vorgeschlagen, der die Einbettung von Attributen in ein Netzwerk und das Clustering gemeinsam durchführt.


We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis.The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g., virtual showrooms, virtual tours and sightseeing, the digital inspection of historical artifacts).A core component of our work is the handling of view-dependent effects.Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object.As input data we are using an RGB video of the object.This video is used to reconstruct a proxy geometry of the object via multi-view stereo.Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts.To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects.Based on these estimations, we are able to convert observed images to diffuse images.These diffuse images can be projected into other views.In the target view, our pipeline reinserts the new view-dependent effects.To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results.Using this image-guided approach, the network does not have to allocate capacity on ``remembering'' object appearance, instead it learns how to combine the appearance of captured images.We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data.

Wir schlagen eine erlernte bildgesteuerte Rendering-Technik vor, die die Vorteile von bildbasiertem Rendering und GAN-basierter Bildsynthese kombiniert und gleichzeitig sichtabhängige Effekte berücksichtigt.

In diesem Beitrag wird eine Methode zur Behandlung von sichtabhängigen Effekten beim neuronalen Rendering vorgeschlagen, die die Robustheit bestehender neuronaler Renderingmethoden verbessert.


We evaluate the distribution learning capabilities of generative adversarial networks by testing them on synthetic datasets.The datasets include common distributions of points in $R^n$ space and images containing polygons of various shapes and sizes.We find that by and large GANs fail to faithfully recreate point datasets which contain discontinous support or sharp bends with noise.Additionally, on image datasets, we find that GANs do not seem to learn to count the number of objects of the same kind in an image.We also highlight the apparent tension between generalization and learning in GANs.

GANs werden anhand synthetischer Datensätze bewertet.


This paper proposes a new approach for step size adaptation in gradient methods.The proposed method called step size optimization (SSO) formulates the step size adaptation as an optimization problem which minimizes the loss function with respect to the step size for the given model parameters and gradients.Then, the step size is optimized based on alternating direction method of multipliers (ADMM).SSO does not require the second-order information or any probabilistic models for adapting the step size, so it is efficient and easy to implement.Furthermore, we also introduce stochastic SSO for stochastic learning environments.In the experiments, we integrated SSO to vanilla SGD and Adam, and they outperformed state-of-the-art adaptive gradient methods including RMSProp, Adam, L4-Adam, and AdaBound on extensive benchmark datasets.

Wir schlagen ein effizientes und effektives Verfahren zur Anpassung der Schrittweite für die Gradientenmethoden vor.

Eine neue Schrittgrößenanpassung in Gradientenmethoden erster Ordnung, die ein neues Optimierungsproblem mit der Erweiterung der Verlustfunktion erster Ordnung und Regularisierung aufstellt, wobei die Schrittgröße als Variable behandelt wird.


Despite the fact that generative models are extremely successful in practice, the theory underlying this phenomenon is only starting to catch up with practice.In this work we address the question of the universality of generative models: is it true that neural networks can approximate any data manifold arbitrarily well?We provide a positive answer to this question and show that under mild assumptions on the activation function one can always find a feedforward neural network that maps the latent space onto a set located within the specified Hausdorff distance from the desired data manifold.We also prove similar theorems for the case of multiclass generative models and cycle generative models, trained to map samples from one manifold to another and vice versa.

Wir haben festgestellt, dass eine große Klasse von Vielfältigkeiten durch ReLU- und Sigmoid-Netzwerke mit beliebiger Genauigkeit erzeugt werden kann.

Diese Arbeit bietet bestimmte grundlegende Garantien dafür, wann Vielfältigkeiten als das Bild einer Karte geschrieben werden können, die durch ein neuronales Netz approximiert wird, und verbindet Theoreme aus der Mannigfaltigkeitsgeometrie und universelle Standard-Approximationsergebnisse.

In diesem Beitrag wird theoretisch gezeigt, dass generative Modelle, die auf neuronalen Netzen basieren, Datenmannigfaltigkeiten annähern können, und es wird nachgewiesen, dass neuronale Netze unter milden Annahmen einen latenten Raum auf eine Menge abbilden können, die der gegebenen Datenmannigfaltigkeit innerhalb einer kleinen Hausdorff-Distanz nahe kommt.


Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL.However, the theoretical understanding of such methods has been rather limited.This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algorithms with theoretical guarantees.We design a meta-algorithm with a theoretical guarantee of monotone improvement to a local maximum of the expected reward.The meta-algorithm iteratively builds a lower bound of the expected reward based on the estimated dynamical model and sample trajectories, and then maximizes the lower bound jointly over the policy and the model.The framework extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires no explicit uncertainty quantification.Instantiating our framework with simplification gives a  variant of model-based RL algorithms Stochastic Lower Bounds Optimization (SLBO).Experiments demonstrate that SLBO achieves the state-of-the-art performance when only 1M or fewer samples are permitted on a range of continuous control benchmark tasks.

Wir entwerfen modellbasierte Reinforcement Learning Algorithmen mit theoretischen Garantien und erzielen die besten Ergebnisse bei Mujuco-Benchmark-Aufgaben, wenn eine Million oder weniger Beispiele zulässig sind.

In der Arbeit wird ein Rahmen für die Entwicklung modellbasierter RL-Algorithmen auf der Grundlage von OFU vorgeschlagen, die eine SOTA-Leistung bei MuJoCo-Aufgaben erreichen.


We study the use of knowledge distillation to compress the U-net architecture.We show that, while standard distillation is not sufficient to reliably train a compressed U-net, introducing other regularization methods, such as batch normalization and class re-weighting, in knowledge distillation significantly improves the training process.This allows us to compress a U-net by over 1000x, i.e., to 0.1% of its original number of parameters, at a negligible decrease in performance.

Wir stellen zusätzliche Techniken zur Wissensdestillation vor, um das U-Netz um mehr als das 1000-fache zu komprimieren.

Die Autoren führten eine modifizierte Destillationsstrategie ein, um eine U-Netz-Architektur um mehr als das 1000-fache zu komprimieren und dabei eine Genauigkeit beizubehalten, die der des ursprünglichen U-Netzes nahe kommt.


Learning neural networks with gradient descent over a long sequence of tasks is problematic as their fine-tuning to new tasks overwrites the network weights that are important for previous tasks.This leads to a poor performance on old tasks â€“ a phenomenon framed as catastrophic forgetting.  While early approaches use task rehearsal and growing networks that both limit the scalability of the task sequence orthogonal approaches build on regularization.  Based on the Fisher information matrix (FIM) changes to parameters that are relevant to old tasks are penalized, which forces the task to be mapped into the available remaining capacity of the network.This requires to calculate the Hessian around a mode, which makes learning tractable.In this paper, we introduce Hessian-free curvature estimates as an alternative method to actually calculating the Hessian.  In contrast to previous work, we exploit the fact that most regions in the loss surface are flat and hence only calculate a Hessian-vector-product around the surface that is relevant for the current task.Our experiments show that on a variety of well-known task sequences we either significantly outperform or are en par with previous work.

Diese Arbeit bietet einen Ansatz zur Bekämpfung des katastrophalen Vergessens durch hessianfreie Krümmungsschätzungen.

Die Arbeit schlägt eine ungefähre Laplace-Methode für das Training neuronaler Netze im Rahmen des kontinuierlichen Lernens mit einer geringen Raumkomplexität vor.


There has been recent interest in improving performance of simple models for multiple reasons such as interpretability, robust learning from small data, deployment in memory constrained settings as well as environmental considerations.In this paper, we propose a novel method SRatio that can utilize information from high performing complex models (viz. deep neural networks, boosted trees, random forests) to reweight a training dataset for a potentially low performing simple model such as a decision tree or a shallow network enhancing its performance.Our method also leverages the per sample hardness estimate of the simple model which is not the case with the prior works which primarily consider the complex model's confidences/predictions and is thus conceptually novel.Moreover, we generalize and formalize the concept of attaching probes to intermediate layers of a neural network, which was one of the main ideas in previous work \citep{profweight}, to other commonly used classifiers and incorporate this into our method.The benefit of these contributions is witnessed in the experiments where on 6 UCI datasets and CIFAR-10 we outperform competitors in a majority (16 out of 27) of the cases and tie for best performance in the remaining cases.In fact, in a couple of cases, we even approach the complex model's performance.We also conduct further experiments to validate assertions and intuitively understand why our method works.Theoretically, we motivate our approach by showing that the weighted loss minimized by simple models using our weighting upper bounds the loss of the complex model.

Methode zur Verbesserung der Leistung einfacher Modelle anhand eines (genauen) komplexen Modells.

In der Arbeit wird eine Methode zur Verbesserung der Vorhersagen eines Modells mit geringer Kapazität vorgeschlagen, die Vorteile gegenüber bestehenden Ansätzen aufweist.


We propose a principled method for kernel learning, which relies on a Fourier-analytic characterization of translation-invariant or rotation-invariant kernels.Our method produces a sequence of feature maps, iteratively refining the SVM margin.We provide rigorous guarantees for optimality and generalization, interpreting our algorithm as online equilibrium-finding dynamics in a certain two-player min-max game.Evaluations on synthetic and real-world datasets demonstrate scalability and consistent improvements over related random features-based methods.

Ein einfacher und praktischer Algorithmus zum Erlernen eines margenmaximierenden translationsinvarianten oder sphärisch symmetrischen Kernels aus Trainingsdaten, unter Verwendung von Werkzeugen der Fourier-Analyse und der Regret-Minimierung.

Die Arbeit schlägt vor, einen benutzerdefinierten übersetzungs- oder drehungsinvarianten Kern in der Fourier-Darstellung zu lernen, um die Marge der SVM zu maximieren.

Die Autoren schlagen einen interessanten Algorithmus für das gemeinsame Lernen des l1-SVM und des Fourier-Kernels vor.

Die Autoren betrachten das Erlernen direkter Fourier-Darstellungen von verschiebungs- und translationsinvarianten Kerneln für Anwendungen des maschinellen Lernens, wobei die Ausrichtung des Kernels auf die Daten die zu optimierende Zielfunktion darstellt.


We elaborate on using importance sampling for causal reasoning, in particular for counterfactual inference.We show how this can be implemented natively in probabilistic programming.By considering the structure of the counterfactual query, one can significantly optimise the inference process.We also consider design choices to enable further optimisations.We introduce MultiVerse, a probabilistic programming prototype engine for approximate causal reasoning.We provide experimental results and compare with Pyro, an existing probabilistic programming framework with some of causal reasoning tools.

Probabilistische Programmierung, die kausale, kontrafaktische Inferenz unterstützt.


We consider the problem of representing collective behavior of large populations and predicting the evolution of a population distribution over a discrete state space.A discrete time mean field game (MFG) is motivated as an interpretable model founded on game theory for understanding the aggregate effect of individual actions and predicting the temporal evolution of population distributions.We achieve a synthesis of MFG and Markov decision processes (MDP) by showing that a special MFG is reducible to an MDP.This enables us to broaden the scope of mean field game theory and infer MFG models of large real-world systems via deep inverse reinforcement learning.Our method learns both the reward function and forward dynamics of an MFG from real data, and we report the first empirical test of a mean field game model of a real-world social media population.

Inferenz eines Mean Field Game (MFG)-Modells für das Verhalten großer Populationen durch eine Synthese von MFG und Markov-Entscheidungsprozessen.

Die Autoren befassen sich mit der Inferenz in Modellen für kollektives Verhalten, indem sie inverses Reinforcement Learning verwenden, um die Belohnungsfunktionen der Agenten im Modell zu lernen.


We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as  offensive basketball gameplay.  When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables.  Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulable way.We present a hierarchical framework that can effectively learn such sequential generative models.  Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime.In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods.We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.

Wir kombinieren tiefe generative Modelle mit programmgesteuerter schwacher Überwachung, um koordinierte Multi-Agenten-Trajektorien von deutlich höherer Qualität zu erzeugen als frühere Basislösungen.

Schlägt sequentielle generative Multi-Agenten-Modelle vor.

Die Arbeit schlägt vor, generative Modelle zu trainieren, die Multi-Agenten-Trajektorien unter Verwendung heuristischer Funktionen erzeugen, die Variablen kennzeichnen, die ansonsten in den Trainingsdaten verborgen wären.


Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, are computationally expensive because they involve training many different model configurations.In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training.In contrast to existing methods, we consider this task as a ranking and transfer learning problem.We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many or very long learning curves.We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without a significant performance degradation of the discovered architecture.In further experiments we analyze the quality of ranking, the influence of different model components as well as the predictive behavior of the model.

Lernen, Lernkurven zu ordnen, um vielversprechende Trainingsaufgaben frühzeitig zu beenden. Neuheit: Verwendung von paarweisen Rangfolgeverlusten zur direkten Modellierung der Wahrscheinlichkeit von Verbesserungs- und Transferlernen über Datensätze hinweg, um die erforderlichen Trainingsdaten zu reduzieren.

In der Arbeit wird eine Methode zur Einstufung von Lernkurven neuronaler Netze vorgeschlagen, die Lernkurven über verschiedene Datensätze hinweg modellieren kann, um höhere Geschwindigkeitssteigerungen bei Bildklassifizierungsaufgaben zu erzielen.


Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster.Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions.This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills.We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence.Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience.While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution - that of using experience replay buffers for all past events - with a mixture of on- and off-policy learning, leveraging behavioral cloning.We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities.When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.

Wir zeigen, dass in kontinuierlichen Lernumgebungen katastrophales Vergessen vermieden werden kann, indem man RL mit einer Mischung aus neuer und wiederholter Erfahrung anwendet, mit einem Verhaltensklonverlust.

Er schlägt eine besondere Variante der Erfahrungswiederholung mit Verhaltensklonen als Methode für kontinuierliches Lernen vor.


We present a method which learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents.Our method is based on a graph-structured variational recurrent neural network, which is trained end-to-end to infer the current state of the (partially observed) world, as well as to forecast future states.We show that our method outperforms various baselines on two sports datasets, one based on real basketball trajectories, and one generated by a soccer game engine.

Wir stellen eine Methode vor, die lernt, zeitliche Informationen und mehrdeutige visuelle Informationen im Kontext von interagierenden Agenten zu integrieren.

Die Autoren schlagen Graph VRNN vor, das die Interaktion mehrerer Agenten modelliert, indem für jeden Agenten ein VRNN eingesetzt wird.

In diesem Beitrag wird eine auf einem neuronalen Graphennetz basierende Architektur vorgestellt, die trainiert wird, um die Interaktionen von Agenten in einer Umgebung direkt aus Pixeln zu lokalisieren und zu modellieren, und die Vorteile des Modells für die Verfolgung von Aufgaben und die Vorhersage von Agentenstandorten aufzeigt.


In this paper we study the problem of learning the weights of a deep convolutional neural network.We consider a network where convolutions are carried out over non-overlapping patches with a single kernel in each layer.We develop an algorithm for simultaneously learning all the kernels from the training data.Our approach dubbed Deep Tensor Decomposition (DeepTD) is based on a rank-1 tensor decomposition.We theoretically investigate DeepTD under a realizable model for the training data where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted convolutional kernels.We show that DeepTD is data-efficient and provably works as soon as the sample size exceeds the total number of convolutional weights in the network.Our numerical experiments demonstrate the effectiveness of DeepTD and verify our theoretical findings.

Wir betrachten ein vereinfachtes Modell eines tiefen Convolutional Neural Networks. Wir zeigen, dass alle Schichten dieses Netzes mit einer geeigneten Anwendung der Tensorzerlegung annähernd gelernt werden können.

Bietet theoretische Garantien für das Lernen von tiefen faConvolutional Neural Networks unter Verwendung der Rang-Eins-Tensorzerlegung.

In dieser Arbeit wird eine Lernmethode für einen eingeschränkten Fall von tiefen Convolutional Networks vorgeschlagen, bei dem die Schichten auf den nicht überlappenden Fall beschränkt sind und nur einen Ausgangskanal pro Schicht haben.

Analysiert das Problem des Lernens einer sehr speziellen Klasse von CNNs: Jede Schicht besteht aus einem einzigen Filter, der auf nicht überlappende Bereiche der Eingabe angewendet wird.


Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively.Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations.The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations.We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10.Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

Neuronale Netze mit Vorwärtskopplung, bei denen die Gewichte nach dem Training pruned werden können, könnten die gleichen Gewichte vor dem Training gepruned worden sein.

Zeigt, dass es spärliche Subnetze gibt, die von Grund auf mit guter Generalisierungsleistung trainiert werden können, und schlägt unpruned, zufällig initialisierte NNs enthaltende Subnetze vor, die von Grund auf mit ähnlicher Generalisierungsgenauigkeit trainiert werden können.

In der Arbeit wird die Hypothese untersucht, dass zufällig initialisierte neuronale Netze Teilnetze enthalten, die gleich schnell oder schneller konvergieren und die gleiche oder eine bessere Klassifizierungsgenauigkeit erreichen können.


We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime.Recent work of \citep{Gale2019, Liu2018} has shown that sparse ResNet-50 architectures trained on ImageNet-2012 dataset converge to solutions that are significantly worse than those found by pruning.We show that, despite the failure of optimizers, there is a linear path with a monotonically decreasing objective from the initialization to the ``good'' solution.Additionally, our attempts to find a decreasing objective path from ``bad'' solutions to the ``good'' ones in the sparse subspace fail.However, if we allow the path to traverse the dense subspace, then we consistently find a path between two solutions.These findings suggest traversing extra dimensions may be needed to escape stationary points found in the sparse subspace.

In diesem Beitrag wird die Schwierigkeit des Trainings spärlicher neuronaler Netze durch Interpolationsexperimente in der Energielandschaft aufgezeigt


Neural network training depends on the structure of the underlying loss landscape, i.e. local minima, saddle points, flat plateaus, and loss barriers.In relation to the structure of the landscape, we study the permutation symmetry of neurons in each layer of a deep neural network, which gives rise not only to multiple equivalent global minima of the loss function but also to critical points in between partner minima.In a network of $d-1$ hidden layers with $n_k$ neurons in layers $k = 1, \ldots, d$, we construct continuous paths between equivalent global minima that lead through a `permutation point' where the input and output weight vectors of two neurons in the same hidden layer $k$ collide and interchange.We show that such permutation points are critical points which lie inside high-dimensional subspaces of equal loss, contributing to the global flatness of the landscape.We also find that a permutation point for the exchange of neurons $i$ and $j$ transits into a flat high-dimensional plateau that enables all $n_k!$permutations of neurons in a given layer $k$ at the same loss value. Moreover, we introduce higher-order permutation points by exploiting the hierarchical structure in the loss landscapes of neural networks, and find that the number of $K$-th order permutation points is much larger than the (already huge) number of equivalent global minima -- at least by a polynomial factor of order $K$. In twotasks, we demonstrate numerically with our path finding method that continuous paths between partner minima exist: first, in a toy network with a single hidden layer on a function approximation task and, second, in a multilayer network on the MNIST task. Our geometricapproach yields a lower bound on the number of critical points generated by weight-space symmetries and provides a simple intuitive link between previous theoretical results and numerical observations.

Die Gewichtsraum-Symmetrie in den Landschaften neuronaler Netze führt zu einer Vielzahl von Sätteln und flachen hochdimensionalen Unterräumen.

In der Arbeit wird eine verlustarme Methode zur Untersuchung der Verlustfunktion in Bezug auf die Parameter eines neuronalen Netzes unter dem Gesichtspunkt der Gewichtsraumsymmetrie vorgestellt.


The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated.We derive, using mean field theory, a set of scalar equations describing how input signals propagate through surrogate networks.The equations reveal that depending on the choice of surrogate model, the networks may or may not exhibit an order to chaos transition, and the presence of depth scales that limit the maximum trainable depth.Specifically, in solving the equations for edge of chaos conditions, we show that surrogates derived using the Gaussian local reparameterisation trick have no critical initialisation, whereas a deterministic surrogates based on analytic Gaussian integration do.The theory is applied to a range of binary neuron and weight design choices, such as different neuron noise models, allowing the categorisation of algorithms in terms of their behaviour at initialisation.Moreover, we predict theoretically and confirm numerically, that common weight initialization schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance.This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to close to $\pm 1$ for deeper networks to be trainable.

Signalausbreitungstheorie angewandt auf kontinuierliche Surrogates binärer Netze; kontraintuitive Initialisierung; Reparameterisierungstrick nicht hilfreich.

Die Autoren untersuchen die Trainingsdynamik binärer neuronaler Netze bei Verwendung kontinuierlicher Surrogates, untersuchen, welche Eigenschaften Netze bei der Initialisierung haben sollten, um optimal zu trainieren, und geben konkrete Ratschläge zu stochastischen Gewichten bei der Initialisierung.

Eine eingehende Untersuchung von stochastischen binären Netzen, kontinuierlichen Surrogaten und ihrer Trainingsdynamik, mit Einblicken in die Initialisierung von Gewichten für beste Leistung.


Semantic dependency parsing, which aims to find rich bi-lexical relationships, allows words to have multiple dependency heads, resulting in graph-structured representations.We propose an approach to semi-supervised learning of semantic dependency parsers based on the CRF autoencoder framework.Our encoder is a discriminative neural semantic dependency parser that predicts the latent parse graph of the input sentence.Our decoder is a generative neural model that reconstructs the input sentence conditioned on the latent parse graph.Our model is arc-factored and therefore parsing and learning are both tractable.Experiments show our model achieves significant and consistent improvement over the supervised baseline.

Wir schlagen einen Ansatz zum halbüberwachten Lernen von semantischen Dependenzparsern vor, der auf dem CRF-Autoencoder-Rahmen basiert.

Diese Arbeit konzentriert sich auf semi-supervised semantische Abhängigkeit Parsing mit dem CRF-Auto-Kodierer, um das Modell in einem semi-supervised Stil zu trainieren, zeigt die Wirksamkeit auf niedrigen Ressource gelabelten Daten Aufgaben.


For sequence models with large word-level vocabularies, a majority of network parameters lie in the input and output layers.In this work, we describe a new method, DeFINE, for learning deep word-level representations efficiently.Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods.DeFINE can be incorporated easily in new or existing sequence models.Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity.On WikiText-103, DeFINE reduces total parameters of Transformer-XL by half with minimal impact on performance.On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters,  achieving comparable performance to state-of-the-art methods with fewer parameters.For machine translation, DeFINE improves a Transformer model by 2% while simultaneously reducing total parameters by 26%

DeFINE verwendet ein tiefes, hierarchisches, spärliches Netzwerk mit neuen Sprungverbindungen, um effizient bessere Worteinbettungen zu lernen.

Dieses Papier beschreibt eine neue Methode zum effizienten Erlernen tiefer Wortrepräsentationen durch die Verwendung einer hierarchischen Struktur mit Skip-Verbindungen für die Verwendung von niedrigdimensionalen Eingabe- und Ausgabeschichten.


In this paper, we present a reproduction of the paper of Bertinetto et al. [2019] "Meta-learning with differentiable closed-form solvers" as part of the ICLR 2019 Reproducibility Challenge.In successfully reproducing the most crucial part of the paper, we reach a performance that is comparable with or superior to the original paper on two benchmarks for several settings.We evaluate new baseline results, using a new dataset presented in the paper.Yet, we also provide multiple remarks and recommendations about reproducibility and comparability.  After we brought our reproducibility work to the authorsâ€™ attention, they have updated the original paper on which this work is based and released code as well.Our contributions mainly consist in reproducing the most important results of their original paper, in giving insight in the reproducibility and in providing a first open-source implementation.

Wir reproduzieren erfolgreich einen Meta-Learning-Ansatz für die Klassifizierung von wenigen Aufnahmen, der durch Backpropagieren der Lösung einer geschlossenen Form funktioniert, und geben Anmerkungen zum Vergleich mit Baselines.


Network pruning has emerged as a powerful technique for reducing the size of deep neural networks.Pruning uncovers high-performance subnetworks by taking a trained dense network and gradually removing unimportant connections.Recently, alternative techniques have emerged for training sparse networks directly without having to train a large dense model beforehand, thereby achieving small memory footprints during both training and inference.These techniques are based on dynamic reallocation of non-zero parameters during training.Thus, they are in effect executing a training-time search for the optimal subnetwork.We investigate a most recent one of these techniques and conduct additional experiments to elucidate its behavior in training sparse deep convolutional networks.Dynamic parameter reallocation converges early during training to a highly trainable subnetwork.We show that neither the structure, nor the initialization of the discovered high-performance subnetwork is sufficient to explain its good performance.Rather, it is the dynamics of parameter reallocation that are responsible for successful learning.Dynamic parameter reallocation thus improves the trainability of deep convolutional networks, playing a similar role as overparameterization, without incurring the memory and computational cost of the latter.

Die dynamische Neuzuweisung von Parametern ermöglicht ein erfolgreiches direktes Training kompakter spärlicher Netze und spielt selbst dann eine unverzichtbare Rolle, wenn wir das optimale spärliche Netz von vornherein kennen.


n this paper we present a thrust in three directions of visual development us- ing supervised and semi-supervised techniques.The first is an implementation of semi-supervised object detection and recognition using the principles of Soft At- tention and Generative Adversarial Networks (GANs).The second and the third are supervised networks that learn basic concepts of spatial locality and quantity respectively using Convolutional Neural Networks (CNNs).The three thrusts to- gether are based on the approach of Experiential Robot Learning, introduced in previous publication.While the results are unripe for implementation, we believe they constitute a stepping stone towards autonomous development of robotic vi- sual modules.

3 Schwerpunkte, die als Sprungbretter für das Roboter-Erfahrungslernen des Bildverarbeitungsmoduls dienen.

Untersucht die Leistung bestehender Bildklassifizierer und Objektdetektoren.


Characterization of the representations learned in intermediate layers of deep networks can provide valuable insight into the nature of a task and can guide the development of well-tailored learning strategies.Here we study convolutional neural network-based acoustic models in the context of automatic speech recognition.Adapting a method proposed by Yosinski et al. [2014], we measure the transferability of each layer between German and English to assess the their language-specifity.We observe three distinct regions of transferability: (1) the first two layers are entirely transferable between languages, (2) layers 2â€“8 are also highly transferable but we find evidence of some language specificity, (3) the subsequent fully connected layers are more language specific but can be successfully finetuned to the target language.To further probe the effect of weight freezing, we performed follow-up experiments using freeze-training [Raghu et al., 2017].Our results are consistent with the observation that CCNs converge 'bottom up' during training and demonstrate the benefit of freeze training, especially for transfer learning.

Mit Ausnahme der ersten beiden Schichten unserer CNN-basierten akustischen Modelle wiesen alle ein gewisses Maß an Sprachspezifität auf, aber das Einfriertraining ermöglichte einen erfolgreichen Transfer zwischen den Sprachen.

Die Arbeit misst die Übertragbarkeit von Merkmalen für jede Schicht in CNN-basierten akustischen Modellen über verschiedene Sprachen hinweg und kommt zu dem Schluss, dass AMs, die mit der Technik des "Freeze-Trainings" trainiert wurden, andere übertragene Modelle übertreffen.


Policy gradients methods often achieve better performance when the change in policy is limited to a small Kullback-Leibler divergence.We derive policy gradients where the change in policy is limited to a small Wasserstein distance (or trust region).This is done in the discrete and continuous multi-armed bandit settings with entropy regularisation.We show that in the small steps limit with respect to the Wasserstein distance $W_2$, policy dynamics are governed by the heat equation, following the Jordan-Kinderlehrer-Otto result.This means that policies undergo diffusion and advection, concentrating near actions with high reward.This helps elucidate the nature of convergence in the probability matching setup, and provides justification for empirical practices such as Gaussian policy priors and additive gradient noise.

Verknüpfung von Wasserstein-Vertrauensgebieten, entropischen politischen Gradienten und der Wärmegleichung.

Die Arbeit untersucht die Verbindungen zwischen Reinforcement Learning und der Theorie des quadratischen optimalen Transports.

Die Autoren untersuchten den Policy-Gradienten mit einem Wechsel der Policies, der durch eine Vertrauensregion mit Wasserstein-Distanz im Multi-Armed-Bandit-Setting begrenzt ist, und zeigten, dass die Policy-Dynamik im Grenzbereich kleiner Schritte durch die Wärmegleichung (Fokker-Planck-Gleichung) bestimmt wird.


The softmax function is widely used to train deep neural networks for multi-class classification.Despite its outstanding performance in classification tasks, the features derived from the supervision of softmax are usually sub-optimal in some scenarios where Euclidean distances apply in feature spaces.To address this issue, we propose a new loss, dubbed the isotropic loss, in the sense that the overall distribution of data points is regularized to approach the isotropic normal one.Combined with the vanilla softmax, we formalize a novel criterion called the isotropic softmax, or isomax for short, for supervised learning of deep neural networks.By virtue of the isomax, the intra-class features are penalized by the isotropic loss while inter-class distances are well kept by the original softmax loss.Moreover, the isomax loss does not require any additional modifications to the network, mini-batches or the training process.Extensive experiments on classification and clustering are performed to demonstrate the superiority and robustness of the isomax loss.

Die Unterscheidungsfähigkeit von Softmax beim Lernen von Merkmalsvektoren von Objekten wird durch die isotrope Normalisierung der globalen Verteilung der Datenpunkte effektiv verbessert.


A fundamental question in reinforcement learning is whether model-free algorithms are sample efficient.Recently,  Jin et al. (2018) proposed a Q-learning algorithm with UCB exploration policy, and proved it has nearly optimal regret bound for finite-horizon episodic MDP.In this paper, we adapt Q-learning with UCB-exploration bonus to infinite-horizon MDP with discounted rewards \emph{without} accessing a generative model.We show that the \textit{sample complexity of exploration} of our algorithm is bounded by $\tilde{O}({\frac{SA}{\epsilon^2(1-\gamma)^7}})$.This improves the previously best known result of $\tilde{O}({\frac{SA}{\epsilon^4(1-\gamma)^8}})$ in this setting achieved by delayed Q-learning (Strehlet al., 2006),, and matches the lower bound in terms of $\epsilon$ as well as $S$ and $A$ up to logarithmic factors.

Wir passen das Q-Lernen mit UCB-Explorationsbonus an ein MDP mit unendlichem Zeithorizont und diskontierten Belohnungen an, ohne auf ein generatives Modell zuzugreifen, und verbessern das bisher beste bekannte Ergebnis.

In dieser Arbeit wurde ein Q-Learning-Algorithmus mit UCB-Explorationspolitik für MDP mit unendlichem Horizont betrachtet.


Backpropagation is driving today's artificial neural networks (ANNs).However, despite extensive research, it remains unclear if the brain implements this algorithm.Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative: neurons can randomly introduce change, and use unspecific feedback signals to observe their effect on the cost and thus approximate their gradient.However, the convergence rate of such learning scales poorly with the number of involved neurons.Here we propose a hybrid learning approach.Each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide.We provide proof that our approach converges to the true gradient for certain classes of networks.In both feedforward and convolutional networks, we empirically show that our approach learns to approximate the gradient, and can match the performance of gradient-based learning.Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-specified learning rules.

Störungen können verwendet werden, um Rückkopplungsgewichte zu trainieren, die in Fully Connected und Convolutional Neural Networks lernen.

In dieser Arbeit wird eine Methode vorgeschlagen, die das Problem des "Gewichtstransports" löst, indem die Gewichte für den Rückwärtsdurchlauf mit Hilfe eines rauschbasierten Schätzers geschätzt werden.


This paper proposes and demonstrates a surprising pattern in the training of neural networks: there is a one to one relation between the values of any pair of losses (such as cross entropy, mean squared error, 0/1 error etc.) evaluated for a model arising at (any point of) a training run.This pattern is universal in the sense that this one to one relationship is identical across architectures (such as VGG, Resnet, Densenet etc.), algorithms (SGD and SGD with momentum) and training loss functions (cross entropy and mean squared error).

Wir identifizieren einige universelle Muster (d.h. über alle Architekturen hinweg) im Verhalten verschiedener Ersatzverluste (CE, MSE, 0-1 Verlust) beim Training neuronaler Netze und präsentieren unterstützende empirische Beweise.