Due to the success of deep learning to solving a variety of challenging machine learning tasks, there is a rising interest in understanding loss functions for training neural networks from a theoretical aspect.Particularly, the properties of critical points and the landscape around them are of importance to determine the convergence performance of optimization algorithms.In this paper, we provide a necessary and sufficient characterization of the analytical forms for the critical points (as well as global minimizers) of the square loss functions for linear neural networks.We show that the analytical forms of the critical points characterize the values of the corresponding loss functions as well as the necessary and sufficient conditions to achieve global minimum.Furthermore, we exploit the analytical forms of the critical points to characterize the landscape properties for the loss functions of linear neural networks and shallow ReLU networks.One particular conclusion is that: While the loss function of linear networks has no spurious local minimum, the loss function of one-hidden-layer nonlinear networks with ReLU activation function does have local minimum that is not global minimum.

Forniamo forme analitiche necessarie e sufficienti per i punti critici delle loss function quadrate per varie reti neurali, e sfruttiamo le forme analitiche per caratterizzare le proprietà del landscape per le loss function di queste reti neurali.


The backpropagation (BP) algorithm is often thought to be biologically implausible in the brain.One of the main reasons is that BP requires symmetric weight matrices in the feedforward and feedback pathways.To address this “weight transport problem” (Grossberg, 1987), two biologically-plausible algorithms, proposed by Liao et al. (2016) and Lillicrap et al. (2016), relax BP’s weight symmetry requirements and demonstrate comparable learning capabilities to that of BP on small datasets.However, a recent study by Bartunov et al. (2018) finds that although feedback alignment (FA) and some variants of target-propagation (TP) perform well on MNIST and CIFAR, they perform significantly worse than BP on ImageNet.Here, we additionally evaluate the sign-symmetry (SS) algorithm (Liao et al., 2016), which differs from both BP and FA in that the feedback and feedforward weights do not share magnitudes but share signs.We examined the performance of sign-symmetry and feedback alignment on ImageNet and MS COCO datasets using different network architectures (ResNet-18 and AlexNet for ImageNet; RetinaNet for MS COCO).Surprisingly, networks trained with sign-symmetry can attain classification performance approaching that of BP-trained networks.These results complement the study by Bartunov et al. (2018) and establish a new benchmark for future biologically-plausible learning algorithms on more difficult datasets and more complex architectures.

Gli algoritmi di apprendimento biologicamente plausibili, in particolare la simmetria dei segni, funzionano bene su ImageNet


We introduce the 2-simplicial Transformer, an extension of the Transformer which includes a form of higher-dimensional attention generalising the dot-product attention, and uses this attention to update entity representations with tensor products of value vectors.We show that this architecture is a useful inductive bias for logical reasoning in the context of deep reinforcement learning.

Introduciamo il 2-simplicial Transformer e mostriamo che questa architettura è un utile bias induttivo per il ragionamento logico nel contesto del deep reinforcement learning.


We present Tensor-Train RNN (TT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics.Long-term forecasting in such systems is highly challenging, since there exist long-term temporal dependencies, higher-order correlations and sensitivity to error propagation.Our proposed tensor recurrent architecture addresses these issues by learning the nonlinear dynamics directly using higher order moments and high-order state transition functions.Furthermore, we decompose the higher-order structure using the tensor-train (TT) decomposition to reduce the number of parameters while preserving the model performance.We theoretically establish the approximation properties of Tensor-Train RNNs for general sequence inputs, and such guarantees are not available for usual RNNs.We also demonstrate significant long-term prediction improvements over general RNN and LSTM architectures on a range of simulated environments with nonlinear dynamics, as well on real-world climate and traffic data.

Previsione accurata su orizzonti temporali molto lunghi usando RNN tensor-train


Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret.We propose a variational message-passing algorithm for variational inference in such models.We make three contributions.First, we propose structured inference networks that incorporate the structure of the graphical model in the inference network of variational auto-encoders (VAE).Second, we establish conditions under which such inference networks enable fast amortized inference similar to VAE.Finally, we derive a variational message passing algorithm to perform efficient natural-gradient inference while retaining the efficiency of the amortized inference.By simultaneously enabling structured, amortized, and natural-gradient inference for deep structured models, our method simplifies and generalizes existing methods.

Proponiamo un algoritmo di message-passing variazionale per modelli che contengono sia il deep model che il graphical model probabilistico.


Modern deep neural networks have a large amount of weights, which make them difficult to deploy on computation constrained devices such as mobile phones.One common approach to reduce the model size and computational cost is to use low-rank factorization to approximate a weight matrix.However, performing standard low-rank factorization with a small rank can hurt the model expressiveness and significantly decrease the performance.In this work, we propose to use a mixture of multiple low-rank factorizations to model a large weight matrix, and the mixture coefficients are computed dynamically depending on its input.We demonstrate the effectiveness of the proposed approach on both language modeling and image classification tasks.Experiments show that our method not only improves the computation efficiency but also maintains (sometimes outperforms) its accuracy compared with the full-rank counterparts.

Una semplice modifica alla fattorizzazione low-rank che migliora le prestazioni (sia in task con immagini sia testuali) pur essendo compatta.


Deep learning training accesses vast amounts of data at high velocity, posing challenges for datasets retrieved over commodity networks and storage devices.We introduce a way to dynamically reduce the overhead of fetching and transporting training data with a method we term Progressive Compressed Records (PCRs).PCRs deviate from previous formats by leveraging progressive compression to split each training example into multiple examples of increasingly higher fidelity, without adding to the total data size.Training examples of similar fidelity are grouped together, which reduces both the system overhead and data bandwidth needed to train a model.We show that models can be trained on aggressively compressed representations of the training data and still retain high accuracy, and that PCRs can enable a 2x speedup on average over baseline formats using JPEG compression.Our results hold across deep learning architectures for a wide range of datasets: ImageNet, HAM10000, Stanford Cars, and CelebA-HQ.

Proponiamo un formato di dati semplice, generale ed efficiente dal punto di vista dello spazio per accelerare il training del deep learning, permettendo che la fedeltà del sample sia selezionata dinamicamente al momento del training


It is fundamental and challenging to train robust and accurate Deep Neural Networks (DNNs) when semantically abnormal examples exist.Although great progress has been made, there is still one crucial research question which is not thoroughly explored yet: What training examples should be focused and how much more should they be emphasised to achieve robust learning?In this work, we study this question and propose gradient rescaling (GR) to solve it.GR modifies the magnitude of logit vector’s gradient to emphasise on relatively easier training data points when noise becomes more severe, which functions as explicit emphasis regularisation to improve the generalisation performance of DNNs.Apart from regularisation, we connect GR to examples weighting and designing robust loss functions.We empirically demonstrate that GR is highly anomaly-robust and outperforms the state-of-the-art by a large margin, e.g., increasing 7% on CIFAR100 with 40% noisy labels.It is also significantly superior to standard regularisers in both clean and abnormal settings.Furthermore, we present comprehensive ablation studies to explore the behaviours of GR under different cases, which is informative for applying GR in real-world scenarios.

APPRENDIMENTO ROBUSTO DELLA RAPPRESENTAZIONE DISCRIMINATIVA ATTRAVERSO IL RESCALING DEL GRADIENTE: UNA PROSPETTIVA SULL' EMPHASIS REGULARISATION


Generative Adversarial Networks (GANs) have achieved remarkable results in the task of generating realistic natural images.In most applications, GAN models share two aspects in common.On the one hand, GANs training involves solving a challenging saddle point optimization problem, interpreted as an adversarial game between a generator and a discriminator functions.On the other hand, the generator and the discriminator are parametrized in terms of deep convolutional neural networks.The goal of this paper is to disentangle the contribution of these two factors to the success of GANs.In particular, we introduce Generative Latent Optimization (GLO), a framework to train deep convolutional generators without using discriminators, thus avoiding the instability of adversarial optimization problems.Throughout a variety of experiments, we show that GLO enjoys many of the desirable properties of GANs: learning from large data, synthesizing visually-appealing samples, interpolating meaningfully between samples, and performing linear arithmetic with noise vectors.

Le GAN hanno successo grazie all'adversarial training o all'uso di ConvNet? Mostriamo che un generatore ConvNet addestrato con una semplice loss di ricostruzione e noise vector learnable porta molte delle proprietà desiderabili di una GAN.


In this paper, we propose a novel kind of kernel, random forest kernel, to enhance the empirical performance of MMD GAN.Different from common forests with deterministic routings, a probabilistic routing variant is used in our innovated random-forest kernel, which is possible to merge with the CNN frameworks.Our proposed random-forest kernel has the following advantages: From the perspective of random forest, the output of GAN discriminator can be viewed as feature inputs to the forest, where each tree gets access to merely a fraction of the features, and thus the entire forest benefits from ensemble learning.In the aspect of kernel method, random-forest kernel is proved to be characteristic, and therefore suitable for the MMD structure.Besides, being an asymmetric kernel, our random-forest kernel is much more flexible, in terms of capturing the differences between distributions.Sharing the advantages of CNN, kernel method, and ensemble learning, our random-forest kernel based MMD GAN obtains desirable empirical performances on CIFAR-10, CelebA and LSUN bedroom data sets.Furthermore, for the sake of completeness, we also put forward comprehensive theoretical analysis to support our experimental results.

Equipaggiare le MMD GAN con un nuovo random-forest kernel.


Reinforcement learning in an actor-critic setting relies on accurate value estimates of the critic.However, the combination of function approximation, temporal difference (TD) learning and off-policy training can lead to an overestimating value function.A solution is to use Clipped Double Q-learning (CDQ), which is used in the TD3 algorithm and computes the minimum of two critics in the TD-target. We show that CDQ induces an underestimation bias and propose a new algorithm that accounts for this by using a weighted average of the target from CDQ and the target coming from a single critic.The weighting parameter is adjusted during training such that the value estimates match the actual discounted return on the most recent episodes and by that it balances over- and underestimation.Empirically, we obtain more accurate value estimates and demonstrate state of the art results on several OpenAI gym tasks.

Un metodo per critic estimate più accurate nel reinforcement learning.


We introduce a systematic framework for quantifying the robustness of classifiers to naturally occurring perturbations of images found in videos.As part of this framework, we construct ImageNet-Vid-Robust, a human-expert--reviewed dataset of 22,668 images grouped into 1,145 sets of perceptually similar images derived from frames in the ImageNet Video Object Detection dataset.We evaluate a diverse array of classifiers trained on ImageNet, including models trained for robustness, and show a median classification accuracy drop of 16\%.Additionally, we evaluate the Faster R-CNN and R-FCN models for detection, and show that natural perturbations induce both classification as well as localization errors, leading to a median drop in detection mAP of 14 points.Our analysis shows that natural perturbations in the real world are heavily problematic for current CNNs, posing a significant challenge to their deployment in safety-critical environments that require reliable, low-latency predictions.

Introduciamo un framework sistematico per quantificare la robustezza dei classificatori alle perturbazioni naturali delle immagini presenti nei video.


Structured tabular data is the most commonly used form of data in industry according to a Kaggle ML and DS Survey.Gradient Boosting Trees, Support Vector Machine, Random Forest, and Logistic Regression are typically used for classification tasks on tabular data.The recent work of Super Characters method using two-dimensional word embedding achieved state-of-the-art results in text classification tasks, showcasing the promise of this new approach.In this paper, we propose the SuperTML method, which borrows the idea of Super Characters method and two-dimensional embedding to address the problem of classification on tabular data.For each input of tabular data, the features are first projected into two-dimensional embedding like an image, and then this image is fed into fine-tuned ImageNet CNN models for classification.Experimental results have shown that the proposed SuperTML method have achieved state-of-the-art results on both large and small datasets.

Deep learning per l'apprendimento automatico di dati tabulari strutturati usando un modello CNN pre-addestrato da ImageNet.


Learning rich representations from predictive learning without labels has been a longstanding challenge in the field of machine learning.Generative pre-training has so far not been as successful as contrastive methods in modeling representations of raw images.In this paper, we propose a neural architecture for self-supervised representation learning on raw images called the PatchFormer which learns to model spatial dependencies across patches in a raw image.Our method learns to model the conditional probability distribution of missing patches given the context of surrounding patches.We evaluate the utility of the learned representations by fine-tuning the pre-trained model on low data-regime classification tasks.Specifically, we benchmark our model on semi-supervised ImageNet classification which has become a popular benchmark recently for semi-supervised and self-supervised learning methods.Our model is able to achieve 30.3% and 65.5% top-1 accuracies when trained only using 1% and 10% of the labels on ImageNet showing the promise for generative pre-training methods.

La decodifica dei pixel può ancora funzionare per representation learning sulle immagini


Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix.Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive.We show how to modify full-matrix adaptive regularization in order to make it practical and effective.We also provide novel theoretical analysisfor adaptive regularization in non-convex optimization settings.The core of our algorithm, termed GGT, consists of efficient inverse computation of square roots of low-rank matrices.Our preliminary experiments underscore improved convergence rate of GGT across a variety of synthetic tasks and standard deep learning benchmarks.

AdaGrad/Adam full-matrix, veloce e truly scalable, con la teoria per l'ottimizzazione adattativa stocastica non convessa


Dialogue systems require a great deal of different but complementary expertise to assist, inform, and entertain humans.For example, different domains (e.g., restaurant reservation, train ticket booking) of goal-oriented dialogue systems can be viewed as different skills, and so does ordinary chatting abilities of chit-chat dialogue systems.In this paper, we propose to learn a dialogue system that independently parameterizes different dialogue skills, and learns to select and combine each of them through Attention over Parameters (AoP).The experimental results show that this approach achieves competitive performance on a combined dataset of MultiWOZ (Budzianowski et al., 2018), In-Car Assistant (Eric et al.,2017), and Persona-Chat (Zhang et al., 2018).Finally, we demonstrate that each dialogue skill is effectively learned and can be combined with other skills to produce selective responses.

In questo articolo, proponiamo di imparare un dialogue system che parametrizza indipendentemente diverse dialogue skill, e impara a selezionare e combinare ciascuna di esse attraverso l'Attention sui Parametri.


Model distillation aims to distill the knowledge of a complex model into a simpler one.In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one.The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data.For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to the original performance, given a fixed network initialization.We evaluate our method in various initialization settings.  Experiments on multiple datasets, MNIST, CIFAR10, PASCAL-VOC, and CUB-200, demonstrate the ad-vantage of our approach compared to alternative methods.  Finally, we include a real-world application of dataset distillation to the continual learning setting: we show that storing distilled images as episodic memory of previous tasks can alleviate forgetting more effectively than real images.

Proponiamo di distillare un grande dataset in un piccolo dataset sintetico che possono addestrare reti vicine alle prestazioni originali.


We relate the minimax game of generative adversarial networks (GANs) to finding the saddle points of the Lagrangian function for a convex optimization problem, where the discriminator outputs and the distribution of generator outputs play the roles of primal variables and dual variables, respectively.This formulation shows the connection between the standard GAN training process and the primal-dual subgradient methods for convex optimization.The inherent connection does not only provide a theoretical convergence proof for training GANs in the function space, but also inspires a novel objective function for training.The modified objective function forces the distribution of generator outputs to be updated along the direction according to the primal-dual subgradient methods.A toy example shows that the proposed method is able to resolve mode collapse, which in this case cannot be avoided by the standard GAN or Wasserstein GAN.Experiments on both Gaussian mixture synthetic data and real-world image datasets demonstrate the performance of the proposed method on generating diverse samples.

Proponiamo un metodo primale-duale con subgradient per il training di GAN e questo metodo allevia efficacemente il mode collapse.


Specifying reward functions is difficult, which motivates the area of reward inference: learning rewards from human behavior.The starting assumption in the area is that human behavior is optimal given the desired reward function, but in reality people have many different forms of irrationality, from noise to myopia to risk aversion and beyond.This fact seems like it will be strictly harmful to reward inference: it is already hard to infer the reward from rational behavior, and noise and systematic biases make actions have less direct of a relationship to the reward.Our insight in this work is that, contrary to expectations, irrationality can actually help rather than hinder reward inference.For some types and amounts of irrationality, the expert now produces more varied policies compared to rational behavior, which help disambiguate among different reward parameters -- those that otherwise correspond to the same rational behavior.We put this to the test in a systematic analysis of the effect of irrationality on reward inference.We start by covering the space of irrationalities as deviations from the Bellman update, simulate expert behavior, and measure the accuracy of inference to contrast the different types and study the gains and losses.We provide a mutual information-based analysis of our findings, and wrap up by discussing the need to accurately model irrationality, as well as to what extent we might expect (or be able to train) real people to exhibit helpful irrationalities when teaching rewards to learners.

Troviamo che l'irrazionalità di un dimostratore esperto può aiutare un learner ad indurre le sue preferenze.


Natural Language Processing models lack a unified approach to robustness testing.In this paper we introduce WildNLP - a framework for testing model stability in a natural setting where text corruptions such as keyboard errors or misspelling occur.We compare robustness of models from 4 popular NLP tasks: Q&A, NLI, NER and Sentiment Analysis by testing their performance on aspects introduced in the framework.In particular, we focus on a comparison between recent state-of-the- art text representations and non-contextualized word embeddings.In order to improve robust- ness, we perform adversarial training on se- lected aspects and check its transferability to the improvement of models with various cor- ruption types.We find that the high perfor- mance of models does not ensure sufficient robustness, although modern embedding tech- niques help to improve it.We release cor- rupted datasets and code for WildNLP frame- work for the community.

Confrontiamo la robustezza dei modelli di 4 popolari task NLP: Q&A, NLI, NER e Sentiment Analysis testando le loro prestazioni su input perturbati.


Training generative models like Generative Adversarial Network (GAN)  is challenging for noisy data.A novel curriculum learning algorithm pertaining to clustering is proposed to address this issue in this paper.The curriculum construction is based on the centrality of underlying clusters in data points.  The data points of high centrality takes priority of being fed into generative models during training.To make our algorithm scalable to large-scale data, the active set is devised, in the sense that every round of training proceeds only on an active subset containing a small fraction of already trained data and the incremental data of lower centrality.Moreover, the geometric analysis is presented to interpret the necessity of cluster curriculum for generative models.The experiments on cat and human-face data validate that our algorithm is able to learn the optimal generative models (e.g. ProGAN) with respect to specified quality metrics for noisy data.An interesting finding is that the optimal cluster curriculum is closely related to the critical point of the geometric percolation process formulated in the paper.

Un nuovo algoritmo di curriculum learning basato su cluster viene proposto per risolvere il training robusto di modelli generativi.


Backdoor attacks aim to manipulate a subset of training data by injecting adversarial triggers such that machine learning models trained on the tampered dataset will make arbitrarily (targeted) incorrect prediction on the testset with the same trigger embedded.While federated learning (FL) is capable of aggregating information provided by different parties for training a better model, its distributed learning methodology and inherently heterogeneous data distribution across parties may bring new vulnerabilities.In addition to recent centralized backdoor attacks on FL where each party embeds the same global trigger during training, we propose the distributed backdoor attack (DBA) --- a novel threat assessment framework developed by fully exploiting the distributed nature of FL.DBA decomposes a global trigger pattern into separate local patterns and embed them into the training set of different adversarial parties respectively.Compared to standard centralized backdoors, we show that DBA is substantially more persistent and stealthy against FL on diverse datasets such as finance and image data.We conduct extensive experiments to show that the attack success rate of DBA is significantly higher than centralized backdoors under different settings.Moreover, we find that distributed attacks are indeed more insidious, as DBA can evade two state-of-the-art robust FL algorithms against centralized backdoors.We also provide explanations for the effectiveness of DBA via feature visual interpretation and feature importance ranking.To further explore the properties of DBA, we test the attack performance by varying different trigger factors, including local trigger variations (size, gap, and location), scaling factor in FL, data distribution, and poison ratio and interval.Our proposed DBA and thorough evaluation results shed lights on characterizing the robustness of FL.

Abbiamo proposto un nuovo attacco backdoor distribuito sul federated learning e dimostriamo che non solo è più efficace rispetto agli attacchi centralizzati standard, ma anche più difficile da difendere con i metodi FL robusti esistenti


Graph networks have recently attracted considerable interest, and in particular in the context of semi-supervised learning.These methods typically work by generating node representations that are propagated throughout a given weighted graph.Here we argue that for semi-supervised learning, it is more natural to consider propagating labels in the graph instead.Towards this end, we propose a differentiable neural version of the classic Label Propagation (LP) algorithm.This formulation can be used for learning edge weights, unlike other methods where weights are set heuristically.Starting from a layer implementing a single iteration of LP, we proceed by adding several important non-linear steps that significantly enhance the label-propagating mechanism.Experiments in two distinct settings demonstrate the utility of our approach.

Rete neurale per l'apprendimento semi-supervised basato sui grafi; rivisita i classici e propaga le label piuttosto che le rappresentazioni delle feature


Neural architecture search (NAS) has made rapid progress incomputervision,wherebynewstate-of-the-artresultshave beenachievedinaseriesoftaskswithautomaticallysearched neural network (NN) architectures.In contrast, NAS has not made comparable advances in natural language understanding (NLU).Corresponding to encoder-aggregator meta architecture of typical neural networks models for NLU tasks (Gong et al. 2018), we re-deﬁne the search space, by splittingitinto twoparts:encodersearchspace,andaggregator search space.Encoder search space contains basic operations such as convolutions, RNNs, multi-head attention and its sparse variants, star-transformers.Dynamic routing is included in the aggregator search space, along with max (avg) pooling and self-attention pooling.Our search algorithm is then fulﬁlled via DARTS, a differentiable neural architecture search framework.We progressively reduce the search space every few epochs, which further reduces the search time and resource costs.Experiments on ﬁve benchmark data-sets show that, the new neural networks we generate can achieve performances comparable to the state-of-the-art models that does not involve language model pre-training.

Neural Architecture Search per una serie di task di Natural Language Understanding. Design dello spazio di ricerca per i task di NLU. Si applica la ricerca dell'architettura differenziabile per scoprire nuovi modelli


Network embedding (NE) methods aim to learn low-dimensional representations of network nodes as vectors, typically in Euclidean space.These representations are then used for a variety of downstream prediction tasks.Link prediction is one of the most popular choices for assessing the performance of NE methods.However, the complexity of link prediction requires a carefully designed evaluation pipeline to provide consistent, reproducible and comparable results.We argue this has not been considered sufficiently in recent works.The main goal of this paper is to overcome difficulties associated with evaluation pipelines and reproducibility of results.We introduce EvalNE, an evaluation framework to transparently assess and compare the performance of NE methods on link prediction.EvalNE provides automation and abstraction for tasks such as hyper-parameter tuning, model validation, edge sampling, computation of edge embeddings and model validation.The framework integrates efficient procedures for edge and non-edge sampling and can be used to easily evaluate any off-the-shelf embedding method.The framework is freely available as a Python toolbox.Finally, demonstrating the usefulness of EvalNE in practice, we conduct an empirical study in which we try to replicate and analyse experimental sections of several influential papers.

In questo articolo introduciamo EvalNE, un toolbox Python per automatizzare la valutazione dei metodi di embedding di rete sulla predizione dei link garantendo la riproducibilità dei risultati.


Deep learning models can be efficiently optimized via stochastic gradient descent, but there is little theoretical evidence to support this.A key question in optimization is to understand when the optimization landscape of a neural network is amenable to gradient-based optimization.We focus on a simple neural network two-layer ReLU network with two hidden units, and show that all local minimizers are global.This combined with recent work of Lee et al. (2017); Lee et al. (2016) show that  gradient descent converges to the global minimizer.

Garanzia di recovery della stochastic gradient descent con inizializzazione casuale per l'apprendimento di una rete neurale a due strati con due hidden unit, pesi a norma unitaria, funzioni di attivazione ReLU e input gaussiani.


Dropout is a simple yet effective technique to improve generalization performance and prevent overfitting in deep neural networks (DNNs).In this paper, we discuss three novel observations about dropout to better understand the generalization of DNNs with rectified linear unit (ReLU) activations: 1) dropout is a smoothing technique that encourages each local linear model of a DNN to be trained on data points from nearby regions; 2) a constant dropout rate can result in effective neural-deactivation rates that are significantly different for layers with different fractions of activated neurons; and 3) the rescaling factor of dropout causes an inconsistency to occur between the normalization during training and testing conditions when batch normalization is also used.  The above leads to three simple but nontrivial improvements to dropout resulting in our proposed method "Jumpout."Jumpout samples the dropout rate using a monotone decreasing distribution (such as the right part of a truncated Gaussian), so the local linear model at each data point is trained, with high probability, to work better for data points from nearby than from more distant regions.Instead of tuning a dropout rate for each layer and applying it to all samples, jumpout moreover adaptively normalizes the dropout rate at each layer and every training sample/batch, so the effective dropout rate applied to the activated neurons are kept the same.Moreover, we rescale the outputs of jumpout for a better trade-off that keeps both the variance and mean of neurons more consistent between training and test phases, which mitigates the incompatibility between dropout and batch normalization.Compared to the original dropout, jumpout shows significantly improved performance on CIFAR10, CIFAR100, Fashion- MNIST, STL10, SVHN, ImageNet-1k, etc., while introducing negligible additional memory and computation costs.

Jumpout applica tre semplici ma efficaci modifiche al dropout, basate su nuove conoscenze sulle prestazioni di generalizzazione della DNN con ReLU nelle regioni locali.


Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse  neural  models for NLP tasks.If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse?Using the Taxi-Euclidean norm to measure sparsity, we find that frequent input words are associated with concentrated or sparse activations, while frequent target words are associated with dispersed activations but concentrated gradients.We find that  gradients associated with function words are more concentrated than the gradients of content words, even controlling for word frequency.

Studiamo l'emergere naturale della sparsità nelle attivazioni e nei gradienti per alcuni layer di un language model LSTM denso, nel corso del training.


The integration of a  Knowledge Base (KB) into a neural dialogue agent is one of the key challenges in Conversational AI.Memory networks has proven to be effective to encode KB information into an external memory to thus generate more fluent and informed responses.Unfortunately, such memory becomes full of latent representations during training, so the most common strategy is to overwrite old memory entries randomly. In this paper, we question this approach and provide experimental evidence showing that conventional memory networks generate many redundant latent vectors resulting in overfitting and the need for larger memories.We introduce memory dropout as an automatic technique that encourages diversity in the latent space by1) Aging redundant memories to increase their probability of being overwritten during training2) Sampling new memories that summarize the knowledge acquired by redundant memories.This technique allows us to incorporate  Knowledge Bases to achieve state-of-the-art dialogue generation in the Stanford Multi-Turn Dialogue dataset.Considering the same architecture, its use provides an improvement of +2.2 BLEU points for the automatic  generation of responses and an increase of +8.1% in the recognition of named entities.

Le memory network convenzionali generano molti vettori latenti ridondanti con conseguente overfitting e la necessità di memorie più grandi. Introduciamo il dropout della memoria come una tecnica automatica che incoraggia la diversità nello spazio latente.


Su-Boyd-Candes (2014) made a connection between Nesterov's method and an ordinary differential equation (ODE).  We show if a Hessian damping term is added to the ODE from Su-Boyd-Candes (2014), then Nesterov's method arises as a straightforward discretization of the modified ODE.Analogously,  in the strongly convex case, a Hessian damping term is added to Polyak's ODE, which is then discretized to yield Nesterov's method for strongly convex functions.   Despite the Hessian term, both second order ODEs can be represented as first order systems.Established Liapunov analysis is used to recover the accelerated rates of convergence in both continuous and discrete time.  Moreover, the Liapunov analysis can be extended to the case of stochastic gradients which allows the full gradient case to be considered as a special case of the stochastic case.  The result is a unified approach to convex acceleration in both continuous and discrete time and in  both the stochastic and full gradient cases.

Il metodo di Nesterov si presenta come una discretizzazione diretta di un'ODE diversa da quella di Su-Boyd-Candes e dimostriamo l'accelerazione del caso stocastico


We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset.L2TL considers joint optimization of vastly-shared weights between models for source and target tasks, and employs adaptive weights for scaling of constituent losses.The adaptation of the weights is based on reinforcement learning, guided with a performance metric on the target validation set.We demonstrate state-of-the-art performance of L2TL given fixed models, consistently outperforming fine-tuning baselines on various datasets.In the regimes of small-scale target datasets and significant label mismatch between source and target datasets, L2TL outperforms previous work by an even larger margin.

Proponiamo di imparare a trasferire l'apprendimento (L2TL) per migliorare il transfer learning su un dataset di destinazione attraverso un'attenta estrazione di informazioni da un dataset di origine.


In many partially observable scenarios, Reinforcement Learning (RL) agents must rely on long-term memory in order to learn an optimal policy.We demonstrate that using techniques from NLP and supervised learning fails at RL tasks due to stochasticity from the environment and from exploration.Utilizing our insights on the limitations of traditional memory methods in RL, we propose AMRL, a class of models that can learn better policies with greater sample efficiency and are resilient to noisy inputs.Specifically, our models use a standard memory module to summarize short-term context, and then aggregate all prior states from the standard model without respect to order.We show that this provides advantages both in terms of gradient decay and signal-to-noise ratio over time.Evaluating in Minecraft and maze environments that test long-term memory, we find that our model improves average return by 19% over a baseline that has the same number of parameters and by 9% over a stronger baseline that has far more parameters.

In Deep RL, le funzioni invarianti rispetto all'ordine possono essere utilizzate insieme ai moduli di memoria standard per migliorare il gradient decay e la resilienza al rumore.


Optimization on manifold has been widely used in machine learning, to handle optimization problems with constraint.Most previous works focus on the case with a single manifold.However, in practice it is quite common that the optimization problem involves more than one constraints, (each constraint corresponding to one manifold).It is not clear in general how to optimize on multiple manifolds effectively and provably especially when the intersection of multiple manifolds is not a manifold or cannot be easily calculated.We propose a unified algorithm framework to handle the optimization on multiple manifolds.Specifically,  we integrate information from multiple manifolds and move along an ensemble direction by viewing the information from each manifold as a drift and adding them together.We prove the convergence properties of the proposed algorithms.We also apply the algorithms into  training neural network with batch normalization layers and achieve preferable empirical results.

Questo articolo introduce un algoritmo per gestire un problema di ottimizzazione con vincoli multipli sotto la visione di manifold.


It has long been assumed that high dimensional continuous control problems cannot be solved effectively by discretizing individual dimensions of the action space due to the exponentially large number of bins over which policies would have to be learned.In this paper, we draw inspiration from the recent success of sequence-to-sequence models for structured prediction problems to develop policies over discretized spaces.Central to this method is the realization that complex functions over high dimensional spaces can be modeled by neural networks that predict one dimension at a time.Specifically, we show how Q-values and policies over continuous spaces can be modeled using a next step prediction model over discretized dimensions.With this parameterization, it is possible to both leverage the compositional structure of action spaces during learning, as well as compute maxima over action spaces (approximately).On a simple example task we demonstrate empirically that our method can perform global search, which effectively gets around the local optimization issues that plague DDPG.We apply the technique to off-policy (Q-learning) methods and show that our method can achieve the state-of-the-art for off-policy methods on several continuous control tasks.

Un metodo per fare Q-learning su spazi d'azione continui prevedendo una sequenza di azioni 1-D discretizzate.


Model-based reinforcement learning (MBRL) aims to learn a dynamic model to reduce the number of interactions with real-world environments.However, due to estimation error, rollouts in the learned model, especially those of long horizon, fail to match the ones in real-world environments.This mismatching has seriously impacted the sample complexity of MBRL.The phenomenon can be attributed to the fact that previous works employ supervised learning to learn the one-step transition models, which has inherent difficulty ensuring the matching of distributions from multi-step rollouts.Based on the claim, we propose to learn the synthesized model by matching the distributions of multi-step rollouts sampled from the synthesized model and the real ones via WGAN.We theoretically show that matching the two can minimize the difference of cumulative rewards between the real transition and the learned one.Our experiments also show that the proposed model imitation method outperforms the state-of-the-art in terms of sample complexity and average return.

Il nostro metodo incorpora WGAN per ottenere occupancy measure matching per transition learning.


Batch Normalization (BN) and its variants have seen widespread adoption in the deep learning community because they improve the training of deep neural networks.Discussions of why this normalization works so well remain unsettled.  We make explicit the relationship between ordinary least squares and partial derivatives computed when back-propagating through BN.We recast the back-propagation of BN as a least squares fit, which zero-centers and decorrelates partial derivatives from normalized activations.This view, which we term {\em gradient-least-squares}, is an extensible and arithmetically accurate description of BN.To further explore this perspective, we motivate, interpret, and evaluate two adjustments to BN.

La normalizzazione gaussiana esegue un adattamento ai minimi quadrati durante la back-propagation, che azzera e decorrela le derivate parziali dalle attivazioni normalizzate.


Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization.While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking.Here theoretical support is provided for one of its conjectured properties, namely, the ability to allow gradient descent to succeed with less tuning of learning rates.It is shown that even if we fix the learning rate of scale-invariant parameters (e.g., weights of each layer with BN) to a constant (say, 0.3), gradient descent still approaches a stationary point (i.e., a solution where gradient is zero) in the rate of T^{−1/2} in T iterations, asymptotically matching the best bound for gradient descent with well-tuned learning rates.A similar result with convergence rate T^{−1/4} is also shown for stochastic gradient descent.

Diamo un'analisi teorica della capacità della batch normalization di sintonizzare automaticamente i learning rate, nel contesto della ricerca di punti stazionari per un obiettivo di deep learning.


Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale.We attempt to carry this success to the field of video modeling by showing that large Generative Adversarial Networks trained on the complex Kinetics-600 dataset are able to produce video samples of substantially higher complexity and fidelity than previous work.  Our proposed model, Dual Video Discriminator GAN (DVD-GAN), scales to longer and higher resolution videos by leveraging a computationally efficient decomposition of its discriminator.We evaluate on the related tasks of video synthesis and video prediction,  and achieve new state-of-the-art Fréchet Inception Distance for prediction for Kinetics-600,  as well as state-of-the-art Inception Score for synthesis on the UCF-101 dataset, alongside establishing a strong baseline for synthesis on Kinetics-600.

Proponiamo DVD-GAN, un modello generativo di video di grandi dimensioni che è allo stato dell'arte su diversi task e produce video altamente complessi quando addestrato su grandi dataset del mondo reale.


Understanding procedural language requires anticipating the causal effects of actions, even when they are not explicitly stated.In this work, we introduce Neural Process Networks to understand procedural text through (neural) simulation of action dynamics.   Our model complements existing memory architectures with dynamic entity tracking by explicitly modeling actions as state transformers.The model updates the states of the entities by executing learned action operators.Empirical results demonstrate that our proposed model can reason about the unstated causal effects of actions, allowing it to provide more accurate contextual information for understanding and generating procedural text, all while offering more interpretable internal representations than existing alternatives.

Proponiamo una nuova architettura di memoria ricorrente che può tracciare i cambiamenti di stato di common sense nelle entità simulando gli effetti causali delle azioni.


There has been a recent trend in training neural networks to replace data structures that have been crafted by hand, with an aim for faster execution, better accuracy, or greater compression.  In this setting, a neural data structure is instantiated by training a network over many epochs of its inputs until convergence.In many applications this expensive initialization is not practical, for example streaming algorithms --- where inputs are ephemeral and can only be inspected a small number of times.  In this paper we explore the learning of approximate set membership over a stream of data in one-shot via meta-learning.We propose a novel memory architecture, the Neural Bloom Filter, which we show to be more compressive than Bloom Filters and several existing memory-augmented neural networks in scenarios of skewed data or structured sets.

Studiamo l'efficienza spaziale delle reti neurali memory-augmented durante il learning di set membership.


We leverage recent insights from second-order optimisation for neural networks to construct a Kronecker factored Laplace approximation to the posterior over the weights of a trained network.Our approximation requires no modification of the training procedure, enabling practitioners to estimate the uncertainty of their models currently used in production without having to retrain them.We extensively compare our method to using Dropout and a diagonal Laplace approximation for estimating the uncertainty of a network.We demonstrate that our Kronecker factored method leads to better uncertainty estimates on out-of-distribution data and is more robust to simple adversarial attacks.Our approach only requires calculating two square curvature factor matrices for each layer.Their size is equal to the respective square of the input and output size of the layer, making the method efficient both computationally and in terms of memory usage.We illustrate its scalability by applying it to a state-of-the-art convolutional network architecture.

Costruiamo un'approssimazione di Laplace con fattore di Kronecker per le reti neurali che porta ad un'efficiente distribuzione normale della matrice dei pesi.


Spectral embedding is a popular technique for the representation of graph data.Several regularization techniques have been proposed to improve the quality of the embedding with respect to downstream tasks like clustering.In this paper, we explain on a simple block model the impact of the complete graph regularization, whereby a constant is added to all entries of the adjacency matrix.Specifically, we show that the regularization forces the spectral embedding  to focus on  the  largest blocks, making the representation less sensitive to noise or outliers.We illustrate these results on both  on both synthetic and real data, showing how regularization improves standard clustering scores.

La regolarizzazione dei grafi costringe l'embedding spettrale a concentrarsi sui cluster più grandi, rendendo la rappresentazione meno sensibile al rumore.


The exposure bias problem refers to the training-inference discrepancy caused by teacher forcing in maximum likelihood estimation (MLE) training for auto-regressive neural network language models (LM).It has been regarded as a central problem for natural language generation (NLG) model training.Although a lot of algorithms have been proposed to avoid teacher forcing and therefore to alleviate exposure bias, there is little work showing how serious the exposure bias problem is.In this work, we first identify the auto-recovery ability of MLE-trained LM, which casts doubt on the seriousness of exposure bias.We then develop a precise, quantifiable definition for exposure bias.However, according to our measurements in controlled experiments, there's only around 3% performance gain when the training-inference discrepancy is completely removed.Our results suggest the exposure bias problem could be much less serious than it is currently assumed to be.

Mostriamo che il bias di esposizione potrebbe essere molto meno grave di quello che si presume attualmente per il training MLE di LM.


The ability of algorithms to evolve or learn (compositional) communication protocols has traditionally been studied in the language evolution literature through the use of emergent communication tasks.Here we scale up this research by using contemporary deep learning methods and by training reinforcement-learning neural network agents on referential communication games.We extend previous work, in which agents were trained in symbolic environments, by developing agents which are able to learn from raw pixel data, a more challenging and realistic input representation.We find that the degree of structure found in the input data affects the nature of the emerged protocols, and thereby corroborate the hypothesis that structured compositional language is most likely to emerge when agents perceive the world as being structured.

Uno studio controllato del ruolo degli ambienti rispetto alle proprietà nei protocolli di comunicazione emergenti.


For understanding generic documents, information like font sizes, column layout, and generally the positioning of words may carry semantic information that is crucial for solving a downstream document intelligence task.Our novel BERTgrid, which is based on Chargrid by Katti et al. (2018), represents a document as a grid of contextualized word piece embedding vectors, thereby making its spatial structure and semantics accessible to the processing neural network.The contextualized embedding vectors are retrieved from a BERT language model.We use BERTgrid in combination with a fully convolutional network on a semantic instance segmentation task for extracting fields from invoices.We demonstrate its performance on tabulated line item and document header field extraction.

Rappresentazione del documento basata su grid con vettori di embedding contestualizzati per documenti con layout 2D


Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers.However, an attacker is not usually able to directly modify another agent's observations.This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial?We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents.The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior.We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent.Videos are available at https://attackingrl.github.io.

Le policy di deep RL possono essere attaccate da altri agenti che compiono azioni in modo da creare osservazioni naturali che sono adversarial.


GloVe and Skip-gram word embedding methods learn word vectors by decomposing a denoised matrix of word co-occurrences into a product of low-rank matrices.In this work, we propose an iterative algorithm for computing word vectors based on modeling word co-occurrence matrices with Generalized Low Rank Models.Our algorithm generalizes both Skip-gram and GloVe as well as giving rise to other embedding methods based on the specified co-occurrence matrix, distribution of co-occurences, and the number of iterations in the iterative algorithm.For example, using a Tweedie distribution with one iteration results in GloVe and using a Multinomial distribution with full-convergence mode results in Skip-gram.Experimental results demonstrate that multiple iterations of our algorithm improves results over the GloVe method on the Google word analogy similarity task.

Presentiamo un nuovo algoritmo iterativo basato su modelli generalizzati di basso rango per il calcolo e l'interpretazione dei modelli di word embedding.


Deterministic models are approximations of reality that are often easier to build and interpret than stochastic alternatives.  Unfortunately, as nature is capricious, observational data can never be fully explained by deterministic models in practice.  Observation and process noise need to be added to adapt deterministic models to behave stochastically, such that they are capable of explaining and extrapolating from noisy data.Adding process noise to deterministic simulators can induce a failure in the simulator resulting in no return value for certain inputs -- a property we describe as ``brittle.''We investigate and address the wasted computation that arises from these failures, and the effect of such failures on downstream inference tasks.We show that performing inference in this space can be viewed as rejection sampling, and train a conditional normalizing flow as a proposal over noise values such that there is a low probability that the simulator crashes, increasing computational efficiency and inference fidelity for a fixed sample budget when used as the proposal in an approximate inference algorithm.

Impariamo un flusso autoregressivo condizionato per proporre perturbazioni che non inducono il fallimento del simulatore, migliorando le prestazioni di inferenza.


Multi-hop question answering requires models to gather information from different parts of a text to answer a question.Most current approaches learn to address this task in an end-to-end way with neural networks, without maintaining an explicit representation of the reasoning process.We propose a method to extract a discrete reasoning chain over the text, which consists of a series of sentences leading to the answer.We then feed the extracted chains to a BERT-based QA model to do final answer prediction.Critically, we do not rely on gold annotated chains or ``supporting facts:'' at training time, we derive pseudogold reasoning chains using heuristics based on named entity recognition and coreference resolution.Nor do we rely on these annotations at test time, as our model learns to extract chains from raw text alone.  We test our approach on two recently proposed large multi-hop question answering datasets: WikiHop and HotpotQA, and achieve state-of-art performance on WikiHop and strong performance on HotpotQA.Our analysis shows the properties of chains that are crucial for high performance: in particular, modeling extraction sequentially is important, as is dealing with each candidate sentence in a context-aware way.Furthermore, human evaluation shows that our extracted chains allow humans to give answers with high confidence, indicating that these are a strong intermediate abstraction for this task.

Miglioriamo il question answering che richiedono un ragionamento multi-hop estraendo una catena intermedia di frasi.


Normalizing constant (also called partition function, Bayesian evidence, or marginal likelihood) is one of the central goals of Bayesian inference, yet most of the existing methods are both expensive and inaccurate.Here we develop a new approach, starting from posterior samples obtained with a standard Markov Chain Monte Carlo (MCMC).We apply a novel Normalizing Flow (NF) approach to obtain an analytic density estimator from these samples, followed by Optimal Bridge Sampling (OBS) to obtain the normalizing constant.We compare our method which we call Gaussianized Bridge Sampling (GBS) to existing methods such as Nested Sampling (NS) and Annealed Importance Sampling (AIS) on several examples, showing our method is both significantly faster and substantially more accurate than these methods, and comes with a reliable error estimation.

Sviluppiamo un nuovo metodo per la stima della costante di normalizzazione (evidenza bayesiana) usando un Optimal Bridge Sampling e un nuovo Normalizing Flow, che dimostra di superare i metodi esistenti in termini di precisione e tempo di calcolo.


We present a large-scale empirical study of catastrophic forgetting (CF) in modern Deep Neural Network (DNN) models that perform sequential (or: incremental) learning.A new experimental protocol is proposed that takes into account typical constraints encountered in application scenarios.As the investigation is empirical, we evaluate CF behavior on the hitherto largest number of visual classification datasets, from each of which we construct a representative number of Sequential Learning Tasks (SLTs) in close alignment to previous works on CF.Our results clearly indicate that there is no model that avoids CF for all investigated datasets and SLTs under application conditions.We conclude with a discussion of potential solutions and workarounds to CF, notably for the EWC and IMM models.

Controlliamo i modelli DNN per la catastrophic forgetting usando un nuovo schema di valutazione che riflette le tipiche condizioni di applicazione, con risultati sorprendenti.


Federated Learning (FL) refers to learning a high quality global model based on decentralized data storage, without ever copying the raw data.A natural scenario arises with data created on mobile phones by the activity of their users.Given the typical data heterogeneity in such situations, it is natural to ask how can the global model be personalized for every such device, individually.In this work, we point out that the setting of Model Agnostic Meta Learning (MAML), where one optimizes for a fast, gradient-based, few-shot adaptation to a heterogeneous distribution of tasks, has a number of similarities with the objective of personalization for FL.We present FL as a natural source of practical applications for MAML algorithms, and make the following observations.1) The popular FL algorithm, Federated Averaging, can be interpreted as a meta learning algorithm.2) Careful fine-tuning can yield a global model with higher accuracy, which is at the same time easier to personalize.However, solely optimizing for the global model accuracy yields a weaker personalization result.3) A model trained using a standard datacenter optimization method is much harder to personalize, compared to one trained using Federated Averaging, supporting the first claim.These results raise new questions for FL, MAML, and broader ML research.

Federated Averaging è già un algoritmo di Meta Learning, mentre i metodi addestrati nel datacenter sono significativamente più difficili da personalizzare.


Memorization of data in deep neural networks has become a subject of significant research interest. In this paper, we link memorization of  images in deep convolutional autoencoders to downsampling through strided convolution.  To analyze this mechanism in a simpler setting, we train linear convolutional autoencoders and show that linear combinations of training data are stored as eigenvectors in the linear operator corresponding to the network when downsampling is used.  On the other hand, networks without downsampling do not memorize training data.  We provide further evidence that the same effect happens in nonlinear networks.  Moreover, downsampling in nonlinear networks causes the model to not only memorize just linear combinations of images, but individual training images.  Since convolutional autoencoder components are building blocks of deep convolutional networks, we envision that our findings will shed light on the important phenomenon of memorization in over-parameterized deep networks.

Identifichiamo il downsampling come un meccanismo per la memorizzazione negli autoencoder convoluzionali.


Reinforcement learning provides a powerful and general framework for decisionmaking and control, but its application in practice is often hindered by the needfor extensive feature and reward engineering.Deep reinforcement learning methodscan remove the need for explicit engineering of policy or value features, butstill require a manually specified reward function.Inverse reinforcement learningholds the promise of automatic reward acquisition, but has proven exceptionallydifficult to apply to large, high-dimensional problems with unknown dynamics.Inthis work, we propose AIRL, a practical and scalable inverse reinforcement learningalgorithm based on an adversarial reward learning formulation that is competitivewith direct imitation learning algorithms.Additionally, we show that AIRL isable to recover portable reward functions that are robust to changes in dynamics,enabling us to learn policies even under significant variation in the environmentseen during training.

Proponiamo un algoritmo di reinforcement learning inverso in grado di apprendere funzioni di reward che possono essere trasferite a nuovi ambienti non visti.


We consider two questions at the heart of machine learning; how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well?Our work responds to \citet{zhang2016understanding}, who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs.We show that the same phenomenon occurs in small linear models.These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization.We also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.We propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large.Interpreting stochastic gradient descent as a stochastic differential equation, we identify the ``noise scale" $g = \epsilon (\frac{N}{B} - 1) \approx \epsilon N/B$, where $\epsilon$ is the learning rate, $N$ the training set size and $B$ the batch size.Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, $B_{opt} \propto \epsilon N$.We verify these predictions empirically.

La generalizzazione è fortemente correlata all'evidenza bayesiana, e il rumore del gradiente spinge SGD verso i minimi la cui evidenza è grande.


In the industrial field, the positron annihilation is not affected by complex environment, and the gamma-ray photon penetration is strong, so the nondestructive detection of industrial parts can be realized.Due to the poor image quality caused by gamma-ray photon scattering, attenuation and short sampling time in positron process, we propose the idea of combining deep learning to generate positron images with good quality and clear details by adversarial nets.The structure of the paper is as follows: firstly, we encode to get the hidden vectors of medical CT images based on transfer Learning, and use PCA to extract positron image features.Secondly, we construct a positron image memory based on attention mechanism as a whole input to the adversarial nets which uses medical hidden variables as a query.Finally, we train the whole model jointly and update the input parameters until convergence.Experiments have proved the possibility of generating rare positron images for industrial non-destructive testing using countermeasure networks, and good imaging results have been achieved.

reti adversarial, meccanismo di attenzione, immagini di positroni, scarsità di dati


We revisit the Recurrent Attention Model (RAM, Mnih et al. (2014)), a recurrent neural network for visual attention, from an active information sampling perspective. We borrow ideas from neuroscience research on the role of active information sampling in the context of visual attention and gaze (Gottlieb, 2018), where the author suggested three types of motives for active information sampling strategies.We find the original RAM model only implements one of them.We identify three key weakness of the original RAM and provide a simple solution by adding two extra terms on the objective function.The modified RAM1) achieves faster convergence,2) allows dynamic decision making per sample without loss of accuracy, and3) generalizes much better on longer sequence of glimpses which is not trained for, compared with the original RAM.

Ispirato dalla ricerca sulle neuroscienze, risolve tre debolezze chiave del modello di attenzione ricorrente ampiamente citato aggiungendo semplicemente due termini alla funzione obiettivo.


Graph Neural Networks (GNNs) for prediction tasks like node classification or edge prediction have received increasing attention in recent machine learning from graphically structured data.However, a large quantity of labeled graphs is difficult to obtain, which significantly limit the true success of GNNs.Although active learning has been widely studied for addressing label-sparse issues with other data types like text, images, etc., how to make it effective over graphs is an open question for research.  In this paper, we present the investigation on active learning with GNNs for node classification tasks.  Specifically, we propose a new method, which uses node feature propagation followed by K-Medoids clustering of the nodes for instance selection in active learning.With a theoretical bound analysis we justify the design choice of our approach.In our experiments on four benchmark dataset, the proposed method outperforms other representative baseline methods consistently and significantly.

Questo articolo introduce un algoritmo di active learning su grafi basato sul clustering.


Continuous Normalizing Flows (CNFs) have emerged as promising deep generative models for a wide range of tasks thanks to their invertibility and exact likelihood estimation.However, conditioning CNFs on signals of interest for conditional image generation and downstream predictive tasks is inefficient due to the high-dimensional latent code generated by the model, which needs to be of the same size as the input data.In this paper, we propose InfoCNF, an efficient conditional CNF that partitions the latent space into a class-specific supervised code and an unsupervised code that shared among all classes for efficient use of labeled information.Since the partitioning strategy (slightly) increases the number of function evaluations (NFEs),  InfoCNF also employs gating networks to learn the error tolerances of its ordinary differential equation (ODE) solvers for better speed and performance.We show empirically that InfoCNF improves the test accuracy over the baseline  while yielding comparable likelihood scores and reducing the NFEs on CIFAR10.Furthermore, applying the same partitioning strategy in InfoCNF on time-series data helps improve extrapolation performance.

Proponiamo l'InfoCNF, un efficiente CNF condizionale che impiega gating networks per imparare le tolleranze di errore dei solutori di ODE


A central goal of unsupervised learning is to acquire representations from unlabeled data or experience that can be used for more effective learning of downstream tasks from modest amounts of labeled data.Many prior unsupervised learning works aim to do so by developing proxy objectives based on reconstruction, disentanglement, prediction, and other metrics.Instead, we develop an unsupervised meta-learning method that explicitly optimizes for the ability to learn a variety of tasks from small amounts of data.To do so, we construct tasks from unlabeled data in an automatic way and run meta-learning over the constructed tasks.Surprisingly, we find that, when integrated with meta-learning, relatively simple task construction mechanisms, such as clustering embeddings, lead to good performance on a variety of downstream, human-specified tasks.Our experiments across four image datasets indicate that our unsupervised meta-learning approach acquires a learning algorithm without any labeled data that is applicable to a wide range of downstream classification tasks, improving upon the embedding learned by four prior unsupervised learning methods.

Un metodo di apprendimento unsupervised che utilizza il meta-learning per consentire l'apprendimento efficiente dei downstream task di classificazione delle immagini, superando i metodi allo stato dell'arte.


Domain transfer is a exciting and challenging branch of machine learning because models must learn to smoothly transfer between domains, preserving local variations and capturing many aspects of variation without labels. However, most successful applications to date require the two domains to be closely related (ex. image-to-image, video-video), utilizing similar or shared networks to transform domain specific properties like texture, coloring, and line shapes. Here, we demonstrate that it is possible to transfer across modalities (ex. image-to-audio) by first abstracting the data with latent generative models and then learning transformations between latent spaces. We find that a simple variational autoencoder is able to learn a shared latent space to bridge between two generative models in an unsupervised fashion, and even between different types of models (ex. variational autoencoder and a generative adversarial network). We can further impose desired semantic alignment of attributes with a linear classifier in the shared latent space. The proposed variation autoencoder enables preserving both locality and semantic alignment through the transfer process, as shown in the qualitative and quantitative evaluations.Finally, the hierarchical structure decouples the cost of training the base generative models and semantic alignments, enabling computationally efficient and data efficient retraining of personalized mapping functions.

VAE condizionale su spazi latenti di modelli generativi pre-trained che permette il transfer tra domini drasticamente diversi preservando la localizzazione e l'allineamento semantico.


We propose Adversarial Inductive Transfer Learning (AITL), a method for addressing discrepancies in input and output spaces between source and target domains.AITL utilizes adversarial domain adaptation and multi-task learning to address these discrepancies.Our motivating application is pharmacogenomics where the goal is to predict drug response in patients using their genomic information.The challenge is that clinical data (i.e. patients) with drug response outcome is very limited, creating a need for transfer learning to bridge the gap between large pre-clinical pharmacogenomics datasets (e.g. cancer cell lines) and clinical datasets.Discrepancies exist between1) the genomic data of pre-clinical and clinical datasets (the input space), and2) the different measures of the drug response (the output space).To the best of our knowledge, AITL is the first adversarial inductive transfer learning method to address both input and output discrepancies.Experimental results indicate that AITL outperforms state-of-the-art pharmacogenomics and transfer learning baselines and may guide precision oncology more accurately.

Un nuovo metodo di inductive transfer learning che impiega l'adversarial learning e il multi-task learning per affrontare la discrepanza nello spazio di input e output


Named entity recognition (NER) and relation extraction (RE) are two important tasks in information extraction and retrieval (IE & IR).Recent work has demonstrated that it is beneficial to learn these tasks jointly, which avoids the propagation of error inherent in pipeline-based systems and improves performance.However, state-of-the-art joint models typically rely on external natural language processing (NLP) tools, such as dependency parsers, limiting their usefulness to domains (e.g. news) where those tools perform well.The few neural, end-to-end models that have been proposed are trained almost completely from scratch.In this paper, we propose a neural, end-to-end model for jointly extracting entities and their relations which does not rely on external NLP tools and which integrates a large, pre-trained language model.Because the bulk of our model's parameters are pre-trained and we eschew recurrence for self-attention, our model is fast to train.On 5 datasets across 3 domains, our model matches or exceeds state-of-the-art performance, sometimes by a large margin.

Una nuova architettura ad alte prestazioni per end-to-end named entity recognition e relation extraction che è veloce da addestrare.


In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks.Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80\%.Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs.We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics.We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks.We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them.We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.

Schema di Bayes variazionale per reti neurali ricorrenti


Over the passage of time Unmanned Autonomous Vehicles (UAVs), especiallyAutonomous flying drones grabbed a lot of attention in Artificial Intelligence.Since electronic technology is getting smaller, cheaper and more efficient, hugeadvancement in the study of UAVs has been observed recently.From monitoringfloods, discerning the spread of algae in water bodies to detecting forest trail, theirapplication is far and wide.Our work is mainly focused on autonomous flyingdrones where we establish a case study towards efficiency, robustness and accuracyof UAVs where we showed our results well supported through experiments.We provide details of the software and hardware architecture used in the study.Wefurther discuss about our implementation algorithms and present experiments thatprovide a comparison between three different state-of-the-art algorithms namelyTrailNet, InceptionResnet and MobileNet in terms of accuracy, robustness, powerconsumption and inference time.In our study, we have shown that MobileNet hasproduced better results with very less computational requirement and power consumption.We have also reported the challenges we have faced during our workas well as a brief discussion on our future work to improve safety features andperformance.

caso di studio sul modello ottimale di deep learning per gli UAV


Music relies heavily on repetition to build structure and meaning.  Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure.  The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence.This suggests that self-attention might also be well-suited to modeling music.In musical composition and performance, however, relative timing is critically important.  Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018).  This is impractical for long sequences such as musical compositions since their memory complexity is quadratic in the sequence length.  We propose an algorithm that reduces the intermediate memory requirements to linear in the sequence length.This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long (thousands of steps) compositions with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies.   We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-competition, and obtain state-of-the-art results on the latter.

Mostriamo il primo utilizzo con successo di Transformer nella generazione di musica che mostra una struttura a lungo termine.


Sequential decision problems for real-world applications often need to be solved in real-time, requiring algorithms to perform well with a restricted computational budget.Width-based lookaheads have shown state-of-the-art performance in classical planning problems as well as over the Atari games with tight budgets.In this work we investigate width-based lookaheads over Stochastic Shortest paths (SSP).We analyse why width-based algorithms perform poorly over SSP problems, and overcome these pitfalls proposing a method to estimate costs-to-go.We formalize width-based lookaheads as an instance of the rollout algorithm, give a definition of width for SSP problems and explain its sample complexity.Our experimental results over a variety of SSP benchmarks show the algorithm to outperform other state-of-the-art rollout algorithms such as UCT and RTDP.

Proponiamo un nuovo algoritmo di Monte Carlo Tree Search / rollout che utilizza la ricerca basata sulla larghezza per costruire un lookahead.


Deep Neural Networks (DNNs) are known for excellent performance in supervised tasks such as classification.Convolutional Neural Networks (CNNs), in particular, can learn effective features and build high-level representations that can be used forclassification, but also for querying and nearest neighbor search.However, CNNs have also been shown to suffer from a performance drop when the distribution of the data changes from training to test data.In this paper we analyze the internalrepresentations of CNNs and observe that the representations of unseen data in each class, spread more (with higher variance) in the embedding space of the CNN compared to representations of the training data.More importantly, this difference is more extreme if the unseen data comes from a shifted distribution.Based on this observation, we objectively evaluate the degree of representation’s variance in each class by applying eigenvalue decomposition on the within-class covariance of the internal representations of CNNs and observe the same behaviour.This can be problematic as larger variances might lead to mis-classification if the sample crosses the decision boundary of its class.We apply nearest neighbor classification on the representations and empirically show that the embeddings with the high variance actually have significantly worse KNN classification performances, although this could not be foreseen from their end-to-end classification results.To tackle this problem, we propose Deep Within-Class Covariance Analysis (DWCCA), a deep neural network layer that significantly reduces the within-class covariance of a DNN’s representation, improving performance on unseen test data from a shifted distribution.We empirically evaluate DWCCA on two datasets for Acoustic Scene Classification (DCASE2016 and DCASE2017).We demonstrate that not only does DWCCA significantly improve the network’s internal representation, italso increases the end-to-end classification accuracy, especially when the test set exhibits a slight distribution shift.By adding DWCCA to a VGG neural network, we achieve around 6 percentage points improvement in the case of a distributionmismatch.

Proponiamo un nuovo layer di deep neural network per normalizzare la covarianza all'interno della classe di una rappresentazione interna in una rete neurale che risulta in un miglioramento significativo della generalizzazione delle rappresentazioni apprese.


Generative models have proven to be an outstanding tool for representing high-dimensional probability distributions and generating realistic looking images.A fundamental characteristic of generative models is their ability to produce multi-modal outputs.However, while training, they are often susceptible to mode collapse, which means that the model is limited in mapping the input noise to only a few modes of the true data distribution.In this paper, we draw inspiration from Determinantal Point Process (DPP) to devise a generative model that alleviates mode collapse while producing higher quality samples.DPP is an elegant probabilistic measure used to model negative correlations within a subset and hence quantify its diversity.We use DPP kernel to model the diversity in real data as well as in synthetic data.Then, we devise a generation penalty term that encourages the generator to synthesize data with a similar diversity to real data.In contrast to previous state-of-the-art generative models that tend to use additional trainable parameters or complex training paradigms, our method does not change the original training scheme.Embedded in an adversarial training and variational autoencoder, our  Generative DPP approach shows a consistent resistance to mode-collapse on a wide-variety of synthetic data and natural image datasets including MNIST, CIFAR10, and CelebA, while outperforming state-of-the-art methods for data-efficiency, convergence-time, and generation quality.Our code will be made publicly available.

L'aggiunta di un criterio di diversità ispirato a DPP nell'obiettivo GAN evita il mode collapse e porta a generazioni migliori.


Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization.In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks.Our capacity bound correlates with the behavior of test error with increasing network sizes (within the range reported in the experiments), and could partly explain the improvement in generalization with over-parametrization.We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.

Suggeriamo un bound di generalizzazione che potrebbe spiegare in parte il miglioramento della generalizzazione con la sovra-parametrizzazione.


We introduce three generic point cloud processing blocks that improve both accuracy and memory consumption of multiple state-of-the-art networks, thus allowing to design deeper and more accurate networks.The novel processing blocks that facilitate efficient information flow are a convolution-type operation block for point sets that blends neighborhood information in a memory-efficient manner; a multi-resolution point cloud processing block; and a crosslink block that efficiently shares information across low- and high-resolution processing branches.Combining these blocks, we design significantly wider and deeper architectures.We extensively evaluate the proposed architectures on multiple point segmentation benchmarks (ShapeNetPart, ScanNet, PartNet) and report systematic improvements in terms of both accuracy and memory consumption by using our generic modules in conjunction with multiple recent architectures (PointNet++, DGCNN, SpiderCNN, PointCNN).We report a 9.7% increase in IoU on the PartNet dataset, which is the most complex, while decreasing memory footprint  by  57%.

Introduciamo tre blocchi generici di elaborazione delle nuvole di punti che migliorano sia la precisione che il consumo di memoria di più reti allo stato dell'arte, permettendo così di progettare reti più profonde e accurate.


End-to-end acoustic-to-word speech recognition models have recently gained popularity because they are easy to train, scale well to large amounts of training data, and do not require a lexicon.In addition, word models may also be easier to integrate with downstream tasks such as spoken language understanding, because inference (search) is much simplified compared to phoneme, character or any other sort of sub-word units.In this paper, we describe methods to construct contextual acoustic word embeddings directly from a supervised sequence-to-sequence acoustic-to-word speech recognition model using the learned attention distribution.On a suite of 16 standard sentence evaluation tasks, our embeddings show competitive performance against a word2vec model trained on the speech transcriptions.In addition, we evaluate these embeddings on a spoken language understanding task and observe that our embeddings match the performance of text-based embeddings in a pipeline of first performing speech recognition and then constructing word embeddings from transcriptions.

Metodi per apprendere word embedding acustici contestuali da un modello di riconoscimento vocale end-to-end, che si comportano in modo competitivo con i word embedding basati sul testo.


Unsupervised monocular depth estimation has made great progress after deeplearning is involved.Training with binocular stereo images is considered as agood option as the data can be easily obtained.However, the depth or disparityprediction results show poor performance for the object boundaries.The mainreason is related to the handling of occlusion areas during the training.In this paper,we propose a novel method to overcome this issue.Exploiting disparity mapsproperty, we generate an occlusion mask to block the back-propagation of the occlusionareas during image warping.We also design new networks with flippedstereo images to induce the networks to learn occluded boundaries.It shows thatour method achieves clearer boundaries and better evaluation results on KITTIdriving dataset and Virtual KITTI dataset.

Questo articolo propone un metodo di masking che risolve i precedenti risultati sfocati della stima della profondità monoculare non supervisionata causata dall'occlusione


Graph classification is currently dominated by graph kernels, which, while powerful, suffer some significant limitations.Convolutional Neural Networks (CNNs) offer a very appealing alternative.However, processing graphs with CNNs is not trivial.To address this challenge, many sophisticated extensions of CNNs have recently been proposed.In this paper, we reverse the problem: rather than proposing yet another graph CNN model, we introduce a novel way to represent graphs as multi-channel image-like structures that allows them to be handled by vanilla 2D CNNs.Despite its simplicity, our method proves very competitive to state-of-the-art graph kernels and graph CNNs, and outperforms them by a wide margin on some datasets.It is also preferable to graph kernels in terms of time complexity.Code and data are publicly available.

Introduciamo un nuovo modo di rappresentare i grafi come strutture multicanale simili alle immagini che permette loro di essere gestiti da CNN 2D standard.


The key attribute that drives the unprecedented success of modern Recurrent Neural Networks (RNNs) on learning tasks which involve sequential data, is their ever-improving ability to model intricate long-term temporal dependencies.However, a well established measure of RNNs' long-term memory capacity is lacking, and thus formal understanding of their ability to correlate data throughout time is limited.Though depth efficiency in convolutional networks is well established by now, it does not suffice in order to account for the success of deep RNNs on inputs of varying lengths, and the need to address their 'time-series expressive power' arises.In this paper, we analyze the effect of depth on the ability of recurrent networks to express correlations ranging over long time-scales.To meet the above need, we introduce a measure of the information flow across time that can be supported by the network, referred to as the Start-End separation rank.Essentially, this measure reflects the distance of the function realized by the recurrent network from a function that models no interaction whatsoever between the beginning and end of the input sequence.We prove that deep recurrent networks support Start-End separation ranks which are exponentially higher than those supported by their shallow counterparts.Moreover, we show that the ability of deep recurrent networks to correlate different parts of the input sequence increases exponentially as the input sequence extends, while that of vanilla shallow recurrent networks does not adapt to the sequence length at all.Thus, we establish that depth brings forth an overwhelming advantage in the ability of recurrent networks to model long-term dependencies, and provide an exemplar of quantifying this key attribute which may be readily extended to other RNN architectures of interest, e.g. variants of LSTM networks.We obtain our results by considering a class of recurrent networks referred to as Recurrent Arithmetic Circuits (RACs), which merge the hidden state with the input via the Multiplicative Integration operation.

Proponiamo una misura della memoria a lungo termine e dimostriamo che le deep recurrent network sono molto più adatte a modellare le dipendenze temporali a lungo termine rispetto a quelle superficiali.


Holistically exploring the perceptual and neural representations underlying animal communication has traditionally been very difficult because of the complexity of the underlying signal.We present here a novel set of techniques to project entire communicative repertoires into low dimensional spaces that can be systematically sampled from, exploring the relationship between perceptual representations, neural representations, and the latent representational spaces learned by machine learning algorithms.We showcase this method in one ongoing experiment studying sequential and temporal maintenance of context in songbird neural and perceptual representations of syllables.We further discuss how studying the neural mechanisms underlying the maintenance of the long-range information content present in birdsong can inform and be informed by machine sequence modeling.

Confrontiamo le rappresentazioni percettive, neurali e modellate della comunicazione animale utilizzando il machine learning, il comportamento e la fisiologia.


The information bottleneck principle (Shwartz-Ziv & Tishby, 2017) suggests that SGD-based training of deep neural networks results in optimally compressed hidden layers, from an information theoretic perspective.However, this claim was established on toy data.The goal of the work we present here is to test these claims in a realistic setting using a larger and deeper convolutional architecture, a ResNet model.We trained PixelCNN++ models as inverse representation decoders to measure the mutual information between hidden layers of a ResNet and input image data, when trained for (1) classification and (2) autoencoding.We find that two stages of learning happen for both training regimes, and that compression does occur, even for an autoencoder.Sampling images by conditioning on hidden layers’ activations offers an intuitive visualisation to understand what a ResNets learns to forget.

Il principio dell'Information Bottleneck applicato alle ResNets, usando i modelli PixelCNN++ per decodificare l'informazione mutua e generare condizionatamente immagini per l'illustrazione delle informazioni


We study the problem of safe adaptation: given a model trained on a variety of past experiences for some task, can this model learn to perform that task in a new situation while avoiding catastrophic failure?This problem setting occurs frequently in real-world reinforcement learning scenarios such as a vehicle adapting to drive in a new city, or a robotic drone adapting a policy trained only in simulation.While learning without catastrophic failures is exceptionally difficult, prior experience can allow us to learn models that make this much easier.These models might not directly transfer to new settings, but can enable cautious adaptation that is substantially safer than na\"{i}ve adaptation as well as learning from scratch.Building on this intuition, we propose risk-averse domain adaptation (RADA).RADA works in two steps: it first trains probabilistic model-based RL agents in a population of source domains to gain experience and capture epistemic uncertainty about the environment dynamics.Then, when dropped into a new environment, it employs a pessimistic exploration policy, selecting actions that have the best worst-case performance as forecasted by the probabilistic model.We show that this simple maximin policy accelerates domain adaptation in a safety-critical driving environment with varying vehicle sizes.We compare our approach against other approaches for adapting to new environments, including meta-reinforcement learning.

L'adattamento di un agente RL in un ambiente di destinazione con dinamiche sconosciute è veloce e sicuro quando trasferiamo l'esperienza precedente in una varietà di ambienti e poi selezioniamo azioni avverse al rischio durante l'adattamento.


We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers.Our model builds an object-based scene representation and translates sentences into executable, symbolic programs.To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation.Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to.Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences.We use curriculum learning to guide the searching over the large compositional space of images and language.Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences.Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains.It also empowers applications including visual question answering and bidirectional image-text retrieval.

Proponiamo il Neuro-Symbolic Concept Learner (NS-CL), un modello che impara concetti visivi, parole e parsing semantico di frasi senza supervisione esplicita su nessuno di essi.


Bayesian inference offers a theoretically grounded and general way to train neural networks and can potentially give calibrated uncertainty.However, it is challenging to specify a meaningful and tractable prior over the network parameters, and deal with the weight correlations in the posterior.To this end, this paper introduces two innovations:(i) a Gaussian process-based hierarchical model for the network parameters based on recently introduced unit embeddings that can flexibly encode weight structures, and(ii) input-dependent contextual variables for the weight prior that can provide convenient ways to regularize the function space being modeled by the network through the use of kernels. We show these models provide desirable test-time uncertainty estimates, demonstrate cases of modeling inductive biases for neural networks with kernels and demonstrate competitive predictive performance on an active learning benchmark.

Introduciamo un Gaussian Process Prior sui pesi in una rete neurale ed esploriamo la sua capacità di modellare i pesi dipendenti dall'input con benefici per vari task, inclusa la stima dell'incertezza e la generalizzazione nei setting low-sample.


We perform an in-depth investigation of the suitability of self-attention models for character-level neural machine translation.We test the standard transformer model, as well as a novel variant in which the encoder block combines information from nearby characters using convolution.We perform extensive experiments on WMT and UN datasets, testing both bilingual and multilingual translation to English using up to three input languages (French, Spanish, and Chinese).Our transformer variant consistently outperforms the standard transformer at the character-level and converges faster while learning more robust character-level alignments.

Eseguiamo un'indagine approfondita sull'idoneità dei modelli di self-attention per character-level neural machine translation.