Determinantal point processes (DPPs) is an effective tool to deliver diversity on multiple machine learning and computer vision tasks.Under deep learning framework, DPP is typically optimized via approximation, which is not straightforward and has some conflict with diversity requirement.We note, however, there has been no deep learning paradigms to optimize DPP directly since it involves matrix inversion which may result in highly computational instability.This fact greatly hinders the wide use of DPP on some specific objectives where DPP serves as a term to measure the feature diversity.In this paper, we devise a simple but effective algorithm to address this issue to optimize DPP term directly expressed with L-ensemble in spectral domain over gram matrix, which is more flexible than learning on parametric kernels.By further taking into account some geometric constraints, our algorithm seeks to generate valid sub-gradients of DPP term in case when the DPP gram matrix is not invertible (no gradients exist in this case).In this sense, our algorithm can be easily incorporated with multiple deep learning tasks.Experiments show the effectiveness of our algorithm, indicating promising performance for practical learning problems.

Abbiamo proposto un metodo specifico di back-propagation attraverso un appropriato sub-gradiente spettrale per integrare il determinantal point process nel framework del deep learning.


The quality of a machine translation system depends largely on the availability of sizable parallel corpora.For the recently popular Neural Machine Translation (NMT) framework, data sparsity problem can become even more severe.With large amount of tunable parameters, the NMT model may overfit to the existing language pairs while failing to understand the general diversity in language.In this paper, we advocate to broadcast every sentence pair as two groups of similar sentences to incorporate more diversity in language expressions, which we name as parallel cluster.Then we define a more general cluster-to-cluster correspondence score and train our model to maximize this score.Since direct maximization is difficult, we derive its lower-bound as our surrogate objective, which is found to generalize point-point Maximum Likelihood Estimation (MLE) and point-to-cluster Reward Augmented Maximum Likelihood (RAML) algorithms as special cases.Based on this novel objective function, we delineate four potential systems to realize our cluster-to-cluster framework and test their performances in three recognized translation tasks, each task with forward and reverse translation directions.In each of the six experiments, our proposed four parallel systems have consistently proved to outperform the MLE baseline, RL (Reinforcement Learning) and RAML systems significantly.Finally, we have performed case study to empirically analyze the strength of the cluster-to-cluster NMT framework.

Inventiamo un nuovo framework cluster-to-cluster per il training NMT, che può comprendere meglio la diversità tra la lingua di origine e quella di destinazione.


The capability of making interpretable and self-explanatory decisions is essential for developing responsible machine learning systems.In this work, we study the learning to explain the problem in the scope of inductive logic programming (ILP).We propose Neural Logic Inductive Learning (NLIL), an efficient differentiable ILP framework that learns first-order logic rules that can explain the patterns in the data.In experiments, compared with the state-of-the-art models, we find NLIL is able to search for rules that are x10 times longer while remaining x3 times faster.We also show that NLIL can scale to large image datasets, i.e. Visual Genome, with 1M entities.

Un efficiente modello ILP differenziabile che impara regole logiche del primo ordine che possono spiegare i dati.


Neural network-based classifiers parallel or exceed human-level accuracy on many common tasks and are used in practical systems.Yet, neural networks are susceptible to adversarial examples, carefully perturbed inputs that cause networks to misbehave in arbitrarily chosen ways.When generated with standard methods, these examples do not consistently fool a classifier in the physical world due to a combination of viewpoint shifts, camera noise, and other natural transformations.Adversarial examples generated using standard techniques require complete control over direct input to the classifier, which is impossible in many real-world systems.We introduce the first method for constructing real-world 3D objects that consistently fool a neural network across a wide distribution of angles and viewpoints.We present a general-purpose algorithm for generating adversarial examples that are robust across any chosen distribution of transformations.We demonstrate its application in two dimensions, producing adversarial images that are robust to noise, distortion, and affine transformation.Finally, we apply the algorithm to produce arbitrary physical 3D-printed adversarial objects, demonstrating that our approach works end-to-end in the real world.Our results show that adversarial examples are a practical concern for real-world systems.

Introduciamo un nuovo metodo per sintetizzare adversarial example robusti nel mondo fisico e lo usiamo per fabbricare i primi oggetti 3D di avversione.


Representations of sets are challenging to learn because operations on sets should be permutation-invariant.To this end, we propose a Permutation-Optimisation module that learns how to permute a set end-to-end.The permuted set can be further processed to learn a permutation-invariant representation of that set, avoiding a bottleneck in traditional set models.We demonstrate our model's ability to learn permutations and set representations with either explicit or implicit supervision on four datasets, on which we achieve state-of-the-art results: number sorting, image mosaics, classification from image mosaics, and visual question answering.

Impara come permutare un insieme, poi codifica l'insieme permutato con RNN per ottenere una rappresentazione dell'insieme.


The physical design of a robot and the policy that controls its motion are inherently coupled.However, existing approaches largely ignore this coupling, instead choosing to alternate between separate design and control phases, which requires expert intuition throughout and risks convergence to suboptimal designs.In this work, we propose a method that jointly optimizes over the physical design of a robot and the corresponding control policy in a model-free fashion, without any need for expert supervision.Given an arbitrary robot morphology, our method maintains a distribution over the design parameters and uses reinforcement learning to train a neural network controller.Throughout training, we refine the robot distribution to maximize the expected reward.This results in an assignment to the robot parameters and neural network policy that are jointly optimal.We evaluate our approach in the context of legged locomotion, and demonstrate that it discovers novel robot designs and walking gaits for several different morphologies, achieving performance comparable to or better than that of hand-crafted designs.

Utilizzare il deep reinforcement learning per progettare gli attributi fisici di un robot insieme ad una policy di controllo.


Deep learning enables training of large and flexible function approximators from scratch at the cost of large amounts of data.Applications of neural networks often consider learning in the context of a single task.However, in many scenarios what we hope to learn is not just a single task, but a model that can be used to solve multiple different tasks.Such multi-task learning settings have the potential to improve data efficiency and generalization by sharing data and representations across tasks.However, in some challenging multi-task learning settings, particularly in reinforcement learning, it is very difficult to learn a single model that can solve all the tasks while realizing data efficiency and performance benefits.Learning each of the tasks independently from scratch can actually perform better in such settings, but it does not benefit from the representation sharing that multi-task learning can potentially provide.In this work, we develop an approach that endows a single model with the ability to represent both extremes: joint training and independent training.To this end, we introduce matrix-interleaving (Mint), a modification to standard neural network models that projects the activations for each task into a different learned subspace, represented by a per-task and per-layer matrix.By learning these matrices jointly with the other model parameters, the optimizer itself can decide how much to share representations between tasks.On three challenging multi-task supervised learning and reinforcement learning problems with varying degrees of shared task structure, we find that this model consistently matches or outperforms joint training and independent training, combining the best elements of both.

Proponiamo un approccio che conferisce a un singolo modello la capacità di rappresentare entrambi gli estremi: joint training e training indipendente, il che porta a un efficace apprendimento multi-task.


Training agents to operate in one environment often yields overfitted models that are unable to generalize to the changes in that environment.However, due to the numerous variations that can occur in the real-world, the agent is often required to be robust in order to be useful.This has not been the case for agents trained with reinforcement learning (RL) algorithms.In this paper, we investigate the overfitting of RL agents to the training environments in visual navigation tasks.Our experiments show that deep RL agents can overfit even when trained on multiple environments simultaneously. We propose a regularization method which combines RL with supervised learning methods by adding a term to the RL objective that would encourage the invariance of a policy to variations in the observations that ought not to affect the action taken.The results of this method, called invariance regularization, show an improvement in the generalization of policies to environments not seen during training.

Proponiamo un termine di regolarizzazione che, quando aggiunto all'obiettivo di reinforcement learning, permette alla policy di massimizzare la reward e contemporaneamente imparare ad essere invariante ai cambiamenti irrilevanti all'interno dell'input.


Though visual information has been introduced for enhancing neural machine translation (NMT), its effectiveness strongly relies on the availability of large amounts of bilingual parallel sentence pairs with manual image annotations.In this paper, we present a universal visual representation learned over the monolingual corpora with image annotations, which overcomes the lack of large-scale bilingual sentence-image pairs, thereby extending image applicability in NMT.In detail, a group of images with similar topics to the source sentence will be retrieved from a light topic-image lookup table learned over the existing sentence-image pairs, and then is encoded as image representations by a pre-trained ResNet.An attention layer with a gated weighting is to fuse the visual information and text information as input to the decoder for predicting target translations.In particular, the proposed method enables the visual information to be integrated into large-scale text-only NMT in addition to the multimodel NMT.Experiments on four widely used translation datasets, including the WMT'16 English-to-Romanian, WMT'14 English-to-German, WMT'14 English-to-French, and Multi30K, show that the proposed approach achieves significant improvements over strong baselines.

Questo lavoro ha proposto una rappresentazione visiva universale per la traduzione automatica neurale (NMT) utilizzando immagini recuperate con argomenti simili alla frase di origine, estendendo l'applicabilità delle immagini nella NMT.


This paper introduces a novel framework for learning algorithms to solve online combinatorial optimization problems.Towards this goal, we introduce a number of key ideas from traditional algorithms and complexity theory.First, we draw a new connection between primal-dual methods and reinforcement learning.Next, we introduce the concept of adversarial distributions (universal and high-entropy training sets), which are distributions that encourage the learner to find algorithms that work well in the worst case.We test our new ideas on a number of optimization problem such as the AdWords problem, the online knapsack problem, and the secretary problem.Our results indicate that the models have learned behaviours that are consistent with the traditional optimal algorithms for these problems.

Combinando idee dalla progettazione di algoritmi tradizionali e dal reinforcement learning, introduciamo un nuovo framework per l'apprendimento di algoritmi che risolvono problemi di ottimizzazione combinatoria online.


Despite their popularity and successes, deep neural networks are poorly understood theoretically and treated as 'black box' systems.Using a functional view of these networks gives us a useful new lens with which to understand them.This allows us us to theoretically or experimentally probe properties of these networks, including the effect of standard initializations, the value of depth, the underlying loss surface, and the origins of generalization.One key result is that generalization results from smoothness of the functional approximation, combined with a flat initial approximation.This smoothness increases with number of units, explaining why massively overparamaterized networks continue to generalize well.

Un approccio funzionale rivela che l'inizializzazione flat, preservata dalla gradient descent, porta alla capacità di generalizzazione.


It is well-known that deeper neural networks are harder to train than shallower ones.In this short paper, we use the (full) eigenvalue spectrum of the Hessian to explore how the loss landscape changes as the network gets deeper, and as residual connections are added to the architecture.Computing a series of quantitative measures on the Hessian spectrum, we show that the Hessian eigenvalue distribution in deeper networks has substantially heavier tails (equivalently, more outlier eigenvalues), which makes the network harder to optimize with first-order methods.We show that adding residual connections mitigates this effect substantially, suggesting a mechanism by which residual connections improve training.

La profondità della rete aumenta gli autovalori outlier nell'Hessiana. Le residual connection mitigano questo fenomeno.


In the context of optimization, a gradient of a neural network indicates the amount a specific weight should change with respect to the loss.Therefore, small gradients indicate a good value of the weight that requires no change and can be kept frozen during training.This paper provides an experimental study on the importance of a neural network weights, and to which extent do they need to be updated.We wish to show that starting from the third epoch, freezing weights which have no informative gradient and are less likely to be changed during training, results in a very slight drop in the overall accuracy (and in sometimes better).We experiment on the MNIST, CIFAR10 and Flickr8k datasets using several architectures (VGG19,ResNet-110 and DenseNet-121).On CIFAR10, we show that freezing 80% of the VGG19 network parameters from the third epoch onwards results in 0.24% drop in accuracy, while freezing 50% of Resnet-110 parameters results in 0.9% drop in accuracy and finally freezing 70% of Densnet-121 parameters results in 0.57% drop in accuracy.Furthermore, to experiemnt with real-life applications, we train an image captioning model with attention mechanism on the Flickr8k dataset using LSTM networks, freezing 60% of the parameters from the third epoch onwards, resulting in a better BLEU-4 score than the fully trained model.Our source code can be found in the appendix.

Un paper sperimentale che dimostra la quantità di pesi ridondanti che possono essere congelati solo a partire dalla terza epoch, con solo un leggerissimo calo di precisione.


Like humans, deep networks learn better when samples are organized and introduced in a meaningful order or curriculum.While conventional approaches to curriculum learning emphasize the difficulty of samples as the core incremental strategy, it forces networks to learn from small subsets of data while introducing pre-computation overheads.In this work, we propose Learning with Incremental Labels and Adaptive Compensation (LILAC), which introduces a novel approach to curriculum learning.LILAC emphasizes incrementally learning labels instead of incrementally learning difficult samples.It works in two distinct phases: first, in the incremental label introduction phase, we unmask ground-truth labels in fixed increments during training, to improve the starting point from which networks learn.In the adaptive compensation phase, we compensate for failed predictions by adaptively altering the target vector to a smoother distribution.We evaluate LILAC against the closest comparable methods in batch and curriculum learning and label smoothing, across three standard image benchmarks, CIFAR-10, CIFAR-100, and STL-10.We show that our method outperforms batch learning with higher mean recognition accuracy as well as lower standard deviation in performance consistently across all benchmarks.We further extend LILAC to state-of-the-art performance across CIFAR-10 using simple data augmentation while exhibiting label order invariance among other important properties.

Un nuovo approccio al curriculum learning attraverso l'apprendimento incrementale delle label e lo smoothing adattivo delle label per i sample mal classificati che aumenta la performance media e diminuisce la deviazione standard.


Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare.Learning representations for words in the ``long tail'' of this distribution requires enormous amounts of data. Representations of rare words trained directly on end tasks are usually poor, requiring us to pre-train embeddings on external data, or treat all rare words as out-of-vocabulary words with a unique representation.We provide a method for predicting embeddings of rare words on the fly from small amounts of auxiliary data with a network trained end-to-end for the downstream task.We show that this improves results against baselines where embeddings are trained on the end task for reading comprehension, recognizing textual entailment and language modeling.

Proponiamo un metodo per trattare le parole rare calcolando la loro embedding dalle definizioni.


The capability of reliably detecting out-of-distribution samples is one of the key factors in deploying a good classifier, as the test distribution always does not match with the training distribution in most real-world applications.In this work, we propose a deep generative classifier which is effective to detect out-of-distribution samples as well as classify in-distribution samples, by integrating the concept of Gaussian discriminant analysis into deep neural networks.Unlike the discriminative (or softmax) classifier that only focuses on the decision boundary partitioning its latent space into multiple regions, our generative classifier aims to explicitly model class-conditional distributions as separable Gaussian distributions.Thereby, we can define the confidence score by the distance between a test sample and the center of each distribution.Our empirical evaluation on multi-class images and tabular data demonstrate that the generative classifier achieves the best performances in distinguishing out-of-distribution samples, and also it can be generalized well for various types of deep neural networks.

Questo articolo propone un deep generative classifier che è efficace per rilevare i sample out-of-distribution così come per classificare i sample in-distribution, integrando il concetto di analisi discriminante gaussiana nelle deep neural network.


One of the most prevalent symptoms among the elderly population, dementia, can be detected by classifiers trained on linguistic features extracted from narrative transcripts.However, these linguistic features are impacted in a similar but different fashion by the normal aging process.Aging is therefore a confounding factor, whose effects have been hard for machine learning classifiers to isolate. In this paper, we show that deep neural network (DNN) classifiers can infer ages from linguistic features, which is an entanglement that could lead to unfairness across age groups.We show this problem is caused by undesired activations of v-structures in causality diagrams, and it could be addressed with fair representation learning.We build neural network classifiers that learn low-dimensional representations reflecting the impacts of dementia yet discarding the effects of age.To evaluate these classifiers, we specify a model-agnostic score $\Delta_{eo}^{(N)}$ measuring how classifier results are disentangled from age.Our best models outperform baseline neural network classifiers in disentanglement, while compromising accuracy by as little as 2.56\% and 2.25\% on DementiaBank and the Famous People dataset respectively.

Mostrare che l'età confonde l'individuazione del deterioramento cognitivo + risolvere con representation learning fair + proporre metriche e modelli.


Much of the focus in the design of deep neural networks had been on improving accuracy, leading to more powerful yet highly complex network architectures that are difficult to deploy in practical scenarios.  As a result, there has been a recent interest in the design of quantitative metrics for evaluating deep neural networks that accounts for more than just model accuracy as the sole indicator of network performance.  In this study, we continue the conversation towards universal metrics for evaluating the performance of deep neural networks for practical on-device edge usage by introducing NetScore, a new metric designed specifically to provide a quantitative assessment of the balance between accuracy, computational complexity, and network architecture complexity of a deep neural network.  In what is one of the largest comparative analysis between deep neural networks in literature, the NetScore metric, the top-1 accuracy metric, and the popular information density metric were compared across a diverse set of 60 different deep convolutional neural networks for image classification on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC 2012) dataset.  The evaluation results across these three metrics for this diverse set of networks are presented in this study to act as a reference guide for practitioners in the field.

Introduciamo NetScore, una nuova metrica progettata per fornire una valutazione quantitativa dell'equilibrio tra precisione, complessità computazionale e complessità dell'architettura di una deep neural network.


Deep Neural Networks (DNNs) are vulnerable to adversarial attacks, especially white-box targeted attacks.This paper studies the problem of how aggressive white-box targeted attacks can be to go beyond widely used Top-1 attacks.We propose to learn ordered Top-k attacks (k>=1), which enforce the Top-k predicted labels of an adversarial example to be the k (randomly) selected and ordered labels (the ground-truth label is exclusive).Two methods are presented.First, we extend the vanilla Carlini-Wagner (C&W) method and use it as a strong baseline.Second, we present an adversarial distillation framework consisting of two components:(i) Computing an adversarial probability distribution for any given ordered Top-$k$ targeted labels.(ii) Learning adversarial examples by minimizing the Kullback-Leibler (KL) divergence between the adversarial distribution and the predicted distribution, together with the perturbation energy penalty.In computing adversarial distributions, we explore how to leverage label semantic similarities, leading to knowledge-oriented attacks.In experiments, we test Top-k (k=1,2,5,10) attacks in the ImageNet-1000 val dataset using two popular DNNs trained with the clean ImageNet-1000  train dataset, ResNet-50 and DenseNet-121.Overall, the adversarial distillation approach obtains the best results, especially by large margin when computation budget is limited..It reduces the perturbation energy consistently with the same attack success rate on all the four k's, and improve the attack success rate by large margin against the modified C&W method for k=10.


Neural message passing algorithms for semi-supervised classification on graphs have recently achieved great success.However, for classifying a node these methods only consider nodes that are a few propagation steps away and the size of this utilized neighborhood is hard to extend.In this paper, we use the relationship between graph convolutional networks (GCN) and PageRank to derive an improved propagation scheme based on personalized PageRank.We utilize this propagation procedure to construct a simple model, personalized propagation of neural predictions (PPNP), and its fast approximation, APPNP.Our model's training time is on par or faster and its number of parameters on par or lower than previous models.It leverages a large, adjustable neighborhood for classification and can be easily combined with any neural network.We show that this model outperforms several recently proposed methods for semi-supervised classification in the most thorough study done so far for GCN-like models.Our implementation is available online.

La propagazione personalizzata delle previsioni neurali (PPNP) migliora le graph neural network separandole in predizione e propagazione tramite PageRank personalizzato.


Most of recent work in cross-lingual word embeddings is severely Anglocentric.The vast majority of lexicon induction evaluation dictionaries are between English and another language, and the English embedding space is selected by default as the hub when learning in a multilingual setting.With this work, however, we challenge these practices.First, we show that the choice of hub language can significantly impact downstream lexicon induction performance.Second, we both expand the current evaluation dictionary collection to include all language pairs using triangulation, and also create new dictionaries for under-represented languages.Evaluating established methods over all these language pairs sheds light into their suitability and presents new challenges for the field.Finally, in our analysis we identify general guidelines for strong cross-lingual embeddings baselines, based on more than just Anglocentric experiments.

La scelta della lingua principale (di destinazione) influisce sulla qualità degli embedding multilinguali, che non dovrebbero essere valutate solo su dizionari anglo-centrici.


Interpreting generative adversarial network (GAN) training as approximate divergence minimization has been theoretically insightful, has spurred discussion, and has lead to theoretically and practically interesting extensions such as f-GANs and Wasserstein GANs.For both classic GANs and f-GANs, there is an original variant of training and a "non-saturating" variant which uses an alternative form of generator update.The original variant is theoretically easier to study, but the alternative variant frequently performs better and is recommended for use in practice.The alternative generator update is often regarded as a simple modification to deal with optimization issues, and it appears to be a common misconception that the two variants minimize the same divergence.In this short note we derive the divergences approximately minimized by the original and alternative variants of GAN and f-GAN training.This highlights important differences between the two variants.For example, we show that the alternative variant of KL-GAN training actually minimizes the reverse KL divergence, and that the alternative variant of conventional GAN training minimizes a "softened" version of the reverse KL.We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.

Il tipico training GAN non ottimizza Jensen-Shannon, ma qualcosa simile ad una divergenza KL inversa.


REINFORCE can be used to train models in structured prediction settings to directly optimize the test-time objective.However, the common case of sampling one prediction per datapoint (input) is data-inefficient.We show that by drawing multiple samples (predictions) per datapoint, we can learn with significantly less data, as we freely obtain a REINFORCE baseline to reduce variance.Additionally we derive a REINFORCE estimator with baseline, based on sampling without replacement.Combined with a recent technique to sample sequences without replacement using Stochastic Beam Search, this improves the training procedure for a sequence model that predicts the solution to the Travelling Salesman Problem.

Mostriamo che estraendo più sample (predizioni) per ogni input (datapoint), possiamo imparare con meno dati poiché otteniamo liberamente una baseline REINFORCE.


Reinforcement learning (RL) is a powerful technique to train an agent to perform a task.  However, an agent that is trained using RL is only capable of achieving the single task that is specified via its reward function.   Such an approach does not scale well to settings in which an agent needs to perform a diverse set of tasks, such as navigating to varying positions in a room or moving objects to varying locations.  Instead, we propose a method that allows an agent to automatically discover the range of tasks that it is capable of performing in its environment.  We use a generator network to propose tasks for the agent to try to achieve, each task being specified as reaching a certain parametrized subset of the state-space.  The generator network is optimized using adversarial training to produce tasks that are always at the appropriate level of difficulty for the agent.  Our method thus automatically produces a curriculum of tasks for the agent to learn.  We show that, by using this framework, an agent can efficiently and automatically learn to perform a wide set of tasks without requiring any prior knowledge of its environment (Videos and code available at: https://sites.google.com/view/goalgeneration4rl).Our method can also learn to achieve tasks with sparse rewards, which pose significant challenges for traditional RL methods.

Risolviamo in modo efficiente i problemi multi-task con un algoritmo di generazione automatica del curriculum basato su un modello generativo che tiene traccia delle prestazioni dell'agente di apprendimento.


A wide range of defenses have been proposed to harden neural networks against adversarial attacks.However, a pattern has emerged in which the majority of adversarial defenses are quickly broken by new attacks.  Given the lack of success at generating robust defenses, we are led to ask a fundamental question:  Are adversarial attacks inevitable?This paper analyzes adversarial examples from a theoretical perspective, and identifies fundamental bounds on the susceptibility of a classifier to adversarial attacks.   We show that, for certain classes of problems, adversarial examples are inescapable.  Using experiments, we explore the implications of theoretical guarantees for real-world problems and discuss how factors such as dimensionality and image complexity limit a classifier's robustness against adversarial examples.

Questo articolo identifica classi di problemi per i quali gli adversarial example sono inevitabili, e deriva limiti fondamentali sulla suscettibilità di qualsiasi classificatore agli adversarial example.


For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks.Activation maps, however, occupy a large memory footprint during both the training and inference step when using mini-batches of inputs.One way to reduce this large memory footprint is to reduce the precision of activations.However, past works have shown that reducing the precision of activations hurts model accuracy.We study schemes to train networks from scratch using reduced-precision activations without hurting accuracy.We reduce the precision of activation maps (along with model parameters) and increase the number of filter maps in a layer, and find that this scheme matches or surpasses the accuracy of the baseline full-precision network.As a result, one can significantly improve the execution efficiency (e.g. reduce dynamic memory footprint, memory band- width and computational energy) and speed up the training and inference process with appropriate hardware support.We call our scheme WRPN -- wide reduced-precision networks.We report results and show that WRPN scheme is better than previously reported accuracies on ILSVRC-12 dataset while being computationally less expensive compared to previously reported reduced-precision networks.

Abbassando la precisione (a 4-bit, 2-bit e persino binario) e allargando i banchi di filtri si ottiene una rete accurata come quelle ottenute con pesi e attivazioni FP32.


We investigate methods for semi-supervised learning (SSL) of a neural linear-chain conditional random field (CRF) for Named Entity Recognition (NER) by treating the tagger as the amortized variational posterior in a generative model of text given tags.We first illustrate how to incorporate a CRF in a VAE, enabling end-to-end training on semi-supervised data.We then investigate a series of increasingly complex deep generative models of tokens given tags enabled by end-to-end optimization, comparing the proposed models against supervised and strong CRF SSL baselines on the Ontonotes5 NER dataset.We find that our best proposed model consistently improves performance by $\approx 1\%$ F1 in low- and moderate-resource regimes and easily addresses degenerate model behavior in a more difficult, partially supervised setting.

Incorporiamo una CRF in una VAE di token e tag NER per l'apprendimento semi-supervised e mostriamo miglioramenti in setting low-resource.


To make deep neural networks feasible in resource-constrained environments (such as mobile devices), it is beneficial to quantize models by using low-precision weights.One common technique for quantizing neural networks is the straight-through gradient method, which enables back-propagation through the quantization mapping.Despite its empirical success, little is understood about why the straight-through gradient method works.Building upon a novel observation that the straight-through gradient method is in fact identical to the well-known Nesterov’s dual-averaging algorithm on a quantization constrained optimization problem, we propose a more principled alternative approach, called ProxQuant , that formulates quantized network training as a regularized learning problem instead and optimizes it via the prox-gradient method.ProxQuant does back-propagation on the underlying full-precision vector and applies an efficient prox-operator in between stochastic gradient steps to encourage quantizedness.For quantizing ResNets and LSTMs, ProxQuant outperforms state-of-the-art results on binary quantization and is on par with state-of-the-art on multi-bit quantization.For binary quantization, our analysis shows both theoretically and experimentally that ProxQuant is more stable than the straight-through gradient method (i.e. BinaryConnect), challenging the indispensability of the straight-through gradient method and providing a powerful alternative.

Un framework teoricamente valido per la quantizzazione del modello usando il metodo del gradiente prossimale.


The vulnerabilities of deep neural networks against adversarial examples have become a significant concern for deploying these models in sensitive domains.Devising a definitive defense against such attacks is proven to be challenging, and the methods relying on detecting adversarial samples are only valid when the attacker is oblivious to the detection mechanism.In this paper, we consider the adversarial detection problem under the robust optimization framework.We partition the input space into subspaces and train adversarial robust subspace detectors using asymmetrical adversarial training (AAT).The integration of the classifier and detectors presents a detection mechanism that provides a performance guarantee to the adversary it considered.We demonstrate that AAT promotes the learning of class-conditional distributions, which further gives rise to generative detection/classification approaches that are both robust and more interpretable.We provide comprehensive evaluations of the above methods, and demonstrate their competitive performances and compelling properties on adversarial detection and robust classification problems.

Una nuova tecnica di modellazione generativa basata sull'adversarial training asimmetrico, e le sue applicazioni al rilevamento di adversarial example e alla classificazione robusta


Exploration is a key component of successful reinforcement learning, but optimal approaches are computationally intractable, so researchers have focused on hand-designing mechanisms based on exploration bonuses and intrinsic reward, some inspired by curious behavior in natural systems.  In this work, we propose a strategy for encoding curiosity algorithms as programs in a domain-specific language and searching, during a meta-learning phase, for algorithms that enable RL agents to perform well in new domains.  Our rich language of programs, which can combine neural networks with other building blocks including nearest-neighbor modules and can choose its own loss functions, enables the expression of highly generalizable programs that perform well in domains as disparate as grid navigation with image input, acrobot, lunar lander, ant and hopper.   To  make this approach feasible, we develop several pruning techniques, including learning to predict a program's success based on its syntactic properties.   We demonstrate the effectiveness of the approach empirically, finding curiosity strategies that are similar to those in published literature, as well as novel strategies that are competitive with them and generalize well.

Il meta-learning con algoritmi di curiosity attraverso la ricerca in un ricco spazio di programmi produce nuovi meccanismi che generalizzano attraverso domini di reinforcement learning molto diversi.


Many machine learning algorithms represent input data with vector embeddings or discrete codes.When inputs exhibit compositional structure (e.g. objects built from parts or procedures from subroutines), it is natural to ask whether this compositional structure is reflected in the the inputs’ learned representations.While the assessment of compositionality in languages has received significant attention in linguistics and adjacent fields, the machine learning literature lacks general-purpose tools for producing graded measurements of compositional structure in more general (e.g. vector-valued) representation spaces.We describe a procedure for evaluating compositionality by measuring how well the true representation-producing model can be approximated by a model that explicitly composes a collection of inferred representational primitives.We use the procedure to provide formal and empirical characterizations of compositional structure in a variety of settings, exploring the relationship between compositionality and learning dynamics, human judgments, representational similarity, and generalization.

Questo articolo propone una semplice procedura per valutare la struttura compositiva nelle rappresentazioni apprese, e usa la procedura per esplorare il ruolo della compositività in quattro problemi di apprendimento.


In this paper, we propose an end-to-end deep learning model, called E2Efold, for RNA secondary structure prediction which can effectively take into account the inherent constraints in the problem.The key idea of E2Efold is to directly predict the RNA base-pairing matrix, and use an unrolled constrained programming algorithm as a building block in the architecture to enforce constraints.With comprehensive experiments on benchmark datasets, we demonstrate the superior performance of E2Efold: it predicts significantly better structures compared to previous SOTA (29.7% improvement in some cases in F1 scores and even larger improvement for pseudoknotted structures) and runs as efficient as the fastest algorithms in terms of inference time.

Un modello DL per la predizione della struttura secondaria dell'RNA, che usa un algoritmo unrolled nell'architettura per far rispettare i vincoli.


Learning in recurrent neural networks (RNNs) is most often implemented by gradient descent using backpropagation through time (BPTT), but BPTT does not model accurately how the brain learns.Instead, many experimental results on synaptic plasticity can be summarized as three-factor learning rules involving eligibility traces of the local neural activity and a third factor.We present here eligibility propagation (e-prop), a new factorization of the loss gradients in RNNs that fits the framework of three factor learning rules when derived for biophysical spiking neuron models.When tested on the TIMIT speech recognition benchmark, it is competitive with BPTT both for training artificial LSTM networks and spiking RNNs.Further analysis suggests that the diversity of learning signals and the consideration of slow internal neural dynamics are decisive to the learning efficiency of e-prop.

Presentiamo la eligibility propagation un'alternativa al BPTT che è compatibile con i dati sperimentali sulla synaptic plasticity e compete con il BPTT sui benchmark di apprendimento automatico.


Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems.RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features.In this paper, we introduce a new technique, Quantized Bottleneck Insertion, to learn finite representations of these vectors and features.The result is a quantized representation of the RNN that can be analyzed to improve our understanding of memory use and general behavior.We present results of this approach on synthetic environments and six Atari games.The resulting finite representations are surprisingly small in some cases, using as few as 3 discrete memory states and 10 observations for a perfect Pong policy.We also show that these finite policy representations lead to improved interpretability.

Estrazione di un automa a stati finiti da una rete neurale ricorrente tramite quantizzazione ai fini dell'interpretabilità con esperimenti su Atari.


Building upon the recent success of deep reinforcement learning methods, we investigate the possibility of on-policy reinforcement learning improvement by reusing the data from several consecutive policies.On-policy methods bring many benefits, such as ability to evaluate each resulting policy.However, they usually discard all the information about the policies which existed before.In this work, we propose adaptation of the replay buffer concept, borrowed from the off-policy learning setting, to the on-policy algorithms.To achieve this, the proposed algorithm generalises the Q-, value and advantage functions for data from multiple policies.The method uses trust region optimisation, while avoiding some of the common problems of the algorithms such as TRPO or ACKTR: it uses hyperparameters to replace the trust region selection heuristics, as well as  the trainable covariance matrix instead of the fixed one.In many cases, the method not only improves the results comparing to the state-of-the-art trust region on-policy learning algorithms such as ACKTR and TRPO, but also with respect to their off-policy counterpart DDPG.

Studiamo l'evidenza teorica e pratica del miglioramento del reinforcement learning on-policy riutilizzando i dati di diverse policy consecutive.


Convolutional Neural Networks (CNN) have been successful in processing data signals that are uniformly sampled in the spatial domain (e.g., images).However, most data signals do not natively exist on a grid, and in the process of being sampled onto a uniform physical grid suffer significant aliasing error and information loss.Moreover, signals can exist in different topological structures as, for example, points, lines, surfaces and volumes.It has been challenging to analyze signals with mixed topologies (for example, point cloud with surface mesh).To this end, we develop mathematical formulations for Non-Uniform Fourier Transforms (NUFT) to directly, and optimally, sample nonuniform data signals of different topologies defined on a simplex mesh into the spectral domain with no spatial sampling error.The spectral transform is performed in the Euclidean space, which removes the translation ambiguity from works on the graph spectrum.Our representation has four distinct advantages: (1) the process causes no spatial sampling error during initial sampling, (2) the generality of this approach provides a unified framework for using CNNs to analyze signals of mixed topologies, (3) it allows us to leverage state-of-the-art backbone CNN architectures for effective learning without having to design a particular architecture for a particular data structure in an ad-hoc fashion, and (4) the representation allows weighted meshes where each element has a different weight (i.e., texture) indicating local properties.We achieve good results on-par with state-of-the-art for 3D shape retrieval task, and new state-of-the-art for point cloud to surface reconstruction task.

Usiamo la trasformazione non euclidea di Fourier delle forme definite da un complesso simpliciale per il deep learning, ottenendo risultati significativamente migliori rispetto alle tecniche di point-based sampling utilizzate nell'attuale letteratura sull'apprendimento 3D.


Discovering causal structure among a set of variables is a fundamental problem in many empirical sciences.Traditional score-based casual discovery methods rely on various local heuristics to search for a Directed Acyclic Graph (DAG) according to a predefined score function.While these methods, e.g., greedy equivalence search, may have attractive results with infinite samples and certain model assumptions, they are less satisfactory in practice due to finite data and possible violation of assumptions.Motivated by recent advances in neural combinatorial optimization, we propose to use Reinforcement Learning (RL) to search for the DAG with the best scoring.Our encoder-decoder model takes observable data as input and generates graph adjacency matrices that are used to compute rewards.The reward incorporates both the predefined score function and two penalty terms for enforcing acyclicity.In contrast with typical RL applications where the goal is to learn a policy, we use RL as a search strategy and our final output would be the graph, among all graphs generated during training, that achieves the best reward.We conduct experiments on both synthetic and real datasets, and show that the proposed approach not only has an improved search ability but also allows for a flexible score function under the acyclicity constraint.

Applichiamo il reinforcement learning alla scoperta causale basata sullo score e otteniamo risultati promettenti su dataset sia sintetici sia reali


The topic modeling discovers the latent topic probability of given the text documents.To generate the more meaningful topic that better represents the given document, we proposed a universal method which can be used in the data preprocessing stage.The method consists of three steps.First, it generates the word/word-pair from every single document.Second, it applies a two way parallel TF-IDF algorithm to word/word-pair for semantic filtering.Third, it uses the k-means algorithm to merge the word pairs that have the similar semantic meaning.Experiments are carried out on the Open Movie Database (OMDb), Reuters Dataset and 20NewsGroup Dataset and use the mean Average Precision score as the evaluation metric.Comparing our results with other state-of-the-art topic models, such as Latent Dirichlet allocation and traditional Restricted Boltzmann Machines.Our proposed data preprocessing can improve the generated topic accuracy by up to 12.99\%.How the number of clusters and the number of word pairs should be adjusted for different type of text document is also discussed.

Abbiamo proposto un metodo universale che può essere utilizzato nella fase di pre-elaborazione dei dati per generare il topic più significativo che rappresenta meglio il documento dato


Multi-task learning has been successful in modeling multiple related tasks with large, carefully curated labeled datasets.By leveraging the relationships among different tasks, multi-task learning framework can improve the performance significantly.However, most of the existing works are under the assumption that the predefined tasks are related to each other.Thus, their applications on real-world are limited, because rare real-world problems are closely related.Besides, the understanding of relationships among tasks has been ignored by most of the current methods.Along this line, we propose a novel multi-task learning framework - Learning To Transfer Via Modelling Multi-level Task Dependency, which constructed attention based dependency relationships among different tasks.At the same time, the dependency relationship can be used to guide what knowledge should be transferred, thus the performance of our model also be improved.To show the effectiveness of our model and the importance of considering multi-level dependency relationship, we conduct experiments on several public datasets, on which we obtain significant improvements over current methods.

Proponiamo un nuovo framework di multi-task learning che estrae automaticamente la relazione di dipendenza multi-view e la usa per guidare il transfer della conoscenza tra diversi task.


We design simple and quantifiable testing of global translation-invariance in deep learning models trained on the MNIST dataset.Experiments on convolutional and capsules neural networks show that both models have poor performance in dealing with global translation-invariance; however, the performance improved by using data augmentation.Although the capsule network is better on the MNIST testing dataset, the convolutional neural network generally has better performance on the translation-invariance.

Verifica dell'invarianza traslazionale globale nelle convolutional e capsule network


Gaussian processes are ubiquitous in nature and engineering.A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes.Here we perturbatively extend this correspondence to finite-width neural networks, yielding non-Gaussian processes as priors.The methodology developed herein allows us to track the flow of preactivation distributions by progressively integrating out random variables from lower to higher layers, reminiscent of renormalization-group flow.We further develop a perturbative prescription to perform Bayesian inference with weakly non-Gaussian priors.

Sviluppiamo un metodo analitico per studiare l'inferenza bayesiana delle reti neurali a larghezza finita e troviamo che l'immagine del flusso del gruppo di rinormalizzazione emerge naturalmente.


Distillation is a method to transfer knowledge from one model to another and often achieves higher accuracy with the same capacity.In this paper, we aim to provide a theoretical understanding on what mainly helps with the distillation.Our answer is "early stopping".Assuming that the teacher network is overparameterized, we argue that the teacher network is essentially harvesting dark knowledge from the data via early stopping.This can be justified by a new concept, Anisotropic In- formation Retrieval (AIR), which means that the neural network tends to fit the informative information first and the non-informative information (including noise) later.Motivated by the recent development on theoretically analyzing overparame- terized neural networks, we can characterize AIR by the eigenspace of the Neural Tangent Kernel(NTK).AIR facilities a new understanding of distillation.With that, we further utilize distillation to refine noisy labels.We propose a self-distillation al- gorithm to sequentially distill knowledge from the network in the previous training epoch to avoid memorizing the wrong labels.We also demonstrate, both theoret- ically and empirically, that self-distillation can benefit from more than just early stopping.Theoretically, we prove convergence of the proposed algorithm to the ground truth labels for randomly initialized overparameterized neural networks in terms of l2 distance, while the previous result was on convergence in 0-1 loss.The theoretical result ensures the learned neural network enjoy a margin on the training data which leads to better generalization.Empirically, we achieve better testing accuracy and entirely avoid early stopping which makes the algorithm more user-friendly.

capire teoricamente l'effetto di regolarizzazione della distillation. Mostriamo che l'early stopping è essenziale in questo processo. Da questa prospettiva, abbiamo sviluppato un metodo di distillation per l'apprendimento con label corrotte con garanzie teoriche.


Lifelong learning poses considerable challenges in terms of effectiveness (minimizing prediction errors for all tasks) and overall computational tractability for real-time performance.  This paper addresses continuous lifelong multitask learning by jointly re-estimating the inter-task relations (\textit{output} kernel) and the per-task model parameters at each round, assuming data arrives in a streaming fashion.We propose a novel algorithm called  \textit{Online Output Kernel Learning Algorithm} (OOKLA) for lifelong learning setting.To avoid the memory explosion, we propose a robust budget-limited versions of the proposed algorithm that efficiently utilize the relationship between the tasks to bound the total number of representative examples in the support set.  In addition, we propose a two-stage budgeted scheme for efficiently tackling the task-specific budget constraints in lifelong learning.Our empirical results over three datasets indicate superior AUC performance for OOKLA and its budget-limited cousins over strong baselines.

un nuovo approccio per lifelong learning online utilizzando kernel di output.


Minecraft is a videogame that offers many interesting challenges for AI systems.In this paper, we focus in construction scenarios where an agent must build a complex structure made of individual blocks.As higher-level objects are formed of lower-level objects, the construction can naturally be modelled as a hierarchical task network.We model a house-construction scenario in classical and HTN planning and compare the advantages and disadvantages of both kinds of models.

Modelliamo uno scenario di costruzione di una casa in Minecraft in pianificazione classica e HTN e confrontiamo i vantaggi e gli svantaggi di entrambi i tipi di modelli.


Attacks on natural language models are difficult to compare due to their different definitions of what constitutes a successful attack.We present a taxonomy of constraints to categorize these attacks.For each constraint, we present a real-world use case and a way to measure how well generated samples enforce the constraint.We then employ our framework to evaluate two state-of-the art attacks which fool models with synonym substitution.These attacks claim their adversarial perturbations preserve the semantics and syntactical correctness of the inputs, but our analysis shows these constraints are not strongly enforced.For a significant portion of these adversarial examples, a grammar checker detects an increase in errors.Additionally, human studies indicate that many of these adversarial examples diverge in semantic meaning from the input or do not appear to be human-written.Finally, we highlight the need for standardized evaluation of attacks that share constraints.Without shared evaluation metrics, it is up to researchers to set thresholds that determine the trade-off between attack quality and attack success.We recommend well-designed human studies to determine the best threshold to approximate human judgement.

Presentiamo un framework per valutare gli adversarial example in NLP e dimostriamo che gli adversarial example generati spesso non preservano la semantica, e sono sintatticamente corretti o sospetti.


In order for machine learning to be deployed and trusted in many applications, it is crucial to be able to reliably explain why the machine learning algorithm makes certain predictions.For example, if an algorithm classifies a given pathology image to be a malignant tumor, then the doctor may need to know which parts of the image led the algorithm to this classification.How to interpret black-box predictors is thus an important and active area of research.  A fundamental question is: how much can we trust the interpretation itself?In this paper, we show that interpretation of deep learning predictions is extremely fragile in the following sense:  two perceptively indistinguishable inputs with the same predicted label can be assigned very different}interpretations.We systematically characterize the fragility of the interpretations generated by several widely-used feature-importance interpretation methods (saliency maps, integrated gradient, and DeepLIFT) on ImageNet and CIFAR-10.Our experiments show that even small random perturbation can change the feature importance and new systematic perturbations can lead to dramatically different interpretations without changing the label.We extend these results to show that interpretations based on exemplars (e.g. influence functions) are similarly fragile.Our analysis of the geometry of the Hessian matrix gives insight on why fragility could be a fundamental challenge to the current interpretation approaches.

Possiamo fidarci della spiegazione di una rete neurale per la sua predizione? Esaminiamo la robustezza di diverse nozioni popolari di interpretabilità delle reti neurali, comprese le mappe di saliency e le funzioni di influenza, e progettiamo adversarial example contro di esse.


Stochastic AUC maximization has garnered an increasing interest due to better fit to imbalanced data classification.However, existing works are limited to stochastic AUC maximization with a linear predictive model, which restricts its predictive power when dealing with extremely complex data.In this paper, we consider stochastic AUC maximization problem with a deep neural network as the predictive model.Building on the saddle point reformulation of a surrogated loss of AUC, the problem can be cast into a {\it non-convex concave} min-max problem.The main contribution made in this paper is to make stochastic AUC maximization more practical for deep neural networks and big data with theoretical insights as well.In particular, we propose to explore Polyak-\L{}ojasiewicz (PL) condition that has been proved and observed in deep learning, which enables us to develop new stochastic algorithms with even faster convergence rate and more practical step size scheme.An AdaGrad-style algorithm is also analyzed under the PL condition with adaptive convergence rate.Our experimental results demonstrate the effectiveness of the proposed algorithms.

L'articolo progetta due algoritmi per il problema di massimizzazione dell'AUC stocastico con complessità allo stato dell'arte quando si usa una deep neural network come modello predittivo, che sono anche verificati da studi empirici.


Designing rewards for Reinforcement Learning (RL) is challenging because it needs to convey the desired task, be efficient to optimize, and be easy to compute.The latter is particularly problematic when applying RL to robotics, where detecting whether the desired configuration is reached might require considerable supervision and instrumentation.Furthermore, we are often interested in being able to reach a wide range of configurations, hence setting up a different reward every time might be unpractical.Methods like Hindsight Experience Replay (HER) have recently shown promise to learn policies able to reach many goals, without the need of a reward.Unfortunately, without tricks like resetting to points along the trajectory, HER might take a very long time to discover how to reach certain areas of the state-space.In this work we investigate different approaches to incorporate demonstrations to drastically speed up the convergence to a policy able to reach any goal, also surpassing the performance of an agent trained with other Imitation Learning algorithms.Furthermore, our method can be used when only trajectories without expert actions are available, which can leverage kinestetic or third person demonstration.

Affrontiamo task condizionati da obiettivi combinando gli algoritmi di Hindsight Experience Replay e di Imitation Learning, mostrando una convergenza più veloce del primo e prestazioni finali più elevate del secondo.


Bayesian neural networks, which both use the negative log-likelihood loss function and average their predictions using a learned posterior over the parameters, have been used successfully across many scientific fields, partly due to their ability to `effortlessly' extract desired representations from many large-scale datasets.However, generalization bounds for this setting is still missing.In this paper, we present a new PAC-Bayesian generalization bound for the negative log-likelihood loss which utilizes the \emph{Herbst Argument} for the log-Sobolev inequality to bound the moment generating function of the learners risk.

Deriviamo un nuovo PAC-Bayesian Bound per loss function senza limiti (per esempio Negative Log-Likelihood).


Data augmentation techniques, e.g., flipping or cropping, which systematically enlarge the training dataset by explicitly generating more training samples, are effective in improving the generalization performance of deep neural networks.In the supervised setting, a common practice for data augmentation is to assign the same label to all augmented samples of the same source.However, if the augmentation results in large distributional discrepancy among them (e.g., rotations), forcing their label invariance may be too difficult to solve and often hurts the performance.To tackle this challenge, we suggest a simple yet effective idea of learning the joint distribution of the original and self-supervised labels of augmented samples.The joint learning framework is easier to train, and enables an aggregated inference combining the predictions from different augmented samples for improving the performance.Further, to speed up the aggregation process, we also propose a knowledge transfer technique, self-distillation, which transfers the knowledge of augmentation into the model itself.We demonstrate the effectiveness of our data augmentation framework on various fully-supervised settings including the few-shot and imbalanced classification scenarios.

Proponiamo una semplice tecnica di data augmentation self-supervised che migliora le prestazioni degli scenari completamente supervisionati, tra cui l'apprendimento few shot e la classificazione imbalanced.


Long short-term memory (LSTM) networks allow to exhibit temporal dynamic behavior with feedback connections and seem a natural choice for learning sequences of 3D meshes.We introduce an approach for dynamic mesh representations as used for numerical simulations of car crashes.To bypass the complication of using 3D meshes, we transform the surface mesh sequences into spectral descriptors that efficiently encode the shape.A two branch LSTM based network architecture is chosen to learn the representations and dynamics of the crash during the simulation.The architecture is based on unsupervised video prediction by an LSTM without any convolutional layer.It uses an encoder LSTM to map an input sequence into a fixed length vector representation.On this representation one decoder LSTM performs the reconstruction of the input sequence, while the other decoder LSTM predicts the future behavior by receiving initial steps of the sequence as seed.The spatio-temporal error behavior of the model is analysed to study how well the model can extrapolate the learned spectral descriptors into the future, that is, how well it has learned to represent the underlying dynamical structural mechanics.Considering that only a few training examples are available, which is the typical case for numerical simulations, the network performs very well.

Un'architettura di rete basata su LSTM a due branch impara la rappresentazione e la dinamica delle mesh 3D delle simulazioni numeriche di crash.


The purpose of an encoding model is to predict brain activity given a stimulus.In this contribution, we attempt at estimating a whole brain encoding model of auditory perception in a naturalistic stimulation setting.We analyze data from an open dataset, in which 16 subjects watched a short movie while their brain activity was being measured using functional MRI.We extracted feature vectors aligned with the timing of the audio from the movie, at different layers of a Deep Neural Network pretrained on the classification of auditory scenes.fMRI data was parcellated using hierarchical clustering on 500 parcels, and encoding models were estimated using a fully connected neural network with one hidden layer, trained to predict the signals for each parcel from the DNN features.Individual encoding models were successfully trained and predicted brain activity on unseen data, in parcels located in the superior temporal lobe, as well as dorsolateral prefrontal regions, which are usually considered as areas involved in auditory and language processing.Taken together, this contribution extends previous attempts on estimating encoding models, by showing the ability to model brain activity using a generic DNN (ie not specifically trained for this purpose) to extract auditory features, suggesting a degree of similarity between internal DNN representations and brain activity in naturalistic settings.

I vettori di feature di SoundNet possono prevedere l'attività cerebrale dei soggetti che guardano un film nelle regioni cerebrali legate all'udito e al linguaggio.


In this paper, we describe the "implicit autoencoder" (IAE), a generative autoencoder in which both the generative path and the recognition path are parametrized by implicit distributions.We use two generative adversarial networks to define the reconstruction and the regularization cost functions of the implicit autoencoder, and derive the learning rules based on maximum-likelihood learning.Using implicit distributions allows us to learn more expressive posterior and conditional likelihood distributions for the autoencoder.Learning an expressive conditional likelihood distribution enables the latent code to only capture the abstract and high-level information of the data, while the remaining information is captured by the implicit conditional likelihood distribution.For example, we show that implicit autoencoders can disentangle the global and local information, and perform deterministic or stochastic reconstructions of the images.We further show that implicit autoencoders can disentangle discrete underlying factors of variation from the continuous factors in an unsupervised fashion, and perform clustering and semi-supervised learning.

Proponiamo un autoencoder generativo che può apprendere distribuzioni posteriori e di likelihood condizionale espressive usando distribuzioni implicite, e addestrare il modello usando una nuova formulazione dell'ELBO.


Strong inductive biases allow children to learn in fast and adaptable ways.Children use the mutual exclusivity (ME) bias to help disambiguate how words map to referents, assuming that if an object has one label then it does not need another.In this paper, we investigate whether or not standard neural architectures have a ME bias, demonstrating that they lack this learning assumption.Moreover, we show that their inductive biases are poorly matched to lifelong learning formulations of classification and translation.We demonstrate that there is a compelling case for designing neural networks that reason by mutual exclusivity, which remains an open challenge.

I bambini usano il bias di mutua esclusività (ME) per imparare nuove parole, mentre le reti neurali standard mostrano il bias opposto, ostacolando l'apprendimento in scenari naturalistici come lifelong learning.


Cortical neurons process and integrate information on multiple timescales.In addition, these timescales or temporal receptive fields display functional and hierarchical organization.For instance, areas important for working memory (WM), such as prefrontal cortex, utilize neurons with stable temporal receptive fields and long timescales to support reliable representations of stimuli.Despite of the recent advances in experimental techniques, the underlying mechanisms for the emergence of neuronal timescales long enough to support WM are unclear and challenging to investigate experimentally.Here, we demonstrate that spiking recurrent neural networks (RNNs) designed to perform a WM task reproduce previously observed experimental findings and that these models could be utilized in the future to study how neuronal timescales specific to WM emerge.

Le spiking recurrent neural network che eseguono un working memory task utilizzano lunghe scale temporali eterogenee, sorprendentemente simili a quelle osservate nella corteccia prefrontale.


Conventional deep learning classifiers are static in the sense that they are trained ona predefined set of classes and learning to classify a novel class typically requiresre-training.In this work, we address the problem of Low-shot network-expansionlearning.We introduce a learning framework which enables expanding a pre-trained(base) deep network to classify novel classes when the number of examples for thenovel classes is particularly small.We present a simple yet powerful distillationmethod where the base network is augmented with additional weights to classifythe novel classes, while keeping the weights of the base network unchanged.Weterm this learning hard distillation, since we preserve the response of the networkon the old classes to be equal in both the base and the expanded network.Weshow that since only a small number of weights needs to be trained, the harddistillation excels for low-shot training scenarios.Furthermore, hard distillationavoids detriment to classification performance on the base classes.Finally, weshow that low-shot network expansion can be done with a very small memoryfootprint by using a compact generative model of the base classes training datawith only a negligible degradation relative to learning with the full training set.

In questo articolo, affrontiamo il problema del network-expansion learning low-shot


People ask questions that are far richer, more informative, and more creative than current AI systems.We propose a neural program generation framework for modeling human question asking, which represents questions as formal programs and generates programs with an encoder-decoder based deep neural network.From extensive experiments using an information-search game, we show that our method can ask optimal questions in synthetic settings, and predict which questions humans are likely to ask in unconstrained settings.We also propose a novel grammar-based question generation framework trained with reinforcement learning, which is able to generate creative questions without supervised data.

Introduciamo un modello di human question asking che combina reti neurali e programmi simbolici, che può imparare a generare buone domande con o senza esempi supervisionati.


The classification of images taken in special imaging environments except air is the first challenge in extending the applications of deep learning.We report on an UW-Net (Underwater Network), a new convolutional neural network (CNN) based network for underwater image classification.In this model, we simulate the visual correlation of background attention with image understanding for special environments, such as fog and underwater by constructing an inception-attention (I-A) module.The experimental results demonstrate that the proposed UW-Net achieves an accuracy of 99.3% on underwater image classification, which is significantly better than other image classification networks, such as AlexNet, InceptionV3, ResNet and Se-ResNet.Moreover, we demonstrate the proposed IA module can be used to boost the performance of the existing object recognition networks.By substituting the inception module with the I-A module, the Inception-ResnetV2 network achieves a 10.7% top1 error rate and a 0% top5 error rate on the subset of ILSVRC-2012, which further illustrates the function of the background attention in the image classifications.

Un meccanismo di comprensione visiva per ambienti speciali


Deep networks have achieved impressive results across a variety of important tasks.However, a known weakness is a failure to perform well when evaluated on data which differ from the training distribution, even if these differences are very small, as is the case with adversarial examples.  We propose \emph{Fortified Networks}, a simple transformation of existing networks, which “fortifies” the hidden layers in a deep network by identifying when the hidden states are off of the data manifold, and maps these hidden states back to parts of the data manifold where the network performs well.Our principal contribution is to show that fortifying these hidden states improves the robustness of deep networks and our experiments (i) demonstrate improved robustness to standard adversarial attacks in both black-box and white-box threat models; (ii) suggest that our improvements are not primarily due to the problem of deceptively good results due to degraded quality in the gradient signal (the gradient masking problem) and (iii) show the advantage of doing this fortification in the hidden layers instead of the input space.  We demonstrate improvements in adversarial robustness on three datasets (MNIST, Fashion MNIST, CIFAR10), across several attack parameters, both white-box and black-box settings, and the most widely studied attacks (FGSM, PGD, Carlini-Wagner).  We show that these improvements are achieved across a wide variety of hyperparameters.

Un migliore adversarial training imparando a mappare il manifold dei dati con autoencoder negli hidden state.


Neural networks could misclassify inputs that are slightly different from their training data, which indicates a small margin between their decision boundaries and the training dataset.In this work, we study the binary classification of linearly separable datasets and show that linear classifiers could also have decision boundaries that lie close to their training dataset if cross-entropy loss is used for training.In particular, we show that if the features of the training dataset lie in a low-dimensional affine subspace and the cross-entropy loss is minimized by using a gradient method, the margin between the training points and the decision boundary could be much smaller than the optimal value.This result is contrary to the conclusions of recent related works such as (Soudry et al., 2018), and we identify the reason for this contradiction.In order to improve the margin, we introduce differential training, which is a training paradigm that uses a loss function defined on pairs of points from each class.We show that the decision boundary of a linear classifier trained with differential training indeed achieves the maximum margin.The results reveal the use of cross-entropy loss as one of the hidden culprits of adversarial examples and introduces a new direction to make neural networks robust against them.

Mostriamo che la minimizzazione della cross-entropy loss utilizzando un metodo con gradiente potrebbe portare a un margine molto povero se le feature del dataset si trovano su un sottospazio a bassa dimensionalità.


The concepts of unitary evolution matrices and associative memory have boosted the field of Recurrent Neural Networks (RNN) to state-of-the-art performance in a variety of sequential tasks.  However, RNN still has a limited capacity to manipulate long-term memory.  To bypass this weakness the most successful applications of RNN use external techniques such as attention mechanisms.In this paper we propose a novel RNN model that unifies the state-of-the-art approaches: Rotational Unit of Memory (RUM).The core of RUM is its rotational operation, which is,  naturally,  a unitary matrix, providing architectures with the power to learn long-term dependencies by overcoming the vanishing and exploding gradients problem.  Moreover,  the rotational unit also serves as associative memory.We evaluate our model on synthetic memorization, question answering and language modeling tasks.   RUM learns the Copying Memory task completely and improves the state-of-the-art result in the Recall task.  RUM’s performance in the bAbI Question Answering task is comparable to that of models with attention mechanism.We also improve the state-of-the-art result to 1.189 bits-per-character (BPC) loss in the Character Level Penn Treebank (PTB) task, which is to signify the applications of RUM to real-world sequential data.The universality of our construction, at the core of RNN, establishes RUM as a promising approach to language modeling, speech recognition and machine translation.

Un nuovo modello RNN che supera significativamente l'attuale frontiera dei modelli in una varietà di task sequenziali.


While many recent advances in deep reinforcement learning rely on model-free methods, model-based approaches remain an alluring prospect for their potential to exploit unsupervised data to learn environment dynamics.One prospect is to pursue hybrid approaches, as in AlphaGo, which combines Monte-Carlo Tree Search (MCTS)—a model-based method—with deep-Q networks (DQNs)—a model-free method.MCTS requires generating rollouts, which is computationally expensive.In this paper, we propose to simulate roll-outs, exploiting the latest breakthroughs in image-to-image transduction, namely Pix2Pix GANs, to predict the dynamics of the environment.Our proposed algorithm, generative adversarial tree search (GATS), simulates rollouts up to a specified depth using both a GAN- based dynamics model and a reward predictor.GATS employs MCTS for planning over the simulated samples and uses DQN to estimate the Q-function at the leaf states.Our theoretical analysis establishes some favorable properties of GATS vis-a-vis the bias-variance trade-off and empirical results show that on 5 popular Atari games, the dynamics and reward predictors converge quickly to accurate solutions.However, GATS fails to outperform DQNs in 4 out of 5 games.Notably, in these experiments, MCTS has only short rollouts (up to tree depth 4), while previous successes of MCTS have involved tree depth in the hundreds.We present a hypothesis for why tree search with short rollouts can fail even given perfect modeling.

Sorprendenti risultati negativi su Model Based + Model deep RL


We present a novel architecture of GAN for a disentangled representation learning.The new model architecture is inspired by Information Bottleneck (IB) theory thereby named IB-GAN.IB-GAN objective is similar to that of InfoGAN but has a crucial difference; a capacity regularization for mutual information is adopted, thanks to which the generator of IB-GAN can harness a latent representation in disentangled and interpretable manner.To facilitate the optimization of IB-GAN in practice, a new variational upper-bound is derived.With experiments on CelebA, 3DChairs, and dSprites datasets, we demonstrate that the visual quality of samples generated by IB-GAN is often better than those by β-VAEs.Moreover, IB-GAN achieves much higher disentanglement metrics score than β-VAEs or InfoGAN on the dSprites dataset.

Ispirati dalla teoria dell'Information Bottleneck, proponiamo una nuova architettura di GAN per l'apprendimento di una rappresentazione disentangled


We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs.As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters.We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramér GAN critic.Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs.In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance.We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.

Spiegare la situazione di bias con le GAN MMD; le GAN MMD funzionano con reti critiche più piccole delle WGAN-GP; nuova metrica di valutazione delle GAN.


We extend the Consensus Network framework to Transductive Consensus Network (TCN), a semi-supervised multi-modal classification framework, and identify its two mechanisms: consensus and classification.By putting forward three variants as ablation studies, we show both mechanisms should be functioning together.Overall, TCNs outperform or align with the best benchmark algorithms when only 20 to 200 labeled data points are available.

Un framework di classificazione multimodale semi-supervised, TCN, che supera vari benchmark.


Separating mixed distributions is a long standing challenge for machine learning and signal processing.Applications include: single-channel multi-speaker separation (cocktail party problem), singing voice separation and separating reflections from images.Most current methods either rely on making strong assumptions on the source distributions (e.g. sparsity, low rank, repetitiveness) or rely on having training samples of each source in the mixture.In this work, we tackle the scenario of extracting an unobserved distribution additively mixed with a signal from an observed (arbitrary) distribution.We introduce a new method: Neural Egg Separation - an iterative method that learns to separate the known distribution from progressively finer estimates of the unknown distribution.In some settings, Neural Egg Separation is initialization sensitive, we therefore introduce GLO Masking which ensures a good initialization.Extensive experiments show that our method outperforms current methods that use the same level of supervision and often achieves similar performance to full supervision.

Un metodo neurale iterativo per l'estrazione di segnali che vengono osservati solo quando mescolati ad altri segnali


In health, machine learning is increasingly common, yet neural network embedding (representation) learning is arguably under-utilized for physiological signals.  This inadequacy stands out in stark contrast to more traditional computer science domains, such as computer vision (CV), and natural language processing (NLP).  For physiological signals, learning feature embeddings is a natural solution to data insufficiency caused by patient privacy concerns -- rather than share data, researchers may share informative embedding models (i.e., representation models), which map patient data to an output embedding.   Here, we present the PHASE (PHysiologicAl Signal Embeddings) framework, which consists of three components: i) learning neural network embeddings of physiological signals, ii) predicting outcomes based on the learned embedding, and iii) interpreting the prediction results by estimating feature attributions in the "stacked" models (i.e., feature embedding model followed by prediction model).  PHASE is novel in three ways: 1) To our knowledge, PHASE is the first instance of transferal of neural networks to create physiological signal embeddings.2) We present a tractable method to obtain feature attributions through stacked models.  We prove that our stacked model attributions can approximate Shapley values -- attributions known to have desirable properties -- for arbitrary sets of models.3) PHASE was extensively tested in a cross-hospital setting including publicly available data.  In our experiments, we show that PHASE significantly outperforms alternative embeddings -- such as raw, exponential moving average/variance, and autoencoder -- currently in use.Furthermore, we provide evidence that transferring neural network embedding/representation learners between distinct hospitals still yields performant embeddings and offer recommendations when transference is ineffective.

Embedding del segnale fisiologico per le prestazioni di predizione e hospital transference con un metodo generale di interpretabilità del valore di Shapley per i modelli stacked.


We consider the dictionary learning problem, where the aim is to model the given data as a linear combination of a few columns of a matrix known as a dictionary, where the sparse weights forming the linear combination are known as coefficients.Since the dictionary and coefficients, parameterizing the linear model are unknown, the corresponding optimization is inherently non-convex.This was a major challenge until recently, when provable algorithms for dictionary learning were proposed.Yet, these provide guarantees only on the recovery of the dictionary, without explicit recovery guarantees on the coefficients.Moreover, any estimation error in the dictionary adversely impacts the ability to successfully localize and estimate the coefficients.This potentially limits the utility of existing provable dictionary learning methods in applications where coefficient recovery is of interest.To this end, we develop NOODL: a simple Neurally plausible alternating Optimization-based Online Dictionary Learning algorithm, which recovers both the dictionary and coefficients exactly at a geometric rate, when initialized appropriately.Our algorithm, NOODL, is also scalable and amenable for large scale distributed implementations in neural architectures, by which we mean that it only involves simple linear and non-linear operations.Finally, we corroborate these theoretical results via experimental evaluation of the proposed algorithm with the current state-of-the-art techniques.

Presentiamo un algoritmo dimostrabile per recuperare esattamente entrambi i fattori del modello di apprendimento del dizionario.


Real-world Relation Extraction (RE) tasks are challenging to deal with, either due to limited training data or class imbalance issues.In this work, we present Data Augmented Relation Extraction (DARE), a simple method to augment training data by properly finetuning GPT2 to generate examples for specific relation types.The generated training data is then used in combination with the gold dataset to train a BERT-based RE classifier.In a series of experiments we show the advantages of our method, which leads in improvements of up to 11 F1 score points compared to a strong baseline.Also, DARE achieves new state-of-the-art in three widely used biomedical RE datasets surpassing the previous best results by 4.7 F1 points on average.

Estrazione di relazioni con data augmentation tramite GPT-2


This paper addresses the problem of representing a system's belief using multi-variate normal distributions (MND) where the underlying model is based on a deep neural network (DNN).The major challenge with DNNs is the computational complexity that is needed to obtain model uncertainty using MNDs.To achieve a scalable method, we propose a novel approach that expresses the parameter posterior in sparse information form.Our inference algorithm is based on a novel Laplace Approximation scheme, which involves a diagonal correction of the Kronecker-factored eigenbasis.As this makes the inversion of the information matrix intractable - an operation that is required for full Bayesian analysis, we devise a low-rank   approximation of this eigenbasis and a memory-efficient sampling scheme.We provide both a theoretical analysis and an empirical evaluation on various benchmark data sets, showing the superiority of our approach over existing methods.

Un algoritmo di inferenza approssimato per deep learning


The ever-increasing size of modern datasets combined with the difficulty of obtaining label information has made semi-supervised learning of significant practical importance in modern machine learning applications.In comparison to supervised learning, the key difficulty in semi-supervised learning is how to make full use of the unlabeled data.In order to utilize manifold information provided by unlabeled data, we propose a novel regularization called the tangent-normal adversarial regularization, which is composed by two parts.The two parts complement with each other and jointly enforce the smoothness along two different directions that are crucial for semi-supervised learning.One is applied along the tangent space of the data manifold, aiming to enforce local invariance of the classifier on the manifold, while the other is performed on the normal space orthogonal to the tangent space, intending to impose robustness on the classifier against the noise causing the observed data deviating from the underlying data manifold.  Both of the two regularizers are achieved by the strategy of virtual adversarial training.Our method has achieved state-of-the-art performance on semi-supervised learning tasks on both artificial dataset and practical datasets.

Proponiamo una nuova strategia di regolarizzazione del manifold basata su adversarial training, che può migliorare significativamente le prestazioni dell'apprendimento semi-supervised.


Universal approximation property of neural networks is one of the motivations to use these models in various real-world problems.However, this property is not the only characteristic that makes neural networks unique as there is a wide range of other approaches with similar property.Another characteristic which makes these models interesting is that they can be trained with the backpropagation algorithm which allows an efficient gradient computation and gives these universal approximators the ability to efficiently learn complex manifolds from a large amount of data in different domains.Despite their abundant use in practice, neural networks are still not well understood and a broad range of ongoing research is to study the interpretability of neural networks.On the other hand, topological data analysis (TDA) relies on strong theoretical framework of (algebraic) topology along with other mathematical tools for analyzing possibly complex datasets.In this work, we leverage a universal approximation theorem originating from algebraic topology to build a connection between TDA and common neural network training framework.We introduce the notion of automatic subdivisioning and devise a particular type of neural networks for regression tasks: Simplicial Complex Networks (SCNs).SCN's architecture is defined with a set of bias functions along with a particular policy during the forward pass which alternates the common architecture search framework in neural networks.We believe the view of SCNs can be used as a step towards building interpretable deep learning models.Finally, we verify its performance on a set of regression problems.

Un nuovo metodo per l'apprendimento supervised attraverso la suddivisione dello spazio di input insieme all'approssimazione della funzione.


The goal of network representation learning is to learn low-dimensional node embeddings that capture the graph structure and are useful for solving downstream tasks.However, despite the proliferation of such methods there is currently no study of their robustness to adversarial attacks.We provide the first adversarial vulnerability analysis on the widely used family of methods based on random walks.We derive efficient adversarial perturbations that poison the network structure and have a negative effect on both the quality of the embeddings and the downstream tasks.We further show that our attacks are transferable since they generalize to many models, and are successful even when the attacker is restricted.

Adversarial attack su embedding di nodi non supervisionati basati sulla teoria della perturbazione degli autovalori.


Hierarchical reinforcement learning methods offer a powerful means of planning flexible behavior in complicated domains.However, learning an appropriate hierarchical decomposition of a domain into subtasks remains a substantial challenge.We present a novel algorithm for subtask discovery, based on the recently introduced multitask linearly-solvable Markov decision process (MLMDP) framework.The MLMDP can perform never-before-seen tasks by representing them as a linear combination of a previously learned basis set of tasks.In this setting, the subtask discovery problem can naturally be posed as finding an optimal low-rank approximation of the set of tasks the agent will face in a domain.We use non-negative matrix factorization to discover this minimal basis set of tasks, and show that the technique learns intuitive decompositions in a variety of domains.Our method has several qualitatively desirable features: it is not limited to learning subtasks with single goal states, instead learning distributed patterns of preferred states; it learns qualitatively different hierarchical decompositions in the same domain depending on the ensemble of tasks the agent will face; and it may be straightforwardly iterated to obtain deeper hierarchical decompositions.

Presentiamo un nuovo algoritmo per la scoperta gerarchica di subtask che sfrutta il framework del processo decisionale lineare di Markov multitask.


This paper introduces a framework for solving combinatorial optimization problems by learning from input-output examples of optimization problems.We introduce a new memory augmented neural model in which the memory is not resettable (i.e the information stored in the memory after processing an input example is kept for the next seen examples).We used deep reinforcement learning to train a memory controller agent to store useful memories.Our model was able to outperform hand-crafted solver on Binary Linear Programming (Binary LP).The proposed model is tested on different Binary LP instances with large number of variables (up to 1000 variables) and constrains (up to 700 constrains).

Proponiamo un modello di memory network per risolvere istanze LP binarie in cui l'informazione della memoria è conservata per l'uso a lungo termine.


Sequence-to-sequence (Seq2Seq) models with attention have excelled at tasks which involve generating natural language sentences such as machine translation, image captioning and speech recognition.Performance has further been improved by leveraging unlabeled data, often in the form of a language model.In this work, we present the Cold Fusion method, which leverages a pre-trained language model during training, and show its effectiveness on the speech recognition task.We show that Seq2Seq models with Cold Fusion are able to better utilize language information enjoyingi) faster convergence and better generalization, andii) almost complete transfer to a new domain while using less than 10% of the labeled training data.

Introduciamo un nuovo metodo per addestrare modelli Seq2Seq con language model che convergono più velocemente, generalizzano meglio e possono trasferirsi quasi completamente a un nuovo dominio usando meno del 10% di dati annotati.


A central capability of intelligent systems is the ability to continuously build upon previous experiences to speed up and enhance learning of new tasks.Two distinct research paradigms have studied this question.Meta-learning views this problem as learning a prior over model parameters that is amenable for fast adaptation on a new task, but typically assumes the set of tasks are available together as a batch.In contrast, online (regret based) learning considers a sequential setting in which problems are revealed one after the other, but conventionally train only a single model without any task-specific adaptation.This work introduces an online meta-learning setting, which merges ideas from both the aforementioned paradigms to better capture the spirit and practice of continual lifelong learning.We propose the follow the meta leader (FTML) algorithm which extends the MAML algorithm to this setting.Theoretically, this work provides an O(logT) regret guarantee for the FTML algorithm.Our experimental evaluation on three different large-scale tasks suggest that the proposed algorithm significantly outperforms alternatives based on traditional online learning approaches.

Introduciamo il setting del problema del meta learning online per catturare meglio lo spirito e la pratica del continual lifelong learning.


We investigate the extent to which individual attention heads in pretrained transformer language models, such as BERT and RoBERTa, implicitly capture syntactic dependency relations.We employ two methods—taking the maximum attention weight and computing the maximum spanning tree—to extract implicit dependency relations from the attention weights of each layer/head, and compare them to the ground-truth Universal Dependency (UD) trees.We show that, for some UD relation types, there exist heads that can recover the dependency type significantly better than baselines on parsed English text, suggesting that some self-attention heads act as a proxy for syntactic structure.We also analyze BERT fine-tuned on two datasets—the syntax-oriented CoLA and the semantics-oriented MNLI—to investigate whether fine-tuning affects the patterns of their self-attention, but we do not observe substantial differences in the overall dependency relations extracted using our methods.Our results suggest that these models have some specialist attention heads that track individual dependency types, but no generalist head that performs holistic parsing significantly better than a trivial baseline, and that analyzing attention weights directly may not reveal much of the syntactic knowledge that BERT-style models are known to learn.

I pesi di attention non espongono completamente ciò che BERT conosce della sintassi.