pwc-archive/papers-with-abstracts huggingface dataset:
Power up! Robust Graph Convolutional Network based on Graph Powering
Authors: Ming Jin, Heng Chang, Wenwu Zhu, Somayeh Sojoudi
Date: 2019-09-25 00:00:00
Tasks: Adversarial Robustness

Graph convolutional networks (GCNs) are powerful tools for graph-structured data. However, they have been recently shown to be vulnerable to topological attacks. To enhance adversarial robustness, we go beyond spectral graph theory to robust graph theory. By challenging the classical graph Laplacian, we propose a new convolution operator that is provably robust in the spectral domain and is incorporated in the GCN architecture to improve expressivity and interpretability. By extending the original graph to a sequence of graphs, we also propose a robust training paradigm that encourages transferability across graphs that span a range of spatial and spectral characteristics. The proposed approaches are demonstrated in extensive experiments to {simultaneously} improve performance in both benign and adversarial situations.
DIME: AN INFORMATION-THEORETIC DIFFICULTY MEASURE FOR AI DATASETS
Authors: Peiliang Zhang, Huan Wang, Nikhil Naik, Caiming Xiong, Richard Socher
Date: 2019-09-25 00:00:00
Proceeding: NeurIPS Workshop DL-IG 2020 12
Tasks: BIG-bench Machine Learning

Evaluating the relative difficulty of widely-used benchmark datasets across time and across data modalities is important for accurately measuring progress in machine learning.  To help tackle this problem, we proposeDIME, an information-theoretic DIfficulty MEasure for datasets, based on conditional entropy estimation of the sample-label distribution.  Theoretically,  we prove a model-agnostic and modality-agnostic lower bound on the 0-1 error by extending Fano’s inequality to the common supervised learning scenario where labels are discrete and features are continuous. Empirically, we estimate this lower bound using a neural network to compute DIME. DIME can be decomposed into components attributable to the data distribution and the number of samples.  DIME can also compute per-class difficulty scores. Through extensive experiments on both vision and language datasets, we show that DIME is well-aligned with empirically observed performance of state-of-the-art machine learning models. We hope that DIME can aid future dataset design and model-training strategies.
Improved Detection of Adversarial Attacks via Penetration Distortion Maximization
Authors: Shai Rozenberg, Gal Elidan, Ran El-Yaniv
Date: 2019-09-25 00:00:00

This paper is concerned with the defense of deep models against adversarial at-
tacks. We develop an adversarial detection method, which is inspired by the cer-
tificate defense approach, and captures the idea of separating class clusters in the
embedding space so as to increase the margin. The resulting defense is intuitive,
effective, scalable and can be integrated into any given neural classification model.
Our method demonstrates state-of-the-art detection performance under all threat
models.
X-Forest: Approximate Random Projection Trees for Similarity Measurement
Authors: Yikai Zhao, Peiqing Chen, Zidong Zhao, Tong Yang, Jie Jiang, Bin Cui, Gong Zhang, Steve Uhlig
Date: 2019-09-25 00:00:00

Similarity measurement plays a central role in various data mining and machine learning tasks. Generally, a similarity measurement solution should, in an ideal state, possess the following three properties: accuracy, efficiency and independence from prior knowledge. Yet unfortunately, vital as similarity measurements are, no previous works have addressed all of them. In this paper, we propose X-Forest, consisting of a group of approximate Random Projection Trees, such that all three targets mentioned above are tackled simultaneously. Our key techniques are as follows. First, we introduced RP Trees into the tasks of similarity measurement such that accuracy is improved. In addition, we enforce certain layers in each tree to share identical projection vectors, such that exalted efficiency is achieved. Last but not least, we introduce randomness into partition to eliminate its reliance on prior knowledge.   We conduct experiments on three real-world datasets, whose results demonstrate that our model, X-Forest, reaches an efficiency of up to 3.5 times higher than RP Trees with negligible compromising on its accuracy, while also being able to outperform traditional Euclidean distance-based similarity metrics by as much as 20% with respect to clustering tasks.   We have released codes in github anonymously so as to meet the demand of reproducibility.
Out-of-Distribution Image Detection Using the Normalized Compression Distance
Authors: Sehun Yu, Donga Lee, Hwanjo Yu
Date: 2019-09-25 00:00:00
Tasks: Out-of-Distribution Detection

On detection of the out-of-distribution images, whose underlying distribution is different from that of the training dataset, we tackle to apply out-of-distribution detection methods to already deployed convolutional neural networks. Most recent approaches have to utilize out-of-distribution samples for validation or retrain the model, which makes it less practical for real-world applications. We propose a novel out-of-distribution detection method MALCOM, which neither uses any out-of-distribution samples nor retrain the model. Inspired by the method using the global average pooling on the feature maps of the convolutional neural networks, the goal of our method is to extract informative sequential patterns from the feature maps. To this end, we introduce a similarity metric which focuses on the shared patterns between two sequences. In short, MALCOM uses both the global average and spatial pattern of the feature maps to accurately identify out-of-distribution samples.
A Deep Recurrent Neural Network via Unfolding Reweighted l1-l1 Minimization
Authors: Huynh Van Luong, Duy Hung Le, Nikos Deligiannis
Date: 2019-09-25 00:00:00

Deep unfolding methods design deep neural networks as learned variations of optimization methods. These networks have been shown to achieve faster convergence and higher accuracy than the original optimization methods. In this line of research, this paper develops a novel deep recurrent neural network (coined reweighted-RNN) by unfolding a reweighted l1-l1 minimization algorithm and applies it to the task of sequential signal reconstruction. To the best of our knowledge, this is the first deep unfolding method that explores reweighted minimization. Due to the underlying reweighted minimization model, our RNN has a different soft-thresholding function (alias, different activation function) for each hidden unit in each layer. Furthermore, it has higher network expressivity than existing deep unfolding RNN models due to the over-parameterizing weights. Moreover, we establish theoretical generalization error bounds for the proposed reweighted-RNN model by means of Rademacher complexity. The bounds reveal that the parameterization of the proposed reweighted-RNN ensures good generalization. We apply the proposed reweighted-RNN to the problem of video-frame reconstruction from low-dimensional measurements, that is, sequential frame reconstruction. The experimental results on the moving MNIST dataset demonstrate that the proposed deep reweighted-RNN significantly outperforms existing RNN models.
Meta-Learning for Variational Inference
Authors: Ruqi Zhang, Yingzhen Li, Chris De Sa, Sam Devlin, Cheng Zhang
Date: 2019-09-25 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference, Computational Efficiency, Meta-Learning, Recommendation Systems, regression, Variational Inference

Variational inference (VI) plays an essential role in approximate Bayesian inference due to its computational efficiency and general applicability. 
Crucial to the performance of VI is the selection of the divergence measure in the optimization objective, as it affects the properties of the approximate posterior significantly. In this paper, we propose a meta-learning algorithm to learn (i) the divergence measure suited for the task of interest to automate the design of the VI method; and (ii) initialization of the variational parameters, which reduces the number of VI optimization steps drastically. We demonstrate the learned divergence outperforms the hand-designed divergence on Gaussian mixture distribution approximation, Bayesian neural network regression, and partial variational autoencoder based recommender systems.
All Simulations Are Not Equal: Simulation Reweighing for Imperfect Information Games
Authors: Qucheng Gong, Yuandong Tian
Date: 2019-09-25 00:00:00
Tasks: All

Imperfect information games are challenging benchmarks for artificial intelligent systems. To reason and plan under uncertainty is a key towards general AI. Traditionally, large amounts of simulations are used in imperfect information games, and they sometimes perform sub-optimally due to large state and action spaces. In this work, we propose a simulation reweighing mechanism using neural networks. It performs backwards verification to public previous actions and assign proper belief weights to the simulations from the information set of the current observation, using an incomplete state solver network (ISSN). We use simulation reweighing in the playing phase of the game contract bridge, and show that it outperforms previous state-of-the-art Monte Carlo simulation based methods, and achieves better play per decision.
Latent Variables on Spheres for Sampling and Inference
Authors: Deli Zhao, Jiapeng Zhu, Bo Zhang
Date: 2019-09-25 00:00:00
Tasks: Variational Inference

Variational inference is a fundamental problem in Variational AutoEncoder (VAE). The optimization with lower bound of marginal log-likelihood results in the distribution of latent variables approximate to a given prior probability, which is the dilemma of employing VAE to solve real-world problems. By virtue of high-dimensional geometry, we propose a very simple algorithm completely different from existing ones to alleviate the variational inference in VAE. We analyze the unique characteristics of random variables on spheres in high dimensions and prove that Wasserstein distance between two arbitrary data sets randomly drawn from a sphere are nearly identical when the dimension is sufficiently large. Based on our theory, a novel algorithm for distribution-robust sampling is devised. Moreover, we reform the latent space of VAE by constraining latent variables on the sphere, thus freeing VAE from the approximate optimization of posterior probability via variational inference. The new algorithm is named Spherical AutoEncoder (SAE). Extensive experiments by sampling and inference tasks validate our theoretical analysis and the superiority of SAE.
Detecting Change in Seasonal Pattern via Autoencoder and Temporal Regularization
Authors: Raphael Fettaya, Dor Bank, Rachel Lemberg, Linoy Barel
Date: 2019-09-25 00:00:00
Tasks: Change Point Detection, Time Series, Time Series Analysis

Change-point detection problem consists of discovering abrupt property changes in the generation process of time-series. Most state-of-the-art models are optimizing the power of a kernel two-sample test, with only a few assumptions on the distribution of the data. Unfortunately, because they presume the samples are distributed i.i.d, they are not able to use information about the seasonality of a time-series. In this paper, we present a novel approach - ATR-CSPD allowing the detection of changes in the seasonal pattern of a time-series. Our method uses an autoencoder together with a temporal regularization, to learn the pattern of each seasonal cycle. Using low dimensional representation of the seasonal patterns, it is possible to accurately and efficiently estimate the existence of a change point using a clustering algorithm. Through experiments on artificial and real-world data sets, we demonstrate the usefulness of the proposed method for several applications.
Universal Safeguarded Learned Convex Optimization with Guaranteed Convergence
Authors: Howard Heaton, Xiaohan Chen, Zhangyang Wang, Wotao Yin
Date: 2019-09-25 00:00:00

Many applications require quickly and repeatedly  solving a certain type of optimization problem, each time with new (but similar) data. However, state of the art general-purpose optimization methods may converge too slowly for real-time use. This shortcoming is addressed by  “learning to optimize” (L2O) schemes, which construct   neural networks from parameterized forms of the update  operations of general-purpose methods. Inferences by each network form solution estimates, and networks are trained to optimize these estimates for a particular distribution of data. This results in task-specific algorithms (e.g., LISTA, ALISTA, and D-LADMM) that can converge order(s) of magnitude faster than general-purpose counterparts.  We provide the first general L2O convergence theory by wrapping all L2O schemes for convex optimization within a single framework. Existing L2O schemes form special cases, and we give a practical guide for applying our L2O framework to other problems. Using safeguarding, our theory proves, as the number of network layers increases, the distance between inferences and the solution set goes to zero, i.e., each cluster point is a solution. Our numerical examples demonstrate the efficacy of our approach for both existing and new L2O methods.
Knockoff-Inspired Feature Selection via Generative Models
Authors: Marco F. Duarte, Siwei Feng
Date: 2019-09-25 00:00:00
Tasks: feature selection, Variable Selection

We propose a feature selection algorithm for supervised learning inspired by the recently introduced 
knockoff framework for variable selection in statistical regression. While variable selection in statistics aims 
to distinguish between true and false predictors, feature selection in machine learning aims to reduce the 
dimensionality of the data while preserving the performance of the learning method. The knockoff framework 
has attracted significant interest due to its strong control of false discoveries while preserving predictive 
power. In contrast to the original approach and later variants that assume a given probabilistic model for the 
variables, our proposed approach relies on data-driven generative models that learn mappings from data 
space to a parametric space that characterizes the probability distribution of the data. Our approach 
requires only the availability of mappings from data space to a distribution in parametric space and from 
parametric space to a distribution in data space; thus, it can be integrated with multiple popular generative 
models from machine learning. We provide example knockoff designs using a variational autoencoder and 
a Gaussian process latent variable model. We also propose a knockoff score metric for a softmax classifier 
that accounts for the contribution of each feature and its knockoff during supervised learning. Experimental 
results with multiple benchmark datasets for feature selection showcase the advantages of our knockoff 
designs and the knockoff framework with respect to existing approaches.
Improving SAT Solver Heuristics with Graph Networks and Reinforcement Learning
Authors: Vitaly Kurin, Saad Godil, Shimon Whiteson, Bryan Catanzaro
Date: 2019-09-25 00:00:00
Tasks: Feature Engineering, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We present GQSAT, a branching heuristic in a Boolean SAT solver trained with value-based reinforcement learning (RL) using Graph Neural Networks for function approximation. Solvers using GQSAT are complete SAT solvers that either provide a satisfying assignment or a proof of unsatisfiability, which is required for many SAT applications. The branching heuristic commonly used in SAT solvers today suffers from bad decisions during their warm-up period, whereas GQSAT has been trained to examine the structure of the particular problem instance to make better decisions at the beginning of the search. Training GQSAT is data efficient and does not require elaborate dataset preparation or feature engineering to train. We train GQSAT on small SAT problems using RL interfacing with an existing SAT solver. We show that GQSAT is able to reduce the number of iterations required to solve SAT problems by 2-3X, and it generalizes to unsatisfiable SAT instances, as well as to problems with 5X more variables than it was trained on. We also show that, to a lesser extent, it generalizes to SAT problems from different domains by evaluating it on graph coloring. Our experiments show that augmenting SAT solvers with agents trained with RL and graph neural networks can improve performance on the SAT search problem.
Toward Understanding The Effect of Loss Function on The Performance of Knowledge Graph Embedding
Authors: Mojtaba Nayyeri, Chengjin Xu, Yadollah Yaghoobzadeh, Hamed Shariat Yazdi, Jens Lehmann
Date: 2019-09-25 00:00:00
Tasks: Graph Embedding, Knowledge Graph Embedding, Knowledge Graphs, Translation

Knowledge graphs (KGs) represent world's facts in structured forms.  KG completion exploits the existing facts in a KG to discover new ones. Translation-based embedding model (TransE) is a prominent formulation to do KG completion. 
Despite the efficiency of TransE in memory and time, it suffers from several limitations in encoding relation patterns such as  symmetric, reflexive etc. To resolve this problem, most of the attempts have circled around the revision of the score function of TransE i.e., proposing a more complicated score function such as Trans(A, D, G, H, R, etc) to mitigate the limitations.  In this paper, we tackle this problem from a different perspective. We show that existing theories corresponding to the limitations of TransE are inaccurate because they ignore the effect of loss function. Accordingly, we pose theoretical investigations of the main limitations of TransE in the light of loss function. To the best of our knowledge, this has not been investigated so far comprehensively. We show that by a proper selection of the loss function for training the TransE model, the main limitations of the model are mitigated. This is explained by setting upper-bound for the scores of positive samples, showing the region of truth (i.e., the region that a triple is considered positive by the model).
Our theoretical proofs with experimental results fill the gap between the capability of translation-based class of embedding models and the loss function. The theories emphasis the importance of the selection of the loss functions for training the models. Our experimental evaluations on different loss functions used for training the models justify our theoretical proofs and confirm the importance of the loss functions on the performance.
R2D2: Reuse & Reduce via Dynamic Weight Diffusion for Training Efficient NLP Models
Authors: Yi Tay, Aston Zhang, Shuai Zhang, Alvin Chan, Luu Anh Tuan, Siu Cheung Hui
Date: 2019-09-25 00:00:00

We propose R2D2 layers, a new neural block for training efficient NLP models. Our proposed method is characterized by a dynamic weight diffusion mechanism which learns to reuse and reduce parameters in the conventional transformation layer, commonly found in popular Transformer/LSTMs models. Our method is inspired by recent Quaternion methods which share parameters via the Hamilton product. This can be interpreted as a neural and learned approximation of the Hamilton product which imbues our method with increased flexibility and expressiveness, i.e., we are no longer restricted by the 4D nature of Quaternion weight sharing. We conduct extensive experiments in the NLP domain, showing that R2D2 (i) enables a parameter savings of up to 2 times to 16 times with minimal degradation of performance and (ii) outperforms other parameter savings alternative such as low-rank factorization and Quaternion methods.
SELF-KNOWLEDGE DISTILLATION ADVERSARIAL ATTACK
Authors: Ma Xiaoxiong[1], Wang Renzhi[1], Tian Cong, Dong Zeqian, Duan Zhenhua
Date: 2019-09-25 00:00:00
Tasks: Adversarial Attack, Knowledge Distillation, Self-Knowledge Distillation

Neural networks show great vulnerability under the threat of adversarial examples.
   By adding small perturbation to a clean image, neural networks with high classification accuracy can be completely fooled.
   One intriguing property of the adversarial examples is transferability.  This property allows adversarial examples to transfer to networks of unknown structure, which is harmful even to the physical world.
   The current way of generating adversarial examples is mainly divided into optimization based and gradient based methods.
   Liu et al. (2017) conjecture that gradient based methods can hardly produce transferable targeted adversarial examples in black-box-attack.
   However, in this paper, we use a simple technique to improve the transferability and success rate of targeted attacks with gradient based methods.
   We prove that gradient based methods can also generate transferable adversarial examples in targeted attacks.
   Specifically, we use knowledge distillation for gradient based methods, and show that the transferability can be improved by effectively utilizing different classes of information.
   Unlike the usual applications of knowledge distillation, we did not train a student network to generate adversarial examples.
   We take advantage of the fact that knowledge distillation can soften the target and obtain higher information, and combine the soft target and hard target of the same network as the loss function.
   Our method is generally applicable to most gradient based attack methods.
Model Comparison of Beer data classification using an electronic nose
Authors: Mohammed Abdi, Aminat Adebiyi, Andrea Fasoli, Alberto Mannari, Ronald Labby, Luisa Bozano
Date: 2019-09-25 00:00:00
Tasks: Classification

Olfaction has been and still is an area which is challenging to the research community. Like other senses of the body, there has been a push to replicate the sense of smell to aid in identifying odorous compounds in the form of an electronic nose. At IBM, our team (Cogniscent) has designed a modular sensor board platform based on the artificial olfaction concept we called EVA (Electronic Volatile Analyzer). EVA is an IoT electronic nose device that aims to reproduce olfaction in living begins by integrating an array of partially specific and uniquely selective smell recognition sensors which are directly exposed to the target chemical analyte or the environment. We are exploring a new technique called temperature-controlled oscillation, which gives us virtual array of sensors to represent our signals/ fingerprint. In our study, we run experiments on identifying different types of beers using EVA. In order to successfully carry this classification task, the entire process starting from preparation of samples, having a consistent protocol of data collection in place all the way to providing the data to be analyzed and input to a machine learning model is very important. On this paper, we will discuss the process of sniffing volatile organic compounds from liquid beer samples and successfully classifying different brands of beers as a pilot test. We researched on different machine learning models in order to get the best classification accuracy for our Beer samples. The best classification accuracy is achieved by using a multi-level perceptron (MLP) artificial neural network (ANN) model, classification of three different brands of beers after splitting one-week data to a training and testing set yielded an accuracy of 97.334. While using separate weeks of data for training and testing set the model yielded an accuracy of 67.812, this is because of drift playing a role in the overall classification process. Using Random forest, the classification accuracy achieved by the model is 0.923. And Decision Tree achieved 0.911.