pwc-archive/papers-with-abstracts huggingface dataset:
Preference-based Reinforcement Learning with Finite-Time Guarantees
Authors: Yichong Xu, Ruosong Wang, Lin F. Yang, Aarti Singh, Artur Dubrawski
Date: 2020-06-16 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Preference-based Reinforcement Learning (PbRL) replaces reward values in traditional reinforcement learning by preferences to better elicit human opinion on the target objective, especially when numerical reward values are hard to design or interpret. Despite promising results in applications, the theoretical understanding of PbRL is still in its infancy. In this paper, we present the first finite-time analysis for general PbRL problems. We first show that a unique optimal policy may not exist if preferences over trajectories are deterministic for PbRL. If preferences are stochastic, and the preference probability relates to the hidden reward values, we present algorithms for PbRL, both with and without a simulator, that are able to identify the best policy up to accuracy $\varepsilon$ with high probability. Our method explores the state space by navigating to under-explored states, and solves PbRL using a combination of dueling bandits and policy search. Experiments show the efficacy of our method when it is applied to real-world problems.
Robust Federated Learning: The Case of Affine Distribution Shifts
Authors: Amirhossein Reisizadeh, Farzan Farnia, Ramtin Pedarsani, Ali Jadbabaie
Date: 2020-06-16 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Federated Learning, image-classification, Image Classification

Federated learning is a distributed paradigm that aims at training models using samples distributed across multiple users in a network while keeping the samples on users' devices with the aim of efficiency and protecting users privacy. In such settings, the training data is often statistically heterogeneous and manifests various distribution shifts across users, which degrades the performance of the learnt model. The primary goal of this paper is to develop a robust federated learning algorithm that achieves satisfactory performance against distribution shifts in users' samples. To achieve this goal, we first consider a structured affine distribution shift in users' data that captures the device-dependent data heterogeneity in federated settings. This perturbation model is applicable to various federated learning problems such as image classification where the images undergo device-dependent imperfections, e.g. different intensity, contrast, and brightness. To address affine distribution shifts across users, we propose a Federated Learning framework Robust to Affine distribution shifts (FLRA) that is provably robust against affine Wasserstein shifts to the distribution of observed samples. To solve the FLRA's distributed minimax problem, we propose a fast and efficient optimization method and provide convergence guarantees via a gradient Descent Ascent (GDA) method. We further prove generalization error bounds for the learnt classifier to show proper generalization from empirical distribution of samples to the true underlying distribution. We perform several numerical experiments to empirically support FLRA. We show that an affine distribution shift indeed suffices to significantly decrease the performance of the learnt classifier in a new test user, and our proposed algorithm achieves a significant gain in comparison to standard federated learning and adversarial training methods.
META-Learning Eligibility Traces for More Sample Efficient Temporal Difference Learning
Authors: Mingde Zhao
Date: 2020-06-16 00:00:00
Tasks: Meta-Learning

Temporal-Difference (TD) learning is a standard and very successful reinforcement learning approach, at the core of both algorithms that learn the value of a given policy, as well as algorithms which learn how to improve policies. TD-learning with eligibility traces provides a way to do temporal credit assignment, i.e. decide which portion of a reward should be assigned to predecessor states that occurred at different previous times, controlled by a parameter $\lambda$. However, tuning this parameter can be time-consuming, and not tuning it can lead to inefficient learning. To improve the sample efficiency of TD-learning, we propose a meta-learning method for adjusting the eligibility trace parameter, in a state-dependent manner. The adaptation is achieved with the help of auxiliary learners that learn distributional information about the update targets online, incurring roughly the same computational complexity per step as the usual value learner. Our approach can be used both in on-policy and off-policy learning. We prove that, under some assumptions, the proposed method improves the overall quality of the update targets, by minimizing the overall target error. This method can be viewed as a plugin which can also be used to assist prediction with function approximation by meta-learning feature (observation)-based $\lambda$ online, or even in the control case to assist policy improvement. Our empirical evaluation demonstrates significant performance improvements, as well as improved robustness of the proposed algorithm to learning rate variation.
Causal Knowledge Extraction from Scholarly Papers in Social Sciences
Authors: Victor Zitian Chen, Felipe Montano-Campos, Wlodek Zadrozny
Date: 2020-06-16 00:00:00
Tasks: Articles, Entity Extraction using GAN, General Classification, Management

The scale and scope of scholarly articles today are overwhelming human researchers who seek to timely digest and synthesize knowledge. In this paper, we seek to develop natural language processing (NLP) models to accelerate the speed of extraction of relationships from scholarly papers in social sciences, identify hypotheses from these papers, and extract the cause-and-effect entities. Specifically, we develop models to 1) classify sentences in scholarly documents in business and management as hypotheses (hypothesis classification), 2) classify these hypotheses as causal relationships or not (causality classification), and, if they are causal, 3) extract the cause and effect entities from these hypotheses (entity extraction). We have achieved high performance for all the three tasks using different modeling techniques. Our approach may be generalizable to scholarly documents in a wide range of social sciences, as well as other types of textual materials.
Depth by Poking: Learning to Estimate Depth from Self-Supervised Grasping
Authors: Ben Goodrich, Alex Kuefler, William D. Richards
Date: 2020-06-16 00:00:00
Tasks: Depth Estimation, Position

Accurate depth estimation remains an open problem for robotic manipulation; even state of the art techniques including structured light and LiDAR sensors fail on reflective or transparent surfaces. We address this problem by training a neural network model to estimate depth from RGB-D images, using labels from physical interactions between a robot and its environment. Our network predicts, for each pixel in an input image, the z position that a robot's end effector would reach if it attempted to grasp or poke at the corresponding position. Given an autonomous grasping policy, our approach is self-supervised as end effector position labels can be recovered through forward kinematics, without human annotation. Although gathering such physical interaction data is expensive, it is necessary for training and routine operation of state of the art manipulation systems. Therefore, this depth estimator comes ``for free'' while collecting data for other tasks (e.g., grasping, pushing, placing). We show our approach achieves significantly lower root mean squared error than traditional structured light sensors and unsupervised deep learning methods on difficult, industry-scale jumbled bin datasets.
DefenseVGAE: Defending against Adversarial Attacks on Graph Data via a Variational Graph Autoencoder
Authors: Ao Zhang, Jinwen Ma
Date: 2020-06-16 00:00:00

Graph neural networks (GNNs) achieve remarkable performance for tasks on graph data. However, recent works show they are extremely vulnerable to adversarial structural perturbations, making their outcomes unreliable. In this paper, we propose DefenseVGAE, a novel framework leveraging variational graph autoencoders(VGAEs) to defend GNNs against such attacks. DefenseVGAE is trained to reconstruct graph structure. The reconstructed adjacency matrix can reduce the effects of adversarial perturbations and boost the performance of GCNs when facing adversarial attacks. Our experiments on a number of datasets show the effectiveness of the proposed method under various threat models. Under some settings it outperforms existing defense strategies. Our code has been made publicly available at https://github.com/zhangao520/defense-vgae.
Acquisition of Channel State Information for mmWave Massive MIMO: Traditional and Machine Learning-based Approaches
Authors: Chenhao Qi, Peihao Dong, Wenyan Ma, Hua Zhang, Zaichen Zhang, Geoffrey Ye Li
Date: 2020-06-16 00:00:00
Tasks: BIG-bench Machine Learning

The accuracy of channel state information (CSI) acquisition directly affects the performance of millimeter wave (mmWave) communications. In this article, we provide an overview on CSI acquisition, including beam training and channel estimation for mmWave massive multiple-input multiple-output systems. The beam training can avoid the estimation of a high-dimension channel matrix while the channel estimation can flexibly exploit advanced signal processing techniques. In addition to introducing the traditional and machine learning-based approaches in this article, we also compare different approaches in terms of spectral efficiency, computational complexity, and overhead.
An Intelligent Group Event Recommendation System in Social networks
Authors: Guoqiong Liao, Xiaomei Huang, Neal N. Xiong, Changxuan Wan
Date: 2020-06-16 00:00:00
Tasks: Recommendation Systems

The importance of contexts has been widely recognized in recommender systems for individuals. However, most existing group recommendation models in Event-Based Social Networks (EBSNs) focus on how to aggregate group members' preferences to form group preferences. In these models, the influence of contexts on groups is considered but simply defined in a manual way, which cannot model the complex and deep interactions between contexts and groups. In this paper, we propose an Attention-based Context-aware Group Event Recommendation model (ACGER) in EBSNs. ACGER models the deep, non-linear influence of contexts on users, groups, and events through multi-layer neural networks. Especially, a novel attention mechanism is designed to enable the influence weights of contexts on users/groups change dynamically with the events concerned. Considering that groups may have completely different behavior patterns from group members, we propose that the preference of a group need to be obtained from indirect and direct perspectives (called indirect preference and direct preference respectively). In order to obtain the indirect preference, we propose a method of aggregating preferences based on attention mechanism. Compared with existing predefined strategies, this method can flexibly adapt the strategy according to the events concerned by the group. In order to obtain the direct preference, we employ neural networks to directly learn it from group-event interactions. Furthermore, to make full use of rich user-event interactions in EBSNs, we integrate the context-aware individual recommendation task into ACGER, which enhances the accuracy of learning of user embeddings and event embeddings. Extensive experiments on two real datasets from Meetup show that our model ACGER significantly outperforms the state-of-the-art models.
Exploiting Visual Semantic Reasoning for Video-Text Retrieval
Authors: Zerun Feng, Zhimin Zeng, Caili Guo, Zheng Li
Date: 2020-06-16 00:00:00
Tasks: Retrieval, Text Retrieval, text similarity, Video Retrieval, Video-Text Retrieval

Video retrieval is a challenging research topic bridging the vision and language areas and has attracted broad attention in recent years. Previous works have been devoted to representing videos by directly encoding from frame-level features. In fact, videos consist of various and abundant semantic relations to which existing methods pay less attention. To address this issue, we propose a Visual Semantic Enhanced Reasoning Network (ViSERN) to exploit reasoning between frame regions. Specifically, we consider frame regions as vertices and construct a fully-connected semantic correlation graph. Then, we perform reasoning by novel random walk rule-based graph convolutional networks to generate region features involved with semantic relations. With the benefit of reasoning, semantic interactions between regions are considered, while the impact of redundancy is suppressed. Finally, the region features are aggregated to form frame-level features for further encoding to measure video-text similarity. Extensive experiments on two public benchmark datasets validate the effectiveness of our method by achieving state-of-the-art performance due to the powerful semantic reasoning.
PAC-Bayesian Generalization Bounds for MultiLayer Perceptrons
Authors: Xinjie Lan, Xin Guo, Kenneth E. Barner
Date: 2020-06-16 00:00:00
Tasks: Generalization Bounds, Variational Inference

We study PAC-Bayesian generalization bounds for Multilayer Perceptrons (MLPs) with the cross entropy loss. Above all, we introduce probabilistic explanations for MLPs in two aspects: (i) MLPs formulate a family of Gibbs distributions, and (ii) minimizing the cross-entropy loss for MLPs is equivalent to Bayesian variational inference, which establish a solid probabilistic foundation for studying PAC-Bayesian bounds on MLPs. Furthermore, based on the Evidence Lower Bound (ELBO), we prove that MLPs with the cross entropy loss inherently guarantee PAC- Bayesian generalization bounds, and minimizing PAC-Bayesian generalization bounds for MLPs is equivalent to maximizing the ELBO. Finally, we validate the proposed PAC-Bayesian generalization bound on benchmark datasets.
DeepCapture: Image Spam Detection Using Deep Learning and Data Augmentation
Authors: Bedeuro Kim, Sharif Abuadbba, Hyoungshick Kim
Date: 2020-06-16 00:00:00
Tasks: Data Augmentation, Deep Learning, Spam detection

Image spam emails are often used to evade text-based spam filters that detect spam emails with their frequently used keywords. In this paper, we propose a new image spam email detection tool called DeepCapture using a convolutional neural network (CNN) model. There have been many efforts to detect image spam emails, but there is a significant performance degrade against entirely new and unseen image spam emails due to overfitting during the training phase. To address this challenging issue, we mainly focus on developing a more robust model to address the overfitting problem. Our key idea is to build a CNN-XGBoost framework consisting of eight layers only with a large number of training samples using data augmentation techniques tailored towards the image spam detection task. To show the feasibility of DeepCapture, we evaluate its performance with publicly available datasets consisting of 6,000 spam and 2,313 non-spam image samples. The experimental results show that DeepCapture is capable of achieving an F1-score of 88%, which has a 6% improvement over the best existing spam detection model CNN-SVM with an F1-score of 82%. Moreover, DeepCapture outperformed existing image spam detection solutions against new and unseen image datasets.
Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation
Authors: Kellie Webster, Emily Pitler
Date: 2020-06-16 00:00:00
Tasks: document understanding, Machine Translation, Sentence, Translation

Machine translation systems with inadequate document understanding can make errors when translating dropped or neutral pronouns into languages with gendered pronouns (e.g., English). Predicting the underlying gender of these pronouns is difficult since it is not marked textually and must instead be inferred from coreferent mentions in the context. We propose a novel cross-lingual pivoting technique for automatically producing high-quality gender labels, and show that this data can be used to fine-tune a BERT classifier with 92% F1 for Spanish dropped feminine pronouns, compared with 30-51% for neural machine translation models and 54-71% for a non-fine-tuned BERT model. We augment a neural machine translation model with labels from our classifier to improve pronoun translation, while still having parallelizable translation models that translate a sentence at a time.
On SCC-recursiveness in Quantitative Argumentation
Authors: Zongshun Wang, Yuping Shen
Date: 2020-06-16 00:00:00
Tasks: Abstract Argumentation

Abstract argumentation is a reasoning model for evaluating arguments based on various semantics. SCC-recursiveness is a sophisticated property of semantics that provides a general schema for characterizing semantics through the decomposition along strongly connected components (SCCs). While this property has been extensively explored in various qualitative frameworks, it has been relatively neglected in quantitative argumentation. To fill this gap, we demonstrate that this property is well-suited to fuzzy extension semantics, which is a quantitative generalization of classical semantics in fuzzy argumentation frameworks (FAF). We tailor the SCC-recursive schema to enable the characterization of fuzzy extension semantics through the recursive decomposition of an FAF along its SCCs. Our contributions are twofold. Theoretically, we show that SCC-recursiveness provides an alternative approach to characterize fuzzy extension semantics, offering a deep understanding and better insight into these semantics. Practically, our schema provides a sound and complete algorithm for computing fuzzy extension semantics, which naturally reduces computational efforts when dealing with a large number of SCCs.
CNN Acceleration by Low-rank Approximation with Quantized Factors
Authors: Nikolay Kozyrskiy, Anh-Huy Phan
Date: 2020-06-16 00:00:00
Tasks: Quantization

The modern convolutional neural networks although achieve great results in solving complex computer vision tasks still cannot be effectively used in mobile and embedded devices due to the strict requirements for computational complexity, memory and power consumption. The CNNs have to be compressed and accelerated before deployment. In order to solve this problem the novel approach combining two known methods, low-rank tensor approximation in Tucker format and quantization of weights and feature maps (activations), is proposed. The greedy one-step and multi-step algorithms for the task of multilinear rank selection are proposed. The approach for quality restoration after applying Tucker decomposition and quantization is developed. The efficiency of our method is demonstrated for ResNet18 and ResNet34 on CIFAR-10, CIFAR-100 and Imagenet classification tasks. As a result of comparative analysis performed for other methods for compression and acceleration our approach showed its promising features.
Practical Quasi-Newton Methods for Training Deep Neural Networks
Authors: Donald Goldfarb, Yi Ren, Achraf Bahamou
Date: 2020-06-16 00:00:00
Proceeding: NeurIPS 2020 12

We consider the development of practical stochastic quasi-Newton, and in particular Kronecker-factored block-diagonal BFGS and L-BFGS methods, for training deep neural networks (DNNs). In DNN training, the number of variables and components of the gradient $n$ is often of the order of tens of millions and the Hessian has $n^2$ elements. Consequently, computing and storing a full $n \times n$ BFGS approximation or storing a modest number of (step, change in gradient) vector pairs for use in an L-BFGS implementation is out of the question. In our proposed methods, we approximate the Hessian by a block-diagonal matrix and use the structure of the gradient and Hessian to further approximate these blocks, each of which corresponds to a layer, as the Kronecker product of two much smaller matrices. This is analogous to the approach in KFAC, which computes a Kronecker-factored block-diagonal approximation to the Fisher matrix in a stochastic natural gradient method. Because the indefinite and highly variable nature of the Hessian in a DNN, we also propose a new damping approach to keep the upper as well as the lower bounds of the BFGS and L-BFGS approximations bounded. In tests on autoencoder feed-forward neural network models with either nine or thirteen layers applied to three datasets, our methods outperformed or performed comparably to KFAC and state-of-the-art first-order stochastic methods.
Model-based Adversarial Meta-Reinforcement Learning
Authors: Zichuan Lin, Garrett Thomas, Guangwen Yang, Tengyu Ma
Date: 2020-06-16 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: continuous-control, Continuous Control, Meta Reinforcement Learning, model, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Meta-reinforcement learning (meta-RL) aims to learn from multiple training tasks the ability to adapt efficiently to unseen test tasks. Despite the success, existing meta-RL algorithms are known to be sensitive to the task distribution shift. When the test task distribution is different from the training task distribution, the performance may degrade significantly. To address this issue, this paper proposes Model-based Adversarial Meta-Reinforcement Learning (AdMRL), where we aim to minimize the worst-case sub-optimality gap -- the difference between the optimal return and the return that the algorithm achieves after adaptation -- across all tasks in a family of tasks, with a model-based approach. We propose a minimax objective and optimize it by alternating between learning the dynamics model on a fixed task and finding the adversarial task for the current model -- the task for which the policy induced by the model is maximally suboptimal. Assuming the family of tasks is parameterized, we derive a formula for the gradient of the suboptimality with respect to the task parameters via the implicit function theorem, and show how the gradient estimator can be efficiently implemented by the conjugate gradient method and a novel use of the REINFORCE estimator. We evaluate our approach on several continuous control benchmarks and demonstrate its efficacy in the worst-case performance over all tasks, the generalization power to out-of-distribution tasks, and in training and test time sample efficiency, over existing state-of-the-art meta-RL algorithms.
GO Hessian for Expectation-Based Objectives
Authors: Yulai Cong, Miaoyun Zhao, Jianqiao Li, Junya Chen, Lawrence Carin
Date: 2020-06-16 00:00:00

An unbiased low-variance gradient estimator, termed GO gradient, was proposed recently for expectation-based objectives $\mathbb{E}_{q_{\boldsymbol{\gamma}}(\boldsymbol{y})} [f(\boldsymbol{y})]$, where the random variable (RV) $\boldsymbol{y}$ may be drawn from a stochastic computation graph with continuous (non-reparameterizable) internal nodes and continuous/discrete leaves. Upgrading the GO gradient, we present for $\mathbb{E}_{q_{\boldsymbol{\boldsymbol{\gamma}}}(\boldsymbol{y})} [f(\boldsymbol{y})]$ an unbiased low-variance Hessian estimator, named GO Hessian. Considering practical implementation, we reveal that GO Hessian is easy-to-use with auto-differentiation and Hessian-vector products, enabling efficient cheap exploitation of curvature information over stochastic computation graphs. As representative examples, we present the GO Hessian for non-reparameterizable gamma and negative binomial RVs/nodes. Based on the GO Hessian, we design a new second-order method for $\mathbb{E}_{q_{\boldsymbol{\boldsymbol{\gamma}}}(\boldsymbol{y})} [f(\boldsymbol{y})]$, with rigorous experiments conducted to verify its effectiveness and efficiency.