pwc-archive/papers-with-abstracts huggingface dataset:
Renyi Differentially Private ADMM for Non-Smooth Regularized Optimization
Authors: Chen Chen, Jaewoo Lee
Date: 2019-09-18 00:00:00
Tasks: feature selection

In this paper we consider the problem of minimizing composite objective functions consisting of a convex differentiable loss function plus a non-smooth regularization term, such as $L_1$ norm or nuclear norm, under R\'enyi differential privacy (RDP). To solve the problem, we propose two stochastic alternating direction method of multipliers (ADMM) algorithms: ssADMM based on gradient perturbation and mpADMM based on output perturbation. Both algorithms decompose the original problem into sub-problems that have closed-form solutions. The first algorithm, ssADMM, applies the recent privacy amplification result for RDP to reduce the amount of noise to add. The second algorithm, mpADMM, numerically computes the sensitivity of ADMM variable updates and releases the updated parameter vector at the end of each epoch. We compare the performance of our algorithms with several baseline algorithms on both real and simulated datasets. Experimental results show that, in high privacy regimes (small $\epsilon$), ssADMM and mpADMM outperform other baseline algorithms in terms of classification and feature selection performance, respectively.
Self-boosted Time-series Forecasting with Multi-task and Multi-view Learning
Authors: Long H. Nguyen, Zhenhe Pan, Opeyemi Openiyi, Hashim Abu-gellban, Mahdi Moghadasi, Fang Jin
Date: 2019-09-17 00:00:00
Tasks: Multi-Task Learning, MULTI-VIEW LEARNING, Time Series, Time Series Analysis, Time Series Forecasting

A robust model for time series forecasting is highly important in many domains, including but not limited to financial forecast, air temperature and electricity consumption. To improve forecasting performance, traditional approaches usually require additional feature sets. However, adding more feature sets from different sources of data is not always feasible due to its accessibility limitation. In this paper, we propose a novel self-boosted mechanism in which the original time series is decomposed into multiple time series. These time series played the role of additional features in which the closely related time series group is used to feed into multi-task learning model, and the loosely related group is fed into multi-view learning part to utilize its complementary information. We use three real-world datasets to validate our model and show the superiority of our proposed method over existing state-of-the-art baseline methods.
Predicting Electricity Consumption using Deep Recurrent Neural Networks
Authors: Anupiya Nugaliyadde, Upeka Somaratne, Kok Wai Wong
Date: 2019-09-18 00:00:00

Electricity consumption has increased exponentially during the past few decades. This increase is heavily burdening the electricity distributors. Therefore, predicting the future demand for electricity consumption will provide an upper hand to the electricity distributor. Predicting electricity consumption requires many parameters. The paper presents two approaches with one using a Recurrent Neural Network (RNN) and another one using a Long Short Term Memory (LSTM) network, which only considers the previous electricity consumption to predict the future electricity consumption. These models were tested on the publicly available London smart meter dataset. To assess the applicability of the RNN and the LSTM network to predict electricity consumption, they were tested to predict for an individual house and a block of houses for a given time period. The predictions were done for daily, trimester and 13 months, which covers short term, mid-term and long term prediction. Both the RNN and the LSTM network have achieved an average Root Mean Square error of 0.1.
Transfer Learning with Dynamic Adversarial Adaptation Network
Authors: Chaohui Yu, Jindong Wang, Yiqiang Chen, Meiyu Huang
Date: 2019-09-18 00:00:00
Tasks: Domain Adaptation, Transfer Learning

The recent advances in deep transfer learning reveal that adversarial learning can be embedded into deep networks to learn more transferable features to reduce the distribution discrepancy between two domains. Existing adversarial domain adaptation methods either learn a single domain discriminator to align the global source and target distributions or pay attention to align subdomains based on multiple discriminators. However, in real applications, the marginal (global) and conditional (local) distributions between domains are often contributing differently to the adaptation. There is currently no method to dynamically and quantitatively evaluate the relative importance of these two distributions for adversarial learning. In this paper, we propose a novel Dynamic Adversarial Adaptation Network (DAAN) to dynamically learn domain-invariant representations while quantitatively evaluate the relative importance of global and local domain distributions. To the best of our knowledge, DAAN is the first attempt to perform dynamic adversarial distribution adaptation for deep adversarial learning. DAAN is extremely easy to implement and train in real applications. We theoretically analyze the effectiveness of DAAN, and it can also be explained in an attention strategy. Extensive experiments demonstrate that DAAN achieves better classification accuracy compared to state-of-the-art deep and adversarial methods. Results also imply the necessity and effectiveness of the dynamic distribution adaptation in adversarial transfer learning.
Learning to Generate Questions with Adaptive Copying Neural Networks
Authors: Xinyuan Lu, Yuhong Guo
Date: 2019-09-17 00:00:00
Tasks: Question Generation, Question-Generation

Automatic question generation is an important problem in natural language processing. In this paper we propose a novel adaptive copying recurrent neural network model to tackle the problem of question generation from sentences and paragraphs. The proposed model adds a copying mechanism component onto a bidirectional LSTM architecture to generate more suitable questions adaptively from the input data. Our experimental results show the proposed model can outperform the state-of-the-art question generation methods in terms of BLEU and ROUGE evaluation scores.
Fiber Nonlinearity Mitigation via the Parzen Window Classifier for Dispersion Managed and Unmanaged Links
Authors: Abdelkerim Amari, Xiang Lin, Octavia A. Dobre, Ramachandran Venkatesan, Alex Alvarado
Date: 2019-09-17 00:00:00
Tasks: BIG-bench Machine Learning

Machine learning techniques have recently received significant attention as promising approaches to deal with the optical channel impairments, and in particular, the nonlinear effects. In this work, a machine learning-based classification technique, known as the Parzen window (PW) classifier, is applied to mitigate the nonlinear effects in the optical channel. The PW classifier is used as a detector with improved nonlinear decision boundaries more adapted to the nonlinear fiber channel. Performance improvement is observed when applying the PW in the context of dispersion managed and dispersion unmanaged systems.
Two Computational Models for Analyzing Political Attention in Social Media
Authors: Libby Hemphill, Angela M. Schöpke-Gonzalez
Date: 2019-09-17 00:00:00
Tasks: Vocal Bursts Valence Prediction

Understanding how political attention is divided and over what subjects is crucial for research on areas such as agenda setting, framing, and political rhetoric. Existing methods for measuring attention, such as manual labeling according to established codebooks, are expensive and can be restrictive. We describe two computational models that automatically distinguish topics in politicians' social media content. Our models---one supervised classifier and one unsupervised topic model---provide different benefits. The supervised classifier reduces the labor required to classify content according to pre-determined topic list. However, tweets do more than communicate policy positions. Our unsupervised model uncovers both political topics and other Twitter uses (e.g., constituent service). These models are effective, inexpensive computational tools for political communication and social media research. We demonstrate their utility and discuss the different analyses they afford by applying both models to the tweets posted by members of the 115th U.S. Congress.
PixelHop: A Successive Subspace Learning (SSL) Method for Object Classification
Authors: Yueru Chen, C. -C. Jay Kuo
Date: 2019-09-17 00:00:00
Tasks: Benchmarking, Decision Making, Dimensionality Reduction, General Classification

A new machine learning methodology, called successive subspace learning (SSL), is introduced in this work. SSL contains four key ingredients: 1) successive near-to-far neighborhood expansion; 2) unsupervised dimension reduction via subspace approximation; 3) supervised dimension reduction via label-assisted regression (LAG); and 4) feature concatenation and decision making. An image-based object classification method, called PixelHop, is proposed to illustrate the SSL design. It is shown by experimental results that the PixelHop method outperforms the classic CNN model of similar model complexity in three benchmarking datasets (MNIST, Fashion MNIST and CIFAR-10). Although SSL and deep learning (DL) have some high-level concept in common, they are fundamentally different in model formulation, the training process and training complexity. Extensive discussion on the comparison of SSL and DL is made to provide further insights into the potential of SSL.
Dual Adversarial Co-Learning for Multi-Domain Text Classification
Authors: Yuan Wu, Yuhong Guo
Date: 2019-09-18 00:00:00
Tasks: Classification, General Classification, Multi-Domain Sentiment Classification, Sentiment Analysis, Sentiment Classification, text-classification, Text Classification

In this paper we propose a novel dual adversarial co-learning approach for multi-domain text classification (MDTC). The approach learns shared-private networks for feature extraction and deploys dual adversarial regularizations to align features across different domains and between labeled and unlabeled data simultaneously under a discrepancy based co-learning framework, aiming to improve the classifiers' generalization capacity with the learned features. We conduct experiments on multi-domain sentiment classification datasets. The results show the proposed approach achieves the state-of-the-art MDTC performance.
Data Mapping and Finite Difference Learning
Authors: Jiangsheng You
Date: 2019-09-18 00:00:00
Tasks: Dimensionality Reduction

Restricted Boltzmann machine (RBM) is a two-layer neural network constructed as a probabilistic model and its training is to maximize a product of probabilities by the contrastive divergence (CD) scheme. In this paper a data mapping is proposed to describe the relationship between the visible and hidden layers and the training is to minimize a squared error on the visible layer by a finite difference learning. This paper presents three new properties in using the RBM: 1) nodes on the visible and hidden layers can take real-valued matrix data without a probabilistic interpretation; 2) the famous CD1 is a finite difference approximation of the gradient descent; 3) the activation can take non-sigmoid functions such as identity, relu and softsign. The data mapping provides a unified framework on the dimensionality reduction, the feature extraction and the data representation pioneered and developed by Hinton and his colleagues. As an approximation of the gradient descent, the finite difference learning is applicable to both directed and undirected graphs. Numerical experiments are performed to verify these new properties on the very low dimensionality reduction, the collinearity of timer series data and the use of flexible activations.
How have German University Tuition Fees Affected Enrollment Rates: Robust Model Selection and Design-based Inference in High-Dimensions
Authors: Konstantin Görgen, Melanie Schienle
Date: 2019-09-18 00:00:00
Tasks: Model Selection

We use official data for all 16 federal German states to study the causal effect of a flat 1000 Euro state-dependent university tuition fee on the enrollment behavior of students during the years 2006-2014. In particular, we show how the variation in the introduction scheme across states and times can be exploited to identify the federal average causal effect of tuition fees by controlling for a large amount of potentially influencing attributes for state heterogeneity. We suggest a stability post-double selection methodology to robustly determine the causal effect across types in the transparently modeled unknown response components. The proposed stability resampling scheme in the two LASSO selection steps efficiently mitigates the risk of model underspecification and thus biased effects when the tuition fee policy decision also depends on relevant variables for the state enrollment rates. Correct inference for the full cross-section state population in the sample requires adequate design -- rather than sampling-based standard errors. With the data-driven model selection and explicit control for spatial cross-effects we detect that tuition fees induce substantial migration effects where the mobility occurs both from fee but also from non-fee states suggesting also a general movement for quality. Overall, we find a significant negative impact of up to 4.5 percentage points of fees on student enrollment. This is in contrast to plain one-step LASSO or previous empirical studies with full fixed effects linear panel regressions which generally underestimate the size and get an only insignificant effect.
Memory-Augmented Neural Networks for Machine Translation
Authors: Mark Collier, Joeran Beel
Date: 2019-09-18 00:00:00
Proceeding: WS 2019 8
Tasks: Decoder, Machine Translation, Translation

Memory-augmented neural networks (MANNs) have been shown to outperform other recurrent neural network architectures on a series of artificial sequence learning tasks, yet they have had limited application to real-world tasks. We evaluate direct application of Neural Turing Machines (NTM) and Differentiable Neural Computers (DNC) to machine translation. We further propose and evaluate two models which extend the attentional encoder-decoder with capabilities inspired by memory augmented neural networks. We evaluate our proposed models on IWSLT Vietnamese to English and ACL Romanian to English datasets. Our proposed models and the memory augmented neural networks perform similarly to the attentional encoder-decoder on the Vietnamese to English translation task while have a 0.3-1.9 lower BLEU score for the Romanian to English task. Interestingly, our analysis shows that despite being equipped with additional flexibility and being randomly initialized memory augmented neural networks learn an algorithm for machine translation almost identical to the attentional encoder-decoder.
From Server-Based to Client-Based Machine Learning: A Comprehensive Survey
Authors: Renjie Gu, Chaoyue Niu, Fan Wu, Guihai Chen, Chun Hu, Chengfei Lyu, Zhihua Wu
Date: 2019-09-18 00:00:00
Tasks: BIG-bench Machine Learning

In recent years, mobile devices have gained increasing development with stronger computation capability and larger storage space. Some of the computation-intensive machine learning tasks can now be run on mobile devices. To exploit the resources available on mobile devices and preserve personal privacy, the concept of client-based machine learning has been proposed. It leverages the users' local hardware and local data to solve machine learning sub-problems on mobile devices and only uploads computation results rather than the original data for the optimization of the global model. Such an architecture can not only relieve computation and storage burdens on servers but also protect the users' sensitive information. Another benefit is the bandwidth reduction because various kinds of local data can be involved in the training process without being uploaded. In this article, we provide a literature review on the progressive development of machine learning from server based to client based. We revisit a number of widely used server-based and client-based machine learning methods and applications. We also extensively discuss the challenges and future directions in this area. We believe that this survey will give a clear overview of client-based machine learning and provide guidelines on applying client-based machine learning to practice.
A Hierarchical Two-tier Approach to Hyper-parameter Optimization in Reinforcement Learning
Authors: Juan Cruz Barsce, Jorge A. Palombarini, Ernesto Martínez
Date: 2019-09-18 00:00:00
Tasks: Bayesian Optimization, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Optimization of hyper-parameters in reinforcement learning (RL) algorithms is a key task, because they determine how the agent will learn its policy by interacting with its environment, and thus what data is gathered. In this work, an approach that uses Bayesian optimization to perform a two-step optimization is proposed: first, categorical RL structure hyper-parameters are taken as binary variables and optimized with an acquisition function tailored for such variables. Then, at a lower level of abstraction, solution-level hyper-parameters are optimized by resorting to the expected improvement acquisition function, while using the best categorical hyper-parameters found in the optimization at the upper-level of abstraction. This two-tier approach is validated in a simulated control task. Results obtained are promising and open the way for more user-independent applications of reinforcement learning.
Advancing subgroup fairness via sleeping experts
Authors: Avrim Blum, Thodoris Lykouris
Date: 2019-09-18 00:00:00
Tasks: Fairness

We study methods for improving fairness to subgroups in settings with overlapping populations and sequential predictions. Classical notions of fairness focus on the balance of some property across different populations. However, in many applications the goal of the different groups is not to be predicted equally but rather to be predicted well. We demonstrate that the task of satisfying this guarantee for multiple overlapping groups is not straightforward and show that for the simple objective of unweighted average of false negative and false positive rate, satisfying this for overlapping populations can be statistically impossible even when we are provided predictors that perform well separately on each subgroup. On the positive side, we show that when individuals are equally important to the different groups they belong to, this goal is achievable; to do so, we draw a connection to the sleeping experts literature in online learning. Motivated by the one-sided feedback in natural settings of interest, we extend our results to such a feedback model. We also provide a game-theoretic interpretation of our results, examining the incentives of participants to join the system and to provide the system full information about predictors they may possess. We end with several interesting open problems concerning the strength of guarantees that can be achieved in a computationally efficient manner.
Laplacian Matrix for Dimensionality Reduction and Clustering
Authors: Laurenz Wiskott, Fabian Schönfeld
Date: 2019-09-18 00:00:00
Tasks: Clustering, Dimensionality Reduction

Many problems in machine learning can be expressed by means of a graph with nodes representing training samples and edges representing the relationship between samples in terms of similarity, temporal proximity, or label information. Graphs can in turn be represented by matrices. A special example is the Laplacian matrix, which allows us to assign each node a value that varies only little between strongly connected nodes and more between distant nodes. Such an assignment can be used to extract a useful feature representation, find a good embedding of data in a low dimensional space, or perform clustering on the original samples. In these lecture notes we first introduce the Laplacian matrix and then present a small number of algorithms designed around it.
A continual learning survey: Defying forgetting in classification tasks
Authors: Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, Tinne Tuytelaars
Date: 2019-09-18 00:00:00
Tasks: Classification, Continual Learning, General Classification, Incremental Learning, Survey

Artificial neural networks thrive in solving the classification problem for a particular rigid task, acquiring knowledge through generalized learning behaviour from a distinct training phase. The resulting network resembles a static entity of knowledge, with endeavours to extend this knowledge without targeting the original task resulting in a catastrophic forgetting. Continual learning shifts this paradigm towards networks that can continually accumulate knowledge over different tasks without the need to retrain from scratch. We focus on task incremental classification, where tasks arrive sequentially and are delineated by clear boundaries. Our main contributions concern 1) a taxonomy and extensive overview of the state-of-the-art, 2) a novel framework to continually determine the stability-plasticity trade-off of the continual learner, 3) a comprehensive experimental comparison of 11 state-of-the-art continual learning methods and 4 baselines. We empirically scrutinize method strengths and weaknesses on three benchmarks, considering Tiny Imagenet and large-scale unbalanced iNaturalist and a sequence of recognition datasets. We study the influence of model capacity, weight decay and dropout regularization, and the order in which the tasks are presented, and qualitatively compare methods in terms of required memory, computation time, and storage.
Intelligent Active Queue Management Using Explicit Congestion Notification
Authors: Cesar A. Gomez, Xianbin Wang, Abdallah Shami
Date: 2019-08-28 00:00:00
Tasks: Management, Reinforcement Learning

As more end devices are getting connected, the Internet will become more congested. Various congestion control techniques have been developed either on transport or network layers. Active Queue Management (AQM) is a paradigm that aims to mitigate the congestion on the network layer through active buffer control to avoid overflow. However, finding the right parameters for an AQM scheme is challenging, due to the complexity and dynamics of the networks. On the other hand, the Explicit Congestion Notification (ECN) mechanism is a solution that makes visible incipient congestion on the network layer to the transport layer. In this work, we propose to exploit the ECN information to improve AQM algorithms by applying Machine Learning techniques. Our intelligent method uses an artificial neural network to predict congestion and an AQM parameter tuner based on reinforcement learning. The evaluation results show that our solution can enhance the performance of deployed AQM, using the existing TCP congestion control mechanisms.