pwc-archive/papers-with-abstracts huggingface dataset:
Polar Embedding
Authors: Ran Iwamoto, Ryosuke Kohita, Akifumi Wachi
Date: 2021-01-01 00:00:00
Proceeding: CoNLL (EMNLP) 2021 11
Tasks: Link Prediction

An efficient representation of a hierarchical structure is essential for developing intelligent systems because most real-world objects are arranged in hierarchies. A distributional representation has brought great success in numerous natural language processing tasks, and a hierarchy is also successfully represented with embeddings. Particularly, the latest approaches such as hyperbolic embeddings showed significant performance by representing essential meanings in a hierarchy (generality and similarity of objects) with spatial properties (distance from the origin and difference of angles). To achieve such an effective connection in commonly used Euclidean space, we propose Polar Embedding that learns representations with the polar coordinate system. In polar coordinates, an object is expressed with two independent variables: radius and angles, which allows us to separately optimize their values on the explicit correspondence of generality and similarity of objects in a hierarchy. Also, we introduce an optimization method combining a loss function controlling gradient and iterative uniformization of distributions.  We overcome the issue resulting from the characteristics that the conventional squared loss function makes distance apart as much as possible for irrelevant pairs in spite of the fact the angle ranges are limited. Our results on a standard link-prediction task indicate that polar embedding outperforms other embeddings in low-dimensional Euclidean space and competitively performs even with hyperbolic embeddings, which possess a geometric advantage.
Dynamic Feature Selection for Efficient and Interpretable Human Activity Recognition
Authors: Randy Ardywibowo, Shahin Boluki, Zhangyang Wang, Bobak J Mortazavi, Shuai Huang, Xiaoning Qian
Date: 2021-01-01 00:00:00
Tasks: Activity Recognition, Combinatorial Optimization, feature selection, Human Activity Recognition, Stochastic Optimization

In many machine learning tasks, input features with varying degrees of predictive capability are usually acquired at some cost. For example, in human activity recognition (HAR) and mobile health (mHealth) applications, monitoring performance should be achieved with a low cost to gather different sensory features, as maintaining sensors incur monetary, computation, and energy cost. We propose an adaptive feature selection method that dynamically selects features for prediction at any given time point. We formulate this problem as an $\ell_0$ minimization problem across time, and cast the combinatorial optimization problem into a stochastic optimization formulation. We then utilize a differentiable relaxation to make the problem amenable to gradient-based optimization. Our evaluations on four activity recognition datasets show that our method achieves a favorable trade-off between performance and the number of features used. Moreover, the dynamically selected features of our approach are shown to be interpretable and associated with the actual activity types.
Deep $k$-NN Label Smoothing Improves Reproducibility of Neural Network Predictions
Authors: Dara Bahri, Heinrich Jiang
Date: 2021-01-01 00:00:00
Tasks: Prediction

Training modern neural networks is an inherently noisy process that can lead to high \emph{prediction churn}-- disagreements between re-trainings of the same model due to factors such as randomization in the parameter initialization and mini-batches-- even when the trained models all attain high accuracies. Such prediction churn can be very undesirable in practice. In this paper, we present several baselines for reducing churn and show that utilizing the $k$-NN predictions to smooth the labels results in a new and principled method that outperforms the baselines on churn while improving accuracy on a variety of benchmark classification tasks and model architectures.
Variational Structured Attention Networks for Dense Pixel-Wise Prediction
Authors: Guanglei Yang, Paolo Rota, Xavier Alameda-Pineda, Dan Xu, Mingli Ding, Elisa Ricci
Date: 2021-01-01 00:00:00
Tasks: Prediction

State-of-the-art performances in dense pixel-wise prediction tasks are obtained with specifically designed convolutional networks. These models often benefit from attention mechanisms that allow better learning of deep representations. Recent works showed the importance of estimating both spatial- and channel-wise attention tensors. In this paper, we propose a unified approach to jointly estimate spatial attention maps and channel attention vectors so as to structure the resulting attention tensor. Moreover, we integrate the estimation of the attention within a probabilistic framework, leading to VarIational STructured Attention networks(VISTA). We implement the inference rules within the neural network, thus allowing for joint learning of the probabilistic and the CNN front-end parameters. Importantly, as demonstrated by our extensive empirical evaluation on six large-scale datasets VISTA outperforms the state-of-the-art in multiple continuous and discrete pixel-level prediction tasks, thus confirming the benefit of structuring the attention tensor and of inferring it within a probabilistic formulation.
Private Image Reconstruction from System Side Channels Using Generative Models
Authors: Yuanyuan Yuan, Shuai Wang, Junping Zhang
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: CPU, Image Reconstruction, Side Channel Analysis

System side channels denote effects imposed on the underlying system and hardware when running a program, such as its accessed CPU cache lines. Side channel analysis (SCA) allows attackers to infer program secrets based on observed side channel signals. Given the ever-growing adoption of machine learning as a service (MLaaS), image analysis software on cloud platforms has been exploited by reconstructing private user images from system side channels. Nevertheless, to date, SCA is still highly challenging, requiring technical knowledge of victim software's internal operations. For existing SCA attacks, comprehending such internal operations requires heavyweight program analysis or manual efforts.

This research proposes an attack framework to reconstruct private user images processed by media software via system side channels. The framework forms an effective workflow by incorporating convolutional networks, variational autoencoders, and generative adversarial networks. Our evaluation of two popular side channels shows that the reconstructed images consistently match user inputs, making privacy leakage attacks more practical. We also show surprising results that even one-bit data read/write pattern side channels, which are deemed minimally informative, can be used to reconstruct quality images using our framework.
RMSprop can converge with proper hyper-parameter
Authors: Naichen Shi, Dawei Li, Mingyi Hong, Ruoyu Sun
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1

Despite the existence of divergence examples, RMSprop remains one of the most popular algorithms in machine learning. Towards closing the gap between theory and practice, we prove that RMSprop can converge with proper choice of hyper-parameters under certain conditions. More specifically, we prove that  when the hyper-parameter $\beta_2$ is large enough, the random shuffling version of RMSprop converges to a bounded region in general, and converges to a stationary point in the interpolation regime. It is worth mentioning that our results do not depend on "bounded gradient"  assumption, which is often the key assumption utilized by existing theoretical work for RMSprop. Removing this assumption allows us to establish a phase transition from divergence to non-divergence for RMSProp. 
Finally, based on our theory, we conjecture that there is a critical threshold in practice, such that RMSprop generates reasonably good results only if $\beta_2\ge {\sf {th}}$. We provide empirical evidence about such a phase transition in our numerical experiments.
Estimating Treatment Effects via Orthogonal Regularization
Authors: Tobias Hatt, Stefan Feuerriegel
Date: 2021-01-01 00:00:00
Tasks: Causal Inference, Decision Making

Decision-making often requires accurate estimation of causal effects from observational data. This is challenging as outcomes of alternative decisions are not observed and have to be estimated. Previous methods estimate outcomes based on unconfoundedness but neglect any constraints that unconfoundedness imposes on the outcomes. In this paper, we propose a novel regularization framework in which we formalize unconfoundedness as an orthogonality constraint. We provide theoretical guarantees that this yields an asymptotically normal estimator for the average causal effect. Compared to other estimators, its asymptotic variance is strictly smaller. Based on our regularization framework, we develop deep orthogonal networks for unconfounded treatments (DONUT) which learn outcomes that are orthogonal to the treatment assignment. Using a variety of benchmark datasets for causal inference, we demonstrate that DONUT outperforms the state-of-the-art substantially.
On Nondeterminism and Instability in Neural Network Optimization
Authors: Cecilia Summers, Michael J. Dinneen
Date: 2021-01-01 00:00:00
Tasks: Diversity

Optimization nondeterminism causes uncertainty when improving neural networks, with small changes in performance difficult to discern from run-to-run variability. While uncertainty can be reduced by training multiple copies of a model with different random seeds, doing so is time-consuming, costly, and makes reproducibility challenging. Despite this, little attention has been paid towards establishing an understanding of this problem. In this work, we establish an experimental protocol for understanding the effect of optimization nondeterminism on model diversity, which allows us to study the independent effects of a variety of sources of nondeterminism. Surprisingly, we find that each source of nondeterminism all have similar effects on multiple measures of model diversity. To explain this intriguing fact, we examine and identify the instability of model training, when taken as an end-to-end procedure, as the key determinant. We show that even one-bit changes in initial model parameters result in models that converge to vastly different values. Last, we demonstrate that recent methods in accelerated model ensembling hold promise for reducing the effects of instability on run-to-run variability.
Deep Q-Learning with Low Switching Cost
Authors: Shusheng Xu, Simon Shaolei Du, Yi Wu
Date: 2021-01-01 00:00:00
Tasks: Atari Games, Deep Reinforcement Learning, Q-Learning, Recommendation Systems, Representation Learning

We initiate the study on deep reinforcement learning problems that require low switching cost, i.e., small number of policy switches during training.  Such a requirement is ubiquitous in many applications, such as medical domains, recommendation systems, education, robotics, dialogue agents, etc, where the deployed policy that actually interacts with the environment cannot change frequently. Our paper investigates different policy switching criteria based on deep Q-networks and further proposes an adaptive approach based on the feature distance between the  deployed  Q-network  and  the  underlying  learning  Q-network.   Through  extensive experiments on a medical treatment environment and a collection of the Atari games, we find our feature-switching criterion substantially decreases the switching cost while maintains a similar sample efficiency to the case without the low-switching-cost constraint.  We also complement this empirical finding with a theoretical justification from a representation learning perspective.
Constraint-Driven Explanations of Black-Box ML Models
Authors: Aditya Aniruddha Shrotri, Nina Narodytska, Alexey Ignatiev, Joao Marques-Silva, Kuldeep S. Meel, Moshe Vardi
Date: 2021-01-01 00:00:00
Tasks: Decision Making

Modern machine learning techniques have enjoyed widespread success, but are plagued by lack of transparency in their decision making, which has led to the emergence of the field of explainable AI.  One popular approach called LIME, seeks to explain an opaque model's behavior, by training a surrogate interpretable model to be locally faithful on perturbed instances. 
Despite being model-agnostic and easy-to-use, it is known that LIME's explanations can be unstable and are susceptible to adversarial attacks as a result of Out-Of-Distribution (OOD) sampling. Quality of explanations is also calculated heuristically, and lacks a strong theoretical foundation. In spite of numerous attempts to remedy some of these issues, making the LIME framework more trustworthy and reliable remains an open problem.

In this work, we demonstrate that the OOD sampling problem stems  from rigidity of the perturbation procedure.  To resolve this issue, we propose a theoretically sound framework based on uniform sampling of user-defined subspaces. Through logical constraints, we afford the end-user the flexibility to delineate the precise subspace of the input domain to be explained. This not only helps mitigate the problem of OOD sampling, but also allow experts to drill down and uncover bugs deep inside the model. For testing the quality of generated explanations, we develop a novel estimation algorithm that is able to certifiably measure the true value of metrics such as fidelity up to any desired degree of accuracy, which can help in building trust in the generated explanations. Our framework called CLIME can be applied to any ML model, and extensive experiments demonstrate its versatility on real-world problems.
Learning to Make Decisions via Submodular Regularization
Authors: Ayya Alieva, Aiden Aceves, Jialin Song, Stephen Mayo, Yisong Yue, Yuxin Chen
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Active Learning, Bayesian Optimization, Combinatorial Optimization, Decision Making, Imitation Learning, Sequential Decision Making

Many sequential decision making tasks can be viewed as combinatorial optimization problems over a large number of actions. When the cost of evaluating an action is high, even a greedy algorithm, which iteratively picks the best action given the history, is prohibitive to run. In this paper, we aim to learn a greedy heuristic for sequentially selecting actions as a surrogate for invoking the expensive oracle when evaluating an action. In particular, we focus on a class of combinatorial problems that can be solved via submodular maximization (either directly on the objective function or via submodular surrogates). We introduce a data-driven optimization framework based on the submodular-norm loss, a novel loss function that encourages the resulting objective to exhibit diminishing returns. Our framework outputs a surrogate objective that is efficient to train, approximately submodular, and can be made permutation-invariant. The latter two properties allow us to prove strong approximation guarantees for the learned greedy heuristic. Furthermore, we show that our model can be easily integrated with modern deep imitation learning pipelines for sequential prediction tasks. We demonstrate the performance of our algorithm on a variety of batched and sequential optimization tasks, including set cover, active learning, and Bayesian optimization for protein engineering.
The large learning rate phase of deep learning
Authors: Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, Guy Gur-Ari
Date: 2021-01-01 00:00:00
Tasks: Deep Learning

The choice of initial learning rate can have a profound effect on the performance of deep networks. We present empirical evidence that networks exhibit sharply distinct behaviors at small and large learning rates. In the small learning rate phase, training can be understood using the existing theory of infinitely wide neural networks. At large learning rates, we find that networks exhibit qualitatively distinct phenomena that cannot be explained by existing theory: The loss grows during the early part of training, and optimization eventually converges to a flatter minimum. Furthermore, we find that the optimal performance is often found in the large learning rate phase. To better understand this behavior we analyze the dynamics of a two-layer linear network and prove that it exhibits these different phases. We find good agreement between our analysis and the training dynamics observed in realistic deep learning settings.
SEQUENCE-LEVEL FEATURES: HOW GRU AND LSTM CELLS CAPTURE N-GRAMS
Authors: Xiaobing Sun, Wei Lu
Date: 2021-01-01 00:00:00
Tasks: Language Modeling, Language Modelling, Rolling Shutter Correction, Sentiment Analysis

Modern recurrent neural networks (RNN) such as Gated Recurrent Units (GRU) and Long Short-term Memory (LSTM) have demonstrated impressive results on tasks involving sequential data in practice. Despite continuous efforts on interpreting their behaviors, the exact mechanism underlying their successes in capturing sequence-level information have not been thoroughly understood.  In this work, we present a study on understanding the essential features captured by GRU/LSTM cells by mathematically expanding and unrolling the hidden states. Based on the closed-form approximations of the hidden states, we argue that the effectiveness of the cells may be attributed to a type of sequence-level representations brought in by the gating mechanism, which enables the cells to encode sequence-level features along with token-level features. Specifically, we show that under certain mild assumptions, the essential components of the cells would consist of such sequence-level features similar to those of N-grams. Based on such a finding, we also found that replacing the standard cells with approximate hidden state representations does not necessarily degrade performance on the sentiment analysis and language modeling tasks, indicating such features may play a significant role for GRU/LSTM cells. We hope that our work can give inspiration on proposing new neural architectures for capturing contextual information within sequences.
UserBERT: Self-supervised User Representation Learning
Authors: Tianyu Li, Ali Cevahir, Derek Cho, Hao Gong, DuyKhuong Nguyen, Bjorn Stenger
Date: 2021-01-01 00:00:00
Tasks: Multi-Task Learning, Representation Learning

This paper extends the BERT model to user data for pretraining user representations in a self-supervised way. By viewing actions (e.g., purchases and clicks) in behavior sequences (i.e., usage history) in an analogous way to words in sentences, we propose methods for the tokenization, the generation of input representation vectors and a novel pretext task to enable the pretraining model to learn from its own input, omitting the burden of collecting additional data. Further, our model adopts a unified structure to simultaneously learn from long-term and short-term user behavior as well as user profiles. Extensive experiments demonstrate that the learned representations  result in significant improvements when transferred to three different real-world tasks, particularly in comparison with task-specific modeling and representations obtained from multi-task learning.
Measuring GAN Training in Real Time
Authors: Yuexiang Zhai, Bai Jiang, Yi Ma, Hao Chen
Date: 2021-01-01 00:00:00
Tasks: Diversity

Generative Adversarial Networks (GAN) are popular generative models of images. Although researchers proposed variants of GAN for different applications, evaluating and comparing GANs is still challenging as GANs may have many failure cases such as low visual quality and model collapse. To alleviate this issue, we propose a novel framework to evaluate the training stability (S), visual quality (Q), and mode diversity (D) of GAN simultaneously. SQD requires only a moderate number of samples, allowing real-time monitoring of the training dynamics of GAN. We showcase the utility of the SQD framework on prevalent GANs and discovered that the gradient penalty (Gulrajani et al., 2017) regularization significantly improves the performance of GAN. We also compare the gradient penalty regularization with other regularization methods and reveal that enforcing the 1-Lipschitz condition of the discriminator network stabilizes GAN training.
Learning to Use Future Information in Simultaneous Translation
Authors: Xueqing Wu, Yingce Xia, Lijun Wu, Shufang Xie, Weiqing Liu, Tao Qin, Tie-Yan Liu
Date: 2021-01-01 00:00:00
Tasks: Decoder, Machine Translation, NMT, Sentence, Translation

Simultaneous neural machine translation (briefly, NMT) has attracted much attention recently. In contrast to standard NMT, where the NMT system can access the full input sentence, simultaneous NMT is a prefix-to-prefix problem, where the system can only utilize the prefix of the input sentence and thus more uncertainty and difficulty are introduced to decoding. Wait-k inference is a simple yet effective strategy for simultaneous NMT, where the decoder generates the output sequence $k$ words behind the input words. For wait-k inference, we observe that wait-m training with $m>k$ in simultaneous NMT (i.e., using more future information for training than inference) generally outperforms wait-k training. Based on this observation, we propose a method that automatically learns how much future information to use in training for simultaneous NMT. Specifically, we introduce a controller to adaptively select wait-m training strategies according to the network status of the translation model and current training sentence pairs, and the controller is jointly trained with the translation model through bi-level optimization. Experiments on four datasets show that our method brings 1 to 3 BLEU point improvement over baselines under the same latency. Our code is available at https://github.com/P2F-research/simulNMT .
Understanding Bias in Anomaly Detection: A Semi-Supervised View with PAC Guarantees
Authors: Ziyu Ye, Yuxin Chen, Haitao Zheng
Date: 2021-01-01 00:00:00
Tasks: Anomaly Detection, Semi-supervised Anomaly Detection, Supervised Anomaly Detection, Unsupervised Anomaly Detection

Anomaly detection presents a unique challenge in machine learning, due to the scarcity of labeled anomaly data. Existing work attempts to mitigate such problems via semi-supervised learning, i.e., augmenting unsupervised anomaly detection models with additional labeled anomaly samples. However, the labeled data often does not align with the target distribution and therefore introduces harmful bias to the trained model. In this paper, we aim to understand the effect of a biased anomaly set for anomaly detection. In particular, we formally state the anomaly detection problem as a semi-supervised learning task. We focus on the anomaly detector's recall at a given false positive rate as the main performance metric. Given two different anomaly score  functions, we formally define their difference in performance as the relative scoring bias of the anomaly detectors. We establish the first finite sample rates for estimating the relative scoring bias for semi-supervised anomaly detection. We then empirically validate our theoretical results on both synthetic and real-world datasets. Furthermore, we provide extensive empirical study on how a biased training anomaly set affects the anomaly score function and therefore the resulting detection performance. Our case study demonstrates scenarios in which the biased anomaly set can be useful, and provides a solid benchmark for future research.