pwc-archive/papers-with-abstracts huggingface dataset:
Learning Blood Oxygen from Respiration Signals
Authors: Hao He, Ying-Cong Chen, Yuan Yuan, Dina Katabi
Date: 2021-01-01 00:00:00

Monitoring blood oxygen is critical in a variety of medical conditions including COVID-19. For almost a century, pulse oximetry has been the only noninvasive method for measuring blood oxygen. While highly useful, pulse oximetry has important limitations. It requires wearable sensors, which can be cumbersome for older patients. It is also known to be biased when used for dark-skinned subjects. In this paper, we demonstrate, for the first time, the feasibility of predicting oxygen saturation from breathing. By eliminating dependency on oximetry, we eliminate bias against skin color. Further, since breathing can be monitored without body contact by analyzing the radio signal in the environment, we show that oxygen too can be monitored without any wearable devices. We introduce a new approach for leveraging auxiliary variables and tasks via a multi-headed neural network model. Empirical results show that our model achieves good accuracy on existing medical datasets and can potentially be used to monitor the recovery of COVID-19 patients.
MSFM: Multi-Scale Fusion Module for Object Detection
Authors: Xuesong Wang, Caisheng Wang
Date: 2021-01-01 00:00:00
Tasks: Object, object-detection, Object Detection, Position

Feature fusion is beneficial to object detection tasks in two folds. On one hand, detail and position information can be combined with semantic information when high and low-resolution features from shallow and deep layers are fused. On the other hand, objects can be detected in different scales, which improves the robustness of the framework. In this work, we present a Multi-Scale Fusion Module (MSFM) that extracts both detail and semantical information from a single input but at different scales within the same layer. Specifically, the input of the module will be resized into different scales on which position and semantic information will be processed, and then they will be rescaled back and combined with the module input. The MSFM is lightweight and can be used as a drop-in layer to many existing object detection frameworks. Experiments show that MSFM can bring +2.5% mAP improvement with only 2.4M extra parameters on Faster R-CNN with ResNet-50 FPN backbone on COCO Object Detection minival set, outperforming that with ResNet-101 FPN backbone without the module which obtains +2.0% mAP with 19.0M extra parameters. The best resulting model achieves a 45.7% mAP on test-dev set. Code will be available.
Feedforward Legendre Memory Unit
Authors: Narsimha Reddy Chilkuri, Chris Eliasmith
Date: 2021-01-01 00:00:00
Tasks: image-classification, Image Classification, Natural Language Inference, QQP, Semantic Similarity, Semantic Textual Similarity, Sentiment Analysis, Sentiment Classification

Recently, a new recurrent neural network (RNN) named the Legendre Memory Unit (LMU) was proposed and shown to achieve state-of-the-art performance on psMNIST and other datasets. Here we consider a modified version of the LMU, named ff-LMU, the core of which is a linear time-invariant (LTI) system.  We first show that the ff-LMU can be trained in a purely feedforward manner and yet executed during inference in a recurrent fashion. Specifically we demonstrate that it trains about 80x faster than LSTM models of the same size.  As a result, it overcomes the well-known limitations of training RNNs on GPUs that make them less scalable than feedforward networks like transformers. Second, to validate its utility, we compare ff-LMU performance against LSTMs on five benchmarks picked from the following categories: sentiment classification, semantic similarity, natural language inference, and image classification. Our models, despite their simplicity, achieve new state-of-the-art results for RNNs on psMNIST and QQP, and exhibit superior performance on the remaining three datasets while using up to 1000x fewer parameters. In general, ff-LMU models are highly parameter efficient. For instance, the first model that beats it on current leaderboards for QQP is a transformer that uses 50,000x more parameters.
VortexNet: Learning Complex Dynamic Systems with Physics-Embedded Networks
Authors: Shiying Xiong, Xingzhe He, Yunjin Tong, Yitong Deng, Bo Zhu
Date: 2021-01-01 00:00:00

In this paper, we present a novel physics-rooted network structure that dramatically facilitates the learning of complex dynamic systems. Our method is inspired by the Vortex Method in fluid dynamics, whose key idea lies in that, given the observed flow field, instead of describing it with a function of space and time, one can equivalently understand the observation as being caused by a number of Lagrangian particles ----- vortices, flowing with the field.  Since the number of such vortices are much smaller than that of the Eulerian, grid discretization, this Lagrangian discretization in essence encodes the system dynamics on a compact physics-based latent space. Our method enforces such Lagrangian discretization with a Encoder---Dynamics---Decode network structure, and trains it with a novel three-stage curriculum learning algorithm. With data generated from the high precision Eulerian DNS method, our alorithm takes advantage of the simplifying power of the Lagrangian method while persisting the physical integrity.
This method fundamentally differs from the current approaches in the field of physics-informed learning, and provides superior results for being more versatile, yielding more physical-correctness with less data sample, and faster to compute at high precision. Beyond providing a viable way of simulating complex fluid at high-precision, our method opens up a brand new horizon for embedding knowledge prior via constructing physically-valid latent spaces, which can be applied to further research areas beyond physical simulation.
Deep Representational Re-tuning using Contrastive Tension
Authors: Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, Magnus Sahlgren
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Semantic Similarity, Semantic Textual Similarity, Sentence, Sentence Embedding, Sentence Embeddings, STS

Extracting semantically useful natural language sentence representations from pre-trained deep neural networks such as Transformers remains a challenge. We first demonstrate that pre-training objectives impose a significant task bias onto the final layers of models with a layer-wise survey of the Semantic Textual Similarity (STS) correlations for multiple common Transformer language models. We then propose a new self-supervised method called Contrastive Tension (CT) to counter such biases. CT frames the training objective as a noise-contrastive task between the final layer representations of two independent models, in turn making the final layer representations suitable for feature extraction. Results from multiple common unsupervised and supervised STS tasks indicate that CT outperforms previous State Of The Art (SOTA), and when combining CT with supervised data we improve upon previous SOTA results with large margins.
Topic-aware Contextualized Transformers
Authors: Ruiying Lu, Bo Chen, Dan dan Guo, Dongsheng Wang, Mingyuan Zhou
Date: 2021-01-01 00:00:00
Tasks: Word Embeddings

Training on disjoint fixed-length segments, Transformers successfully transform static word embeddings into contextualized word representations. However, they often restrict the context of a token to the segment it resides in and hence neglect the flow of contextual information across segments, failing to capture longer-term dependencies beyond the predefined segment length. This paper uses a probabilistic deep topic model to provide contextualized embeddings at both the token and segment levels. It also introduces topic self-attention and a contextual next-word embedding guided topic select-attention, injecting contextualized topic information into Transformer-based architectures. Moving beyond conventional Transformers that ignore longer-range word dependencies and contextualize their word representations at the segment level, the proposed method not only captures global semantic coherence of all segments and global word concurrence patterns, but also enriches the representation of each token by adapting it to its local context, which is not limited to the segment it resides in and can be flexibly defined according to the task. Experiments on various corpora show that adding only a few extra parameters, the proposed topic-aware contextualized transformers consistently outperform their conventional counterparts, and can be used to generate coherent sentences and paragraphs.
Cluster-Former: Clustering-based Sparse Transformer for Question Answering
Authors: Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng, Jingjing Liu
Date: 2021-01-01 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Clustering, Question Answering

Transformer has become ubiquitous in the deep learning field. One of the key ingredients that destined its success is the self-attention mechanism, which allows fully-connected contextual encoding over input tokens. 
However, despite its effectiveness in modeling short sequences, self-attention suffers when handling inputs with extreme long-range dependencies, as its complexity grows quadratically with respect to the sequence length.
Therefore, long sequences are often encoded by Transformer in chunks using a sliding window.
In this paper, we propose Cluster-Former, a novel clustering-based sparse Transformer to perform attention across chunked sequences. The proposed framework is pivoted on two unique types of Transformer layer: Sliding-Window Layer and Cluster-Former Layer, which encode local sequence information and global context jointly and iteratively.
This new design allows information integration beyond local windows, which is especially beneficial for question answering (QA) tasks that rely on long-range dependencies. Experiments show that Cluster-Former achieves state-of-the-art performance on several major QA benchmarks.
Dataset Meta-Learning from Kernel-Ridge Regression
Authors: Timothy Nguyen, Zhourong Chen, Jaehoon Lee
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Dataset Distillation, Meta-Learning, regression

One of the most fundamental aspects of any machine learning algorithm is the training data used by the algorithm. 
We introduce the novel concept of $\epsilon$-approximation of datasets, obtaining datasets which are much smaller than or are significant corruptions of the original training data while maintaining similar performance. We introduce a meta-learning algorithm Kernel Inducing Points (KIP) for obtaining such remarkable datasets, drawing inspiration from recent developments in the correspondence between infinitely-wide neural networks and kernel ridge-regression (KRR). For KRR tasks, we demonstrate that KIP can compress datasets by one or two orders of magnitude, significantly improving previous dataset distillation and subset selection methods while obtaining state of the art results for MNIST and CIFAR10 classification. Furthermore, our KIP-learned datasets are transferable to the training of finite-width neural networks even beyond the lazy-training regime. Consequently, we obtain state of the art results for neural network dataset distillation with potential applications to privacy-preservation.
A Large-scale Study on Training Sample Memorization in Generative Modeling
Authors: Ching-Yuan Bai, Hsuan-Tien Lin, Colin Raffel, Wendy Kan
Date: 2021-01-01 00:00:00
Tasks: Benchmarking, Memorization

Many recent developments on generative models for natural images have relied on heuristically-motivated metrics that can be easily gamed by memorizing a small sample from the true distribution or training a model directly to improve the metric.
In this work, we critically evaluate the gameability of the benchmarking procedure by running a competition which ultimately resulted in participants attempting to cheat. Our competition received over 11000 submitted models which allowed us to investigate memorization-aware metrics for measuring generative model performance. Specifically, we propose the Memorization-Informed Frechet Inception Distance (MiFID) and discuss ways to ensure that winning submissions were based on genuine improvements in perceptual quality. We evaluate the effectiveness of our benchmark by manually inspecting the code for the 1000 top-performing models and labeling different forms of memorization that were intentionally or unintentionally used. To facilitate future work on benchmarking generative models, we release generated images and our labels for these models as well as code to compute the MiFID metric.
Meta Attention Networks: Meta-Learning Attention to Modulate Information Between Recurrent Independent Mechanisms
Authors: Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, Yoshua Bengio
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Meta-Learning

Decomposing knowledge into interchangeable pieces promises a generalization advantage when there are changes in distribution. A learning agent interacting with the environment is likely to be faced with situations requiring novel combinations of existing pieces of knowledge. We hypothesize that such a decomposition of knowledge is particularly relevant for being able to generalize in a systematic way to out of distribution changes. To study these ideas, we propose a particular training framework in which we assume that the pieces of knowledge an agent needs, as well as its reward function are stationary and can be re-used across tasks. The attention mechanisms dynamically select which modules should be adapted, and the parameters of the \textit{selected} modules are changed quickly as a learner is confronted with variations in what it experiences, while the parameters of the attention mechanisms act as slowly changing meta-parameters. We focus on pieces of knowledge captured by an ensemble of  modules sparsely communicating with each other via a bottleneck of attention. We find that meta-learning the  modular aspects of the proposed system greatly help in achieving faster learning, in experiments with a reinforcement learning setup involving navigation in a partially observed grid world with image-level input.  We also find that reversing the role of parameters and meta-parameters does not work nearly as well, suggesting a particular role for fast adaptation of the dynamically selected modules.
Symmetry-Augmented Representation for Time Series
Authors: Amine Mohamed Aboussalah, Chi-Guhn Lee
Date: 2021-01-01 00:00:00
Tasks: Management, Time Series, Time Series Analysis

We examine the hypothesis that the concept of symmetry augmentation is fundamentally linked to learning. Our focus in this study is on the augmentation of symmetry embedded in 1-dimensional time series (1D-TS). Motivated by the duality between 1D-TS and networks, we augment the symmetry by converting 1D-TS into three 2-dimensional representations: temporal correlation (GAF), transition dynamics (MTF), and recurrent events (RP). This conversion does not require a priori knowledge of the types of symmetries hidden in the 1D-TS. We then exploit the equivariance property of CNNs to learn the hidden symmetries in the augmented 2-dimensional data. We show that such conversion only increases the amount of symmetry, which may lead to more efficient learning. Specifically, we prove that a direct sum based augmentation will never decrease the amount of symmetry. We also attempt to measure the amount of symmetry in the original 1D-TS and augmented representations using the notion of persistent homology, which reveals symmetry increase after augmentation quantitatively. We present empirical studies to confirm our findings using two cases: reinforcement learning for financial portfolio management and classification with the CBF data set. Both cases demonstrate significant improvements in learning efficiency.
Provable Robustness by Geometric Regularization of ReLU Networks
Authors: Chester Holtz, Changhao Shi, Gal Mishne
Date: 2021-01-01 00:00:00

Recent work has demonstrated that neural networks are vulnerable to small, adversarial perturbations of their input. In this paper, we propose an efficient regularization scheme inspired by convex geometry and barrier methods to improve the robustness of feedforward ReLU networks. Since such networks are piecewise linear, they partition the input space into polyhedral regions (polytopes). Our regularizer is designed to minimize the distance between training samples and the \textit{analytical centers} of their respective polytopes so as to push points away from the boundaries. Our regularizer \textit{provably} improves a lower bound on the necessary adversarial perturbation required to switch an example's label. The addition of a second regularizer that encourages linear decision boundaries improves robustness while avoiding over-regularization of the classifier. We demonstrate the robustness of our approach with respect to $\ell_\infty$ and $\ell_2$ adversarial perturbations on multiple datasets. Our method is competitive with state-of-the-art algorithms for learning robust networks. Moreover, applying our algorithm in conjunction with adversarial training boosts the robustness of classifiers even further.
Online Continual Learning Under Domain Shift
Authors: Quang Pham, Chenghao Liu, Steven Hoi
Date: 2021-01-01 00:00:00
Tasks: Continual Learning

Existing continual learning benchmarks often assume each task's training and test data are from the same distribution, which may not hold in practice. Towards making continual learning practical, in this paper, we introduce a novel setting of online continual learning under conditional domain shift, in which domain shift exists between training and test data of all tasks: $P^{tr}(X, Y) \neq P^{te}(X,Y)$, and the model is required to generalize to unseen domains at test time. To address this problem, we propose \emph{Conditional Invariant Experience Replay (CIER)} that can simultaneously retain old knowledge, acquire new information, and generalize to unseen domains. CIER employs an adversarial training to correct the shift in $P(X,Y)$ by matching $P(X|Y)$, which results in an invariant representation that can generalize to unseen domains during inference. Our extensive experiments show that CIER can bridge the domain gap in continual learning and significantly outperforms state-of-the-art methods. We will release our benchmarks and implementation upon acceptance.
Contextual Transformation Networks for Online Continual Learning
Authors: Quang Pham, Chenghao Liu, Doyen Sahoo, Steven Hoi
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Continual Learning, Transfer Learning

Continual learning methods with fixed architectures rely on a single network to learn models that can perform well on all tasks.
As a result, they often only accommodate common features of those tasks but neglect each task's specific features. On the other hand, dynamic architecture methods can have a separate network for each task, but they are too expensive to train and not scalable in practice, especially in online settings.
To address this problem, we propose a novel online continual learning method named ``Contextual Transformation Networks” (CTN) to efficiently model the \emph{task-specific features} while enjoying neglectable complexity overhead compared to other fixed architecture methods. 
Moreover, inspired by the Complementary Learning Systems (CLS) theory, we propose a novel dual memory design and an objective to train CTN that can address both catastrophic forgetting and knowledge transfer simultaneously. 
Our extensive experiments show that CTN is competitive with a large scale dynamic architecture network and consistently outperforms other fixed architecture methods under the same standard backbone. We will release our implementation upon acceptance.
Continual Lifelong Causal Effect Inference with Real World Evidence
Authors: Zhixuan Chu, Stephen Rathbun, Sheng Li
Date: 2021-01-01 00:00:00
Tasks: counterfactual, Representation Learning, Selection bias

The era of real world evidence has witnessed an increasing availability of observational data, which much facilitates the development of causal effect inference. Although significant advances have been made to overcome the challenges in causal effect estimation, such as missing counterfactual outcomes and selection bias, they only focus on source-specific and stationary observational data. In this paper, we investigate a new research problem of causal effect inference from incrementally available observational data, and present three new evaluation criteria accordingly, including extensibility, adaptability, and accessibility. We propose a Continual Causal Effect Representation Learning method for estimating causal effect with observational data, which are incrementally available from non-stationary data distributions. Instead of having access to all seen observational data, our method only stores a limited subset of feature representations learned from previous data. Combining the selective and balanced representation learning, feature representation distillation, and feature transformation, our method achieves the continual causal effect estimation for new data without compromising the estimation capability for original data. Extensive experiments demonstrate the significance of continual causal effect inference and the effectiveness of our method.
Improving Learning to Branch via Reinforcement Learning
Authors: Haoran Sun, Wenbo Chen, Hui Li, Le Song
Date: 2021-01-01 00:00:00
Proceeding: NeurIPS Workshop LMCA 2020 12
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Branch-and-Bound~(B\&B) is a general and widely used algorithm paradigm for solving Mixed Integer Programming~(MIP). 
Recently there is a surge of interest in designing learning-based branching policies as a fast approximation of strong branching, a human-designed heuristic. In this work, we argue strong branching is not a good expert to imitate for its poor decision quality when turning off its side effects in solving linear programming. To obtain more effective and non-myopic policies than a local heuristic, we formulate the branching process in MIP as reinforcement learning~(RL) and design a policy characterization for the B\&B process to improve our agent by novelty search evolutionary strategy. Across a range of NP-hard problems, our trained RL agent significantly outperforms expert-designed branching rules and the state-of-the-art learning-based branching methods in terms of both speed and effectiveness. Our results suggest that with carefully designed policy networks and learning algorithms, reinforcement learning has the potential to advance algorithms for solving MIPs.
Offline Policy Optimization with Variance Regularization
Authors: Riashat Islam, Samarth Sinha, Homanga Bharadhwaj, Samin Yeasar Arnob, Zhuoran Yang, Zhaoran Wang, Animesh Garg, Lihong Li, Doina Precup
Date: 2021-01-01 00:00:00
Tasks: continuous-control, Continuous Control, Offline RL, Reinforcement Learning (RL)

Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications. This is often because off-policy RL algorithms suffer from distributional shift, due to mismatch between dataset and the target policy, leading to high variance and over-estimation of value functions. In this work, we propose variance regularization for offline RL algorithms, using stationary distribution corrections. We show that by using Fenchel duality, we can avoid double sampling issues for computing the gradient of the variance regularizer. The proposed algorithm for offline variance regularization can be used to augment  any existing offline policy optimization algorithms. We  show that the regularizer leads to a lower bound to the offline policy optimization objective, which can help avoid over-estimation errors, and explains the benefits of our approach  across a range of continuous control domains when compared to existing algorithms.