pwc-archive/papers-with-abstracts huggingface dataset:
Proper Straight-Through Estimator: Breaking symmetry promotes convergence to true minimum
Authors: Shinya Gongyo, Kohta Ishikawa
Date: 2021-09-29 00:00:00

In the quantized network, its gradient shows either vanishing or diverging. The network thus cannot be learned by the standard back-propagation, so that an alternative approach called Straight Through Estimator (STE), which replaces the part of the gradient with a simple differentiable function, is used. While STE is known to work well for learning the quantized network empirically, it has not been established theoretically. A recent study by Yin et. al. (2019) has provided theoretical support for STE. However, its justification is still limited to the model in the one-hidden layer network with the binary activation where  Gaussian generates the input data, and the true labels are output from the teacher network with the same binary network architecture. In this paper, we discuss the effectiveness of STEs in more general situations without assuming the shape of the input distribution and the labels. By considering the scale symmetry of the network and specific properties of the STEs, we find that STE with clipped Relu is superior to STEs with identity function and vanilla Relu. The clipped Relu STE, which breaks the scale symmetry, may pick up one of the local minima degenerated in scales, while the identity STE and vanilla Relu STE, which keep the scale symmetry, may not pick it up. To confirm this observation, we further present an analysis of a simple misspecified model as an example. We find that all the stationary points are identical with the vanishing points of the cRelu STE gradient, while some of them are not identical with the vanishing points of the identity and Relu STE.
FLAME-in-NeRF: Neural control of Radiance Fields for Free View Face Animation
Authors: ShahRukh Athar, Zhixin Shu, Dimitris Samaras
Date: 2021-09-29 00:00:00
Tasks: NeRF, Neural Rendering, Novel View Synthesis

This paper presents a neural rendering method for controllable portrait video synthesis.Recent advances in volumetric neural rendering, such as neural radiance fields (NeRF), have enabled the photorealistic novel view synthesis of static scenes with impressive results. However, modeling dynamic and controllable objects as part of a scene with such scene representations is still challenging. 
In this work, we design a system that enables 1) novel view synthesis for portrait video, of both the human subject and the scene they are in and 2) explicit control of the facial expressions through a low-dimensional expression representation. 
We represent the distribution of human facial expressions using the expression parameters of a 3D Morphable Model (3DMMs) and condition the NeRF volumetric function on them. 
Furthermore, we impose a spatial prior, brought by 3DMM fitting,  to guide the network to learn disentangled control for static scene appearance and dynamic facial actions. We show the effectiveness of our method on free view synthesis of portrait videos with expression controls. To train a scene, our method only requires a short video of a subject captured by a mobile device.
A Simple and Debiased Sampling Method for Personalized Ranking
Authors: Lu Yu, Shichao Pei, Chuxu Zhang, Xiangliang Zhang
Date: 2021-09-29 00:00:00

Pairwise ranking models have been widely used to address  various problems, such as recommendation. The basic idea is to learn the rank of users' preferred items through  separating items into positive samples if user-item interactions exist, and negative samples otherwise. Due to the limited number of observed interactions, pairwise ranking models face  serious class-imbalance issue. Our theoretical analysis shows that current sampling-based methods cause the vertex-level imbalance problem, which makes the norm of  learned item embeddings towards infinite after a certain training iterations, and consequently results in vanishing gradient and affects the model performance. To this end, we propose VINS, an efficient \emph{\underline{Vi}tal \underline{N}egative \underline{S}ampler}, to alleviate the class-imbalance issue for pairwise ranking models optimized by gradient methods. The core of VINS is a bias sampler with reject probability that will tend to accept a negative candidate with a larger popularity than the given positive item. Evaluation results on several real datasets demonstrate that the proposed sampling method speeds up the training procedure 30\% to 50\% for ranking models ranging from shallow to deep, while maintaining and even improving the quality of ranking results in top-N item recommendation.
Value-aware transformers for 1.5d data
Authors: James F Cann, Timothy J Roberts, Amy R Tso, Amy Nelson, Parashkev Nachev
Date: 2021-09-29 00:00:00
Tasks: Form, Length-of-Stay prediction, Representation Learning

Sparse sequential highly-multivariate data of the form characteristic of hospital in-patient investigation and treatment poses a considerable challenge for representation learning. Such data is neither faithfully reducible to 1d nor dense enough to constitute multivariate series. Conventional models compromise their data by requiring these forms at the point of input. Building on contemporary sequence-modelling architectures we design a value-aware transformer, prompting a reconceptualisation of our data as 1.5-dimensional: a token-value form both respecting its sequential nature and augmenting it with a quantifier. Experiments focused on sequential in-patient laboratory data up to 48hrs after hospital admission show that the value-aware transformer performs favourably versus competitive baselines on in-hospital mortality and length-of-stay prediction within the MIMIC-III dataset.
Vicinal Counting Networks
Authors: Viresh Ranjan, Minh Hoai
Date: 2021-09-29 00:00:00
Tasks: Crowd Counting, Data Augmentation

We tackle the task of Few-Shot Counting. Given an image containing multiple objects of a novel visual category and few exemplar bounding boxes depicting the visual category of interest, we want to count all of the instances of the desired visual category in the image. A key challenge in building an accurate few-shot visual counter is the scarcity of annotated training data due to the laborious effort needed for collecting and annotating the data. To address this challenge, we propose Vicinal Counting Networks, which learn to augment the existing training data along with learning to count. A Vicinal Counting Network consists of a generator and a counting network. The generator takes as input an image along with a random noise vector and generates an augmented version of the input image. The counting network learns to count the objects in the original and augmented images. The training signal for the generator comes from the counting loss of the counting network, and the generator aims to synthesize images which result in a small counting loss. Unlike GANs which are trained in an adversarial setting, Vicinal Counting Networks are trained in a cooperative setting where the generator aims to help the counting network in achieving accurate predictions on the synthesized images. We also show that our proposed data augmentation framework can be extended to other counting tasks like crowd counting. Our code and trained model will be released for research usage.
Spatiotemporal Representation Learning on Time Series with Dynamic Graph ODEs
Authors: Ming Jin, Yuan-Fang Li, Yu Zheng, Bin Yang, Shirui Pan
Date: 2021-09-29 00:00:00
Tasks: Graph structure learning, Representation Learning, Time Series, Time Series Analysis

Spatiotemporal representation learning on multivariate time series has received tremendous attention in forecasting traffic and energy data. Recent works either rely on complicated discrete neural architectures or graph priors, hindering their effectiveness and applications in the real world. In this paper, inspired by neural ordinary differential equations and graph structure learning, we propose a fully continuous model named Dynamic Graph ODE (DyG-ODE) to capture both long-range spatial and temporal dependencies to learn expressive representations on arbitrary multivariate time series data without being restricted by rigid preconditions (e.g., graph priors). For modeling the continuous dynamics of spatiotemporal clues, we design a simple yet powerful dynamic graph ODE by coupling the proposed spatial and temporal ODEs, which not only allows the model to obtain infinite spatial and temporal receptive fields but also reduces numerical errors and model complexity significantly. Our empirical evaluations demonstrate the superior effectiveness and efficiency of DyG-ODE on a number of benchmark datasets.
EfficientPhys: Enabling Simple, Fast, and Accurate Camera-Based Vitals Measurement
Authors: Xin Liu, Brian L. Hill, Ziheng Jiang, Shwetak Patel, Daniel McDuff
Date: 2021-09-29 00:00:00
Tasks: Face Detection

Camera-based physiological measurement is a growing field with neural models providing state-the-art-performance. Prior research have explored various "end-to-end'' models; however these methods still require several preprocessing steps. These additional operations are often non-trivial to implement making replication and deployment difficult and can even have a higher computational budget than the "core'' network itself. In this paper, we propose two novel and efficient neural models for camera-based physiological measurement called EfficientPhys that remove the need for face detection, segmentation, normalization, color space transformation or any other preprocessing steps. Using an input of raw video frames, our models achieve state-of-the-art accuracy on three public datasets. We show that this is the case whether using a transformer or convolutional backbone. We further evaluate the latency of the proposed networks and show that our most light weight network also achieves a 33\% improvement in efficiency.
Online graph nets
Authors: Hojin Kang, Jou-Hui Ho, Diego Mesquita, Jorge Pérez, Amauri H Souza
Date: 2021-09-29 00:00:00
Tasks: Link Prediction, Prediction

Temporal graph neural networks (T-GNNs) sequentially update node states and use temporal message passing to predict events in continuous-time dynamic graphs. While node states rest in the memory, the message-passing operations must be computed on-demand for each prediction. In practice, these operations are the computational bottleneck of state-of-the-art T-GNNs as they require topologically exploring large temporal graphs. To circumvent this caveat, we propose Online Graph Nets (OGNs). To avoid temporal message passing, OGN maintains a summary of the temporal neighbors of each node in a latent variable and updates it as events unroll, in an online fashion. At prediction time, OGN simply combines node states and their latents to obtain node-level representations. Consequently, the memory cost of OGN is constant with respect to the number of previous events. Remarkably, OGN outperforms most existing T-GNNs on temporal link prediction benchmarks while running orders of magnitude faster. For instance, OGN performs similarly to the best-known T-GNN on Reddit, with a $374\times$ speedup. Also, since OGNs do not explore temporal graphs at prediction time, they are well-suited for on-device predictions (e.g., on mobile phones).
On the interventional consistency of autoencoders
Authors: Giulia Lanzillotta, Felix Leeb, Stefan Bauer, Bernhard Schölkopf
Date: 2021-09-29 00:00:00
Tasks: Disentanglement, Representation Learning

Autoencoders have played a crucial role in the field of representation learning since its inception, proving to be a flexible learning scheme able to accommodate various notions of optimality of the representation. The now established idea of disentanglement and the recently popular perspective of causality in representation learning identify modularity and robustness to be essential characteristics of the optimal representation. In this work, we show that the current conceptual tools available to assess the quality of the representation against these criteria (e.g. latent traversals or disentanglement metrics)  are inadequate. In this regard, we introduce the notion of \emph{interventional consistency} of a representation and argue that it is a desirable property of any disentangled representation. We develop a general training scheme for autoencoders that takes into account interventional consistency in the optimality condition. We present empirical evidence toward the validity of the approach on three different autoencoders, namely standard autoencoders (AE), variational autoencoders (VAE) and structural autoencoders (SAE).
Another key finding in this work is that differentiating between information and structure in the latent space of autoencoders can increase the modularity and interpretability of the resulting representation.
Mitigating Dataset Bias Using Per-Sample Gradients From A Biased Classifier
Authors: Sumyeong Ahn, Se-Young Yun
Date: 2021-09-29 00:00:00

The performance of deep neural networks (DNNs) primarily depends on the configuration of the training set. Specifically, biased training sets can make the trained model have unintended prejudice, which causes severe errors in the inference. Although several studies have addressed biased training using human supervision, few studies have been conducted without human knowledge because biased information cannot be easily extracted without human involvement. This study proposes a simple method to remove prejudice from a biased model without additional information and reconstruct a balanced training set based on the biased training set. The novel training method consists of three steps: (1) training biased DNNs, (2) measuring the contribution to the prejudicial training and generating balanced data batches to prevent the prejudice, (3) training de-biased DNNs with the balanced data. We test the training method based on various synthetic and real-world biased sets and discuss how gradients can efficiently detect minority samples. The experiment demonstrates that the detection method based on the gradients helps erase prejudice, resulting in improved inference accuracy by up to 19.58\% compared to the other state-of-the-art algorithm.
An evaluation of quality and robustness of smoothed explanations
Authors: Ahmad Ajalloeian, Seyed-Mohsen Moosavi-Dezfooli, Michalis Vlachos, Pascal Frossard
Date: 2021-09-29 00:00:00

Explanation methods play a crucial role in helping to understand the decisions of deep neural networks (DNNs) to develop trust that is critical for the adoption of predictive models. However, explanation methods are easily manipulated through visually imperceptible perturbations that generate misleading explanations. The geometry of the decision surface of the DNNs has been identified as the main cause of this phenomenon and several \emph{smoothing} approaches have been proposed to build more robust explanations.
In this work, we provide a thorough evaluation of the quality and robustness of the explanations derived by smoothing approaches. Their different properties are evaluated with extensive experiments, which reveal the settings where the smoothed explanations are better, and also worse than the explanations derived by the common Gradient method. By making the connection with the literature on adversarial attacks, we further show that such smoothed explanations are robust primarily against additive $\ell_p$-norm attacks. However, a combination of additive and non-additive attacks can still manipulate these explanations, which reveals shortcomings in their robustness properties.
Identifying Interactions among Categorical Predictors with Monte-Carlo Tree Search
Authors: Tan Zhu, Fei Do, Chloe Becquey, Jinbo Bi
Date: 2021-09-29 00:00:00

Identifying interpretable interactions among categorical predictors for predictive modeling is crucial in various research fields. Recent studies have examined interpretable interactions using decision tree (DT) learning methods, which construct DTs by greedy rules due to the high memory and time complexity of building and evaluating DTs, resulting in a local optimal solution. This paper formulates the selection of quadratic and higher order interactive terms into a LASSO problem and then relaxes it into multiple DT learning problems. A Monte Carlo Tree Search-based interaction selection (MCTs-IS) method is proposed to identify the optimal DT in an online learning manner. A DT pruning strategy is developed based on LASSO that can easily be applied to MCTs. We prove that  MCTs-IS converges with high probability to the optimal solution of the DT learning problem. Extensive experiments have been conducted to demonstrate the effectiveness of the proposed algorithm on real-world datasets.
Transferable Visual Control Policies Through Robot-Awareness
Authors: Edward S. Hu, Kun Huang, Oleh Rybkin, Dinesh Jayaraman
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4

Training visual control policies from scratch on a new robot typically requires generating large amounts of robot-specific data. Could we leverage data previously collected on another robot to reduce or even completely remove this need for robot-specific data? We propose a “robot-aware control” paradigm that achieves this by exploiting readily available knowledge about the robot. We then instantiate this in a robot-aware model-based RL policy by training modular dynamics models that couple a transferable, robot-agnostic world dynamics module with a robot- specific, potentially analytical, robot dynamics module. This also enables us to set up visual planning costs that separately consider the robot agent and the world. Our experiments on tabletop manipulation tasks with simulated and real robots demonstrate that these plug-in improvements dramatically boost the transferability of visual model-based RL policies, even permitting zero-shot transfer onto new robots for the very first time. Project website: https://sites.google.com/view/rac-iclr22
Lottery Tickets can have Structural Sparsity
Authors: Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, Zhangyang Wang
Date: 2021-09-29 00:00:00

The lottery ticket hypothesis (LTH) has shown that dense models contain highly sparse subnetworks (i.e., $\textit{winning tickets}$) that can be trained in isolation to match full accuracy. Despite many exciting efforts being made, there is one  "commonsense" seldomly challenged: a winning ticket is found by iterative magnitude pruning (IMP) and hence the resultant pruned subnetworks have only unstructured sparsity. That gap limits the appeal of winning tickets in practice, since the highly irregular sparse patterns are challenging to accelerate on hardware. Meanwhile, directly substituting structured pruning for unstructured pruning in IMP damages performance more severely and is usually unable to locate winning tickets. 

In this paper, we demonstrate $\textbf{the first positive result}$ that a structurally sparse winning ticket can be effectively found in general. The core idea is to append ``post-processing techniques" after each round of (unstructured) IMP, to enforce the formation of structural sparsity. Specifically, we first ``re-fill" pruned elements back in some channels deemed to be important, and then ``re-group" non-zero elements to create flexible group-wise structural patterns. Both our identified channel- and group-wise structural subnetworks win the lottery, with substantial inference speedups readily supported by practical hardware. Extensive experiments, conducted on diverse datasets across multiple network backbones, consistently validate our proposal, showing that the hardware acceleration roadblock of LTH is now removed. Specifically, the structural winning tickets obtain up to $\{64.93\%, 64.84\%, 64.84\%\}$ running time savings at $\{36\%\sim 80\%, 74\%, 58\%\}$ sparsity on CIFAR, Tiny-ImageNet, ImageNet, while maintaining comparable accuracy. All the codes and pre-trained models will be publicly released.
Sample Complexity of Deep Active Learning
Authors: Zhao Song, Baocheng Sun, Danyang Zhuo
Date: 2021-09-29 00:00:00
Tasks: Active Learning, BIG-bench Machine Learning, Fraud Detection, Medical Diagnosis

Many machine learning algorithms require large numbers of labeled training data to deliver state-of-the-art results. However, in many domains of AI, there are abundant unlabeled data but it is costly to get data labeled by experts, such as medical diagnosis and fraud detection. In these domains, active learning, where an algorithm maximizes model accuracy while requiring the least number of labeled data, is appealing.
Active learning uses both labeled and unlabeled data to train models, and the learning algorithm decides which subset of data should acquire labels.
Due to the costly label acquisition, it is interesting to know whether it is possible from a theoretical perspective to understand how many labeled data are actually needed to train a machine learning model. This question is known as the sample complexity problem, and it has been extensively explored for training linear machine learning models (e.g., linear regression). Today, deep learning has become the de facto method for machine learning, but the sample complexity problem for deep active learning remains unsolved. This problem is challenging due to the non-linear nature of neural networks.
In this paper, we present the first deep active learning algorithm which has a provable sample complexity. Using this algorithm, we have derived the first upper bound on the number of required labeled data for training neural networks. 
Our upper bound shows that the minimum number of labeled data a neural net needs does not depend on the data distribution or the width of the neural network but is determined by the smoothness of non-linear activation and the dimension of the input data.
Theoretical Analysis of Consistency Regularization with Limited Augmented Data
Authors: Shuo Yang, Yijun Dong, Rachel Ward, Inderjit S Dhillon, Sujay Sanghavi, Qi Lei
Date: 2021-09-29 00:00:00
Tasks: Data Augmentation, Generalization Bounds, regression

Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to use augmented data. In this paper, we take a small step in this direction; we present a simple new statistical framework to analyze data augmentation - specifically, one that captures what it means for one input sample to be an augmentation of another, and also the richness of the augmented set. We use this to interpret consistency regularization as a way to reduce function class complexity, and characterize its generalization performance. Specializing this analysis for linear regression shows that consistency regularization has strictly better sample efficiency as compared to empirical risk minimization on the augmented set. In addition, we also provide generalization bounds under consistency regularization for logistic regression and two-layer neural networks. We perform experiments that make a clean and apples-to-apples comparison (i.e. with no extra modeling or data tweaks) between ERM and consistency regularization using CIFAR-100 and WideResNet; these demonstrate the superior efficacy of consistency regularization.