pwc-archive/papers-with-abstracts huggingface dataset:
Closed-form Sample Probing for Learning Generative Models in Zero-shot Learning
Authors: Samet Cetin, Orhun Buğra Baran, Ramazan Gokberk Cinbis
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Form, Generalized Zero-Shot Learning, Meta-Learning, Sample Probing, Zero-Shot Learning

Generative model based approaches have led to significant advances in zero-shot learning (ZSL) over the past few-years. These approaches typically aim to learn a conditional generator that synthesizes training samples of classes conditioned on class definitions. The final zero-shot learning model is then obtained by training a supervised classification model over the real and/or synthesized training samples of seen and unseen classes, combined. Therefore, naturally, the generative model needs to produce not only relevant samples, but also those that are sufficiently rich for classifier training purposes, which is handled by various heuristics in existing works. In this paper, we introduce a principled approach for training generative models directly for training data generation purposes. Our main observation is that the use of closed-form models opens doors to end-to-end training thanks to differentiability of the solvers. In our approach, at each generative model update step, we fit a task-specific closed-form ZSL model from generated samples, and measure its loss on novel samples all within the compute graph, a procedure that we refer to as sample probing. In this manner, the generator receives feedback directly based on the value of its samples for model training purposes. Our experimental results show that the proposed sample probing approach improves the ZSL results even when integrated into state-of-the-art generative models.
Reconstruction for disentanglement, Contrast for invariance
Authors: Jiageng Zhu, Hanchen Xie, Wael AbdAlmgaeed
Date: 2021-09-29 00:00:00
Tasks: Disentanglement, Representation Learning

 Disentangled and invariant representation are two vital goals for representation learning and many approaches have been proposed to achieve one of them. However, those two goals are actually complementary to each other and we propose a framework to accomplish both of them together. We introduce weakly supervised signals to learn disentangled representation and use contrastive methods to enforce invariant representation. By experimenting on state-of-the-art datasets, the results show that our framework outperforms previous works on both tasks.
Edge Partition Modulated Graph Convolutional Networks
Authors: Yilin He, Chaojie Wang, Hao Zhang, Bo Chen, Mingyuan Zhou
Date: 2021-09-29 00:00:00
Tasks: Relation, Variational Inference

Graph convolutional networks (GCNs), which propagate the node features through the edges and learn how to transform the aggregated features under label supervision, have achieved great success in supervised feature extraction for both graph-level and node-level classification tasks. However, GCNs typically treat the graph adjacency matrix as given and ignore how the edges could be formed by different latent inter-node relations. In this paper, we introduce a relational graph generative process to model how the observed edges are generated by aggregating the node interactions over multiple overlapping node communities, each of which represents a particular type of relation that contributes to the edges via a logical OR mechanism. Based on this relational generative model, we partition each edge into the summation of multiple relation-specific weighted edges, and use the weighted edges in each community to define a relation-specific GCN. We introduce a variational inference framework to jointly learn how to partition the edges into different communities and combine relation-specific GCNs for the end classification tasks. Extensive evaluations on real-world datasets have demonstrated the working mechanisms of the edge partition modulated GCNs and their efficacy in learning both node and graph-level representations.
BAM: Bayes Augmented with Memory
Authors: Josue Nassar, Jennifer Rogers Brennan, Ben Evans, Kendall Lowrey
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4

Online learning via Bayes' theorem allows new data to be continuously integrated into an agent's current beliefs. However, a naive application of Bayesian methods in non-stationary environments leads to slow adaptation and results in state estimates that may converge confidently to the wrong parameter value.  A common solution when learning in changing environments is to discard/downweight past data; however, this simple mechanism of ``forgetting'' fails to account for the fact that many real-world environments involve revisiting similar states. We propose a new framework, Bayes augmented with memory (BAM), that takes advantage of past experience by allowing the agent to choose which past observations to remember and which to forget. We demonstrate that BAM generalizes many popular Bayesian update rules for non-stationary environments. Through a variety of experiments, we demonstrate the ability of BAM to continuously adapt in an ever-changing world.
Efficient Point Transformer for Large-scale 3D Scene Understanding
Authors: Chunghyun Park, Yoonwoo Jeong, Minsu Cho, Jaesik Park
Date: 2021-09-29 00:00:00
Tasks: 3D Semantic Segmentation, Quantization, Scene Understanding, Semantic Segmentation

The recent success of neural networks has enabled a better interpretation of 3D point clouds, but processing a large-scale 3D scene remains a challenging problem. Most approaches divide a large-scale 3D scene into multiple regions and combine the local predictions, but this inevitably increases inference time and involves preprocessing stages, such as k-nearest neighbor search. An alternative is to quantize the point cloud to voxels and process them with sparse convolution. Although sparse convolution is efficient and scalable for large 3D scenes, the quantization artifacts impair geometric details and degrade prediction accuracy. This paper proposes an Efficient Point Transformer (EPT) that effectively relieves the quantization artifacts and avoids expensive resource requirements. Each layer of EPT implements the local self-attention mechanism for analyzing continuous 3D coordinates and offers fast inference time using a voxel hashing-based architecture. The proposed method can be adopted for various 3D vision applications, such as 3D semantic segmentation and 3D detection. In experiments, the proposed EPT model outperforms the state-of-the-art on large-scale 3D semantic segmentation benchmarks and also shows better performance on 3D detection benchmarks than point-based or voxel-based baseline methods.
Rethinking Again the Value of Network Pruning -- A Dynamical Isometry Perspective
Authors: Huan Wang, Can Qin, Yue Bai, Yun Fu
Date: 2021-09-29 00:00:00
Tasks: Network Pruning

Several recent works questioned the value of inheriting weight in structured neural network pruning because they empirically found training from scratch can match or even outperform finetuning a pruned model. In this paper, we present evidences that this argument is actually \emph{inaccurate} because of using improperly small finetuning learning rates. With larger learning rates, our results consistently suggest pruning outperforms training from scratch on multiple networks (ResNets, VGG11) and datasets (MNIST, CIFAR10, ImageNet) over most pruning ratios. To deeply understand why finetuning learning rate holds such a critical role, we examine the theoretical reason behind through the lens of \emph{dynamical isometry}, a nice property of networks that can make the gradient signals preserve norm during propagation. Our results suggest that weight removal in pruning breaks dynamical isometry, \emph{which fundamentally answers for the performance gap between a large finetuning LR and~a small one}. Therefore, it is necessary to recover the dynamical isometry before finetuning. In this regard, we also present a regularization-based technique to do so, which is rather simple-to-implement yet effective in dynamical isometry recovery on modern residual convolutional neural networks.
CSQ: Centered Symmetric Quantization for Extremely Low Bit Neural Networks
Authors: Faaiz Asim, Jaewoo Park, Azat Azamat, Jongeun Lee
Date: 2021-09-29 00:00:00
Tasks: Quantization

Recent advances in quantized neural networks (QNNs) are closing the performance gap with the full precision neural networks. However at very low precision (i.e., $\le 3$-bits), QNNs often still suffer significant performance degradation. The conventional uniform symmetric quantization scheme allocates unequal numbers of positive and negative quantization levels. We show that this asymmetry in the number of positive and negative quantization levels can result in significant quantization error and performance degradation at low precision. We propose and analyze a quantizer called centered symmetric quantizer (CSQ), which preserves the symmetry of latent distribution by providing equal representations to the negative and positive sides of the distribution. We also propose a novel method to efficiently map CSQ to binarized neural network hardware using bitwise operations. Our analyses and experimental results using state-of-the-art quantization methods on ImageNet and CIFAR-10 show the importance of using CSQ for weight in place of the conventional quantization scheme at extremely low-bit precision (2$\sim$3 bits).
The Details Matter: Preventing Class Collapse in Supervised Contrastive Learning
Authors: Daniel Yang Fu, Mayee F Chen, Michael Zhang, Kayvon Fatahalian, Christopher Ré
Date: 2021-09-29 00:00:00
Tasks: Contrastive Learning, Transfer Learning

Supervised contrastive learning optimizes a loss that pushes together embeddings of points from the same class while pulling apart embeddings of points from different classes. Class collapse—when every point from the same class has the same embedding—minimizes this loss but loses critical information that is not encoded in the class labels. For instance, the “cat” label does not capture unlabeled categories such as breeds, poses, or backgrounds (which we call “strata”). As a result, class collapse produces embeddings that are less useful for downstream applications such as transfer learning and achieves sub-optimal generalization error when there are strata. We explore a simple modification to supervised contrastive loss that prevents class collapse by uniformly pulling apart individual points from the same class. More importantly, we introduce a theoretical framing to analyze this loss through a view of how it embeds strata of different sizes. We show that our loss maintains distinctions between strata in embedding space, even though it does not explicitly use strata labels. We empirically explore several downstream implications of this insight. Our loss produces embeddings that achieve lift on three downstream applications by distinguishing strata: 4.4 points on coarse-to-fine transfer learning, 2.5 points on worst-group robustness, and 1.0 points on minimal coreset construction. Our loss also produces more accurate models, with up to 4.0 points of lift across 9 tasks.
Universal Controllers with Differentiable Physics for Online System Identification
Authors: Michelle Guo, Wenhao Yu, Daniel Ho, Jiajun Wu, Yunfei Bai, Karen Liu, Wenlong Lu
Date: 2021-09-29 00:00:00
Tasks: Domain Adaptation

Creating robots that can handle changing or unknown environments is a critical step towards real-world robot applications. Existing methods tackle this problem by training controllers robust to large ranges of environment parameters (Domain Randomization), or by combining ``Universal'' Controllers (UC) conditioned on environment parameters with learned identification modules that (implicitly or explicitly) identify the environment parameters from sensory inputs (Domain Adaptation). However, these methods can lead to over-conservative behaviors or poor generalization outside the training distribution. In this work, we present a domain adaptation approach that improves generalization of the identification module by leveraging prior knowledge in physics. Our proposed algorithm, UC-DiffOSI, combines a UC trained on a wide range of environments with an Online System Identification module based on a differentiable physics engine (DiffOSI). We evaluate UC-DiffOSI on articulated rigid body control tasks, including a wiping task that requires contact-rich environment interaction.
Compared to previous works, UC-DiffOSI outperforms domain randomization baselines and is more robust than domain adaptation methods that rely on learned identification models. In addition, we perform two studies showing that UC-DiffOSI operates well in environments with changing or unknown dynamics. These studies test sudden changes in the robot's mass and inertia, and they evaluate in an environment (PyBullet) whose dynamics differs from training (NimblePhysics).
Efficient Semi-Discrete Optimal Transport Using the Maximum Relative Error between Distributions
Authors: Huidong Liu, Ke Ma, Lei Zhou, Dimitris Samaras
Date: 2021-09-29 00:00:00

Semi-Discrete Optimal Transport (SDOT) transforms a continuous distribution to a discrete distribution. However, existing SDOT algorithms for high dimensional distributions have two limitations. 1) It is difficult to evaluate the quality of the transport maps produced by SDOT algorithms, because computing a high-dimensional Wasserstein distance for SDOT is intractable and 2) The transport map cannot guarantee that all target points have the corresponding source points that are mapped to them. To address these limitations, we introduce the Maximum Relative Error (\texttt{MRE}) between the target distribution and the transported distribution computed by an SDOT map. If the \texttt{MRE} is smaller than 1, then every target point is guaranteed to have an area in the source distribution that is mapped to it. We propose a statistical method to compute the lower and upper bounds of the \texttt{MRE} given a confidence threshold and a precision. The gap between the lower bound and the upper bound approaches 0 as the number of samples goes to infinity. We present an efficient Epoch Gradient Descent algorithm for SDOT (SDOT-EGD) that computes the learning rate, number of iterations, and number of epochs in order to guarantee an arbitrarily small \texttt{MRE} in expectation. Experiments on both low and high-dimensional data show that SDOT-EGD is much faster and converges much better than state-of-the-art SDOT algorithms. We also show our method's potential to improve GAN training by avoiding the oscillation caused by randomly changing the association between noise and the real images.
Intervention-based Recurrent Casual Model for Non-stationary Video Causal Discovery
Authors: Yuke Li, Kenneth Li, Pin Wang, Donglai Wei, Hanspeter Pfister, Ching-Yao Chan
Date: 2021-09-29 00:00:00
Tasks: Causal Discovery, counterfactual, Counterfactual Reasoning

Non-stationary casual structures are prevalent in real-world physical systems. For example, the stacked blocks interacted with one another until they fall apart, while the billiard balls are moving independently until they collide. However, most video causal discovery methods can not discover such non-stationary casual structures due to the lack of modeling for the instantaneous change and the dynamics of the casual structure. In this work, we propose the Intervention-based Recurrent Casual Model (IRCM) for non-stationary video casual discovery. First, we extend the existing intervention-based casual discovery framework for videos to formulate the instantaneous change of the casual structure in a principled manner. Then, we use a recurrent model to sequentially predict the causal structure model based on previous observations to capture the non-stationary dynamic of the casual structure. We evaluate our method on two popular physical system simulation datasets with various types of multi-body interactions. Experimental results show that the proposed IRCM achieves the state-of-the-art performance on both the counterfactual reasoning and future forecasting tasks.
Contrastive Pre-training for Zero-Shot Information Retrieval
Authors: Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, Edouard Grave
Date: 2021-09-29 00:00:00
Tasks: Contrastive Learning, Fact Checking, Information Retrieval, Question Answering, Retrieval

Information retrieval is an important component in natural language processing, for knowledge intensive tasks such as question answering and fact checking. Recently, information retrieval has seen the emergence of dense retrievers, based on neural networks, as an alternative to classical sparse methods based on term-frequency. Neural retrievers work well on the problems for which they were specifically trained, but they do not generalize as well as term-frequency methods to new domains or applications. By contrast, in many other NLP tasks, conventional self-supervised pre-training based on masking leads to strong generalization with small number of training examples. We believe this is not yet the case for information retrieval, because these pre-training methods are not well adapted to this task. In this work, we consider contrastive learning as a more natural pre-training technique for retrieval and show that it leads to models that are competitive with BM25 on many domains or applications, even without training on supervised data. Our dense pre-trained models also compare favorably against BERT pre-trained models in the few-shot setting, and achieves state-of-the-art performance on the BEIR benchmark when fine-tuned on MS-MARCO.
Structured Energy Network as a dynamic loss function. Case study. A case study with multi-label Classification
Authors: Jay-Yoon Lee, Dhruvesh Patel, Purujit Goyal, Andrew McCallum
Date: 2021-09-29 00:00:00
Tasks: Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, Structured Prediction

We propose SEAL which utilizes this energy network as a trainable loss function for a simple feedfoward network. Structured prediction energy networks (SPENs) (Belanger & McCallum, 2016; Gygli et al., 2017) have shown that a neural network (i.e. energy network) can learn a reasonable energy function over the candidate structured outputs. We find that rather than using SPEN as a prediction network, using it as a trainable loss function is not only computationally efficient but also results in higher performance. compared to SPENs in both training and inference time. As the energy loss function is trainable, we propose SEAL to be dynamic which can adapt energy function to focus on the region where feedforward model will be affected most. We find this to be effective in ablation study comparing SEAL to the static version (§4) where energy function is fixed after pretraining. We show the relation to previous work on the joint optimization model of energy network and feedforward model (INFNET) as we show that it is equivalent to SEAL using margin-based loss if INFNET relaxes their loss function. Based on the unique architecture of SEAL, we further propose a variant of SEAL that utilizes noise contrastive ranking (NCE) loss that by itself does not perform well as a structured energy network, but embodied in SEAL, it shows the greatest performance among the variants we study. We demonstrate the effectiveness of SEAL on 7 feature-based and 3 text-based multi-label classification datasets. The best version of SEAL that uses NCE ranking method achieves close to +2.85, +2.23 respective F1 point gain in average over cross-entropy and INFNET on the feature-based datasets, excluding one outlier that has an excessive gain of +50.0 F1 points. Lastly, examining whether the proposed framework is effective on a large pre-trained model as well, we observe SEAL achieving +0.87 F1 point gain in average on top of BERT-based adapter model o text datasets.
OVD-Explorer: A General Information-theoretic Exploration Approach for Reinforcement Learning
Authors: Jinyi Liu, Zhi Wang, Yan Zheng, Jianye Hao, Junjie Ye, Chenjia Bai, Pengyi Li
Date: 2021-09-29 00:00:00
Tasks: MuJoCo, reinforcement-learning, Reinforcement Learning (RL)

Many exploration strategies are built upon the optimism in the face of the uncertainty (OFU) principle for reinforcement learning. However, without considering the aleatoric uncertainty, existing methods may over-explore the state-action pairs with large randomness and hence are non-robust. In this paper, we explicitly capture the aleatoric uncertainty from a distributional perspective and propose an information-theoretic exploration method named Optimistic Value Distribution Explorer (OVD-Explorer). OVD-Explorer follows the OFU principle, but more importantly, it avoids exploring the areas with high aleatoric uncertainty through maximizing the mutual information between policy and the upper bounds of policy's returns. Furthermore, to make OVD-Explorer tractable for continuous RL, we derive a closed form solution, and integrate it with SAC, which, to our knowledge, for the first time alleviates the negative impact on exploration caused by aleatoric uncertainty for continuous RL. Empirical evaluations on the commonly used Mujoco benchmark and a novel GridChaos task demonstrate that OVD-Explorer can alleviate over-exploration and outperform state-of-the-art methods.
Design and Evaluation for Robust Continual Learning
Authors: Yeu-Shin Fu, Josh Milthorpe
Date: 2021-09-29 00:00:00
Tasks: Continual Learning, Experimental Design

Continual learning is the ability to learn from new experiences without forgetting
previous experiences. Different continual learning methods are each motivated
by their own interpretation of the continual learning scenario, resulting in a wide
variety of experiment protocols, which hinders understanding and comparison of
results. Existing works emphasize differences in accuracy without considering
the effects of experimental settings. However, understanding the effects of experimental
assumptions is the most crucial part of any evaluation, as the experimental
protocol may supply implicit information. We propose six rules as a guideline for
experimental design and execution to conduct robust continual learning evaluation
for better understanding of the methods. Using these rules, we demonstrate the
importance of experimental choices regarding the sequence of incoming data and
the sequence of the task oracle. Even when task oracle-based methods are desired,
the rules can guide experimental design to support better evaluation and understanding
of the continual learning methods. Consistent application of these rules
in evaluating continual learning methods makes explicit the effect and validity of
many assumptions, thereby avoiding misleading conclusions.
Cross-Domain Lossy Compression as Optimal Transport with an Entropy Bottleneck
Authors: Huan Liu, George Zhang, Jun Chen, Ashish J Khisti
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Decoder, Denoising, Super-Resolution

We study the problem of cross-domain lossy compression where the reconstruction distribution is different from the source distribution in order to account for distributional shift due to processing. We formulate this as a generalization of optimal transport with an entropy bottleneck to account for the rate constraint due to compression. We provide expressions for the tradeoff between  compression rate and the achievable distortion with and without shared common randomness between the encoder and decoder and demonstrate using the example of a binary source that shared randomness can strictly improve the tradeoff. For the case without common randomness and squared-Euclidean distortion, we show that the optimal solution partially decouples into the problem of optimal compression and transport and also characterize the penalty associated with fully decoupling them.  We provide experimental results by training deep learning end-to-end compression systems for performing denoising on SVHN and super-resolution on MNIST, and demonstrate consistency with our theoretical results.