pwc-archive/papers-with-abstracts huggingface dataset:
Few-shot Learning with Big Prototypes
Authors: Ning Ding, Yulin Chen, Xiaobin Wang, Hai-Tao Zheng, Zhiyuan Liu, Pengjun Xie
Date: 2021-09-29 00:00:00
Tasks: Few-Shot Learning

Using dense vectors, i.e., prototypes, to represent abstract information of classes has become a common approach in low-data machine learning scenarios. Typically, prototypes are mean output embeddings over the instances for each class. In this case, prototypes have the same dimension of example embeddings, and such tensors could be regarded as ``points'' in the feature space from the geometrical perspective. But these points may lack the expressivity of the whole class-level information due to the biased sampling.
In this paper, we propose to use tensor fields (``areas'') to model prototypes to enhance the expressivity of class-level information. Specifically, we present \textit{big prototypes}, where prototypes are represented by hyperspheres with dynamic sizes. A big prototype could be effectively modeled by two sets of learnable parameters, one is the center of the hypersphere, which is an embedding with the same dimension of training examples. The other is the radius of the sphere, which is a constant. Compared with atactic manifolds with complex boundaries, representing hypersphere with parameters is immensely easier. Moreover, it is convenient to perform metric-based classification with big prototypes in few-shot learning, where we only need to calculate the distance from a data point to the surface of the hypersphere.
Extensive experiments on few-shot learning tasks across NLP and CV demonstrate the effectiveness of big prototypes.
MOG: Molecular Out-of-distribution Generation with Energy-based Models
Authors: Seul Lee, Dong Bok Lee, Sung Ju Hwang
Date: 2021-09-29 00:00:00
Tasks: Drug Discovery, Out of Distribution (OOD) Detection

Recent advances of deep generative models opened up a new horizon for de novo drug discovery. However, a well-known problem of existing works on molecule generation is that the generated molecules highly resemble those in the training set. Models that do not require training molecules such as RL-based models circumvent this problem, but they lack information about existing molecules. In this paper, we propose Molecular Out-of-distribution Generation (MOG), a novel framework that explicitly generates OOD molecules with respect to given molecules by combining two aspects of energy-based models (EBMs): generation and out-of-distribution (OOD) detection. This can be done by introducing multiple energy pivots to Langevin dynamics in generation and increase energy instead of minimizing it. We also utilize a property predictor to provide the property gradient of molecules to the modified Langevin dynamics. To validate the ability to explore the chemical space beyond the known molecular distribution, we experiment with MOG to generate molecules of high absolute values of docking score, which is the affinity score based on a physical binding simulation between a target protein and a given molecule. Docking score is a strong proxy to drug activity unlike penalized logP or QED and requires stronger exploration as it is nonlinear to local molecular structures and has many local optima. MOG is able to generate molecules with high docking scores compared to existing methods. Moreover, we further show the energy-increasing strategy based on EBMs can be universally applied to existing models and enhance their resulting novelty.
Discovering Latent Network Topology in Contextualized Representations with Randomized Dynamic Programming
Authors: Yao Fu, Mirella Lapata
Date: 2021-09-29 00:00:00
Tasks: Paraphrase Generation

The discovery of large-scale discrete latent structures is crucial for understanding the fundamental generative processes of language. In this work, we use structured latent variables to study the representation space of contextualized embeddings and gain insight into the hidden topology of pretrained language models. However, existing methods are severely limited by issues of scalability and efficiency as working with large combinatorial spaces requires expensive memory consumption. We address this challenge by proposing a Randomized Dynamic Programming (RDP) algorithm for the approximate inference of structured models with DP-style exact computation (e.g., Forward-Backward). Our technique samples a subset of DP paths reducing memory complexity to as small as one percent. We use RDP to analyze the representation space of pretrained language models, discovering a large-scale latent network in a fully unsupervised way. The induced latent states not only serve as anchors marking the topology of the space (neighbors and connectivity), but also reveal linguistic properties related to syntax, morphology, and semantics. We also show that traversing this latent network yields unsupervised paraphrase generation.
On Reward Maximization and Distribution Matching for Fine-Tuning Language Models
Authors: Tomasz Korbak, Hady Elsahar, Germán Kruszewski, Marc Dymetman
Date: 2021-09-29 00:00:00
Tasks: Language Modelling, Reinforcement Learning (RL), Text Generation

The availability of large pre-trained models is changing the landscape of Machine Learning research and practice, moving from a "training from scratch" to a "fine-tuning'' paradigm. While in some applications the goal is to "nudge'' the pre-trained distribution towards preferred outputs, in others it is to steer it towards a different distribution over the sample space. Two main paradigms have emerged to tackle this challenge: Reward Maximization (RM) and, more recently, Distribution Matching (DM). RM applies standard Reinforcement Learning (RL) techniques, such as Policy Gradients, to gradually increase the reward signal. DM prescribes to first make explicit the target distribution that the model is fine-tuned to approximate. Here we explore the intimate connections between the two paradigms and show that methods such as KL-control developed in the RM paradigm can also be construed as belonging to DM. We further observe that while DM differs from RM, it can suffer from similar training difficulties, such as high gradient variance. We leverage connections between the two paradigms to import the concept of baseline into DM methods. We empirically validate the benefits of adding a baseline on an array of controllable language generation tasks such as constraining topic, sentiment, and gender distributions in texts sampled from a language model. We observe superior performance in terms of constraint satisfaction, stability, and sample efficiency.
Momentum Conserving Lagrangian Neural Networks
Authors: Ravinder Bhattoo, Sayan Ranu, N M Anoop Krishnan
Date: 2021-09-29 00:00:00
Tasks: Inductive Bias

Realistic models of physical world rely on differentiable symmetries that, in turn, correspond to conservation laws. Recent works on Lagrangian and Hamiltonian neural networks show that the underlying symmetries of a system can be easily learned by a neural network when provided with an appropriate inductive bias. However, these models still suffer from issues such as inability to generalize to arbitrary system sizes, poor interpretability, and most importantly, inability to learn translational and rotational symmetries, which lead to the conservation laws of linear and angular momentum, respectively. Here, we present a momentum conserving Lagrangian neural network (MCLNN) that learns the Lagrangian of a system, while also preserving the translational and rotational symmetries. We test our approach on linear and non-linear spring systems, and a gravitational system, demonstrating the energy and momentum conservation. We also show that the model developed can generalize to systems of any arbitrary size. Finally, we discuss the interpretability of the MCLNN, which directly provides physical insights into the interactions of multi-particle systems.
Contrastive Learning of 3D Shape Descriptor with Dynamic Adversarial Views
Authors: Shuaihang Yuan, Yi Fang
Date: 2021-09-29 00:00:00
Tasks: 3D Shape Classification, 3D Shape Recognition, 3D Shape Representation, Contrastive Learning, Retrieval, Self-Supervised Learning

View-based deep learning models have shown the capability to learn 3D shape descriptors with superior performance on 3D shape recognition, classification, and retrieval. Most popular techniques often leverage the class label to train deep neural networks under supervision to learn to extract 3D deep representation by aggregating information from a static and pre-selected set of different views used for all shapes. Those approaches, however, often face challenges posed by the requirement of a large amount of annotated training data and the lack of a mechanism for the adaptive selection of shape-instance-dependent views towards the learning of more informative 3D shape representation. This paper addresses those two challenging issues by introducing the concept of adversarial views and developing a new mechanism to generate views for adversarial training of a self-supervised contrastive model for 3D shape descriptor, denoted as CoLAV. In particular, compared to the recent advances in multi-view approaches, our proposed CoLAV gains advantages by leveraging the contrastive learning techniques for self-supervised learning of 3D shape representations without the need for labeled data. In addition, CoLAV introduces a novel mechanism for the dynamic generation of shape-instance-dependent adversarial views as positive pairs to adversarially train robust contrastive learning models towards the learning of more informative 3D shape representation. Comprehensive experimental results on 3D shape classification demonstrate that the 3D shape descriptor learned by CoLAV exhibits superior performance for 3D shape recognition over other state-of-the-art techniques, even though CoLAV is completely self-trained with unlabeled 3D datasets (e.g., ModelNet40).
RitzNet: A Deep Neural Network Method for Linear Stress Problems
Authors: Min Liu, Zhiqiang Cai, Karthik Ramani
Date: 2021-09-29 00:00:00

Learning based method for physics related computation has attracted significant attention recently. Effort has been devoted into learning a surrogate model which simulates system behavior from existing data. This paper presents RitzNet, an unsupervised learning method which takes any point in the computation domain as input, and learns a neural network model to output its corresponding function value satisfying the underlying governing PDEs. We focus on the linear elastic boundary value problem and formulate it as the natural minimization of its associated energy functional, whose discrete version is further utilized as the loss function of RitzNet. A standard fully connected deep neural network structure is explored in this study to model the solutions of a system of elliptic PDEs. Numerical studies on problems with analytical solutions or unknown solutions show that the proposed RitzNet is capable of approximating linear elasticity problems accurately. A parametric sensitivity study sheds light on the potential of RitzNet due to its meshless characteristics.
Source-Free Few-Shot Domain Adaptation
Authors: Wenyu Zhang, Li Shen, Chuan-Sheng Foo, Wanyue Zhang
Date: 2021-09-29 00:00:00
Tasks: Domain Adaptation, domain classification, Test-time Adaptation

Deep models are prone to performance degradation when there is a domain shift between the source (training) data and target (test) data. Test-time adaptation of pre-trained source models with streaming unlabelled target data is an attractive setting that protects the privacy of source data, but it has mini-batch size and class-distribution requirements on the streaming data which might not be desirable in practice. In this paper, we propose the source-free few-shot adaptation setting to address these practical challenges in deploying test-time adaptation. Specifically, we propose a constrained optimization of source model batch normalization layers by finetuning linear combination coefficients between training and support statistics. The proposed method is easy to implement and improves source model performance with as little as one labelled target sample per class. We evaluate on different multi-domain classification datasets. Experiments demonstrate that our proposed method achieves comparable or better performance than test-time adaptation, while not constrained by streaming conditions.
Robust and Data-efficient Q-learning by Composite Value-estimation
Authors: Gabriel Kalweit, Maria Kalweit, Joschka Boedecker
Date: 2021-09-29 00:00:00
Tasks: Q-Learning

In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency and is susceptible to stochasticity or noise in transitions and reward, which is limiting with regard to real-world applications. We alleviate these problems by proposing two novel off-policy Temporal-Difference formulations: (1) Truncated Q-functions which represent the return for the first n steps of a target-policy rollout w.r.t. the full action-value and (2) Shifted Q-functions, acting as the farsighted return after this truncated rollout. This decomposition allows us to optimize both parts with their individual learning rates, achieving significant learning speedup and robustness to variance in the reward signal, leading to the Composite Q-learning algorithm. We employ Composite Q-learning within TD3 and compare Composite TD3 with TD3 and TD3(Delta), which we introduce as an off-policy variant of TD(Delta). Moreover, we show that Composite TD3 outperforms TD3 as well as TD3(Delta) significantly in terms of data-efficiency in multiple simulated robot tasks and that Composite Q-learning is robust to stochastic environments and reward functions.
GenTAL: Generative Denoising Skip-gram Transformer for Unsupervised Binary Code Similarity Detection
Authors: Litao Li, Steven Ding, Philippe Charland, Hanbo Yu, Christopher James Molloy
Date: 2021-09-29 00:00:00
Tasks: Contrastive Learning, Denoising, Feature Engineering, Language Modeling, Language Modelling, Malware Analysis, Vulnerability Detection

Binary code similarity detection serves a critical role in cybersecurity. It alleviates the huge manual effort required in the reverse engineering process for malware analysis and vulnerability detection, where often the original source code is not available for analysis. Most of the existing solutions focus on a manual feature engineering process and customized code matching algorithms that are inefficient and inaccurate. Recent deep-learning-based solutions embed the semantics of binary code into a latent space through supervised contrastive learning. However, one cannot cover all its possible forms in the training set to learn the variance of the same semantic. In this paper, we propose an unsupervised model aiming to learn the intrinsic representation of assembly code semantics. Specifically, we propose a Transformer-based auto-encoder-like language model for the low-level assembly code grammar to capture the abstract semantic representation. By coupling a Transformer encoder and a skip-gram-style loss design, it can learn a compact representation that is robust against different compilation options. We conduct experiments on four different block-level code similarity tasks. It shows that our method is more robust compared to the state-of-the-art.
Deep Fusion of Multi-attentive Local and Global Features with Higher Efficiency for Image Retrieval
Authors: Baorong Shi
Date: 2021-09-29 00:00:00
Tasks: Image Retrieval, Re-Ranking, Retrieval

Image retrieval is to search images similar to the given query image by extracting features. Previously, methods that firstly search by global features then re-rank images using local feature matching were proposed, which has an excellent performance on many datasets. However, their drawbacks are also obvious. For example, the local feature matching consumes time and space greatly, the re-ranking process weakens the influence of global features, and the local feature learning is not accurate enough and semantic enough because of the trivial design. In this work, we proposed a Unifying Global and Attention-based Local Features Retrieval method (referred to as UGALR), which is an end-to-end and single-stage pipeline. Particularly, UGALR benefits from two aspects: 1) it accelerates extraction speed and reduces memory consumption by removing the re-ranking process and learning local feature matching with convolutional neural networks instead of RANSAC algorithm; 2) it learns more accurate and semantic local information through combining spatial and channel attention with the aid of intermediate supervision. Experiments on Revisited Oxford and Paris datasets validate the effectiveness of our approach, and we achieved state-of-the-art performance compared to other popular methods. The codes will be available soon.
Target-Side Data Augmentation for Sequence Generation
Authors: Shufang Xie, Ang Lv, Yingce Xia, Lijun Wu, Tao Qin, Rui Yan, Tie-Yan Liu
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Abstractive Text Summarization, Data Augmentation, Decoder, Machine Translation, Translation

Autoregressive sequence generation, a prevalent task in machine learning and natural language processing, generates every target token conditioned on both a source input and previously generated target tokens. Previous data augmentation methods, which have been shown to be effective for the task, mainly enhance source inputs (e.g., injecting noise into the source sequence by random swapping or masking, back translation, etc.) while overlooking the target-side augmentation. In this work, we propose a target-side augmentation method for sequence generation. In training, we use the decoder output probability distributions as soft indicators, which are multiplied with target token embeddings, to build pseudo tokens. These soft pseudo tokens are then used as target tokens to enhance the training. We conduct comprehensive experiments on various sequence generation tasks, including dialog generation, machine translation, and abstractive summarization. Without using any extra labeled data or introducing additional model parameters, our method significantly outperforms strong baselines. The code is available at https://github.com/TARGET-SIDE-DATA-AUG/TSDASG.
Using Graph Representation Learning with Schema Encoders to Measure the Severity of Depressive Symptoms
Authors: Simin Hong, Anthony Cohn, David Crossland Hogg
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Decision Making, Diagnostic, Graph Representation Learning, Representation Learning, Sentiment Analysis

Graph neural networks (GNNs) are widely used in regression and classification problems applied to text, in areas such as sentiment analysis and medical decision-making processes. We propose a novel form for node attributes within a GNN based model that capture node-specific embeddings for every word in the vocabulary. This provides a global representation at each node, coupled with node-level updates according to associations among words in a transcript. We demonstrate the efficacy of the approach by augmenting the accuracy of measuring major depressive disorder (MDD). Prior research has sought to make a diagnostic prediction of depression levels from patient data using several modalities, including audio, video, and text. On the DAIC-WOZ benchmark, our method outperforms state-of-art methods by a substantial margin, including those using multiple modalities.
You May Need both Good-GAN and Bad-GAN for Anomaly Detection
Authors: Riqiang Gao, Zhoubing Xu, Guillaume Chabin, Awais Mansoor, Florin-Cristian Ghesu, Bogdan Georgescu, Bennett A. Landman, Sasa Grbic
Date: 2021-09-29 00:00:00
Tasks: Anatomy, Anomaly Detection

Generative adversarial nets (GAN) have been successfully adapted for anomaly detection, where end-to-end anomaly scoring by so-called Bad-GAN has shown promising results. A Bad-GAN generates pseudo anomalies at the low-density area of inlier distribution, and thus the inlier/outlier distinction can be approximated. However, the generated pseudo anomalies from existing Bad-GAN approaches may (1) converge to certain patterns with limited diversity, and (2) differ from the real anomalies, making the anomaly detection hard to generalize. In this work, we propose a new model called Taichi-GAN to address the aforementioned issues of a conventional Bad-GAN. First, a new orthogonal loss is proposed to regularize the cosine distance of decentralized generated samples in a Bad-GAN. Second, we utilize few anomaly samples (when available) with a conventional GAN, i.e., so-called Good-GAN, to draw the generated pseudo anomalies closer to the real anomalies. Our Taichi-GAN incorporates Good-GAN and Bad-GAN in an adversarial manner; which generates pseudo anomalies that contributing to a more robust discriminator for anomaly scoring, and thus anomaly detection. Substantial improvements can be observed from our proposed model on multiple simulated and real-life anatomy detection tasks.
L0-Sparse Canonical Correlation Analysis
Authors: Ofir Lindenbaum, Moshe Salhov, Amir Averbuch, Yuval Kluger
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4

Canonical Correlation Analysis (CCA) models are powerful for studying the associations between two sets of variables. The canonically correlated representations, termed \textit{canonical variates} are widely used in unsupervised learning to analyze unlabeled multi-modal registered datasets. Despite their success, CCA models may break (or overfit) if the number of variables in either of the modalities exceeds the number of samples. Moreover, often a significant fraction of the variables measures modality-specific information, and thus removing them is beneficial for identifying the \textit{canonically correlated variates}. Here, we propose $\ell_0$-CCA, a method for learning correlated representations based on sparse subsets of variables from two observed modalities.
Sparsity is obtained by multiplying the input variables by stochastic gates, whose parameters are learned together with the CCA weights via an $\ell_0$-regularized correlation loss. 
We further propose $\ell_0$-Deep CCA for solving the problem of non-linear sparse CCA by modeling the correlated representations using deep nets. We demonstrate the efficacy of the method using several synthetic and real examples. Most notably, by gating nuisance input variables, our approach improves the extracted representations compared to other linear, non-linear and sparse CCA-based models.