pwc-archive/papers-with-abstracts huggingface dataset:
MULTI-LEVEL APPROACH TO ACCURATE AND SCALABLE HYPERGRAPH EMBEDDING
Authors: Sepideh Maleki, Donya Saless, Dennis Wall, Keshav Pingali
Date: 2021-09-29 00:00:00
Tasks: Graph Embedding, hypergraph embedding, Link Prediction, Node Classification

Many problems such as node classification and link prediction in network data
can be solved using graph embeddings, and a number of algorithms are known for constructing
such embeddings. However, it is difficult to use graphs to capture 
non-binary relations such as communities of nodes. These kinds of 
complex relations are expressed more naturally as hypergraphs.
While hypergraphs are a generalization of graphs, state-of-the-art graph
embedding techniques are not adequate for solving prediction and
classification tasks on large hypergraphs accurately in reasonable time.
In this paper, we introduce NetVec, a novel multi-level framework for scalable unsupervised hypergraph embedding, which outperforms state-of-the-art hypergraph embedding systems by up to 
15% in accuracy. 
We also show that NetVec is capable of generating high quality embeddings for real-world hypergraphs with millions of nodes and hyperedges in only a couple of minutes while existing hypergraph systems either fail for such large hypergraphs or may take days to produce the embeddings.
DictFormer: Tiny Transformer with Shared Dictionary
Authors: Qian Lou, Ting Hua, Yen-Chang Hsu, Yilin Shen, Hongxia Jin
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Abstractive Text Summarization, Language Modeling, Language Modelling, Machine Translation, Translation

We introduce DictFormer with efficient shared dictionary to provide a compact, fast, and accurate transformer model. DictFormer significantly reduces the redundancy in the transformer's parameters by replacing the prior transformer's parameters with compact, shared dictionary, a few unshared coefficients, and indices. Also, DictFormer enables faster computations since expensive weights multiplications are converted into cheap shared look-ups on dictionary and a few linear projections. Training dictionary and coefficients are not trivial since indices used for looking up dictionary are not differentiable. We adopt a sparse-constraint training with $l_1\,\,norm$ relaxation to learn coefficients and indices in DictFormer. DictFormer is flexible to support different model sizes by dynamically changing dictionary size. Compared to existing lightweight Transformers, DictFormer consistently improves performance on multiple tasks, e.g., machine translation, abstractive summarization, and language modeling. Extensive experiments show that DictFormer outperforms prior light-weight transformer by $\sim2$ BLEU for machine translation task and achieves $\sim 1.7$ lower perplexity for the language modeling task, when matching or reducing the model size of the original transformer. Also, DictFormer reduces $6.3\times$ to $8.9\times$ model size on average without losing accuracy.
Exploiting Knowledge Distillation for Few-Shot Image Generation
Authors: Xingzhong Hou, Boxiao Liu, Fang Wan, Haihang You
Date: 2021-09-29 00:00:00
Tasks: Diversity, Image Generation, Knowledge Distillation, Relation

Few-shot image generation, which trains generative models on limited examples, is of practical importance. The existing pipeline is first pretraining a source model (which contains a generator and a discriminator) on a large-scale dataset and finetuning it on a target domain with limited samples. The main challenge is that the few-shot model easily becomes overfitting. It can be attributed to two aspects: the lack of sample diversity for the generator and the failure of fidelity discrimination for the discriminator. In this paper, we treat the diversity and fidelity in the source model as a kind of knowledge and propose to improve the generation results via exploring knowledge distillation. The source model trained on the large-scale dataset is regarded as teacher model and the target model (student) is learned by introducing momentum relation distillation module to produce diverse samples and source discrimination distillation to ensure the fidelity discrimination. With the momentum relation distillation and source discrimination distillation modules, the proposed method outperforms the state-of-the-art of by a large margin, i.e., 10% for FFHQ to Sketches, while achieving better diversity.
Nonlinear ICA Using Volume-Preserving Transformations
Authors: Xiaojiang Yang, Yi Wang, Jiacheng Sun, Xing Zhang, Shifeng Zhang, Zhenguo Li, Junchi Yan
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4

Nonlinear ICA is a fundamental problem in machine learning, aiming to identify the underlying independent components (sources) from data which is assumed to be a nonlinear function (mixing function) of these sources. Recent works prove that if the sources have some particular structures (e.g. temporal structure), they are theoretically identifiable even if the mixing function is arbitrary. However, in many cases such restrictions on the sources are difficult to satisfy or even verify, hence it inhibits the applicability of the proposed methods. Different from these works, we propose a general framework for nonlinear ICA, in which the mixing function is assumed to be a volume-preserving transformation, and meanwhile the conditions on the sources can be much looser. We provide an insightful proof of the identifiability of the proposed framework. We implement the framework by volume-preserving Flow-based models, and verify our theory by experiments on artificial data and synthesized images. Moreover, results on real-world images indicate that our framework can disentangle interpretable features.
Count-GNN: Graph Neural Networks for Subgraph Isomorphism Counting
Authors: Xingtong Yu, Zemin Liu, Yuan Fang, Xinming Zhang
Date: 2021-09-29 00:00:00
Tasks: Navigate

The prevalence of graph structures has attracted a surge of research interest in graph data. As many graph-based tasks exploit recurring subgraph patterns on graphs, subgraph isomorphism counting becomes an important problem. Classical methods usually boil down to a backtracking framework that needs to navigate a huge search space with prohibitive computational cost due to the NP-completeness of the problem. Some recent studies resort to graph neural networks (GNNs) to learn a low-dimensional representation for both the query subgraph and the input graph, in order to predict the number of query subgraph isomorphisms on the input graph. However, typical GNNs employ a node-centric message passing mechanism that receives and aggregates messages on nodes. While effective on node-oriented tasks, they become inadequate in complex structure matching for isomorphism counting. Moreover, given an input graph, the space of query subgraph is enormous, and thus expecting a single model to fit the diverse range of query subgraphs is unrealistic. In this paper, we propose a novel GNN called Count-GNN for subgraph isomorphic counting, to deal with the above challenges at two levels. At the edge level, we resort to an edge-centric message passing scheme, where messages on edges are propagated and aggregated based on the edge adjacency. By treating edges as first-class citizens, Count-GNN is able to preserve finer-grained structural information, given that an edge is an atomic unit of encoding graph structures. At the graph level, we modulate the graph representation conditioned on the query subgraph, so that the model can be adapted to each unique query for better matching with the input graph. To demonstrate the effectiveness and efficiency of Count-GNN, we conduct extensive experiments on a number of benchmark graphs. Results show that Count-GNN achieves superior performance in comparison to the state-of-the-art baselines.
FCause: Flow-based Causal Discovery
Authors: Tomas Geffner, Emre Kiciman, Angus Lamb, Martin Kukla, Miltiadis Allamanis, Cheng Zhang
Date: 2021-09-29 00:00:00
Tasks: Causal Discovery, Missing Values

Current causal discovery methods either fail to scale, model only limited forms of functional relationships, or cannot handle missing values. This limits their reliability and applicability. We propose FCause, a new flow-based causal discovery method that addresses these drawbacks.  Our method is scalable to both high dimensional as well as large volume of data, is able to model complex nonlinear relationships between variables, and can perform causal discovery under partially observed data. Furthermore, our formulation generalizes existing continuous optimization based causal discovery methods, providing a unified view of such models. We perform an extensive empirical evaluation, and show that FCause achieves state of the art results in several causal discovery benchmarks under different conditions reflecting real-world application needs.
The Essential Elements of Offline RL via Supervised Learning
Authors: Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, Sergey Levine
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Offline RL, reinforcement-learning, Reinforcement Learning (RL)

Offline reinforcement learning (RL) is typically tackled with value-based TD methods. However, a number of approaches have been proposed that aim to simplify offline RL by reducing it to weighted, filtered, or conditional behavioral cloning problems. These methods coerce suboptimal data into a form that allows supervised learning methods to acquire optimal policies. For example, any experience is optimal for learning to reach the final state in that experience. However, it remains unclear which ingredients are essential for such methods to work well, and when these methods outperform value-based approximate dynamic programming algorithms. These methods, which we collectively refer to as reinforcement learning via supervised learning (RvS), involve a number of design decisions, such as policy architectures and how the conditioning variable is constructed. Through extensive experiments, this paper studies the importance of these design decisions. The most important design decisions boil down to carefully choosing model capacity (e.g., via regularization or architecture) and choosing which information to condition on (e.g., goals or rewards). Our experiments find that more complex design choices, such as the large sequence models and value-based weighting schemes used in some prior works, are generally not necessary. Our results show that carefully designed RvS methods can attain results that match or exceed the best prior methods across a range of different offline RL benchmarks, including datasets with little or no optimal data. Our results help to identify the limits of current RvS methods and identify some important open problems.
Evolutionary perspective on model fine-tuning
Authors: Andrei Kucharavy, Ljiljana Dolamic, Rachid Guerraoui
Date: 2021-09-29 00:00:00
Tasks: BIG-bench Machine Learning, Image Generation, model, Text Generation

Be it in natural language generation or in the image generation, massive performances gains have been achieved in the last years. While a substantial part of these advances can be attributed to improvement in machine learning architectures, an important role has also been played by the ever-increasing parameter number of machine learning models, which made from-scratch retraining of the models prohibitively expensive for a large number of users.
In response to that, model fine-tuning - starting with an already good model and further training it on the data relevant to a new, related problem, gained in popularity. This fine-tuning is formally similar to the natural evolution of genetic codes in response to shifting environment. 
Here, we formalize this similarity in the framework of Fisher Geometric model and extreme value theory and present a set of tricks used by naturally evolving organisms to accelerate their adaptation, applicable to model fine-tuning.
SplitRegex: Faster Regex Synthesis via Neural Example Splitting
Authors: Su-Hyeon Kim, Hyunjoon Cheon, Yo-Sub Han, Sang-Ki Ko
Date: 2021-09-29 00:00:00

Due to the practical importance of regular expressions (regexes, for short), there has been a lot of research to automatically generate regexes from positive and negative string examples. A basic idea of learning a regex is a search-and-repair; search for a correct regex and repair it if incorrect. The problem is known to be PSPACE-complete and the main issue is to obtain a regex quickly within a time limit. 
While classical regex learning methods do not perform well, recent approaches using deep neural networks show better performance
with respect to the accuracy of the resulting regexes. However, all these approaches including SOTA models are often extremely slow because of the slow searching mechanism, and do not produce desired regexes within a given time limit. 
We tackle the problem of learning regexes faster from positive and negative strings by relying on a novel approach called `neural example splitting'. Our approach essentially split up example strings into multiple parts using a neural network trained to group similar substrings from positive strings. This helps to learn a regex faster and, thus, more accurately since we now learn from several short-length strings.
We propose an effective regex synthesis framework called `SplitRegex' that synthesizes subregexes from `split' positive substrings and produces the final regex by concatenating the synthesized subregexes. For the negative sample, we exploit pre-generated subregexes during the subregex synthesis process and perform the matching against negative strings. Then the final regex becomes consistent with all negative strings. SplitRegex is a divided-and-conquer framework for learning target regexes; split (=divide) positive strings and infer partial regexes for multiple parts, which is much more accurate than the whole string inferring, and concatenate (=conquer) inferred regexes while satisfying negative strings. We empirically demonstrate that the proposed SplitRegex framework substantially improves the previous regex synthesis approaches over four benchmark datasets.
Benchmarking Machine Learning Robustness in Covid-19 Spike Sequence Classification
Authors: Sarwan Ali, Bikram Sahoo, Pin-Yu Chen, Murray Patterson
Date: 2021-09-29 00:00:00
Tasks: Benchmarking, BIG-bench Machine Learning, Classification

The rapid spread of the COVID-19 pandemic has resulted in an unprecedented amount of sequence data of the SARS-CoV-2 viral genome --- millions of sequences and counting.  This amount of data, while being orders of magnitude beyond the capacity of traditional approaches to understanding the diversity, dynamics and evolution of viruses, is nonetheless a rich resource for machine learning (ML) and deep learning (DL) approaches as alternatives for extracting such important information from these data.  It is of hence utmost importance to design a framework for testing and benchmarking the robustness of these ML and DL approaches.

This paper the first (to our knowledge) to explore such a framework. In this paper, we introduce several ways to perturb SARS-CoV-2 spike protein sequences in ways that mimic the error profiles of common sequencing platforms such as Illumina and PacBio.  We show from experiments on a wide array of ML approaches from naive Bayes to logistic regression, that DL approaches are more robust (and accurate) to such adverarial attacks to the input sequences, while $k$-mer based feature vector representations are more robust than the baseline one-hot embedding.  Our benchmarking framework may developers of futher ML and DL techniques to properly assess their approaches towards understanding the behaviour of the SARS-CoV-2 virus, or towards avoiding possible future pandemics.
Transformer-based Transform Coding
Authors: Yinhao Zhu, Yang Yang, Taco Cohen
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Computational Efficiency, Data Compression, Image Compression, Quantization, Video Compression

Neural data compression based on nonlinear transform coding has made great progress over the last few years, mainly due to improvements in prior models, quantization methods and nonlinear transforms. A general trend in many recent works pushing the limit of rate-distortion performance is to use ever more expensive prior models that can lead to prohibitively slow decoding. Instead, we focus on more expressive transforms that result in a better rate-distortion-computation trade-off. Specifically, we show that nonlinear transforms built on Swin-transformers can achieve better compression efficiency than transforms built on convolutional neural networks (ConvNets), while requiring fewer parameters and shorter decoding time. Paired with a compute-efficient Channel-wise Auto-Regressive Model prior, our SwinT-ChARM model outperforms VTM-12.1 by $3.68\%$ in BD-rate on Kodak with comparable decoding speed. In P-frame video compression setting, we are able to outperform the popular ConvNet-based scale-space-flow model by $12.35\%$ in BD-rate on UVG. We provide model scaling studies to verify the computational efficiency of the proposed solutions and conduct several analyses to reveal the source of coding gain of transformers over ConvNets, including better spatial decorrelation, flexible effective receptive field, and more localized response of latent pixels during progressive decoding.
CrowdPlay: Crowdsourcing human demonstration data for offline learning in Atari games
Authors: Matthias Gerstgrasser, Rakshit Trivedi, David C. Parkes
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Atari Games, Decision Making, Imitation Learning, Offline RL, OpenAI Gym, Sequential Decision Making

Human demonstrations of video game play can serve as vital surrogate representations of real-world behaviors, access to which would facilitate rapid progress in several complex learning settings (e.g. behavior classification, imitation learning, offline RL etc.). The ability to build such human demonstration datasets is compelling as it alleviates the pitfalls associated with building simulators or synthetic data collection pipelines which are often impractical for real-world domains, either due to cost of building and running them or very large sim-to-real gap. Crowdsourcing has emerged as a standardized tool to collect useful datasets that require human participation. However, its applicability has been under-explored in the context of collecting human behaviors on sequential decision making tasks. To this end, we present CrowdPlay -- a complete crowdsourcing pipeline for OpenAI Gym and Gym-like MDP environments, a large-scale publicly available crowdsourced dataset of human gameplay demonstrations on single and multi-player Atari 2600 games, a collection of imitation and offline reinforcement learning benchmarks, and a detailed discussion of crowdsourcing and incentivization methodology. We hope that this will drive the improvement in design of algorithms that can account for the intricacies in the dataset and thereby, enable a step forward in direction of effective learning in real-world settings.
Continual Learning Using Pseudo-Replay via Latent Space Sampling
Authors: Gyuhak Kim, Sepideh Esmaeilpour, Zixuan Ke, Tatsuya Konishi, Bing Liu
Date: 2021-09-29 00:00:00
Tasks: class-incremental learning, Class Incremental Learning, Continual Learning, Incremental Learning

This paper investigates continual learning in the setting of class-incremental learning (CIL). Although numerous techniques have been proposed, CIL remains to be a highly challenging problem due to catastrophic forgetting (CF). However, so far few existing techniques have made use of pre-trained image feature extractors. In this paper, we propose to use a recently reported strong pre-trained feature extractor called CLIP and also propose a novel and yet simple pseudo-replay method to deal with CF. The proposed method is called PLS. Unlike the popular pseudo-replay approach that builds data generators to generate pseudo previous task data, PLS works in the latent space by sampling pseudo feature representations of previous tasks from the last layer of the pre-trained feature extractor. PLS is not only simple and efficient but also does not invade data privacy due to the fact that it works in the latent feature space. Experimental results show that the proposed method PLS outperforms state-of-the-art baselines by a large margin, where both PLS and the baselines leverage the CLIP pre-trained image feature extractor.
Trading Coverage for Precision: Conformal Prediction with Limited False Discoveries
Authors: Adam Fisch, Tal Schuster, Tommi S. Jaakkola, Regina Barzilay
Date: 2021-09-29 00:00:00
Tasks: Computational chemistry, Conformal Prediction, Drug Discovery, Prediction

In this paper, we develop a new approach to conformal prediction in which we aim to output a precise set of promising prediction candidates that is guaranteed to contain a limited number of incorrect answers. Standard conformal prediction provides the ability to adapt to model uncertainty by constructing a calibrated candidate set in place of a single prediction, with guarantees that the set contains the correct answer with high probability. In order to obey this coverage property, however, conformal sets can often become inundated with noisy candidates---which can render them unhelpful in practice. This is particularly relevant to large-scale settings where the cost (monetary or otherwise) of false positives is substantial, such as for in-silico screening for drug discovery, where any positively identified molecular compound is then manufactured and tested. We propose to trade coverage for precision by enforcing that the presence of incorrect candidates in the predicted conformal sets (i.e., the total number of false discoveries) is bounded according to a user-specified tolerance. Subject to this constraint, our algorithm then optimizes for a generalized notion of set coverage (i.e., the true discovery rate) that allows for any number of true answers for a given query (including zero). We demonstrate the effectiveness of this approach across a number of classification tasks in natural language processing, computer vision, and computational chemistry.
$f$-Divergence Thermodynamic Variational Objective: a Deformed Geometry Perspective
Authors: Jun Li, Ping Li
Date: 2021-09-29 00:00:00
Tasks: Variational Inference

In this paper, we propose a $f$-divergence Thermodynamic Variational Objective ($f$-TVO). $f$-TVO generalizes the Thermodynamic Variational Objective (TVO) by replacing Kullback–Leibler (KL) divergence with arbitary differeitiable $f$-divergence. In particular, $f$-TVO approximates dual function of model evidence $f^*(p(x))$ rather than the log model evidence $\log p(x)$ in TVO. $f$-TVO is derived from a deformed $\chi$-geometry perspective. By defining $\chi$-exponential family exponential, we are able to integral $f$-TVO along the $\chi$-path, which is the deformed geodesic between variational posterior distribution and true posterior distribution. Optimizing scheme of $f$-TVO includes reparameterization trick and Monte Carlo approximation. Experiments on VAE and Bayesian neural network show that the proposed $f$-TVO performs better than cooresponding baseline $f$-divergence variational inference.