pwc-archive/papers-with-abstracts huggingface dataset:
Generalization to Out-of-Distribution transformations
Authors: Shanka Subhra Mondal, Zack Dulberg, Jonathan Cohen
Date: 2021-09-29 00:00:00
Tasks: Diversity, Translation

Humans understand a set of canonical geometric transformations (such as translation, rotation and scaling) that support generalization by being untethered to any
specific object. We explored inductive biases that allowed artificial neural networks to learn these transformations in pixel space in a way that could generalize
out-of-distribution (OOD). Unsurprisingly, we found that convolution and high
training diversity were important contributing factors to OOD generalization of
translation to untrained shapes, sizes, time-points and locations, however these
weren’t sufficient for rotation and scaling. To remedy this we show that two more
principle components are needed 1) iterative training where outputs are fed back
as inputs 2) applying convolutions after conversion to log-polar space. We propose POLARAE which exploits all four components and outperforms standard
convolutional autoencoders and variational autoencoders trained iteratively with
high diversity wrt OOD generalization to larger shapes in larger grids and new
locations.
HydraSum - Disentangling Stylistic Features in Text Summarization using Multi-Decoder Models
Authors: Tanya Goyal, Nazneen Rajani, Wenhao Liu, Wojciech Maciej Kryscinski
Date: 2021-09-29 00:00:00
Tasks: Abstractive Text Summarization, Decoder, Mixture-of-Experts, Specificity, Text Summarization

Existing abstractive summarization models lack explicit control mechanisms that would allow users to influence the stylistic features of the model outputs. This results in generating generic summaries that do not cater to the users needs or preferences. To address this issue we introduce HydraSum, a new summarization architecture that extends the single decoder framework of current models, e.g. BART, to a mixture-of-experts version consisting of multiple decoders. Our proposed model encourages each expert, i.e. decoder, to learn and generate stylistically-distinct summaries along dimensions such as abstractiveness, length, specificity, and others. At each time step, HydraSum employs a gating mechanism that decides the contribution of each individual decoder to the next token's output probability distribution. Through experiments on three summarization datasets (CNN, Newsroom, XSum), we demonstrate that this gating mechanism automatically learns to assign contrasting summary styles to different HydraSum decoders under the standard training objective without the need for additional supervision. We further show that a guided version of the training process can explicitly govern which summary style is partitioned between decoders, e.g. high abstractiveness vs. low abstractiveness or high specificity vs. low specificity, and also increase the stylistic-difference between individual decoders. Finally, our experiments demonstrate that our decoder framework is highly flexible: during inference, we can sample from individual decoders or mixtures of different subsets of the decoders to yield a diverse set of summaries and enforce single- and multi-style control over summary generation.
Variational Inference for Discriminative Learning with Generative Modeling of Feature Incompletion
Authors: Kohei Miyaguchi, Takayuki Katsuki, Akira Koseki, Toshiya Iwamori
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Imputation, Missing Values, regression, Variational Inference

We are concerned with the problem of distributional prediction with incomplete features: The goal is to estimate the distribution of target variables given feature vectors with some of the elements missing. A typical approach to this problem is to perform missing-value imputation and regression, simultaneously or sequentially, which we call the generative approach. Another approach is to perform regression after appropriately encoding missing values into the feature, which we call the discriminative approach. In comparison, the generative approach is more robust to the feature corruption while the discriminative approach is more favorable to maximize the performance of prediction. 
In this study, we propose a hybrid method to take the best of both worlds. Our method utilizes the black-box variational inference framework so that it can be applied to a wide variety of modern machine learning models, including the variational autoencoders. We also confirmed the effectiveness of the proposed method empirically.
Learning Invariant Reward Functions through Trajectory Interventions
Authors: Ivan Ovinnikov, Eugene Bykovets, Joachim M. Buhmann
Date: 2021-09-29 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Inverse reinforcement learning methods aim to retrieve the reward function of
a Markov decision process based on a dataset of expert demonstrations. The
commonplace scarcity of such demonstrations potentially leads to the absorption of
spurious correlations in the data by the learning model, which as a result, exhibits
behavioural overfitting to the expert dataset when trained on the obtained reward
function. We study the generalization properties of the maximum entropy method
for solving the inverse reinforcement learning problem for both exact and approximate
formulations and demonstrate that by applying an instantiation of the invariant
risk minimization principle, we can recover reward functions which induce better
performing policies across domains in the transfer setting.
Learning Context-Adapted Video-Text Retrieval by Attending to User Comments
Authors: Laura Hanu, Yuki M Asano, James Thewlis, Christian Rupprecht
Date: 2021-09-29 00:00:00
Tasks: Retrieval, Text Retrieval, Video-Text Retrieval

Learning strong representations for multi-modal retrieval is an important problem for many applications, such as recommendation and search. Current benchmarks and even datasets are often manually constructed and consist of mostly clean samples where all modalities are well-correlated with the content. Thus, current video-text retrieval literature largely focuses on video titles or audio transcripts, while ignoring user comments, since users often tend to discuss topics only vaguely related to the video.
In this paper we present a novel method that learns meaningful representations from videos, titles and comments, which are abundant on the internet. Due to the nature of user comments, we introduce an attention-based mechanism that allows the model to disregard text with irrelevant content. 
In our experiments, we demonstrate that, by using comments, our method is able to learn better, more contextualised, representations, while also achieving competitive results on standard video-text retrieval benchmarks.
Fast and Sample-Efficient Domain Adaptation for Autoencoder-Based End-to-End Communication
Authors: Jayaram Raghuram, Yijing Zeng, Dolores Garcia, Somesh Jha, Suman Banerjee, Joerg Widmer, Rafael Ruiz
Date: 2021-09-29 00:00:00
Tasks: Decoder, Domain Adaptation

The problem of domain adaptation conventionally considers the setting where a source domain has plenty of labeled data, and a target domain (with a different data distribution) has plenty of unlabeled data but none or very limited labeled data. In this paper, we address the setting where the target domain has only limited labeled data from a distribution that is expected to change frequently. We first propose a fast and light-weight method for adapting a Gaussian mixture density network (MDN) using only a small set of target domain samples. This method is well-suited for the setting where the distribution of target data changes rapidly (e.g., a wireless channel), making it challenging to collect a large number of samples and retrain. We then apply the proposed MDN adaptation method to the problem of end-of-end learning of a communication autoencoder, which jointly learns the encoder, decoder, and a channel networks to minimize the decoding error rate. However, the error rate of an autoencoder trained on a particular (source) channel distribution can degrade as the channel distribution changes frequently, not allowing enough time for data collection and retraining of the autoencoder to the target channel distribution. We propose a method for adapting the autoencoder without modifying the encoder and decoder neural networks, and adapting only the MDN model of the channel. The method utilizes feature transformations at the decoder to compensate for changes in the channel distribution, and effectively present to the decoder samples close to the source distribution. Experimental evaluation on simulated datasets and real mmWave wireless channels demonstrate that the proposed methods can adapt the MDN model using very limited number of samples, and improve or maintain the error rate of the autoencoder under changing channel conditions.
FEATURE-AUGMENTED HYPERGRAPH NEURAL NETWORKS
Authors: Xueqi Ma, Pan Li, Qiong Cao, James Bailey, Yue Gao
Date: 2021-09-29 00:00:00
Tasks: Node Classification, Representation Learning

Graph neural networks (GNNs) and their variants have demonstrated superior performance in learning graph representations by aggregating features based on graph or hypergraph structures. However, it is becoming evident that most exist- ing graph-based GNNs are susceptible to over-smoothing and are non-robust to perturbations. For representation learning tasks, hypergraphs usually have more expressive power than graphs through their ability to encode higher-order data correlations. In this paper, we propose Feature-Augmented Hypergraph Neural Networks (FAHGNN) focusing on hypergraph structures. In FAHGNN, we explore the influence of node features for the expressive power of GNNs and augment features by introducing common features and personal features to model information. Specifically, for a node, the common features contain the shared information with other nodes in hyperedges, while the personal features represent its special information. In this way, the feature types each possess different distinguishing powers. Considering the different properties of these two kinds of features, we design different propagation strategies for information aggregation on hypergraphs. Furthermore, during the propagation process, we further augment features by randomly dropping node features. We leverage consistency regularization across different data augmentations of the two feature types to optimize the prediction consistency for the model. Extensive experiments on several benchmarks show that FAHGNN significantly outperforms other state-of-the-art methods for node classification tasks. Our theoretical study and experimental results further support the effectiveness of FAHGNN for mitigating issues of over-smoothing and enhancing the robustness of the model.
Coherent and Consistent Relational Transfer Learning with Autoencoders
Authors: Harald Stromfelt, Luke Dickens, Artur Garcez, Alessandra Russo
Date: 2021-09-29 00:00:00
Tasks: Decoder, Relation, Transfer Learning

Human defined concepts are inherently transferable, but it is not clear under what conditions they can be modelled effectively by non-symbolic artificial learners.
This paper argues that for a transferable concept to be learned, the system of relations that define it must be coherent across domains.
This is to say that the learned concept-specific relations ought to be consistent with respect to a theory that constrains their semantics and that such consistency must extend beyond the representations encountered in the source domain.
To demonstrate this, we first present formal definitions for consistency and coherence, and a proposed Dynamic Comparator relation-decoder model designed around these principles. 
We then perform a proposed Partial Relation Transfer learning task on a novel data set, using a neural-symbolic autoencoder architecture that combines sub-symbolic representations with modular relation-decoders.
By comparing against several existing relation-decoder models, our experiments show that relation-decoders which maintain consistency over unobserved regions of representational space retain coherence across domains, whilst achieving better transfer learning performance.
Revisit Kernel Pruning with Lottery Regulated Grouped Convolutions
Authors: Shaochen Zhong, Guanqun Zhang, Ningjia Huang, Shuai Xu
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Clustering, Network Pruning

Structured pruning methods which are capable of delivering a densely pruned network are among the most popular techniques in the realm of neural network pruning, where most methods prune the original network at a filter or layer level. Although such methods may provide immediate compression and acceleration benefits, we argue that the blanket removal of an entire filter or layer may result in undesired accuracy loss. In this paper, we revisit the idea of kernel pruning (to only prune one or several $k \times k$ kernels out of a 3D-filter), a heavily overlooked approach under the context of structured pruning due to it will naturally introduce sparsity to filters within the same convolutional layer—thus, making the remaining network no longer dense. We address this problem by proposing a versatile grouped pruning framework where we first cluster filters from each convolutional layer into equal-sized groups, prune the grouped kernels we deem unimportant from each filter group, then permute the remaining filters to form a densely grouped convolutional architecture (which also enables the parallel computing capability) for fine-tuning. Specifically, we consult empirical findings from a series of literature regarding $\textit{Lottery Ticket Hypothesis}$ to determine the optimal clustering scheme per layer, and develop a simple yet cost-efficient greedy approximation algorithm to determine which group kernels to keep within each filter group. Furthermore, we discuss how this pruning framework will potentially work with more advanced unsupervised clustering schemes and inductive biases on weights shifting discovered in the future. Extensive experiments also demonstrate our method often outperforms comparable SOTA methods with lesser data argumentation needed, smaller fine-tuning budget required, and sometimes even with much simpler procedure executed (e.g., one-shot pruning v. iterative pruning, standard fine-tuning v. custom fine-tuning).
Less data is more: Selecting informative and diverse subsets with balancing constraints
Authors: Srikumar Ramalingam, Daniel Glasner, Kaushal Patel, Raviteja Vemulapalli, Sadeep Jayasumana, Sanjiv Kumar
Date: 2021-09-29 00:00:00
Tasks: Diversity

Deep learning has yielded extraordinary results in vision and natural language processing, but this achievement comes at a cost. Most models require enormous resources during training, both in terms of computation and in human labeling effort. We show that we can identify informative and diverse subsets of data that lead to deep learning models with similar performance as the ones trained with the original dataset. Prior methods have exploited diversity and uncertainty in submodular objective functions for choosing subsets. In addition to these measures, we show that balancing constraints on predicted class labels and decision boundaries are beneficial. We propose a novel formulation of these constraints using matroids, an algebraic structure that generalizes linear independence in vector spaces, and present an efficient greedy algorithm with constant approximation guarantees. We outperform competing baselines on standard classification datasets such as CIFAR-10, CIFAR-100, ImageNet, as well as long-tailed datasets such as CIFAR-100-LT.
Revisiting the Lottery Ticket Hypothesis: A Ramanujan Graph Perspective
Authors: BITHIKA PAL, Arindam Biswas, Pabitra Mitra, BISWAJIT BASU
Date: 2021-09-29 00:00:00

Neural networks often yield to weight pruning resulting in a sparse subnetwork that is adequate for a given task. Retraining these `lottery ticket' subnetworks from their initialization minimizes the computational burden while preserving the test set accuracy of the original network. Based on our knowledge, the existing literature only confirms that pruning is needed and it can be achieved up to certain sparsity. We analyze the pruned network in the context of the properties of Ramanujan expander graphs. We consider the feed-forward network (both multi-layer perceptron and convolutional network) as a series of bipartite graphs which establish the connection from input to output. Now, as the fraction of remaining weights reduce with increasingly aggressive pruning two distinct regimes are observed: initially, no significant decrease in accuracy is demonstrated, and then the accuracy starts dropping rapidly. We empirically show that in the first regime the pruned lottery ticket sub-network remains a Ramanujan graph. Subsequently, with the loss of Ramanujan graph property, accuracy begins to reduce sharply. This characterizes an absence of resilient connectivity in the pruned sub-network. We also propose a new magnitude-based pruning algorithm to preserve the above property. We perform experiments on MNIST and CIFAR10 datasets using different established feed-forward architectures and show that the winning ticket obtained from the proposed algorithm is much more robust.
Reachability Traces for Curriculum Design in Reinforcement Learning
Authors: Thommen Karimpanal George, Majid Abdolshah, Hung Le, Santu Rana, Sunil Gupta, Truyen Tran, Svetha Venkatesh
Date: 2021-09-29 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

The objective in goal-based reinforcement learning is to learn a policy to reach a particular goal state within the environment. However, the underlying reward function may be too sparse for the agent to efficiently learn useful behaviors. Recent studies have demonstrated that reward sparsity can be overcome by instead learning a curriculum of simpler subtasks. In this work, we design an agent's curriculum by focusing on the aspect of goal reachability, and introduce the idea of a reachability trace, which is used as a basis to determine a sequence of intermediate subgoals to guide the agent towards its primary goal. We discuss several properties of the trace function, and in addition, validate our proposed approach empirically in a range of environments, while comparing its performance against appropriate baselines.
Bridging Recommendation and Marketing via Recurrent Intensity Modeling
Authors: Yifei Ma, Ge Liu, Anoop Deoras
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: Marketing

Personalized item-recommendation (ItemRec) and targeted item-marketing are two sides of the same coin. The key in the latter problem, which we call UserRec, is to recommend users in their temporal contexts on behalf of the item providers. As ItemRec progresses to using end-to-end sequence models which directly ingest the item-consumption history of each user to build real-time hidden states, we study whether we can turn these models around to recommend users in their real-time contexts. Our main challenge is to compare these item-consumption sequences of variable lengths and duration. To overcome it, we discover a novel connection to Temporal Point Process (TPP), where we use its intensity parameter to predict the user's likelihood of future engagements, based on but not biased by the past observations. We develop a Recurrent Intensity Modeling (RIM) trick to convert sequential ItemRec models to UserRec based on Marked-TPP decomposition. RIM allows us to rethink recommendation in a Matching (Mtch) scenario, where the benefits of the users (e.g., ItemRec relevance) and item providers (e.g., item-exposure guarantees) are considered at the same time. Notably, as RIM is based on sequence models, we extend to Online Matching (OnlnMtch) to serve ItemRec in real-time while satisfying the item guarantees by approximate ratios in the long-term, borrowing techniques from online dual decomposition in convex optimization. Finally, to complement our discussions on RIM and Mtch, we provide empirical simulations on real-world datasets with open-source codes.
How Did the Model Change? Efficiently Assessing Machine Learning API Shifts
Authors: Lingjiao Chen, Matei Zaharia, James Zou
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: BIG-bench Machine Learning

ML prediction APIs from providers like Amazon and Google have made it simple to use ML in applications. A challenge for users is that such APIs continuously change over time as the providers update models, and changes can happen silently without users knowing. It is thus important to monitor when and how much the MLAPIs’ performance shifts. To provide detailed change assessment, we model MLAPI shifts as confusion matrix differences, and propose a principled algorithmic framework, MASA, to provably assess these shifts efficiently given a sample budget constraint.MASAemploys an upper-confidence bound based approach to adaptively determine on which data point to query the ML API to estimate shifts. Empirically, we observe significant ML API shifts from 2020 to 2021 among 12 out of 36 applications using commercial APIs from Google, Microsoft, Amazon, and other providers. These real-world shifts include both improvements and reductions in accuracy. Extensive experiments show that MASA can estimate such API shifts more accurately than standard approaches given the same budget
Feature Kernel Distillation
Authors: Bobby He, Mete Ozay
Date: 2021-09-29 00:00:00
Proceeding: ICLR 2022 4
Tasks: image-classification, Image Classification, Knowledge Distillation

Trained Neural Networks (NNs) can be viewed as data-dependent kernel machines, with predictions determined by the inner product of last-layer representations across inputs, referred to as the feature kernel. We explore the relevance of the feature kernel for Knowledge Distillation (KD), using a mechanistic understanding of an NN's optimisation process. We extend the theoretical analysis of Allen-Zhu & Li (2020) to show that a trained NN's feature kernel is highly dependent on its parameter initialisation, which biases different initialisations of the same architecture to learn different data attributes in a multi-view data setting. This enables us to prove that KD using only pairwise feature kernel comparisons can improve NN performance in such settings, with both single & ensemble teacher models, whereas standard training without KD fails to generalise. We further use our theory to motivate practical considerations for improving student generalisation when using distillation with feature kernels, which allows us to propose a novel approach: Feature Kernel Distillation (FKD). Finally, we experimentally corroborate our theory in the image classification setting, showing that FKD is amenable to ensemble distillation, can transfer knowledge across datasets, and outperforms both vanilla KD & other feature kernel based KD baselines across a range of architectures & datasets.