pwc-archive/papers-with-abstracts huggingface dataset:
Constellation Nets for Few-Shot Learning
Authors: Weijian Xu, Yifan Xu, Huaijin Wang, Zhuowen Tu
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Clustering, Few-Shot Image Classification, Few-Shot Learning

The success of deep convolutional neural networks builds on top of the learning of effective convolution operations, capturing a hierarchy of structured features via filtering, activation, and pooling. However, the explicit structured features, e.g. object parts, are not expressive in the existing CNN frameworks. In this paper, we tackle the few-shot learning problem and make an effort to enhance structured features by expanding CNNs with a constellation model, which performs cell feature clustering and encoding with a dense part representation; the relationships among the cell features are further modeled by an attention mechanism. With the additional constellation branch to increase the awareness of object parts, our method is able to attain the advantages of the CNNs while making the overall internal representations more robust in the few-shot learning setting. Our approach attains a significant improvement over the existing methods in few-shot learning on the CIFAR-FS, FC100, and mini-ImageNet benchmarks.
Sparse Recovery via Bootstrapping: Collaborative or Independent?
Authors: Anonymous
Date: 2021-01-01 00:00:00
Tasks: Image Reconstruction, regression, Sparse Representation-based Classification

Sparse regression problems have traditionally been solved using all available measurements simultaneously. However, this approach fails in challenging scenarios such as when the noise level is high or there are missing data / adversarial samples. 
We propose JOBS (Joint-Sparse Optimization via Bootstrap Samples) -- a \emph{collaborative} sparse-regression framework on bootstrapped samples from the pool of available measurements via a joint-sparse constraint to ensure support consistency. In comparison to traditional bagging which solves sub-problems in an \emph{independent} fashion across bootstrapped samples, JOBS achieves state-of-the-art performance with the added advantage of having a sparser solution while requiring a lower number of observation samples.

Analysis of theoretical performance limits is employed to determine critical optimal parameters: the number of bootstrap samples $K$ and the number of elements $L$ in each bootstrap sample. Theoretical results indicate a better bound than Bagging (i.e. higher probability of achieving the same or better performance). Simulation results are used to validate this parameter selection. JOBS is robust to adversarial samples that fool the baseline method, as shown by better generalization in an image reconstruction task where the adversary has similar occlusions or alignment as the test sample. Furthermore, JOBS also improves discriminative performance in a facial recognition task in a sparse-representation-based classification setting.
Information-theoretic Probing Explains Reliance on Spurious Features
Authors: Charles Lovering, Rohan Jha, Tal Linzen, Ellie Pavlick
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1

Most current NLP systems are based on a pre-train-then-fine-tune paradigm, in which a large neural network is first trained in a self-supervised way designed to encourage the network to extract broadly-useful linguistic features, and then fine-tuned for a specific task of interest. Recent work attempts to understand why this recipe works and explain when it fails. Currently, such analyses have produced two sets of apparently-contradictory results. Work that analyzes the representations that result from pre-training (via "probing classifiers") finds evidence that rich features of linguistic structure can be decoded with high accuracy, but work that analyzes model behavior after fine-tuning (via "challenge sets") indicates that decisions are often not based on such structure but rather on spurious heuristics specific to the training set. In this work, we test the hypothesis that the extent to which a feature influences a model's decisions can be predicted using a combination of two factors: The feature's "extractability" after pre-training (measured using information-theoretic probing techniques), and the "evidence" available during fine-tuning (defined as the feature's co-occurrence rate with the label). In experiments with both synthetic and natural language data, we find strong evidence (statistically significant correlations) supporting this hypothesis.
A Theory of Self-Supervised Framework for Few-Shot Learning
Authors: Zhong Cao, Jiang Lu, Jian Liang, ChangShui Zhang
Date: 2021-01-01 00:00:00
Tasks: Few-Shot Learning, Self-Supervised Learning

Recently, self-supervised learning (SSL) algorithms have been applied to Few-shot learning(FSL). FSL aims at distilling transferable knowledge on existing classes with large-scale labeled data to cope with novel classes for which only a few labeled data are available. Due to the limited number of novel classes, the initial embedding network becomes an essential component and can largely affect the performance in practice. But almost no one analyzes why a pre-trained embedding network with self-supervised training can provide representation for downstream FSL tasks in theory. In this paper, we first summarized the supervised FSL methods and explained why SSL is suitable for FSL. Then we further analyzed the main difference between supervised training and self-supervised training on FSL and obtained the bound for the gap between self-supervised loss and supervised loss. Finally, we proposed potential ways to improve the test accuracy under the setting of self-supervised FSL.
HyperReal: Complex-Valued Layer Functions For Complex-Valued Scaling Invariance
Authors: Utkarsh Singhal, Yifei Xing, Stella Yu
Date: 2021-01-01 00:00:00

Complex-valued measurements in MRI and SAR imaging often have complex-valued scaling ambiguity, calling for models that are invariant to complex-valued scaling of pixels. Deep Complex Networks (DCN) extends real-valued algebra to complex-valued algebra in neural networks, but it does not address the issue of complex-valued scaling. SurReal complex-valued networks adopt a manifold view of complex numbers and derive a distance metric that is invariant to complex scaling. With distance features, it achieves complex-scaling invariance. However, rich complex-valued information is lost in this representation, and additionally,  SurReal is also prevented from using complex-valued non-linearity, limiting its expressive power. We simplify the manifold formulation of SurReal and propose a new layer function that achieves complex-scaling invariance within the complex domain.  We can then build hierarchical complex-valued features with complex-scaling invariance.  Our so-called HyperReal model results in a much leaner model with better generalization. Benchmarked on MSTAR, HyperReal beats DCN (and matches SurReal) with only 3%(40%) of their respective parameter counts.
Model-Free Counterfactual Credit Assignment
Authors: Thomas Mesnard, Theophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Tom Stepleton, Nicolas Heess, Marcus Hutter, Lars Holger Buesing, Remi Munos
Date: 2021-01-01 00:00:00
Tasks: counterfactual, model, valid

Credit assignment in reinforcement learning is the problem of measuring an action’s influence on future rewards. 
In particular, this requires separating \emph{skill} from \emph{luck}, ie.\ disentangling the effect of an action on rewards from that of external factors and subsequent actions. To achieve this, we adapt the notion of counterfactuals from causality theory to a model-free RL setup. 
The key idea is to condition value functions on \emph{future} events, by learning to extract relevant information from a trajectory. We then propose to use these as future-conditional baselines and critics in policy gradient algorithms and we develop a valid, practical variant with provably lower variance, while achieving unbiasedness by constraining the hindsight information not to contain information about the agent’s actions. We demonstrate the efficacy and validity of our algorithm on a number of illustrative problems.
FSV: Learning to Factorize Soft Value Function for Cooperative Multi-Agent Reinforcement Learning
Authors: Yueheng Li, Tianhao Zhang, Chen Wang, Jinan Sun, Shikun Zhang, Guangming Xie
Date: 2021-01-01 00:00:00
Tasks: Multi-agent Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Starcraft, Starcraft II

We explore energy-based solutions for cooperative multi-agent reinforcement learning (MARL) using the idea of function factorization in centralized training with decentralized execution (CTDE). Existing CTDE based factorization methods are susceptible to the relative overgeneralization, where finding a suboptimal Nash Equilibrium, which is a well-known game-theoretic pathology. To resolve this issue, we propose a novel factorization method for cooperative MARL, named FSV, which learns to factorize the joint soft value function into individual ones for decentralized execution. Theoretical analysis shows that FSV solves a rich class of factorization tasks. Our experiment for the well-known task of the Max of Two Quadratics game shows that FSV fully converges to global optima in the joint action space in the continuous tasks by local searching in the joint action space. We evaluate FSV on a challenging set of StarCraft II micromanagement tasks, and show that FSV significantly outperforms existing factorization multi-agent reinforcement learning methods.
Hellinger Distance Constrained Regression
Authors: Egor Rotinov
Date: 2021-01-01 00:00:00
Tasks: MuJoCo, regression, Reinforcement Learning (RL)

This paper introduces the off-policy reinforcement learning method that uses the Hellinger distance between sampling policy and current policy as a constraint. Hellinger distance squared multiplied by two is greater than or equal to total variation distance squared and less than or equal to Kullback-Leibler divergence, therefore lower bound for expected discounted return for the new policy is improved comparing to KL. Also, Hellinger distance is less than or equal to 1, so there is a policy-independent lower bound for expected discounted return. HDCR is capable of training with Experience Replay, a common setting for distributed RL when collecting trajectories using different policies and learning from this data centralized. HDCR shows results comparable to or better than Advantage-weighted Behavior Model and Advantage-Weighted Regression on MuJoCo tasks using offline datasets collected by random agents and datasets obtained during the first iterations of online training of HDCR agent.
Collaborative Filtering with Smooth Reconstruction of the Preference Function
Authors: Ali Shirali, Reza Kazemi, Arash Amini
Date: 2021-01-01 00:00:00
Tasks: Collaborative Filtering, Quantization, Recommendation Systems, Representation Learning

The problem of predicting the rating of a set of users to a set of items in a recommender system based on partial knowledge of the ratings is widely known as collaborative filtering. In this paper, we consider a mapping of the items into a vector space and study the prediction problem by assuming an underlying smooth preference function for each user, the quantization at each given vector yields the associated rating. To estimate the preference functions, we implicitly cluster the users with similar ratings to form dominant types. Next, we associate each dominant type with a smooth preference function; i.e., the function values for items with nearby vectors shall be close to each other. 
The latter is accomplished by a rich representation learning in a so-called frequency domain. In this framework, we propose two approaches for learning user and item representations. First, we use an alternating optimization method in the spirit of $k$-means to cluster users and map items. We further make this approach less prone to overfitting by a boosting technique. 
Second, we present a feedforward neural network architecture consisting of interpretable layers which implicitely clusters the users. The performance of the method is evaluated on two benchmark datasets (ML-100k and ML-1M). Albeit the method benefits from simplicity, it shows a remarkable performance and opens a venue for future research. All codes are publicly available on the GitLab.
Learned residual Gerchberg-Saxton network for computer generated holography
Authors: Lennart Schlieder, Heiner Kremer, Valentin Volchkov, Kai Melde, Peer Fischer, Bernhard Schölkopf
Date: 2021-01-01 00:00:00

Computer generated holography (CGH) aims to generate phase plates that create an intensity pattern at a certain distance behind the holography plate when illuminated. Since only the intensity and not the phase of the wave is of interest, this is an ill-defined inverse problem. Usually these problems are tackled by iterative optimization algorithms which are part of the convex optimization framework. These algorithms essentially minimize a loss using a forward model. Even though many of the tackled inverse problems are non-convex, these algorithms reach acceptable solutions by finding a local minimum. The ability of Deep Neural Networks to estimate a large range of functions has made a different approach to these problems possible. Instead of an iterative optimization algorithm that converges to a (sub-)optimal solution, the inverse problem can be solved by training a neural network to directly estimate the inverse operator. However simple convolutional neural networks tend to overfit when learning the inverse operator and do not generalize well outside the training distribution. Therefore this paper introduces a hybrid approach that can be interpreted as an unrolled Gerchberg-Saxton algorithm, which we term Learned Residual Gerchberg-Saxton (LRGS) network. We train this network for the generation of multi-focus computer generated holograms, and beat state-of-the-art existing methods.
SCoRe: Pre-Training for Context Representation in Conversational Semantic Parsing
Authors: Tao Yu, Rui Zhang, Alex Polozov, Christopher Meek, Ahmed Hassan Awadallah
Date: 2021-01-01 00:00:00
Proceeding: NeurIPS Workshop CAP 2020 12
Tasks: Dialogue State Tracking, Language Modeling, Language Modelling, Multi-domain Dialogue State Tracking, Natural Language Queries, Semantic Parsing, Text-To-SQL

Conversational Semantic Parsing (CSP) is the task of converting a sequence of natural language queries to formal language (e.g., SQL, SPARQL) that can be executed against a structured ontology (e.g.  databases, knowledge bases).  To accomplish  this  task,  a  CSP  system  needs  to  model  the  relation  between  the unstructured language utterance and the structured ontology while representing the multi-turn dynamics of the dialog. Pre-trained language models (LMs) are the state-of-the-art for various natural language processing tasks. However, existing pre-trained LMs that use language modeling training objectives over free-form text have limited ability to represent natural language references to contextual structural data. In this work, we present SCORE, a new pre-training approach for CSP tasks designed to induce representations that capture the alignment between the dialogue flow and the structural context. We demonstrate the broad applicability of SCORE to CSP tasks by combining SCORE with strong base systems on four different tasks (SPARC, COSQL, MWOZ, and SQA). We show that SCORE can improve the performance over all these base systems by a significant margin and achieves state-of-the-art results on three of them. Our implementation and checkpoints of the model will be available at Anonymous URL.
High-Likelihood Area Matters --- Rewarding Near-Correct Predictions Under Imbalanced Distributions
Authors: Guangxiang Zhao, Lei LI, Xuancheng Ren, Xu sun, Bin He
Date: 2021-01-01 00:00:00
Tasks: Vocal Bursts Intensity Prediction

Learning from natural datasets poses significant challenges for traditional classification methods based on the cross-entropy objective due to imbalanced class distributions, in particular, the long-tailed class distributions. It is intuitive to assume that the examples from tail classes are harder to learn so that the systems should be uncertain of the prediction, where the low-likelihood area establishes and the systems are driven more actively to predict correctly. However, this assumption is one-sided and could be misleading. We find in practice that the high-likelihood area contains correct predictions for tail classes and it plays a vital role in learning imbalanced class distributions. In light of this finding, we propose the encourage loss, which rewards the system when the examples belonging to tailed classes in the high-likelihood area are correctly predicted. In contrast to traditional methods that focus on predicting right or wrong, the encourage loss puts weights on strengthening the correct predictions. Experiments on the large-scale long-tailed iNaturalist 2018 classification dataset, and the ImageNet-LT benchmark both validate the proposed approach. We further analyze in detail the influence of the encourage loss on diverse data distributions, including both computer vision and natural language processing tasks.
Composite Adversarial Training for Multiple Adversarial Perturbations and Beyond
Authors: Xinyang Zhang, Zheng Zhang, Ting Wang
Date: 2021-01-01 00:00:00

One intriguing property of deep neural networks (DNNs) is their vulnerability to adversarial perturbations. Despite the plethora of work on defending against individual perturbation models, improving DNN robustness against the combinations of multiple perturbations is still fairly under-studied. In this paper, we propose \underline{c}omposite \underline{a}dversarial \underline{t}raining (CAT), a novel training method that flexibly integrates and optimizes multiple adversarial losses, leading to significant robustness improvement with respect to individual perturbations as well as their ``compositions''. Through empirical evaluation on benchmark datasets and models, we show that CAT outperforms existing adversarial training methods by large margins in defending against the compositions of pixel perturbations and spatial transformations, two major classes of adversarial perturbation models, while incurring limited impact on clean inputs.
Learning Representation in Colour Conversion
Authors: Arash Akbarinia, Raquel Gil-Rodriguez, Alban Flachot, Matteo Toscani
Date: 2021-01-01 00:00:00
Tasks: image-classification, Image Classification, Scene Segmentation

Colours can be represented in an infinite set of spaces highlighting distinct features. In this work, we study the structure of colour representation in variational autoencoders (VAEs) and investigate whether a specific organisation of colours yields higher encoding efficiency. To this end, we propose a novel unsupervised task: colour space conversion (ColourConvNets). We trained several instances of VAEs whose input and output are in different colour spaces, e.g. from RGB to CIE L*a*b* (in total five colour spaces were examined). This allows us to systematically study the influence of input-output colour spaces on the representation learnt in VAEs. We thoroughly analysed the finite embedding space of vector quantised VAEs with three different methods (single feature, hue shift and linear transformation). The interpretations reached with these techniques are in agreement suggesting that (i) luminance and chromatic information are encoded in separate embedding vectors, and (ii) the structure of network's embedding space is determined by the output colour space. Evaluation of a large number of networks demonstrates that ColourConvNets with decorrelated output colour spaces produce higher quality images with a lower pixel-wise colour difference (1-2 DeltaE).  We further assess the ColourConvNets capacity in reconstructing the global content of an image in two downstream tasks: image classification (ImageNet) and scene segmentation networks (COCO). Our results show, with respect to the baseline network (whose input and output are RGB) 5-10% higher classification accuracy is obtained with decorrelating ColourConvNets.
Gradient-based tuning of Hamiltonian Monte Carlo hyperparameters
Authors: Andrew Campbell, Wenlong Chen, Vincent Stimper, José Miguel Hernández-Lobato, Yichuan Zhang
Date: 2021-01-01 00:00:00

Hamiltonian Monte Carlo (HMC) is one of the most successful sampling methods in machine learning. However, its performance is significantly affected by the choice of hyperparameter values, which require careful tuning. Existing approaches for automating this task either optimise a proxy for mixing speed or consider the HMC chain as an implicit variational distribution and optimize a tractable lower bound that is too loose to be useful in practice. Instead, we propose to optimize an objective that quantifies directly the speed of convergence to the target distribution. Our objective can be easily optimized using stochastic gradient descent.  We evaluate our proposed method and compare to baselines on a variety of problems including synthetic 2D distributions, the posteriors of variational autoencoders and the Boltzmann distribution for molecular configurations of a 22 atom molecule. We find our method is competitive with or improves upon alternative baselines on all problems we consider.
Representation and Bias in Multilingual NLP: Insights from Controlled Experiments on Conditional Language Modeling
Authors: Ada Wan
Date: 2021-01-01 00:00:00
Tasks: Fairness, Language Modeling, Language Modelling, Machine Translation, Multilingual NLP

Inspired by the phenomenon of performance disparity between languages in machine translation, we investigate whether and to what extent languages are equally hard to "conditional-language-model". Our goal is to improve our understanding and expectation of the relationship between language, data representation, size, and performance in one-to-one conditional language modeling through a series of systematically controlled experiments with the Transformer and parallel data on the 6 diverse, official languages of the United Nations --- in 30 directions, 5 sizes, and 3 primary representation types in character, byte, and word, along with 5 alternate variants for a secondary set of controls. We observe indications suggesting a script bias on the character level, a length bias on the byte level, and a word bias that gives rise to a hierarchy in performance across languages. We also identify two types of sample-wise non-monotonicity --- while word-based representations are prone to exhibit Double Descent, length can induce unstable performance across the size range studied in a novel meta phenomenon which we term "erraticity". By eliminating statistically significant performance disparity on the character and byte levels, we show that, in the context of computing with the Transformer, there is no complexity intrinsic to languages other than that related to their statistical attributes and that performance disparity is not a necessary condition but a byproduct of word segmentation. Our application of statistical comparisons as a fairness measure also serves as a novel rigorous method for the intrinsic evaluation of languages, resolving a decades-long debate on language complexity. We hope our work helps open up new directions in the area of language and computing that would be fairer and more flexible.