pwc-archive/papers-with-abstracts huggingface dataset:
Physics-Aware Difference Graph Networks for Sparsely-Observed Dynamics
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5

Sparsely available data points cause numerical error on finite differences which hinders us from modeling the dynamics of physical systems. The discretization error becomes even larger when the sparse data are irregularly distributed or defined on an unstructured grid, making it hard to build deep learning models to handle physics-governing observations on the unstructured grid. In this paper, we propose a novel architecture, Physics-aware Difference Graph Networks (PA-DGN), which exploits neighboring information to learn finite differences inspired by physics equations. PA-DGN leverages data-driven end-to-end learning to discoverunderlying dynamical relations between the spatial and temporal differences in given sequential observations. We demonstrate the superiority of PA-DGN in theapproximation of directional derivatives and the prediction of graph signals on the synthetic data and the real-world climate observations from weather stations.
Diffusion Models Using a Single Equation
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Denoising

In this work I present a novel viewpoint and a simplistic implementation and explanation of denoising diffusion models, and also the intuition that we force these models to sample from the data distribution by misleading them. The aim of this blogpost is to lower the barrier of entry to the field of diffusion models, by providing an explanation of their inner workings that is mathematically very light (only uses a single equation, with carefully meaningfully named variables), and try to understand them from a different point of view compared to previous work. I also discuss new intuitions about these models, and provide an end-to-end, simple to use implementation for diffusion models, that is easy to customize and is intended be a good starting point for future projects on this subject.
Evaluation of Feature-based explanations
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Attribute, Feature Importance

This blog post describes contribution of the paper titled "EVALUATIONS AND METHODS FOR EXPLANATION THROUGH ROBUSTNESS ANALYSIS" by [Cheng et. al](https://openreview.net/forum?id=Hye4KeSYDr) that discusses assessing robustness in a novel way and coming up with more robust explanation in the specific area of insertion and removal of explanations. This is because such explanations face two drawbacks:
1.	 When the feature importance is estimated by removing a feature by setting it to a baseline value, it has higher chance to attribute high importance if some values deflect a lot from baseline. An example would be setting RGB pixels to black, that will give bright pixels more importance.
2.	When the feature importance is estimated by removing a feature by giving it some value sampled from the distribution (using a generative model) , there is an inherent bias that goes from the generative model to this process and not all domains can have a proper generative models.
MODALS: Data augmentation that works for everyone
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Data Augmentation

The usefulness of data augmentation has led to the development of specific techniques of augmentation unique to each modality of data. The techniques developed for one modality usually suit the type of data in that particular modality. For image data some commonly used augmentation techniques are rotation, cropping, applying affine transforms, random flips, contrast and color augmentations and adding Gaussian blur. More recent techniques like CutMix[3] and MixUp[4] apply data augmentation on both the image and label space. Many robust data augmentation techniques for image data already exist and are used widely. However modalities like tables and graph data don’t have as many robust augmentation techniques. Since the techniques developed for images were made taking into consideration the nature of image data, they usually cannot be applied to other modalities. If a generalized, modality agnostic framework for augmentation could be developed then standard, robust augmentation techniques can be applied across many modalities. This is exactly what the authors of the paper propose.
On Dyadic Fairness: Exploring and Mitigating Bias in Graph Connections
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Fairness, Link Prediction, Prediction

This blog post discusses the ICLR 2021 paper "On Dyadic Fairness: Exploring and Mitigating Bias in Graph Connections" by Li et al., highlighting the importance of its theoretical results while critiquing the notions and applications of dyadic fairness provided. This paper presents a beautiful proof that can be followed to analyze representation disparities produced by various message-passing algorithms, and an algorithmic skeleton for improving the fairness of many message-passing algorithms. At the same time, it is essential that, as a community, we critically analyze for which applications a fair link prediction algorithm can distribute justice and contextualize our understandings of the politics and limitations of different notions of fairness in link prediction applications.
Do KG-augmented Models Leverage Knowledge as Humans Do?
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Knowledge Graphs, Question Answering, Recommendation Systems

Knowledge Graphs (KGs) can help neural-symbolic models to improve performance on various knowledge-intensive tasks, like recommendation systems and question answering. Concretely, neural reasoning over KGs may "explain" which information is relevant for inference. However, as an old saying goes, "seeing is not believing," it is natural to ask the question, "do KG-augmented models really behave as we expect？" This post presents the historical perspectives of KG-augmented models and discusses a recent work raising this question. Interestingly, empirical results demonstrate that perturbed KGs can maintain the downstream performance, which subvert our cognition over KG-augmented models' ability. We believe this topic is necessary and important for neural-symbolic reasoning and can guide future work on designing KG-augmented models.
On The Effects of Learning Views on Neural Representations in Self-Supervised Learning
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Representation Learning, Self-Supervised Learning

Contrastive self-supervised learning (SSL) methods require domain expertise to select a set of data transformations that work well on a specific downstream task, modality, and domain. This creates a bottleneck for the utilization of SSL strategies on new datasets, particularly those sampled from different domains. This blog post details the Viewmaker Networks, a method that attempts to overcome this issue by using constrained perturbations and an adversarial training objective to synthesize novel views. This method can be broadly extended to many domains, as it learns views directly from the training data. We cover the details of the methodology and related concepts as they apply to contrastive visual representation learning. This blog post also develops further insights into the visual representations learned using the Viewmaker's augmented views by conducting additional experiments. Specifically, we investigate the dimensional collapse and representational similarity between differently trained models (handcrafted views and viewmaker views). We observe that training models using Viewmaker views not only make better use of the embedding space but also learn a different function of the data when compared to training models using handcrafted views, something one should carefully consider when choosing one over the other. We feel these observations will encourage further discussions on learned views in self-supervised learning.
Are CNNs biased towards texture rather than object shape?
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Face Recognition, Object

Although we are seeing so many exciting research papers with advancements in CNN architectures and their application domains, we still have little to no understanding as to why these systems decide as they do. That’s why we consider these systems ‘black box', we don’t know the ‘reasoning’ behind a particular decision. And such behaviors cannot be overlooked just because they are scoring high on predefined metrics. For example, the Gender Shapes project shows that various face recognition systems perform worse on minority classes(accuracy difference of up to 34% between lighter-skinned males and darker-skinned females). Now if such systems are used for law enforcement, airport, or employment screenings, this bias can have major repercussions. This highlights the importance of ‘explainability’ in computer vision systems.

‘Adversarial attacks’ demonstrates one such counter-intuitive behavior of CNNs. These examples are specially devised to fool the CNNs into predicting wrong labels, just by altering the image by a noise indistinguishable to the human eye.

One such behavior is captured by the paper ‘ImageNet-trained CNNs are biased towards texture’. Let’s dive in…
Why Does DARTS Miss the Target, and How Do We Aim to Fix It?
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Network Pruning

This blog post is based on 'Rethinking Architecture Selection in Differentiable NAS' from ICLR 2021 (Wang et al., 2021). The post establishes context by explaining DARTS (Liu et al., 2019) and summarizes an analysis of the failure modes of DARTS (Zela et al., 2020) before returning to focus on the main work. The post compares potential causes for failure of DARTS presented by (Zela et al., 2020) and (Wang et al., 2021) before describing the perturbation algorithm presented in (Wang et al., 2021) in order to help remedy failings of DARTS. Finally, the post reflects on the perturbation algorithm's downside as a step away from differentiable NAS (and toward discrete NAS) and relates the supernet-based pruning of DARTS to broader literature in network pruning.
Does Adam Converge and When?
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5

In this blog post, we revisit the (non-)convergence behavior of Adam. Especially, we briefly review the non-convergence results by Reddi et al'19 and the convergence results by Shi et al.'20. Their results take important steps forward to understand Adam better. However, the convergence analysis by Shi et al.'20 requires $\beta_1$ to be either 0 or small enough ($\beta_1$ is the momentum hyperparameter in Adam). Is this a reasonable requirement?  If not, how large is the gap between theory and practice? In this blog, we will discuss these questions from multiple different perspectives. We will show that the gap is actually non-negligible, and the discussion on the convergence of Adam is far from being concluded.
Deep dive into CoCon - A Self Supervised approach for Controlled Text Generation
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Language Modeling, Language Modelling, Story Generation, Text Generation

Transformer-based language models ([1] Vaswani et.al, 2017) have stirred transfer-based learning in NLP and have improved the performance of several NLP tasks. The preliminary step involves pretraining a language model on a large amount of text on the web. Research on steering a pretrained language model to enable fine-grained control over the content and sentiment of output is still under active exploration and has great potential in various applications such as story generation, search engines, etc. This blog post discusses a paper which proposes a content conditioner when trained auto-regressively alongside a Large pretrained language model (like GPT-2) provides the capability to control text at a fine-grained level.
Discovering Non-Monotonic Autoregressive Ordering for Text Generation Models using Sinkhorn Distributions
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Code Generation, Decoder, Image Captioning, Machine Translation, Text Generation, Text Summarization, Translation

In this blog, we discuss an important but less researched topic - Discovering non-monotonic orderings for guiding models to obtain high-quality texts. We specifically discuss the model proposed in the ICLR 2021 paper by Li 2021. This model uses Gumbel-Sinkhorn distributions to assist a decoder model by providing good-quality generation orders during training. The trained models help in generating high-quality outputs for four important NLG tasks: (a) Image Captioning (b) Code Generation (c) Text Summarization and (d) Machine Translation. Interestingly, the model behavior replicates human behavior in some sense - Considering what to write about, before figuring out how to write about it.
FrLove : Could a Frenchman rapidly identify Lovecraft?
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Cross-Domain Few-Shot, cross-domain few-shot learning, Few-Shot Learning

This post examines the work in 'Self-training For Few-shot Transfer Across Extreme Task Differences'), accepted as an oral presentation at ICLR 2021. In this post, we break down the task of cross-domain few shot learning, present a bird's eye overview of the techniques involved, and describe the proposed method - STARTUP. In the latter section of the post, we also attempt to extend the experiments to language. To the best of our knowledge, cross-domain few shot learning in a multilingual setting has not been explored before and it raises interesting questions. Our code is available at https://anonymous.4open.science/r/frlove-5918/
Classification or Generation? Understanding Paradigm Shift for Knowledge-Intensive Tasks
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Classification, Entity Retrieval, Retrieval, Text Generation

Knowledge-intensive tasks such as entity retrieval are challenging for even cutting edge NLP models since they require models to apply knowledge about the world. Previous studies typically treat this task as classification. Recently, a new paradigm has emerged, which reformats knowledge-intensive tasks as natural language generation. This post summarizes the paradigm shift and reviews the new generative methodology for the ICLR community, providing philosophical questions and new directions.
The token parser and manipulator, next-generation Deep Learning architecture
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Contrastive Learning, Deep Learning

Deep Learning is an excellently scalable approach for processing unstructured, high-dimensional, raw sensory signal. It is so good that these properties also becomes its most popular criticism. At the moment, deep learning is mostly just a giant correlation machine, devouring enormous amount of data to recognise hidden pattern in data, but still lacking in human-like systematic generalisation required in many reasoning tasks. Symbolic AI on the other hand possesses these abilities by design, but relied on handcrafted symbols that has already been abstracted away from the raw information. Among many approach to combine the best of both worlds, I am most excited about the end-to-end trainable architecture with a perception module that structurised the raw input and a reasoning module operates on top of these symbol-like vectors. While there are still a lot of work before such a system becomes practically relevant, in this blog post we will take a look at the paper Contrastive Learning of Structured World Model, an early paper that offer a glimpse into such architecture through a concrete implementation.
Prior Knowledge for Few-shot Learning—Inductive Reasoning and Distribution Calibration
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Few-Shot Learning, Knowledge Graphs, Meta-Learning, Transfer Learning

Few-shot learning is an important technique that can improve the learning capabilities of machine intelligence and practical adaptive applications. Previous researchers apply the meta-learning strategy to endow the new model with the ability or leverage transfer learning to alleviate the challenge of data-hungry. Moreover, prior knowledge such as knowledge graphs can also be modeled under the few-shot setting. This post gives an overview of recent works about how prior knowledge can address the problem of few-shot learning, and discusses a simple and efficient few-shot learning approach that estimates the novel class distributions derived inductively from the base classes.
The Risks and Rewards of Invariant Risk Minimization
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: BIG-bench Machine Learning

Spurious correlations are one of the most prominent pain points for building and deploying machine learning models. Invariant Risk Minimization (IRM) is a learning algorithm designed to mitigate the effect of spurious features and perform well despite shifts in the test distribution. In this blog post, we discuss the motivation and details of IRM as well as it's criticisms and shortcomings.
DISTRIBUTIONALLY ROBUST NEURAL NETWORKS FOR GROUP SHIFTS ON THE IMPORTANCE OF REGULARIZATION FOR WORST-CASE GENERALIZATION
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Natural Language Inference, Stochastic Optimization

Overparameterized neural networks can be highly accurate on average on an i.i.d. test set yet consistently fail on atypical groups of the data (e.g., by learning spurious correlations that hold on average but not in such groups). Distributionally robust optimization (DRO) allows us to learn models that instead minimize the worst-case training loss over a set of pre-defined groups. However, we find that naively applying group DRO to overparameterized neural networks fails: these models can perfectly fit the training data, and any model with vanishing average training loss also already has vanishing worst-case training loss. Instead, the poor worst-case performance arises from poor generalization on some groups. By coupling group DRO models with increased regularization—a stronger-than-typical
$l_{2}$ penalty or early stopping—we achieve substantially higher worst-group accuracies, with 10–40 percentage point improvements on a natural language inference task and two image tasks, while maintaining high average accuracies. Our results suggest that regularization is important for worst-group generalization in the overparameterized regime, even if it is not needed for average generalization. Finally, we introduce a stochastic optimization algorithm, with convergence guarantees, to efficiently train group DRO models.
From Molecular Dynamics to MeshGraphNets
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Graph Learning

In this blog, we discuss the MeshGraphNets paper and its predecessor paper through the lens of the graph-learning paradigm. We claim that molecular dynamics and smoothed particle hydrodynamics are the ancestors of all graph-based, learned particle simulators and show how graph-based approaches naturally extend to meshes. Then, we compare MeshGraphNets to other approaches, both graph-based and not. Last but not least, we conclude by presenting the strengths and weaknesses of the model, directions for future work, and a code snippet of the core algorithm written in JAX.
Understanding Few-Shot Multi-Task Representation Learning Theory
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Few-Shot Learning, Learning Theory, Machine Translation, Meta-Learning, Multi-Task Learning, object-detection, Object Detection, Representation Learning

Multi-Task Representation Learning (MTR) is a popular paradigm to learn shared representations from multiple related tasks. It has demonstrated its efficiency for solving different problems, ranging from machine translation for natural language processing to object detection in computer vision. On the other hand, Few-Shot Learning is a recent problem that seeks to mimic the human capability to quickly learn how to solve a target task with little supervision. For this topic, researchers have turned to meta-learning that learns to learn a new task by training a model on a lot of small tasks. As meta-learning still suffers from a lack of theoretical understanding for its success in few-shot tasks, an intuitively appealing approach would be to bridge the gap between it and multi-task learning to better understand the former using the results established for the latter. In this post, we dive into a recent ICLR 2021 paper by Du et al. that demonstrated novel learning bounds for multi-task learning in the few-shot setting and go beyond it by establishing the connections that allow to better understand the inner workings of meta-learning algorithms as well.
Behavior of Mini-Batch Optimization for Training Deep Neural Networks on Large Datasets
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: image-classification, Image Classification

Stochastic Weight Averaging in Parallel (SWAP) is a method that enables the training of deep neural networks on large datasets using large mini-batch sizes while also not sacrificing good generalization behavior. The algorithm uses large mini-batches to calculate the approximate model weights. The final model weight is the average of refined weights through parallel small mini-batch training of the approximate weights. This post provides a summary of the paper presenting SWAP, in addition to providing simple explanations of both related and foundational concepts upon which SWAP builds. Important concepts such as convexity, generalizability and gradient descent are explained. Related approaches that aim to obtain good generalization properties for large mini batches, like, ensemble of model parameters and local updating methods are discussed.  Model performance of SWAP is presented for the task of image classification by deep learning models using popular computer vision benchmark datasets, such as, CIFAR 10, CFAIR 100, and Image Net. Further possible improvements identified by the authors are elucidated upon and additional future directions are identified and explained by the post authors.
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Image-to-Image Translation, Translation

Co-modulated GANs link image-conditional GANs and unconditional modulated models to address large-scale image completion tasks. Co-modulation brings stochastic and conditional style representations together. To improve existing metrics for image completion, the proposed Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS) is robust to sampling size, captures subtle differences well, and correlates with human preferences. Experiments using co-modulated GANs lead to high quality and diverse results in free-form image completion and image-to-image translation tasks. We extend the findings by Zhao et al. by performing new image completion experiments to examine the biases of co-modulated GANs.