pwc-archive/papers-with-abstracts huggingface dataset:
Prototypical Representation Learning for Low-resource Knowledge Extraction: Summary and Perspective
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Contrastive Learning, Few-Shot Learning, Relation Extraction, Representation Learning

Recent years have witnessed the success of prototypical representation in widespread low-resource tasks, since "Prototypical Networks for Few-shot Learning (NeurIPS 2017)" proposed to represent each class as a prototype by the mean of its instance embeddings and learn a metric space in which classification can be performed by computing distances to prototypes. A recent paper "*Prototypical Representation Learning for Relation Extraction*" accepted by ICLR 2021, as a member of the growing zoo of prototypical networks, has addressed **prototypical representation learning for low-resource knowledge extraction**. 
In this post, we briefly summarize this issue by highlighting the ICLR paper. Different from vanilla prototypical networks, this ICLR paper has proposed to tackle low-resource knowledge extraction (1) considering both *compactness intra each prototype* and *separability inter prototypes*, (2) by leveraging *contrastive learning* and projecting prototypes into *geometric space*. 
Furthermore, we also point out some shortcomings of this paper and put forward some promising directions.
Recent Advances in Deep Learning for Routing Problems
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Combinatorial Optimization, Deep Learning

Developing neural network-driven solvers for combinatorial optimization problems such as the Travelling Salesperson Problem have seen a surge of academic interest recently. This blogpost presents a Neural Combinatorial Optimization pipeline that unifies several recently proposed model architectures and learning paradigms into one single framework. Through the lens of the pipeline, we analyze recent advances in deep learning for routing problems, and provide new directions to stimulate future research towards practical impact.
BERT vs ALBERT explained
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Representation Learning, Self-Supervised Learning

Implementing Machine Learning and Deep Learning models at scale require an immense amount of training time and computational resources. Particularly in the context of language representation learning, studies have shown that full network pre-training which is large is of crucial importance for achieving state-of-the-art performance. But, we know that increasing the model size results in an increase in the number of model parameters, which significantly increases the training and computation requirements. This can be a huge challenge in the domain of large scale computing. In this blog, we provide a brief summary of the ICLR paper “ALBERT: A LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS.” This paper talks about two parameter reduction techniques to lower memory consumption and increase the training speed of the BERT (Bidirectional Encoder Representations from Transformers) architecture. The proposed methods in the paper led to models that scale much better compared to the original BERT.
Knowledge Distillation as Self-Supervised Learning
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Knowledge Distillation, Self-Supervised Learning

Self-supervised learning (SSL) methods have been shown to effectively train large neural networks with unlabeled data. These networks can produce useful image representations that can exceed the performance of supervised pretraining on downstream tasks. However, SSL is not effective with smaller models. This limits applications where computational power is limited, such as edge devices. Knowledge distillation (KD) is a popular method to train a smaller student network from a larger and more powerful teacher network. The [SEED](https://arxiv.org/abs/2101.04731) paper by Fang et al., published in ICLR 2021, applies knowledge distillation to self-supervised learning to pretrain smaller neural networks without supervision. In this post, we will discuss self-supervised learning and knowledge distillation and how they are unified in SEED.
Rethinking ValueDice: Does It Really Improve Performance?
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Imitation Learning

Since the introduction of GAIL, adversarial imitation learning (AIL) methods attract lots of research interests. Among these methods, ValueDice has achieved significant improvements: it beats the classical approach Behavioral Cloning (BC) in the offline setting, and it requires fewer interactions than GAIL in the online setting. Are these improvements benefited from more advanced algorithm designs? We answer this question with the following conclusions.

First, we show that the ValueDice could reduce to BC in the offline setting. Second, we verify that overfitting exists and regularization matters. Specifically, we demonstrate that with weight decay, BC also nearly matches the expert performance as ValueDice does. The first two claims explain the superior offline performance of ValueDice. Third, we establish that ValueDice does not work at all when the expert trajectory is subsampled. Instead, the mentioned success holds when the expert trajectory is complete, in which ValueDice is closely related to BC that performs well as mentioned. Finally, we discuss the implications of our research for imitation learning studies beyond ValueDice.
A Deeper Look at Zero-Cost Proxies for Lightweight NAS
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Neural Architecture Search

While early algorithms for neural architecture search (NAS) used extreme computational budgets, recent techniques have aimed to lower the computation time. Very recently, a new family of techniques were introduced to approximate the performance of neural architectures in just five seconds: so-called "zero-cost proxies" (Mellor et al. 2020; Abdelfattah et al. 2021). However, two recent papers have showed that simple baselines such as "number of parameters" are competitive, casting doubt on the efficacy of zero-cost proxies (Ning et al. 2021, Chen et al. 2021). In this blog post, we take a deeper look at zero-cost proxies for NAS.  We survey prior work and then run new experiments using the recent NAS-Bench-360 and TransNAS-Bench-101 benchmarks, which give a much more diverse set of datasets and tasks than all prior work. We find that there are no clear "best" zero-cost proxies across all tasks, and we confirm that simple baselines are consistently competitive with proposed zero-cost proxies. However, we also conclude that zero-cost proxies have great potential in NAS, especially when used in combination with other methods or to improve the performance of existing methods at very little additional cost. Overall, we provide a landscape overview of this promising area, highlight strengths and weaknesses, and shed light on future research in this direction.
Normalisation is dead, long live normalisation!
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Form

Since the advent of Batch Normalisation (BN) almost every state-of-the-art (SOTA) method uses some form of normalisation.
After all, normalisation generally speeds up learning and leads to models that generalise better than their unnormalised counterparts.
This turns out to be especially useful when using some form of skip connections, which are prominent in Residual Networks (ResNets), for example.
However, Brock et al. (2021a) suggest that SOTA performance can also be achieved using ResNets without normalisation!
Knowledge Graph Papers @ ICLR 2021
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Complex Query Answering, Entity Linking, Knowledge Graphs, Question Answering

This post aims at providing an overview of ICLR 2021 papers focusing on knowledge graphs (KGs). In particular, we highlight the research in four wide areas: complex query answering and reasoning in KGs, temporal logics and KGs, NLP point of view and entity linking, multimodal question answering with KGs. We hope this post could be of use for researchers and practitioners from adjacent areas to demonstrate the benefits of employing knowledge graphs in those various domains solving important problems.
Looking at the Performer from a Hopfield Point of View
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5

The recent paper Rethinking Attention with Performers constructs a new efficient attention mechanism in an elegant way. It strongly reduces the computational cost for long sequences, while keeping the intriguing properties of the original attention mechanism. In doing so, Performers have a complexity only linear in the input length, in contrast to the quadratic complexity of standard Transformers. This is a major breakthrough in the strive of improving Transformer models. In this blog post, we look at the Performer from a Hopfield Network point of view and relate aspects of the Performer architecture to findings in the field of associative memories and Hopfield Networks. This blog post sheds light on the Performer from three different directions: (i) Performers resemble classical Hopfield Networks, (ii) Sparseness increases memory capacity, and (iii) Performer normalization relates to the activation function of continuous Hopfield Networks.
Why Are Kronecker Products So Effective?
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5

In this blog post we will review the layer scheme proposed by the authors of the paper "Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $$1/n$$ Parameters", which links quaternion-based Neural Networks with the Kronecker Product, and later explain how the Kronecker Product provides a connection between the paper and a parallel line of research into parameter efficient Neural Networks for Computer Vision based on Singular Value Decomposition.
How to Play the PCA Game
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5

The paper EigenGame: PCA as a Nash Equilibrium was published at ICLR 2021. The authors, Ian Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel, introduced a decentralized algorithm for PCA via a game-theoretic analysis by showing that with the right utility functions, PCA is the same as finding the Nash equilibrium. This blog reviews the key ideas for the argument, such as the hierarchy of eigenvectors and objectives for PCA.
Not All Lotteries Are Made Equal
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: All

The Lottery Ticket Hypothesis (LTH) states that for a reasonably sized neural network, there exists a subnetwork within the same network that yields no less performance than the dense counterpart when trained from the same initialization. To the best of our knowledge, prior work regarding the LTH has only investigated overparameterized models, and the emergence of the LTH is often attributed to the initial model being large, i.e., a dense sampling of tickets. In this blog post, we present evidence that challenges this notion of the LTH. We investigate the effect of model size and the ease of finding winning tickets. Through this work, we show that winning tickets is easier to find for smaller models.
Real-Time Anomaly Detection for Multivariate Data Stream
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Anomaly Detection

The paper titled “Probabilistic reasoning for streaming anomaly detection” from MIT CSAIL proposed a framework for performing online anomaly detection on univariate data. Unfortunately, most of the data in the real world are multivariate. Hence, mandating the need for more research into performing online anomaly detection in multivariate data. We have been inspired by their work and extended their framework to support multivariate data with some clever optimizations to build a scalable system.
Incorporating Bayesian approaches in Deep Learning Research
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5
Tasks: Bayesian Inference, Deep Learning, Survey

The blog will take the form of a survey paper as we summarize the proceedings from the [Symposium on Approximate Bayesian Inference](http://approximateinference.org/), during [NeurIPS](https://nips.cc/), 2019 in Vancouver. Rightly so, I focused on the Bayesian workshop as it emphasized the fundamentals, rather than bleeding-edge results. The knowledge that can lead to progress results from understanding how things work on a foundational level.
Contrasting LMU with LSTM
Authors: Anonymous
Date: 2022-01-17 00:00:00
Proceeding: ICLR Track Blog 2022 5

Both Hidden Markov Model (HMM) and Recurrent Neural Network (RNN) suffer from disappearing transitions and (vanishing \& exploding) gradient problems. LSTM maintains a long time-range dependency on a sequencing task. However, information flow in the network tends to saturate once the number of time steps exceeds a few thousand. Legendre Memory Unit (LMU) is a revolutionary evolution on the design of RNN that can conveniently handle extremely long-range dependency. Let's try to figure out why the LMU exceeds the performance of the LSTM in this blog.
An Intelligent Pedagogical Agent to Foster Computational Thinking in Open-Ended Game Design Activities
Authors: Anonymous
Date: 2022-02-08 00:00:00
Tasks: Form, Game Design, Unity

Free-form Game-Design (GD) environments show promise in fostering Computational Thinking (CT) skills at a young age. However, such environments can be challenging to some students due to their highly open-ended nature. Our long-term goal is to alleviate this difficulty via agents that can monitor the student interaction with the environment, detect when the student needs help and provide personalized support accordingly. In this paper, we present a preliminary evaluation of one such agent deployed in a real-word free-form GD learning environment to foster CT in the early K-12 education, Unity-CT. We focus on the effect of repetition by comparing student behaviors between no intervention, 1-shot intervention, and repeated intervention groups for two different errors that are known to be challenging in the online lessons of Unity-CT environment. Our findings showed that the agents were perceived very positively by the students and the repeated intervention showed promising results in terms of helping students making less errors and more correct behaviors, albeit only for one of the two target errors. Based on these results, we provide insights on how to improve the delivery of the IPA interventions in free-form GD environments.
What Is Thrown? Lightweight Precise Extraction of Exceptional Preconditions in Java Methods
Authors: Anonymous
Date: 2022-01-31 00:00:00

When a method throws an exception—its exceptional precondition—is a crucial element of the method’s documentation that clients should know to properly use it. Unfortunately, exceptional behavior is often poorly documented, and sensitive to changes in a project’s implementation details that can be onerous to keep synchronized with the documentation.

We present wit, an automated technique that extracts the exceptional preconditions of Java methods. wit uses static analysis to analyze the paths in a method’s implementation that lead to throwing an exception. wit’s analysis is precise, in that it only reports exceptional preconditions that are correct and correspond to feasible exceptional behavior. It is also lightweight: it only needs the source code of the class (or classes) to be analyzed—without building or running the whole project. To this end, its design uses heuristics that give up some completeness (wit cannot infer all exceptional preconditions) in exchange for precision and ease of applicability.

We ran wit on 46 Java projects, where it discovered 11 875 exceptional preconditions in 10 234 methods, taking just 1 second per method on average out of a total of 388 000. A manual analysis of a significant sample of these exceptional preconditions confirmed that wit is 100% precise, and demonstrated that it can accurately and automatically document the exceptional behavior of Java methods.
Exploring Example Selection for Few-shot Text-to-SQL Semantic Parsing
Authors: Anonymous
Date: 2022-01-16 00:00:00
Tasks: Few-Shot Learning, In-Context Learning, Semantic Parsing, Text to SQL, Text-To-SQL

We study example selection methods for few-shot text-to-SQL tasks with unseen databases. Annotating natural language questions with corresponding SQL queries is expensive, but we can use abundant unlabeled questions to efficiently select examples to annotate and then use them to adapt models. Many previous works only randomly sample a few instances for few-shot learning, but this random selection is not sufficient to select representative and informative examples that provide specific domain knowledge. We thus explore methods to efficiently choose annotation examples. We identify two important factors: the diversity of selected instances and the dissimilarity to the source training data if any. A diverse training set contains more domain knowledge, while dissimilar examples are selected to fill in the domain gap between the source and target. We show that our best example selection approach substantially improves few-shot text-to-SQL performance in both finetuning using T5 and in-context learning with Codex: average execution accuracy gains of 8.7% and 4.3% over random selection. Our extensive analysis demonstrates the importance of the similarity metric and the embedding method for example representations. We also find that effective example selection reduces syntax errors on the target domains. Our results encourage future work to further explore example selection for efficient adaptation of text-to-SQL models.
Analysis and Mitigation of Dataset Artifacts in OpenAI GPT-3
Authors: Mingi Ryu
Date: 2021-12-19 00:00:00
Tasks: Natural Language Inference

With the recent release of public beta, we took full advantage of OpenAI's Models-as-a-Service (MaaS) offering of GPT-3 to analyze and mitigate dataset artifacts in a model that has one of the highest number of parameters. Recent studies in dataset artifacts and adversarial attacks suggest that state of the art (SoTA) NLP models are susceptible to spurious correlations in training datasets. We decided to investigate GPT-3 on dataset artifacts taking advantage of its large scale and task-agnostic pre-training. We began by verifying few-shot capabilities of GPT-3 in order to lay the groundwork for analysis. Furthermore, we employed our approach to fine-tuning for the natural language inference (NLI) task. Using SNLI as a baseline, we carried out several experiments with Adversarial NLI (ANLI) to evaluate the performance and robustness of GPT-3. Our findings suggest that using adversarial datasets could mitigate dataset artifacts in GPT-3 at a negligible overall performance cost.
Prompt-free and Efficient Language Model Fine-Tuning
Authors: Anonymous
Date: 2021-12-02 00:00:00
Tasks: Few-Shot Learning, Language Modeling, Language Modelling, model

Current methods for few-shot fine-tuning of pretrained masked language model (PLM) require carefully engineered prompts and verbalizers for each new task, to convert examples into a cloze-format that the PLM can score. In this work, we propose Perfect, a simple and efficient method for few-shot fine-tuning of PLMs without relying on any such handcrafting, which is highly effective given as few as 32 data points. Perfect makes two key design choices: First, we show that manually engineered task prompts can be replaced with task-specific adapters that enable sample-efficient fine-tuning and reduce memory and storage costs by roughly factors of 5 and 100, respectively. Second, instead of using handcrafted verbalizers, we learn a new multi-token label embedding during fine-tuning which are not tied to the model vocabulary and which allow us to avoid complex auto-regressive decoding. These embeddings are not only learnable from limited data but also enable nearly 100x faster training and inference. 
Experiments on a wide range of few shot NLP tasks demonstrate that Perfect, while being simple and efficient, also outperforms existing state-of-the-art few-shot learning methods. We will release our code publicly to facilitate future work.
Motion Correction in low SNR MRI using an approximate Rician log-likelihood
Authors: Anonymous
Date: 2022-02-04 00:00:00
Proceeding: WBIR Workshop Biomedical_Imaging_Registration 2022 7

Some MRI acquisitions, such as Sodium imaging, produce data with very low signal-to-noise ratio (SNR) and meaningful analysis may require several images to be averaged. As the data contains substantial noise, motion correction using standard registration tools may not be effective. This paper employs a simple generative model for the data, where the error is described as following a Rician distribution, which more accurately characterised the image noise. Maximum a posteriori inference is enabled by a differentiable approximation to the Rician log-likelihood function. We find that this approach substantially outperforms a Gaussian log-likelihood baseline on synthetic data that has been corrupted by Rician noise of varying degrees. We show results of our approach on real Sodium MRI data, and demonstrate that we can reduces the effects of substantial motion.
Weighted Metamorphosis for registration of images with different topology.
Authors: Anonymous
Date: 2022-02-04 00:00:00
Proceeding: WBIR Workshop Biomedical_Imaging_Registration 2022 7
Tasks: Disentanglement

We present an extension of the Metamorphosis algorithm to align images with a different topology and/or appearance. We propose to restrict/limit the metamorphic intensity additions using a time-varying spatial weight function. It can be used to model prior knowledge about the topological/appearance changes (e.g., tumour/oedema). We show that our method improves the disentanglement between anatomical (i.e., shape) and topological (i.e., appearance) changes, thus improving the registration interpretability and its clinical usefulness. As clinical application, we validated our method using MR brain tumour images from the BraTS 2021 dataset. We showed that our method can better align healthy brain templates to images with brain tumours than existing state-of-the-art methods. Our PyTorch code is freely available here: https://anonymous.4open.science/r/weighted_metamorphosis-1DC7/.
Cross domain knowledge compression in realtime optical flow prediction on ultrasound sequences
Authors: Anonymous
Date: 2022-02-04 00:00:00
Proceeding: WBIR Workshop Biomedical_Imaging_Registration 2022 7
Tasks: Knowledge Distillation, Optical Flow Estimation

In this paper, we propose an approach for realtime optical flow estimation in ultrasound sequences of vein and arteries based on knowledge
distillation. Knowledge distillation is a technique to train a faster, smaller model by learning from cues of other models. Mobile devices with limited resources could be key in providing effective point-of-care healthcare and motivate the search of more lightweight solutions in the deep learning based image analysis. For ultrasound video analysis motion correspondences of image contents (anatomies) have to be computed for temporal context and for real time application, fast solutions are required. We use a PWC-Net's optical flow estimation output as soft targets to train a lightweight optical flow estimator. We analyse how well it works on the challenging task of fast segmentation propagation of vein and arteries in ultrasound images. Experiments show that even though we did not fine-tune the teachers on this task, a model trained with soft targets outperformed a model trained directly with labels and without a teacher.