pwc-archive/papers-with-abstracts huggingface dataset:
Learning from Executions for Semantic Parsing
Authors: Bailin Wang, Mirella Lapata, Ivan Titov
Date: 2021-04-12 00:00:00
Proceeding: NAACL 2021 4
Tasks: Semantic Parsing

Semantic parsing aims at translating natural language (NL) utterances onto machine-interpretable programs, which can be executed against a real-world environment. The expensive annotation of utterance-program pairs has long been acknowledged as a major bottleneck for the deployment of contemporary neural models to real-life applications. In this work, we focus on the task of semi-supervised learning where a limited amount of annotated data is available together with many unlabeled NL utterances. Based on the observation that programs which correspond to NL utterances must be always executable, we propose to encourage a parser to generate executable programs for unlabeled utterances. Due to the large search space of executable programs, conventional methods that use approximations based on beam-search such as self-training and top-k marginal likelihood training, do not perform as well. Instead, we view the problem of learning from executions from the perspective of posterior regularization and propose a set of new training objectives. Experimental results on Overnight and GeoQuery show that our new objectives outperform conventional methods, bridging the gap between semi-supervised and supervised learning.
Paragraph-level Simplification of Medical Texts
Authors: Ashwin Devaraj, Iain J. Marshall, Byron C. Wallace, Junyi Jessy Li
Date: 2021-04-12 00:00:00
Proceeding: NAACL 2021 4
Tasks: Decoder, Language Modeling, Language Modelling

We consider the problem of learning to simplify medical texts. This is important because most reliable, up-to-date information in biomedicine is dense with jargon and thus practically inaccessible to the lay audience. Furthermore, manual simplification does not scale to the rapidly growing body of biomedical literature, motivating the need for automated approaches. Unfortunately, there are no large-scale resources available for this task. In this work we introduce a new corpus of parallel texts in English comprising technical and lay summaries of all published evidence pertaining to different clinical topics. We then propose a new metric based on likelihood scores from a masked language model pretrained on scientific texts. We show that this automated measure better differentiates between technical and lay summaries than existing heuristics. We introduce and evaluate baseline encoder-decoder Transformer models for simplification and propose a novel augmentation to these in which we explicitly penalize the decoder for producing "jargon" terms; we find that this yields improvements over baselines in terms of readability.
Few-shot Intent Classification and Slot Filling with Retrieved Examples
Authors: Dian Yu, Luheng He, Yuan Zhang, Xinya Du, Panupong Pasupat, Qi Li
Date: 2021-04-12 00:00:00
Proceeding: NAACL 2021 4
Tasks: Classification, Few-Shot Learning, General Classification, intent-classification, Intent Classification, Intent Classification and Slot Filling, Natural Language Understanding, Retrieval, slot-filling, Slot Filling

Few-shot learning arises in important practical scenarios, such as when a natural language understanding system needs to learn new semantic labels for an emerging, resource-scarce domain. In this paper, we explore retrieval-based methods for intent classification and slot filling tasks in few-shot settings. Retrieval-based methods make predictions based on labeled examples in the retrieval index that are similar to the input, and thus can adapt to new domains simply by changing the index without having to retrain the model. However, it is non-trivial to apply such methods on tasks with a complex label space like slot filling. To this end, we propose a span-level retrieval method that learns similar contextualized representations for spans with the same label via a novel batch-softmax objective. At inference time, we use the labels of the retrieved spans to construct the final structure with the highest aggregated score. Our method outperforms previous systems in various few-shot settings on the CLINC and SNIPS benchmarks.
Fighting the COVID-19 Infodemic with a Holistic BERT Ensemble
Authors: Giorgos Tziafas, Konstantinos Kogkalidis, Tommaso Caselli
Date: 2021-04-12 00:00:00
Proceeding: NAACL (NLP4IF) 2021 6
Tasks: Misinformation

This paper describes the TOKOFOU system, an ensemble model for misinformation detection tasks based on six different transformer-based pre-trained encoders, implemented in the context of the COVID-19 Infodemic Shared Task for English. We fine tune each model on each of the task's questions and aggregate their prediction scores using a majority voting approach. TOKOFOU obtains an overall F1 score of 89.7%, ranking first.
A Replication Study of Dense Passage Retriever
Authors: Xueguang Ma, Kai Sun, Ronak Pradeep, Jimmy Lin
Date: 2021-04-12 00:00:00
Tasks: Open-Domain Question Answering, Question Answering, Retrieval, Text Retrieval

Text retrieval using learned dense representations has recently emerged as a promising alternative to "traditional" text retrieval using sparse bag-of-words representations. One recent work that has garnered much attention is the dense passage retriever (DPR) technique proposed by Karpukhin et al. (2020) for end-to-end open-domain question answering. We present a replication study of this work, starting with model checkpoints provided by the authors, but otherwise from an independent implementation in our group's Pyserini IR toolkit and PyGaggle neural text ranking library. Although our experimental results largely verify the claims of the original paper, we arrived at two important additional findings that contribute to a better understanding of DPR: First, it appears that the original authors under-report the effectiveness of the BM25 baseline and hence also dense--sparse hybrid retrieval results. Second, by incorporating evidence from the retriever and an improved answer span scoring technique, we are able to improve end-to-end question answering effectiveness using exactly the same models as in the original work.
Variational Autoencoder Analysis of Ising Model Statistical Distributions and Phase Transitions
Authors: David Yevick
Date: 2021-04-13 00:00:00
Tasks: Decoder

Variational autoencoders employ an encoding neural network to generate a probabilistic representation of a data set within a low-dimensional space of latent variables followed by a decoding stage that maps the latent variables back to the original variable space. Once trained, a statistical ensemble of simulated data realizations can be obtained by randomly assigning values to the latent variables that are subsequently processed by the decoding section of the network. To determine the accuracy of such a procedure when applied to lattice models, an autoencoder is here trained on a thermal equilibrium distribution of Ising spin realizations. When the output of the decoder for synthetic data is interpreted probabilistically, spin realizations can be generated by randomly assigning spin values according to the computed likelihood. The resulting state distribution in energy-magnetization space then qualitatively resembles that of the training samples. However, because correlations between spins are suppressed, the computed energies are unphysically large for low-dimensional latent variable spaces. The features of the learned distributions as a function of temperature, however, provide a qualitative indication of the presence of a phase transition and the distribution of realizations with characteristic cluster sizes.
Bayesian Optimisation for a Biologically Inspired Population Neural Network
Authors: Mahak Kothari, Swapna Sasi, Jun Chen, Elham Zareian, Basabdatta Sen Bhattacharya
Date: 2021-04-13 00:00:00
Tasks: Bayesian Optimisation, Rhythm, Time Series, Time Series Analysis

We have used Bayesian Optimisation (BO) to find hyper-parameters in an existing biologically plausible population neural network. The 8-dimensional optimal hyper-parameter combination should be such that the network dynamics simulate the resting state alpha rhythm (8 - 13 Hz rhythms in brain signals). Each combination of these eight hyper-parameters constitutes a 'datapoint' in the parameter space. The best combination of these parameters leads to the neural network's output power spectral peak being constraint within the alpha band. Further, constraints were introduced to the BO algorithm based on qualitative observation of the network output time series, so that high amplitude pseudo-periodic oscillations are removed. Upon successful implementation for alpha band, we further optimised the network to oscillate within the theta (4 - 8 Hz) and beta (13 - 30 Hz) bands. The changing rhythms in the model can now be studied using the identified optimal hyper-parameters for the respective frequency bands. We have previously tuned parameters in the existing neural network by the trial-and-error approach; however, due to time and computational constraints, we could not vary more than three parameters at once. The approach detailed here, allows an automatic hyper-parameter search, producing reliable parameter sets for the network.
An Adaptive Synaptic Array using Fowler-Nordheim Dynamic Analog Memory
Authors: Darshit Mehta, Kenji Aono, Shantanu Chakrabartty
Date: 2021-04-13 00:00:00

In this paper we present a synaptic array that uses dynamical states to implement an analog memory for energy-efficient training of machine learning (ML) systems. Each of the analog memory elements is a micro-dynamical system that is driven by the physics of Fowler-Nordheim (FN) quantum tunneling, whereas the system level learning modulates the state trajectory of the memory ensembles towards the optimal solution. We show that the extrinsic energy required for modulation can be matched to the dynamics of learning and weight decay leading to a significant reduction in the energy-dissipated during ML training. With the energy-dissipation as low as 5 fJ per memory update and a programming resolution up to 14 bits, the proposed synapse array could be used to address the energy-efficiency imbalance between the training and the inference phases observed in artificial intelligence (AI) systems.
Optimal scaling of random-walk Metropolis algorithms using Bayesian large-sample asymptotics
Authors: Sebastian M Schmon, Philippe Gagnon
Date: 2021-04-13 00:00:00

High-dimensional limit theorems have been shown useful to derive tuning rules for finding the optimal scaling in random-walk Metropolis algorithms. The assumptions under which weak convergence results are proved are however restrictive: the target density is typically assumed to be of a product form. Users may thus doubt the validity of such tuning rules in practical applications. In this paper, we shed some light on optimal-scaling problems from a different perspective, namely a large-sample one. This allows to prove weak convergence results under realistic assumptions and to propose novel parameter-dimension-dependent tuning guidelines. The proposed guidelines are consistent with previous ones when the target density is close to having a product form, and the results highlight that the correlation structure has to be accounted for to avoid performance deterioration if that is not the case, while justifying the use of a natural (asymptotically exact) approximation to the correlation matrix that can be employed for the very first algorithm run.
COVID-19 case data for Italy stratified by age class
Authors: Giuseppe Calafiore, Giulia Fracastoro
Date: 2021-04-13 00:00:00

The dataset described in this paper contains daily data about COVID-19 cases that occurred in Italy over the period from Jan. 28, 2020 to March 20, 2021, divided into ten age classes of the population, the first class being 0-9 years, the tenth class being 90 years and over. The dataset contains eight columns, namely: date (day), age class, number of new cases, number of newly hospitalized patients, number of patients entering intensive care, number of deceased patients, number of recovered patients, number of active infected patients. This data has been officially released for research purposes by the Italian authority for COVID-19 epidemiologic surveillance (Istituto Superiore di Sanit\`a - ISS), upon formal request by the authors, in accordance with the Ordonnance of the Chief of the Civil Protection Department n. 691 dated Aug. 4 2020. A separate file contains the numerosity of the population in each age class, according to the National Institute of Statistics (ISTAT) data of the resident population of Italy as of Jan. 2020. This data has potential use, for instance, in epidemiologic studies of the effects of the COVID-19 contagion in Italy, in mortality analysis by age class, and in the development and testing of dynamical models of the contagion.
The computational asymptotics of Gaussian variational inference and the Laplace approximation
Authors: Zuheng Xu, Trevor Campbell
Date: 2021-04-13 00:00:00
Tasks: Bayesian Inference, Stochastic Optimization, Variational Inference

Gaussian variational inference and the Laplace approximation are popular alternatives to Markov chain Monte Carlo that formulate Bayesian posterior inference as an optimization problem, enabling the use of simple and scalable stochastic optimization algorithms. However, a key limitation of both methods is that the solution to the optimization problem is typically not tractable to compute; even in simple settings the problem is nonconvex. Thus, recently developed statistical guarantees -- which all involve the (data) asymptotic properties of the global optimum -- are not reliably obtained in practice. In this work, we provide two major contributions: a theoretical analysis of the asymptotic convexity properties of variational inference with a Gaussian family and the maximum a posteriori (MAP) problem required by the Laplace approximation; and two algorithms -- consistent Laplace approximation (CLA) and consistent stochastic variational inference (CSVI) -- that exploit these properties to find the optimal approximation in the asymptotic regime. Both CLA and CSVI involve a tractable initialization procedure that finds the local basin of the optimum, and CSVI further includes a scaled gradient descent algorithm that provably stays locally confined to that basin. Experiments on nonconvex synthetic and real-data examples show that compared with standard variational and Laplace approximations, both CSVI and CLA improve the likelihood of obtaining the global optimum of their respective optimization problems.
Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap
Authors: Alexander D'Amour, Alexander Franks
Date: 2021-04-12 00:00:00
Tasks: Dimensionality Reduction

A key condition for obtaining reliable estimates of the causal effect of a treatment is overlap (a.k.a. positivity): the distributions of the features used to perform causal adjustment cannot be too different in the treated and control groups. In cases where overlap is poor, causal effect estimators can become brittle, especially when they incorporate weighting. To address this problem, a number of proposals (including confounder selection or dimension reduction methods) incorporate feature representations to induce better overlap between the treated and control groups. A key concern in these proposals is that the representation may introduce confounding bias into the effect estimator. In this paper, we introduce deconfounding scores, which are feature representations that induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. As a proof of concept, we characterize a family of deconfounding scores in a simplified setting with Gaussian covariates, and show that in some simple simulations, these scores can be used to construct estimators with good finite-sample properties. In particular, we show that this technique could be an attractive alternative to standard regularizations that are often applied to IPW and balancing weights.
An Adversarial Imitation Click Model for Information Retrieval
Authors: Xinyi Dai, Jianghao Lin, Weinan Zhang, Shuai Li, Weiwen Liu, Ruiming Tang, Xiuqiang He, Jianye Hao, Jun Wang, Yong Yu
Date: 2021-04-13 00:00:00
Tasks: Imitation Learning, Information Retrieval, model, Recommendation Systems, Retrieval

Modern information retrieval systems, including web search, ads placement, and recommender systems, typically rely on learning from user feedback. Click models, which study how users interact with a ranked list of items, provide a useful understanding of user feedback for learning ranking models. Constructing "right" dependencies is the key of any successful click model. However, probabilistic graphical models (PGMs) have to rely on manually assigned dependencies, and oversimplify user behaviors. Existing neural network based methods promote PGMs by enhancing the expressive ability and allowing flexible dependencies, but still suffer from exposure bias and inferior estimation. In this paper, we propose a novel framework, Adversarial Imitation Click Model (AICM), based on imitation learning. Firstly, we explicitly learn the reward function that recovers users' intrinsic utility and underlying intentions. Secondly, we model user interactions with a ranked list as a dynamic system instead of one-step click prediction, alleviating the exposure bias problem. Finally, we minimize the JS divergence through adversarial training and learn a stable distribution of click sequences, which makes AICM generalize well across different distributions of ranked lists. A theoretical analysis has indicated that AICM reduces the exposure bias from $O(T^2)$ to $O(T)$. Our studies on a public web search dataset show that AICM not only outperforms state-of-the-art models in traditional click metrics but also achieves superior performance in addressing the exposure bias and recovering the underlying patterns of click sequences.
On the instability of embeddings for recommender systems: the case of Matrix Factorization
Authors: Giovanni Gabbolini, Edoardo D'Amico, Cesare Bernardis, Paolo Cremonesi
Date: 2021-04-12 00:00:00
Tasks: Recommendation Systems

Most state-of-the-art top-N collaborative recommender systems work by learning embeddings to jointly represent users and items. Learned embeddings are considered to be effective to solve a variety of tasks. Among others, providing and explaining recommendations. In this paper we question the reliability of the embeddings learned by Matrix Factorization (MF). We empirically demonstrate that, by simply changing the initial values assigned to the latent factors, the same MF method generates very different embeddings of items and users, and we highlight that this effect is stronger for less popular items. To overcome these drawbacks, we present a generalization of MF, called Nearest Neighbors Matrix Factorization (NNMF). The new method propagates the information about items and users to their neighbors, speeding up the training procedure and extending the amount of information that supports recommendations and representations. We describe the NNMF variants of three common MF approaches, and with extensive experiments on five different datasets we show that they strongly mitigate the instability issues of the original MF versions and they improve the accuracy of recommendations on the long-tail.
A Non-sequential Approach to Deep User Interest Model for CTR Prediction
Authors: Keke Zhao, Xing Zhao, Qi Cao, Linjian Mo
Date: 2021-04-05 00:00:00
Tasks: Click-Through Rate Prediction

Click-Through Rate (CTR) prediction plays an important role in many industrial applications, and recently a lot of attention is paid to the deep interest models which use attention mechanism to capture user interests from historical behaviors. However, most current models are based on sequential models which truncate the behavior sequences by a fixed length, thus have difficulties in handling very long behavior sequences. Another big problem is that sequences with the same length can be quite different in terms of time, carrying completely different meanings. In this paper, we propose a non-sequential approach to tackle the above problems. Specifically, we first represent the behavior data in a sparse key-vector format, where the vector contains rich behavior info such as time, count and category. Next, we enhance the Deep Interest Network to take such rich information into account by a novel attention network. The sparse representation makes it practical to handle large scale long behavior sequences. Finally, we introduce a multidimensional partition framework to mine behavior interactions. The framework can partition data into custom designed time buckets to capture the interactions among information aggregated in different time buckets. Similarly, it can also partition the data into different categories and capture the interactions among them. Experiments are conducted on two public datasets: one is an advertising dataset and the other is a production recommender dataset. Our models outperform other state-of-the-art models on both datasets.
SetConv: A New Approach for Learning from Imbalanced Data
Authors: Yang Gao, Yi-Fan Li, Yu Lin, Charu Aggarwal, Latifur Khan
Date: 2021-04-03 00:00:00
Proceeding: EMNLP 2020 11
Tasks: BIG-bench Machine Learning, Classification, General Classification, Sentiment Analysis, Sentiment Classification

For many real-world classification problems, e.g., sentiment classification, most existing machine learning methods are biased towards the majority class when the Imbalance Ratio (IR) is high. To address this problem, we propose a set convolution (SetConv) operation and an episodic training strategy to extract a single representative for each class, so that classifiers can later be trained on a balanced class distribution. We prove that our proposed algorithm is permutation-invariant despite the order of inputs, and experiments on multiple large-scale benchmark text datasets show the superiority of our proposed framework when compared to other SOTA methods.
Deep Monocular Visual Odometry for Ground Vehicle
Authors: Xiangwei Wang, HUI ZHANG
Date: 2020-09-21 00:00:00
Tasks: Camera Calibration, Dimensionality Reduction, Monocular Visual Odometry, Visual Odometry

Monocular visual odometry, with the ability to help robots to locate themselves in unexplored
environments, has been a crucial research problem in robotics. Though the existed learning-based endto-end methods can reduce engineering efforts such as accurate camera calibration and tedious case-by-case
parameter tuning, the accuracy is still limited. One of the main reasons is that previous works aim to learn sixdegrees-of-freedom motions despite the constrained motion of a ground vehicle by its mechanical structure
and dynamics. To push the limit, we analyze the motion pattern of a ground vehicle and focus on learning
two-degrees-of-freedom motions by proposed motion focusing and decoupling. The experiments on KITTI
dataset show that the proposed motion focusing and decoupling approach can improve the visual odometry
performance by reducing the relative pose error. Moreover, with the dimension reduction of the learning
objective, our network is much lighter with only four convolution layers, which can quickly converge during
the training stage and run in real-time at over 200 frames per second during the testing stage.