pwc-archive/papers-with-abstracts huggingface dataset:
Recommending Burgers based on Pizza Preferences: Addressing Data Sparsity with a Product of Experts
Authors: Martin Milenkoski, Diego Antognini, Claudiu Musat
Date: 2021-04-26 00:00:00
Tasks: Collaborative Filtering

In this paper, we describe a method to tackle data sparsity and create recommendations in domains with limited knowledge about user preferences. We expand the variational autoencoder collaborative filtering from a single-domain to a multi-domain setting. The intuition is that user-item interactions in a source domain can augment the recommendation quality in a target domain. The intuition can be taken to its extreme, where, in a cross-domain setup, the user history in a source domain is enough to generate high-quality recommendations in a target one. We thus create a Product-of-Experts (POE) architecture for recommendations that jointly models user-item interactions across multiple domains. The method is resilient to missing data for one or more of the domains, which is a situation often found in real life. We present results on two widely-used datasets - Amazon and Yelp, which support the claim that holistic user preference knowledge leads to better recommendations. Surprisingly, we find that in some cases, a POE recommender that does not access the target domain user representation can surpass a strong VAE recommender baseline trained on the target domain.
Universal Off-Policy Evaluation
Authors: Yash Chandak, Scott Niekum, Bruno Castro da Silva, Erik Learned-Miller, Emma Brunskill, Philip S. Thomas
Date: 2021-04-26 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: counterfactual, Decision Making, Off-policy evaluation, Sequential Decision Making

When faced with sequential decision-making problems, it is often useful to be able to predict what would happen if decisions were made using a new policy. Those predictions must often be based on data collected under some previously used decision-making rule. Many previous methods enable such off-policy (or counterfactual) estimation of the expected value of a performance measure called the return. In this paper, we take the first steps towards a universal off-policy estimator (UnO) -- one that provides off-policy estimates and high-confidence bounds for any parameter of the return distribution. We use UnO for estimating and simultaneously bounding the mean, variance, quantiles/median, inter-quantile range, CVaR, and the entire cumulative distribution of returns. Finally, we also discuss Uno's applicability in various settings, including fully observable, partially observable (i.e., with unobserved confounders), Markovian, non-Markovian, stationary, smoothly non-stationary, and discrete distribution shifts.
Online parameter inference for the simulation of a Bunsen flame using heteroscedastic Bayesian neural network ensembles
Authors: Maximilian L. Croci, Ushnish Sengupta, Matthew P. Juniper
Date: 2021-04-26 00:00:00

This paper proposes a Bayesian data-driven machine learning method for the online inference of the parameters of a G-equation model of a ducted, premixed flame. Heteroscedastic Bayesian neural network ensembles are trained on a library of 1.7 million flame fronts simulated in LSGEN2D, a G-equation solver, to learn the Bayesian posterior distribution of the model parameters given observations. The ensembles are then used to infer the parameters of Bunsen flame experiments so that the dynamics of these can be simulated in LSGEN2D. This allows the surface area variation of the flame edge, a proxy for the heat release rate, to be calculated. The proposed method provides cheap and online parameter and uncertainty estimates matching results obtained with the ensemble Kalman filter, at less computational cost. This enables fast and reliable simulation of the combustion process.
Model-centric Data Manifold: the Data Through the Eyes of the Model
Authors: Luca Grementieri, Rita Fioresi
Date: 2021-04-26 00:00:00
Tasks: model, valid

We discover that deep ReLU neural network classifiers can see a low-dimensional Riemannian manifold structure on data. Such structure comes via the local data matrix, a variation of the Fisher information matrix, where the role of the model parameters is taken by the data variables. We obtain a foliation of the data domain and we show that the dataset on which the model is trained lies on a leaf, the data leaf, whose dimension is bounded by the number of classification labels. We validate our results with some experiments with the MNIST dataset: paths on the data leaf connect valid images, while other leaves cover noisy images.
Infinitesimal gradient boosting
Authors: Clément Dombry, Jean-Jil Duchamps
Date: 2021-04-26 00:00:00

We define infinitesimal gradient boosting as a limit of the popular tree-based gradient boosting algorithm from machine learning. The limit is considered in the vanishing-learning-rate asymptotic, that is when the learning rate tends to zero and the number of gradient trees is rescaled accordingly. For this purpose, we introduce a new class of randomized regression trees bridging totally randomized trees and Extra Trees and using a softmax distribution for binary splitting. Our main result is the convergence of the associated stochastic algorithm and the characterization of the limiting procedure as the unique solution of a nonlinear ordinary differential equation in a infinite dimensional function space. Infinitesimal gradient boosting defines a smooth path in the space of continuous functions along which the training error decreases, the residuals remain centered and the total variation is well controlled.
To mock a Mocking bird : Studies in Biomimicry
Authors: Inavamsi Enaganti, Bud Mishra
Date: 2021-04-26 00:00:00

This paper dwells on certain novel game-theoretic investigations in bio-mimicry, discussed from the perspectives of information asymmetry, individual utility and its optimization via strategic interactions involving co-evolving preys (e.g., insects) and predators (e.g., reptiles) who learn. Formally, we consider a panmictic ecosystem, occupied by species of prey with relatively short lifespan, which evolve mimicry signals over generations as they interact with predators with relatively longer lifespans, thus endowing predators with the ability to learn prey signals. Every prey sends a signal and provides utility to the predator. The prey can be either nutritious or toxic to the predator, but the prey may signal (possibly) deceptively without revealing its true "type." The model is used to study the situation where multi-armed bandit predators with zero prior information are introduced into the ecosystem. As a result of exploration and exploitation the predators naturally select the prey that result in the evolution of those signals. This co-evolution of strategies produces a variety of interesting phenomena which are subjects of this paper.
Quantum causal inference in the presence of hidden common causes: An entropic approach
Authors: Mohammad Ali Javidian, Vaneet Aggarwal, Zubin Jacob
Date: 2021-04-24 00:00:00
Tasks: Causal Inference

Quantum causality is an emerging field of study which has the potential to greatly advance our understanding of quantum systems. In this paper, we put forth a theoretical framework for merging quantum information science and causal inference by exploiting entropic principles. For this purpose, we leverage the tradeoff between the entropy of hidden cause and the conditional mutual information of observed variables to develop a scalable algorithmic approach for inferring causality in the presence of latent confounders (common causes) in quantum systems. As an application, we consider a system of three entangled qubits and transmit the second and third qubits over separate noisy quantum channels. In this model, we validate that the first qubit is a latent confounder and the common cause of the second and third qubits. In contrast, when two entangled qubits are prepared and one of them is sent over a noisy channel, there is no common confounder. We also demonstrate that the proposed approach outperforms the results of classical causal inference for the Tubingen database when the variables are classical by exploiting quantum dependence between variables through density matrices rather than joint probability distributions. Thus, the proposed approach unifies classical and quantum causal inference in a principled way.
Tracking Peaceful Tractors on Social Media -- XAI-enabled analysis of Red Fort Riots 2021
Authors: Ajay Agarwal, Basant Agarwal
Date: 2021-04-24 00:00:00
Tasks: Explainable Artificial Intelligence (XAI), Misinformation

On 26 January 2021, India witnessed a national embarrassment from the demographic least expected from - farmers. People across the nation watched in horror as a pseudo-patriotic mob of farmers stormed capital Delhi and vandalized the national pride- Red Fort. Investigations that followed the event revealed the existence of a social media trail that led to the likes of such an event. Consequently, it became essential and necessary to archive this trail for social media analysis - not only to understand the bread-crumbs that are dispersed across the trail but also to visualize the role played by misinformation and fake news in this event. In this paper, we propose the tractor2twitter dataset which contains around 0.05 million tweets that were posted before, during, and after this event. Also, we benchmark our dataset with an Explainable AI ML model for classification of each tweet into either of the three categories - disinformation, misinformation, and opinion.
Scalable Microservice Forensics and Stability Assessment Using Variational Autoencoders
Authors: Prakhar Sharma, Phillip Porras, Steven Cheung, James Carpenter, Vinod Yegneswaran
Date: 2021-04-23 00:00:00
Tasks: CPU

We present a deep learning based approach to containerized application runtime stability analysis, and an intelligent publishing algorithm that can dynamically adjust the depth of process-level forensics published to a backend incident analysis repository. The approach applies variational autoencoders (VAEs) to learn the stable runtime patterns of container images, and then instantiates these container-specific VAEs to implement stability detection and adaptive forensics publishing. In performance comparisons using a 50-instance container workload, a VAE-optimized service versus a conventional eBPF-based forensic publisher demonstrates 2 orders of magnitude (OM) CPU performance improvement, a 3 OM reduction in network transport volume, and a 4 OM reduction in Elasticsearch storage costs. We evaluate the VAE-based stability detection technique against two attacks, CPUMiner and HTTP-flood attack, finding that it is effective in isolating both anomalies. We believe this technique provides a novel approach to integrating fine-grained process monitoring and digital-forensic services into large container ecosystems that today simply cannot be monitored by conventional techniques
Predicting Adversary Lateral Movement Patterns with Deep Learning
Authors: Nathan Danneman, James Hyde
Date: 2021-04-23 00:00:00
Tasks: Deep Learning

This paper develops a predictive model for which host, in an enterprise network, an adversary is likely to compromise next in the course of a campaign. Such a model might support dynamic monitoring or defenses. We generate data for this model using simulated networks, with hosts, users, and adversaries as first-class entities. We demonstrate the predictive accuracy of the model on out-of-sample simulated data, and validate the findings against data captured from a Red Team event on a live enterprise network
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Authors: Artidoro Pagnoni, Vidhisha Balachandran, Yulia Tsvetkov
Date: 2021-04-27 00:00:00
Proceeding: NAACL 2021 4
Tasks: Abstractive Text Summarization

Modern summarization models generate highly fluent but often factually unreliable outputs. This motivated a surge of metrics attempting to measure the factuality of automatically generated summaries. Due to the lack of common benchmarks, these metrics cannot be compared. Moreover, all these methods treat factuality as a binary concept and fail to provide deeper insights into the kinds of inconsistencies made by different systems. To address these limitations, we devise a typology of factual errors and use it to collect human annotations of generated summaries from state-of-the-art summarization systems for the CNN/DM and XSum datasets. Through these annotations, we identify the proportion of different categories of factual errors in various summarization models and benchmark factuality metrics, showing their correlation with human judgment as well as their specific strengths and weaknesses.
Question-Aware Memory Network for Multi-hop Question Answering in Human-Robot Interaction
Authors: Xinmeng Li, Mamoun Alazab, Qian Li, Keping Yu, Quanjun Yin
Date: 2021-04-27 00:00:00
Tasks: Graph Embedding, Graph Question Answering, Knowledge Graph Embedding, Multi-hop Question Answering, Question Answering

Knowledge graph question answering is an important technology in intelligent human-robot interaction, which aims at automatically giving answer to human natural language question with the given knowledge graph. For the multi-relation question with higher variety and complexity, the tokens of the question have different priority for the triples selection in the reasoning steps. Most existing models take the question as a whole and ignore the priority information in it. To solve this problem, we propose question-aware memory network for multi-hop question answering, named QA2MN, to update the attention on question timely in the reasoning process. In addition, we incorporate graph context information into knowledge graph embedding model to increase the ability to represent entities and relations. We use it to initialize the QA2MN model and fine-tune it in the training process. We evaluate QA2MN on PathQuestion and WorldCup2014, two representative datasets for complex multi-hop question answering. The result demonstrates that QA2MN achieves state-of-the-art Hits@1 accuracy on the two datasets, which validates the effectiveness of our model.
UoT-UWF-PartAI at SemEval-2021 Task 5: Self Attention Based Bi-GRU with Multi-Embedding Representation for Toxicity Highlighter
Authors: Hamed Babaei Giglou, Taher Rahgooy, Mostafa Rahgouy, Jafar Razmara
Date: 2021-04-27 00:00:00
Proceeding: SEMEVAL 2021
Tasks: Toxic Spans Detection

Toxic Spans Detection(TSD) task is defined as highlighting spans that make a text toxic. Many works have been done to classify a given comment or document as toxic or non-toxic. However, none of those proposed models work at the token level. In this paper, we propose a self-attention-based bidirectional gated recurrent unit(BiGRU) with a multi-embedding representation of the tokens. Our proposed model enriches the representation by a combination of GPT-2, GloVe, and RoBERTa embeddings, which led to promising results. Experimental results show that our proposed approach is very effective in detecting span tokens.
Semi-Supervised Joint Estimation of Word and Document Readability
Authors: Yoshinari Fujinuma, Masato Hagiwara
Date: 2021-04-27 00:00:00
Proceeding: NAACL (TextGraphs) 2021 6

Readability or difficulty estimation of words and documents has been investigated independently in the literature, often assuming the existence of extensive annotated resources for the other. Motivated by our analysis showing that there is a recursive relationship between word and document difficulty, we propose to jointly estimate word and document difficulty through a graph convolutional network (GCN) in a semi-supervised fashion. Our experimental results reveal that the GCN-based method can achieve higher accuracy than strong baselines, and stays robust even with a smaller amount of labeled data.
Shellcode_IA32: A Dataset for Automatic Shellcode Generation
Authors: Pietro Liguori, Erfan Al-Hossami, Domenico Cotroneo, Roberto Natella, Bojan Cukic, Samira Shaikh
Date: 2021-04-27 00:00:00
Proceeding: ACL (NLP4Prog) 2021 8
Tasks: Code Generation, Machine Translation, NMT, Translation

We take the first step to address the task of automatically generating shellcodes, i.e., small pieces of code used as a payload in the exploitation of a software vulnerability, starting from natural language comments. We assemble and release a novel dataset (Shellcode_IA32), consisting of challenging but common assembly instructions with their natural language descriptions. We experiment with standard methods in neural machine translation (NMT) to establish baseline performance levels on this task.
LAST at CMCL 2021 Shared Task: Predicting Gaze Data During Reading with a Gradient Boosting Decision Tree Approach
Authors: Yves Bestgen
Date: 2021-04-27 00:00:00
Proceeding: NAACL (CMCL) 2021 6
Tasks: All

A LightGBM model fed with target word lexical characteristics and features obtained from word frequency lists, psychometric data and bigram association measures has been optimized for the 2021 CMCL Shared Task on Eye-Tracking Data Prediction. It obtained the best performance of all teams on two of the five eye-tracking measures to predict, allowing it to rank first on the official challenge criterion and to outperform all deep-learning based systems participating in the challenge.
SE-DAE: Style-Enhanced Denoising Auto-Encoder for Unsupervised Text Style Transfer
Authors: Jicheng Li, Yang Feng, Jiao Ou
Date: 2021-04-27 00:00:00
Tasks: Denoising, Sentence, Style Transfer, Text Style Transfer, Unsupervised Text Style Transfer

Text style transfer aims to change the style of sentences while preserving the semantic meanings. Due to the lack of parallel data, the Denoising Auto-Encoder (DAE) is widely used in this task to model distributions of different sentence styles. However, because of the conflict between the target of the conventional denoising procedure and the target of style transfer task, the vanilla DAE can not produce satisfying enough results. To improve the transferability of the model, most of the existing works combine DAE with various complicated unsupervised networks, which makes the whole system become over-complex. In this work, we design a novel DAE model named Style-Enhanced DAE (SE-DAE), which is specifically designed for the text style transfer task. Compared with previous complicated style-transfer models, our model do not consist of any complicated unsupervised networks, but only relies on the high-quality pseudo-parallel data generated by a novel data refinement mechanism. Moreover, to alleviate the conflict between the targets of the conventional denoising procedure and the style transfer task, we propose another novel style denoising mechanism, which is more compatible with the target of the style transfer task. We validate the effectiveness of our model on two style benchmark datasets. Both automatic evaluation and human evaluation show that our proposed model is highly competitive compared with previous strong the state of the art (SOTA) approaches and greatly outperforms the vanilla DAE.
Extractive and Abstractive Explanations for Fact-Checking and Evaluation of News
Authors: Ashkan Kazemi, Zehua Li, Verónica Pérez-Rosas, Rada Mihalcea
Date: 2021-04-27 00:00:00
Proceeding: NAACL (NLP4IF) 2021 6
Tasks: Fact Checking, Language Modeling, Language Modelling, Misinformation

In this paper, we explore the construction of natural language explanations for news claims, with the goal of assisting fact-checking and news evaluation applications. We experiment with two methods: (1) an extractive method based on Biased TextRank -- a resource-effective unsupervised graph-based algorithm for content extraction; and (2) an abstractive method based on the GPT-2 language model. We perform comparative evaluations on two misinformation datasets in the political and health news domains, and find that the extractive method shows the most promise.
Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention
Authors: Soo Hyun Ryu, Richard L. Lewis
Date: 2021-04-26 00:00:00
Proceeding: NAACL (CMCL) 2021 6
Tasks: Retrieval, Sentence

We advance a novel explanation of similarity-based interference effects in subject-verb and reflexive pronoun agreement processing, grounded in surprisal values computed from a pretrained large-scale Transformer model, GPT-2. Specifically, we show that surprisal of the verb or reflexive pronoun predicts facilitatory interference effects in ungrammatical sentences, where a distractor noun that matches in number with the verb or pronoun leads to faster reading times, despite the distractor not participating in the agreement relation. We review the human empirical evidence for such effects, including recent meta-analyses and large-scale studies. We also show that attention patterns (indexed by entropy and other measures) in the Transformer show patterns of diffuse attention in the presence of similar distractors, consistent with cue-based retrieval models of parsing. But in contrast to these models, the attentional cues and memory representations are learned entirely from the simple self-supervised task of predicting the next word.
Morph Call: Probing Morphosyntactic Content of Multilingual Transformers
Authors: Vladislav Mikhailov, Oleg Serikov, Ekaterina Artemova
Date: 2021-04-26 00:00:00
Proceeding: NAACL (SIGTYP) 2021 6
Tasks: Common Sense Reasoning, MORPH, POS, POS Tagging, Sentence, World Knowledge

The outstanding performance of transformer-based language models on a great variety of NLP and NLU tasks has stimulated interest in exploring their inner workings. Recent research has focused primarily on higher-level and complex linguistic phenomena such as syntax, semantics, world knowledge, and common sense. The majority of the studies are anglocentric, and little remains known regarding other languages, precisely their morphosyntactic properties. To this end, our work presents Morph Call, a suite of 46 probing tasks for four Indo-European languages of different morphology: English, French, German and Russian. We propose a new type of probing task based on the detection of guided sentence perturbations. We use a combination of neuron-, layer- and representation-level introspection techniques to analyze the morphosyntactic content of four multilingual transformers, including their less explored distilled versions. Besides, we examine how fine-tuning for POS-tagging affects the model knowledge. The results show that fine-tuning can improve and decrease the probing performance and change how morphosyntactic knowledge is distributed across the model. The code and data are publicly available, and we hope to fill the gaps in the less studied aspect of transformers.
Teaching a Massive Open Online Course on Natural Language Processing
Authors: Ekaterina Artemova, Murat Apishev, Veronika Sarkisyan, Sergey Aksenov, Denis Kirjanov, Oleg Serikov
Date: 2021-04-26 00:00:00
Proceeding: NAACL (TeachingNLP) 2021 6
Tasks: Language Modeling, Language Modelling, Sentence

This paper presents a new Massive Open Online Course on Natural Language Processing, targeted at non-English speaking students. The course lasts 12 weeks; every week consists of lectures, practical sessions, and quiz assignments. Three weeks out of 12 are followed by Kaggle-style coding assignments. Our course intends to serve multiple purposes: (i) familiarize students with the core concepts and methods in NLP, such as language modeling or word or sentence representations, (ii) show that recent advances, including pre-trained Transformer-based models, are built upon these concepts; (iii) introduce architectures for most demanded real-life applications, (iv) develop practical skills to process texts in multiple languages. The course was prepared and recorded during 2020, launched by the end of the year, and in early 2021 has received positive feedback.