pwc-archive/papers-with-abstracts huggingface dataset:
Contrastive Representation Learning: A Framework and Review
Authors: Phuc H. Le-Khac, Graham Healy, Alan F. Smeaton
Date: 2020-10-10 00:00:00
Tasks: BIG-bench Machine Learning, Contrastive Learning, Metric Learning, Representation Learning

Contrastive Learning has recently received interest due to its success in self-supervised representation learning in the computer vision domain. However, the origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields and domains including Metric Learning and natural language processing. In this paper we provide a comprehensive literature review and we propose a general Contrastive Representation Learning framework that simplifies and unifies many different contrastive learning methods. We also provide a taxonomy for each of the components of contrastive learning in order to summarise it and distinguish it from other forms of machine learning. We then discuss the inductive biases which are present in any contrastive learning system and we analyse our framework under different views from various sub-fields of Machine Learning. Examples of how contrastive learning has been applied in computer vision, natural language processing, audio processing, and others, as well as in Reinforcement Learning are also presented. Finally, we discuss the challenges and some of the most promising future research directions ahead.
Hierarchical Evidence Set Modeling for Automated Fact Extraction and Verification
Authors: Shyam Subramanian, Kyumin Lee
Date: 2020-10-10 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Claim Verification, Sentence

Automated fact extraction and verification is a challenging task that involves finding relevant evidence sentences from a reliable corpus to verify the truthfulness of a claim. Existing models either (i) concatenate all the evidence sentences, leading to the inclusion of redundant and noisy information; or (ii) process each claim-evidence sentence pair separately and aggregate all of them later, missing the early combination of related sentences for more accurate claim verification. Unlike the prior works, in this paper, we propose Hierarchical Evidence Set Modeling (HESM), a framework to extract evidence sets (each of which may contain multiple evidence sentences), and verify a claim to be supported, refuted or not enough info, by encoding and attending the claim and evidence sets at different levels of hierarchy. Our experimental results show that HESM outperforms 7 state-of-the-art methods for fact extraction and claim verification. Our source code is available at https://github.com/ShyamSubramanian/HESM.
AEGD: Adaptive Gradient Descent with Energy
Authors: Hailiang Liu, Xuping Tian
Date: 2020-10-10 00:00:00

We propose AEGD, a new algorithm for first-order gradient-based optimization of non-convex objective functions, based on a dynamically updated energy variable. The method is shown to be unconditionally energy stable, irrespective of the step size. We prove energy-dependent convergence rates of AEGD for both non-convex and convex objectives, which for a suitably small step size recovers desired convergence rates for the batch gradient descent. We also provide an energy-dependent bound on the stationary convergence of AEGD in the stochastic non-convex setting. The method is straightforward to implement and requires little tuning of hyper-parameters. Experimental results demonstrate that AEGD works well for a large variety of optimization problems: it is robust with respect to initial data, capable of making rapid initial progress. The stochastic AEGD shows comparable and often better generalization performance than SGD with momentum for deep neural networks.
Localizing Open-Ontology QA Semantic Parsers in a Day Using Machine Translation
Authors: Mehrad Moradshahi, Giovanni Campagna, Sina J. Semnani, Silei Xu, Monica S. Lam
Date: 2020-10-10 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Machine Translation, NMT, Overall - Test, Question Answering, Translation

We propose Semantic Parser Localizer (SPL), a toolkit that leverages Neural Machine Translation (NMT) systems to localize a semantic parser for a new language. Our methodology is to (1) generate training data automatically in the target language by augmenting machine-translated datasets with local entities scraped from public websites, (2) add a few-shot boost of human-translated sentences and train a novel XLMR-LSTM semantic parser, and (3) test the model on natural utterances curated using human translators. We assess the effectiveness of our approach by extending the current capabilities of Schema2QA, a system for English Question Answering (QA) on the open web, to 10 new languages for the restaurants and hotels domains. Our models achieve an overall test accuracy ranging between 61% and 69% for the hotels domain and between 64% and 78% for restaurants domain, which compares favorably to 69% and 80% obtained for English parser trained on gold English data and a few examples from validation set. We show our approach outperforms the previous state-of-the-art methodology by more than 30% for hotels and 40% for restaurants with localized ontologies for the subset of languages tested. Our methodology enables any software developer to add a new language capability to a QA system for a new domain, leveraging machine translation, in less than 24 hours.
On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks
Authors: Stephen Mussmann, Robin Jia, Percy Liang
Date: 2020-10-10 00:00:00
Proceeding: Findings of the Association for Computational Linguistics 2020
Tasks: Active Learning, Open-Domain Question Answering, QQP, Question Answering

Many pairwise classification tasks, such as paraphrase detection and open-domain question answering, naturally have extreme label imbalance (e.g., $99.99\%$ of examples are negatives). In contrast, many recent datasets heuristically choose examples to ensure label balance. We show that these heuristics lead to trained models that generalize poorly: State-of-the art models trained on QQP and WikiQA each have only $2.4\%$ average precision when evaluated on realistically imbalanced test data. We instead collect training data with active learning, using a BERT-based embedding model to efficiently retrieve uncertain points from a very large pool of unlabeled utterance pairs. By creating balanced training data with more informative negative examples, active learning greatly improves average precision to $32.5\%$ on QQP and $20.1\%$ on WikiQA.
Diagnosing and Preventing Instabilities in Recurrent Video Processing
Authors: Thomas Tanay, Aivar Sootla, Matteo Maggioni, Puneet K. Dokania, Philip Torr, Ales Leonardis, Gregory Slabaugh
Date: 2020-10-10 00:00:00
Tasks: Denoising, Diagnostic, Super-Resolution, Video Denoising, Video Enhancement

Recurrent models are a popular choice for video enhancement tasks such as video denoising or super-resolution. In this work, we focus on their stability as dynamical systems and show that they tend to fail catastrophically at inference time on long video sequences. To address this issue, we (1) introduce a diagnostic tool which produces input sequences optimized to trigger instabilities and that can be interpreted as visualizations of temporal receptive fields, and (2) propose two approaches to enforce the stability of a model during training: constraining the spectral norm or constraining the stable rank of its convolutional layers. We then introduce Stable Rank Normalization for Convolutional layers (SRN-C), a new algorithm that enforces these constraints. Our experimental results suggest that SRN-C successfully enforces stability in recurrent video processing models without a significant performance loss.
Leveraging Spatial Information in Radiology Reports for Ischemic Stroke Phenotyping
Authors: Surabhi Datta, Shekhar Khanpara, Roy F. Riascos, Kirk Roberts
Date: 2020-10-10 00:00:00

Classifying fine-grained ischemic stroke phenotypes relies on identifying important clinical information. Radiology reports provide relevant information with context to determine such phenotype information. We focus on stroke phenotypes with location-specific information: brain region affected, laterality, stroke stage, and lacunarity. We use an existing fine-grained spatial information extraction system--Rad-SpatialNet--to identify clinically important information and apply simple domain rules on the extracted information to classify phenotypes. The performance of our proposed approach is promising (recall of 89.62% for classifying brain region and 74.11% for classifying brain region, side, and stroke stage together). Our work demonstrates that an information extraction system based on a fine-grained schema can be utilized to determine complex phenotypes with the inclusion of simple domain rules. These phenotypes have the potential to facilitate stroke research focusing on post-stroke outcome and treatment planning based on the stroke location.
Semi-supervised Formality Style Transfer using Language Model Discriminator and Mutual Information Maximization
Authors: Kunal Chawla, Diyi Yang
Date: 2020-10-10 00:00:00
Proceeding: Findings of the Association for Computational Linguistics 2020
Tasks: Formality Style Transfer, Language Modeling, Language Modelling, Semi-Supervised Formality Style Transfer, Sentence, Style Transfer, Text Style Transfer, Unsupervised Text Style Transfer

Formality style transfer is the task of converting informal sentences to grammatically-correct formal sentences, which can be used to improve performance of many downstream NLP tasks. In this work, we propose a semi-supervised formality style transfer model that utilizes a language model-based discriminator to maximize the likelihood of the output sentence being formal, which allows us to use maximization of token-level conditional probabilities for training. We further propose to maximize mutual information between source and target styles as our training objective instead of maximizing the regular likelihood that often leads to repetitive and trivial generated responses. Experiments showed that our model outperformed previous state-of-the-art baselines significantly in terms of both automated metrics and human judgement. We further generalized our model to unsupervised text style transfer task, and achieved significant improvements on two benchmark sentiment style transfer datasets.
Noise in Classification
Authors: Maria-Florina Balcan, Nika Haghtalab
Date: 2020-10-10 00:00:00
Tasks: Classification, General Classification

This chapter considers the computational and statistical aspects of learning linear thresholds in presence of noise. When there is no noise, several algorithms exist that efficiently learn near-optimal linear thresholds using a small amount of data. However, even a small amount of adversarial noise makes this problem notoriously hard in the worst-case. We discuss approaches for dealing with these negative results by exploiting natural assumptions on the data-generating process.
Maximin Optimization for Binary Regression
Authors: Nisan Chiprut, Amir Globerson, Ami Wiesel
Date: 2020-10-10 00:00:00
Tasks: regression

We consider regression problems with binary weights. Such optimization problems are ubiquitous in quantized learning models and digital communication systems. A natural approach is to optimize the corresponding Lagrangian using variants of the gradient ascent-descent method. Such maximin techniques are still poorly understood even in the concave-convex case. The non-convex binary constraints may lead to spurious local minima. Interestingly, we prove that this approach is optimal in linear regression with low noise conditions as well as robust regression with a small number of outliers. Practically, the method also performs well in regression with cross entropy loss, as well as non-convex multi-layer neural networks. Taken together our approach highlights the potential of saddle-point optimization for learning constrained models.
Unsupervised Neural Networks for Quantum Eigenvalue Problems
Authors: Henry Jin, Marios Mattheakis, Pavlos Protopapas
Date: 2020-10-10 00:00:00

Eigenvalue problems are critical to several fields of science and engineering. We present a novel unsupervised neural network for discovering eigenfunctions and eigenvalues for differential eigenvalue problems with solutions that identically satisfy the boundary conditions. A scanning mechanism is embedded allowing the method to find an arbitrary number of solutions. The network optimization is data-free and depends solely on the predictions. The unsupervised method is used to solve the quantum infinite well and quantum oscillator eigenvalue problems.
Semi-supervised Neural Networks solve an inverse problem for modeling Covid-19 spread
Authors: Alessandro Paticchio, Tommaso Scarlatti, Marios Mattheakis, Pavlos Protopapas, Marco Brambilla
Date: 2020-10-10 00:00:00

Studying the dynamics of COVID-19 is of paramount importance to understanding the efficiency of restrictive measures and develop strategies to defend against upcoming contagion waves. In this work, we study the spread of COVID-19 using a semi-supervised neural network and assuming a passive part of the population remains isolated from the virus dynamics. We start with an unsupervised neural network that learns solutions of differential equations for different modeling parameters and initial conditions. A supervised method then solves the inverse problem by estimating the optimal conditions that generate functions to fit the data for those infected by, recovered from, and deceased due to COVID-19. This semi-supervised approach incorporates real data to determine the evolution of the spread, the passive population, and the basic reproduction number for different countries.
Exathlon: A Benchmark for Explainable Anomaly Detection over Time Series
Authors: Vincent Jacob, Fei Song, Arnaud Stiegler, Bijan Rad, Yanlei Diao, Nesime Tatbul
Date: 2020-10-10 00:00:00
Tasks: Anomaly Detection, Time Series, Time Series Analysis

Access to high-quality data repositories and benchmarks have been instrumental in advancing the state of the art in many experimental research domains. While advanced analytics tasks over time series data have been gaining lots of attention, lack of such community resources severely limits scientific progress. In this paper, we present Exathlon, the first comprehensive public benchmark for explainable anomaly detection over high-dimensional time series data. Exathlon has been systematically constructed based on real data traces from repeated executions of large-scale stream processing jobs on an Apache Spark cluster. Some of these executions were intentionally disturbed by introducing instances of six different types of anomalous events (e.g., misbehaving inputs, resource contention, process failures). For each of the anomaly instances, ground truth labels for the root cause interval as well as those for the extended effect interval are provided, supporting the development and evaluation of a wide range of anomaly detection (AD) and explanation discovery (ED) tasks. We demonstrate the practical utility of Exathlon's dataset, evaluation methodology, and end-to-end data science pipeline design through an experimental study with three state-of-the-art AD and ED techniques.
Hybrid-S2S: Video Object Segmentation with Recurrent Networks and Correspondence Matching
Authors: Fatemeh Azimi, Stanislav Frolov, Federico Raue, Joern Hees, Andreas Dengel
Date: 2020-10-10 00:00:00
Tasks: One-shot visual object segmentation, Segmentation, Semantic Segmentation, Video Object Segmentation, Video Semantic Segmentation

One-shot Video Object Segmentation~(VOS) is the task of pixel-wise tracking an object of interest within a video sequence, where the segmentation mask of the first frame is given at inference time. In recent years, Recurrent Neural Networks~(RNNs) have been widely used for VOS tasks, but they often suffer from limitations such as drift and error propagation. In this work, we study an RNN-based architecture and address some of these issues by proposing a hybrid sequence-to-sequence architecture named HS2S, utilizing a dual mask propagation strategy that allows incorporating the information obtained from correspondence matching. Our experiments show that augmenting the RNN with correspondence matching is a highly effective solution to reduce the drift problem. The additional information helps the model to predict more accurate masks and makes it robust against error propagation. We evaluate our HS2S model on the DAVIS2017 dataset as well as Youtube-VOS. On the latter, we achieve an improvement of 11.2pp in the overall segmentation accuracy over RNN-based state-of-the-art methods in VOS. We analyze our model's behavior in challenging cases such as occlusion and long sequences and show that our hybrid architecture significantly enhances the segmentation quality in these difficult scenarios.
Fairness-aware Agnostic Federated Learning
Authors: Wei Du, Depeng Xu, Xintao Wu, Hanghang Tong
Date: 2020-10-10 00:00:00
Tasks: Fairness, Federated Learning

Federated learning is an emerging framework that builds centralized machine learning models with training data distributed across multiple devices. Most of the previous works about federated learning focus on the privacy protection and communication cost reduction. However, how to achieve fairness in federated learning is under-explored and challenging especially when testing data distribution is different from training distribution or even unknown. Introducing simple fairness constraints on the centralized model cannot achieve model fairness on unknown testing data. In this paper, we develop a fairness-aware agnostic federated learning framework (AgnosticFair) to deal with the challenge of unknown testing distribution. We use kernel reweighing functions to assign a reweighing value on each training sample in both loss function and fairness constraint. Therefore, the centralized model built from AgnosticFair can achieve high accuracy and fairness guarantee on unknown testing data. Moreover, the built model can be directly applied to local sites as it guarantees fairness on local data distributions. To our best knowledge, this is the first work to achieve fairness in federated learning. Experimental results on two real datasets demonstrate the effectiveness in terms of both utility and fairness under data shift scenarios.
TOTOPO: Classifying univariate and multivariate time series with Topological Data Analysis
Authors: Polina Pilyugina, Rodrigo Rivera-Castro, Eugeny Burnaev
Date: 2020-10-10 00:00:00
Proceeding: NeurIPS Workshop TDA_and_Beyond 2020 12
Tasks: Benchmarking, Time Series, Time Series Analysis, Topological Data Analysis

This work is devoted to a comprehensive analysis of topological data analysis fortime series classification. Previous works have significant shortcomings, such aslack of large-scale benchmarking or missing state-of-the-art methods. In this work,we propose TOTOPO for extracting topological descriptors from different types ofpersistence diagrams. The results suggest that TOTOPO significantly outperformsexisting baselines in terms of accuracy. TOTOPO is also competitive with thestate-of-the-art, being the best on 20% of univariate and 40% of multivariate timeseries datasets. This work validates the hypothesis that TDA-based approaches arerobust to small perturbations in data and are useful for cases where periodicity andshape help discriminate between classes.
Symbolic Parallel Adaptive Importance Sampling for Probabilistic Program Analysis
Authors: Yicheng Luo, Antonio Filieri, Yuan Zhou
Date: 2020-10-10 00:00:00
Tasks: Probabilistic Programming

Probabilistic software analysis aims at quantifying the probability of a target event occurring during the execution of a program processing uncertain incoming data or written itself using probabilistic programming constructs. Recent techniques combine symbolic execution with model counting or solution space quantification methods to obtain accurate estimates of the occurrence probability of rare target events, such as failures in a mission-critical system. However, they face several scalability and applicability limitations when analyzing software processing with high-dimensional and correlated multivariate input distributions. In this paper, we present SYMbolic Parallel Adaptive Importance Sampling (SYMPAIS), a new inference method tailored to analyze path conditions generated from the symbolic execution of programs with high-dimensional, correlated input distributions. SYMPAIS combines results from importance sampling and constraint solving to produce accurate estimates of the satisfaction probability for a broad class of constraints that cannot be analyzed by current solution space quantification methods. We demonstrate SYMPAIS's generality and performance compared with state-of-the-art alternatives on a set of problems from different application domains.
A Predictive Autoscaler for Elastic Batch Jobs
Authors: Peng Gao
Date: 2020-10-10 00:00:00
Tasks: Scheduling, Time Series, Time Series Analysis

Large batch jobs such as Deep Learning, HPC and Spark require far more computational resources and higher cost than conventional online service. Like the processing of other time series data, these jobs possess a variety of characteristics such as trend, burst, and seasonality. Cloud providers offer short-term instances to achieve scalability, stability, and cost-efficiency. Given the time lag caused by joining into the cluster and initialization, crowded workloads may lead to a violation in the scheduling system. Based on the assumption that there are infinite resources and ideal placements available for users to require in the cloud environment, we propose a predictive autoscaler to provide an elastic interface for the customers and overprovision instances based on the trained regression model. We contribute to a method to embed heterogeneous resource requirements in continuous space into discrete resource buckets and an autoscaler to do predictive expand plans on the time series of resource bucket counts. Our experimental evaluation of the production resources usage data validates the solution and the results show that the predictive autoscaler relieves the burden of making scaling plans, avoids long launching time at lower cost and outperforms other prediction methods with fine-tuned settings.
Interpreting Multivariate Shapley Interactions in DNNs
Authors: Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, Quanshi Zhang
Date: 2020-10-10 00:00:00

This paper aims to explain deep neural networks (DNNs) from the perspective of multivariate interactions. In this paper, we define and quantify the significance of interactions among multiple input variables of the DNN. Input variables with strong interactions usually form a coalition and reflect prototype features, which are memorized and used by the DNN for inference. We define the significance of interactions based on the Shapley value, which is designed to assign the attribution value of each input variable to the inference. We have conducted experiments with various DNNs. Experimental results have demonstrated the effectiveness of the proposed method.