pwc-archive/papers-with-abstracts huggingface dataset:
Modern Hopfield Networks and Attention for Immune Repertoire Classification
Authors: Michael Widrich, Bernhard Schäfl, Hubert Ramsauer, Milena Pavlović, Lukas Gruber, Markus Holzleitner, Johannes Brandstetter, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, Günter Klambauer
Date: 2020-07-16 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: BIG-bench Machine Learning, Classification, General Classification, Immune Repertoire Classification, Interpretable Machine Learning, Multiple Instance Learning

A central mechanism in machine learning is to identify, store, and recognize patterns. How to learn, access, and retrieve such patterns is crucial in Hopfield networks and the more recent transformer architectures. We show that the attention mechanism of transformer architectures is actually the update rule of modern Hopfield networks that can store exponentially many patterns. We exploit this high storage capacity of modern Hopfield networks to solve a challenging multiple instance learning (MIL) problem in computational biology: immune repertoire classification. Accurate and interpretable machine learning methods solving this problem could pave the way towards new vaccines and therapies, which is currently a very relevant research topic intensified by the COVID-19 crisis. Immune repertoire classification based on the vast number of immunosequences of an individual is a MIL problem with an unprecedentedly massive number of instances, two orders of magnitude larger than currently considered problems, and with an extremely low witness rate. In this work, we present our novel method DeepRC that integrates transformer-like attention, or equivalently modern Hopfield networks, into deep learning architectures for massive MIL such as immune repertoire classification. We demonstrate that DeepRC outperforms all other methods with respect to predictive performance on large-scale experiments, including simulated and real-world virus infection data, and enables the extraction of sequence motifs that are connected to a given disease class. Source code and datasets: https://github.com/ml-jku/DeepRC
Add a SideNet to your MainNet
Authors: Adrien Morisot
Date: 2020-07-14 00:00:00
Tasks: General Classification, Knowledge Distillation, text-classification, Text Classification

As the performance and popularity of deep neural networks has increased, so too has their computational cost. There are many effective techniques for reducing a network's computational footprint (quantisation, pruning, knowledge distillation), but these lead to models whose computational cost is the same regardless of their input. Our human reaction times vary with the complexity of the tasks we perform: easier tasks (e.g. telling apart dogs from boat) are executed much faster than harder ones (e.g. telling apart two similar looking breeds of dogs). Driven by this observation, we develop a method for adaptive network complexity by attaching a small classification layer, which we call SideNet, to a large pretrained network, which we call MainNet. Given an input, the SideNet returns a classification if its confidence level, obtained via softmax, surpasses a user determined threshold, and only passes it along to the large MainNet for further processing if its confidence is too low. This allows us to flexibly trade off the network's performance with its computational cost. Experimental results show that simple single hidden layer perceptron SideNets added onto pretrained ResNet and BERT MainNets allow for substantial decreases in compute with minimal drops in performance on image and text classification tasks. We also highlight three other desirable properties of our method, namely that the classifications obtained by SideNets are calibrated, complementary to other compute reduction techniques, and that they enable the easy exploration of compute accuracy space.
Dynamic Relational Inference in Multi-Agent Trajectories
Authors: Ruichao Xiao, Manish Kumar Singh, Rose Yu
Date: 2020-07-16 00:00:00

Inferring interactions from multi-agent trajectories has broad applications in physics, vision and robotics. Neural relational inference (NRI) is a deep generative model that can reason about relations in complex dynamics without supervision. In this paper, we take a careful look at this approach for relational inference in multi-agent trajectories. First, we discover that NRI can be fundamentally limited without sufficient long-term observations. Its ability to accurately infer interactions degrades drastically for short output sequences. Next, we consider a more general setting of relational inference when interactions are changing overtime. We propose an extension ofNRI, which we call the DYnamic multi-AgentRelational Inference (DYARI) model that can reason about dynamic relations. We conduct exhaustive experiments to study the effect of model architecture, under-lying dynamics and training scheme on the performance of dynamic relational inference using a simulated physics system. We also showcase the usage of our model on real-world multi-agent basketball trajectories.
Second Order PAC-Bayesian Bounds for the Weighted Majority Vote
Authors: Andrés R. Masegosa, Stephan S. Lorenzen, Christian Igel, Yevgeny Seldin
Date: 2020-07-01 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Binary Classification, Classification, General Classification

We present a novel analysis of the expected risk of weighted majority vote in multiclass classification. The analysis takes correlation of predictions by ensemble members into account and provides a bound that is amenable to efficient minimization, which yields improved weighting for the majority vote. We also provide a specialized version of our bound for binary classification, which allows to exploit additional unlabeled data for tighter risk estimation. In experiments, we apply the bound to improve weighting of trees in random forests and show that, in contrast to the commonly used first order bound, minimization of the new bound typically does not lead to degradation of the test error of the ensemble.
Cognitive Biomarker Prioritization in Alzheimer's Disease using Brain Morphometric Data
Authors: Bo Peng, Xiaohui Yao, Shannon L. Risacher, Andrew J. Saykin, Li Shen, Xia Ning
Date: 2020-02-18 00:00:00
Tasks: Learning-To-Rank

Background:Cognitive assessments represent the most common clinical routine for the diagnosis of Alzheimer's Disease (AD). Given a large number of cognitive assessment tools and time-limited office visits, it is important to determine a proper set of cognitive tests for different subjects. Most current studies create guidelines of cognitive test selection for a targeted population, but they are not customized for each individual subject. In this manuscript, we develop a machine learning paradigm enabling personalized cognitive assessments prioritization. Method: We adapt a newly developed learning-to-rank approach PLTR to implement our paradigm. This method learns the latent scoring function that pushes the most effective cognitive assessments onto the top of the prioritization list. We also extend PLTR to better separate the most effective cognitive assessments and the less effective ones. Results: Our empirical study on the ADNI data shows that the proposed paradigm outperforms the state-of-the-art baselines on identifying and prioritizing individual-specific cognitive biomarkers. We conduct experiments in cross validation and level-out validation settings. In the two settings, our paradigm significantly outperforms the best baselines with improvement as much as 22.1% and 19.7%, respectively, on prioritizing cognitive features. Conclusions: The proposed paradigm achieves superior performance on prioritizing cognitive biomarkers. The cognitive biomarkers prioritized on top have great potentials to facilitate personalized diagnosis, disease subtyping, and ultimately precision medicine in AD.
A unified survey of treatment effect heterogeneity modeling and uplift modeling
Authors: Weijia Zhang, Jiuyong Li, Lin Liu
Date: 2020-07-14 00:00:00
Tasks: Marketing, Survey

A central question in many fields of scientific research is to determine how an outcome would be affected by an action, or to measure the effect of an action (a.k.a treatment effect). In recent years, a need for estimating the heterogeneous treatment effects conditioning on the different characteristics of individuals has emerged from research fields such as personalized healthcare, social science, and online marketing. To meet the need, researchers and practitioners from different communities have developed algorithms by taking the treatment effect heterogeneity modeling approach and the uplift modeling approach, respectively. In this paper, we provide a unified survey of these two seemingly disconnected yet closely related approaches under the potential outcome framework. We then provide a structured survey of existing methods by emphasizing on their inherent connections with a set of unified notations to make comparisons of the different methods easy. We then review the main applications of the surveyed methods in personalized marketing, personalized medicine, and social studies. Finally, we summarize the existing software packages and present discussions based on the use of methods on synthetic, semi-synthetic and real world data sets and provide some general guidelines for choosing methods.
Model-based Clustering using Automatic Differentiation: Confronting Misspecification and High-Dimensional Data
Authors: Siva Rajesh Kasa, Vaibhav Rajan
Date: 2020-07-08 00:00:00
Tasks: Clustering, Model Selection

We study two practically important cases of model based clustering using Gaussian Mixture Models: (1) when there is misspecification and (2) on high dimensional data, in the light of recent advances in Gradient Descent (GD) based optimization using Automatic Differentiation (AD). Our simulation studies show that EM has better clustering performance, measured by Adjusted Rand Index, compared to GD in cases of misspecification, whereas on high dimensional data GD outperforms EM. We observe that both with EM and GD there are many solutions with high likelihood but poor cluster interpretation. To address this problem we design a new penalty term for the likelihood based on the Kullback Leibler divergence between pairs of fitted components. Closed form expressions for the gradients of this penalized likelihood are difficult to derive but AD can be done effortlessly, illustrating the advantage of AD-based optimization. Extensions of this penalty for high dimensional data and for model selection are discussed. Numerical experiments on synthetic and real datasets demonstrate the efficacy of clustering using the proposed penalized likelihood approach.
Energy-based View of Retrosynthesis
Authors: Ruoxi Sun, Hanjun Dai, Li Li, Steven Kearnes, Bo Dai
Date: 2020-07-14 00:00:00
Tasks: Drug Discovery, Retrosynthesis, Single-step retrosynthesis

Retrosynthesis -- the process of identifying a set of reactants to synthesize a target molecule -- is of vital importance to material design and drug discovery. Existing machine learning approaches based on language models and graph neural networks have achieved encouraging results. In this paper, we propose a framework that unifies sequence- and graph-based methods as energy-based models (EBMs) with different energy functions. This unified perspective provides critical insights about EBM variants through a comprehensive assessment of performance. Additionally, we present a novel dual variant within the framework that performs consistent training over Bayesian forward- and backward-prediction by constraining the agreement between the two directions. This model improves state-of-the-art performance by 9.6% for template-free approaches where the reaction type is unknown.
Learning Common Harmonic Waves on Stiefel Manifold -- A New Mathematical Approach for Brain Network Analyses
Authors: Jiazhou Chen, Guoqiang Han, Hongmin Cai, Defu Yang, Paul J. Laurienti, Martin Styner, Guorong Wu, Alzheimer's Disease Neuroimaging Initiative ADNI
Date: 2020-07-01 00:00:00

Converging evidence shows that disease-relevant brain alterations do not appear in random brain locations, instead, its spatial pattern follows large scale brain networks. In this context, a powerful network analysis approach with a mathematical foundation is indispensable to understand the mechanism of neuropathological events spreading throughout the brain. Indeed, the topology of each brain network is governed by its native harmonic waves, which are a set of orthogonal bases derived from the Eigen-system of the underlying Laplacian matrix. To that end, we propose a novel connectome harmonic analysis framework to provide enhanced mathematical insights by detecting frequency-based alterations relevant to brain disorders. The backbone of our framework is a novel manifold algebra appropriate for inference across harmonic waves that overcomes the limitations of using classic Euclidean operations on irregular data structures. The individual harmonic difference is measured by a set of common harmonic waves learned from a population of individual Eigen systems, where each native Eigen-system is regarded as a sample drawn from the Stiefel manifold. Specifically, a manifold optimization scheme is tailored to find the common harmonic waves which reside at the center of Stiefel manifold. To that end, the common harmonic waves constitute the new neuro-biological bases to understand disease progression. Each harmonic wave exhibits a unique propagation pattern of neuro-pathological burdens spreading across brain networks. The statistical power of our novel connectome harmonic analysis approach is evaluated by identifying frequency-based alterations relevant to Alzheimer's disease, where our learning-based manifold approach discovers more significant and reproducible network dysfunction patterns compared to Euclidian methods.
PowerGAN: Synthesizing Appliance Power Signatures Using Generative Adversarial Networks
Authors: Alon Harell, Richard Jones, Stephen Makonin, Ivan V. Bajic
Date: 2020-07-20 00:00:00
Tasks: Generative Adversarial Network, Non-Intrusive Load Monitoring

Non-intrusive load monitoring (NILM) allows users and energy providers to gain insight into home appliance electricity consumption using only the building's smart meter. Most current techniques for NILM are trained using significant amounts of labeled appliances power data. The collection of such data is challenging, making data a major bottleneck in creating well generalizing NILM solutions. To help mitigate the data limitations, we present the first truly synthetic appliance power signature generator. Our solution, PowerGAN, is based on conditional, progressively growing, 1-D Wasserstein generative adversarial network (GAN). Using PowerGAN, we are able to synthesise truly random and realistic appliance power data signatures. We evaluate the samples generated by PowerGAN in a qualitative way as well as numerically by using traditional GAN evaluation methods such as the Inception score.
AMR-To-Text Generation with Graph Transformer
Authors: Tianming Wang, Xiaojun Wan, Hanqi Jin
Date: 2020-01-01 00:00:00
Proceeding: TACL 2020 1
Tasks: Abstract Meaning Representation, AMR-to-Text Generation, Graph-to-Sequence, Text Generation

Abstract meaning representation (AMR)-to-text generation is the challenging task of generating natural language texts from AMR graphs, where nodes represent concepts and edges denote relations. The current state-of-the-art methods use graph-to-sequence models; however, they still cannot significantly outperform the previous sequence-to-sequence models or statistical approaches. In this paper, we propose a novel graph-to-sequence model (Graph Transformer) to address this task. The model directly encodes the AMR graphs and learns the node representations. A pairwise interaction function is used for computing the semantic relations between the concepts. Moreover, attention mechanisms are used for aggregating the information from the incoming and outgoing neighbors, which help the model to capture the semantic information effectively. Our model outperforms the state-of-the-art neural approach by 1.5 BLEU points on LDC2015E86 and 4.8 BLEU points on LDC2017T10 and achieves new state-of-the-art performances.
Target-Guided Structured Attention Network for Target-Dependent Sentiment Analysis
Authors: Ji Zhang, Chengyao Chen, PengFei Liu, Chao He, Cane Wing-Ki Leung
Date: 2020-01-01 00:00:00
Proceeding: TACL 2020 1
Tasks: Sentence, Sentiment Analysis, Sentiment Classification

Target-dependent sentiment analysis (TDSA) aims to classify the sentiment of a text towards a given target. The major challenge of this task lies in modeling the semantic relatedness between a target and its context sentence. This paper proposes a novel Target-Guided Structured Attention Network (TG-SAN), which captures target-related contexts for TDSA in a fine-to-coarse manner. Given a target and its context sentence, the proposed TG-SAN first identifies multiple semantic segments from the sentence using a target-guided structured attention mechanism. It then fuses the extracted segments based on their relatedness with the target for sentiment classification. We present comprehensive comparative experiments on three benchmarks with three major findings. First, TG-SAN outperforms the state-of-the-art by up to 1.61{\%} and 3.58{\%} in terms of accuracy and Marco-F1, respectively. Second, it shows a strong advantage in determining the sentiment of a target when the context sentence contains multiple semantic segments. Lastly, visualization results show that the attention scores produced by TG-SAN are highly interpretable
Acoustic-Prosodic and Lexical Cues to Deception and Trust: Deciphering How People Detect Lies
Authors: Xi (Leslie) Chen, Sarah Ita Levitan, Michelle Levine, M, Marko ic, Julia Hirschberg
Date: 2020-01-01 00:00:00
Proceeding: TACL 2020 1
Tasks: Deception Detection

Humans rarely perform better than chance at lie detection. To better understand human perception of deception, we created a game framework, LieCatcher, to collect ratings of perceived deception using a large corpus of deceptive and truthful interviews. We analyzed the acoustic-prosodic and linguistic characteristics of language trusted and mistrusted by raters and compared these to characteristics of actual truthful and deceptive language to understand how perception aligns with reality. With this data we built classifiers to automatically distinguish trusted from mistrusted speech, achieving an F1 of 66.1{\%}. We next evaluated whether the strategies raters said they used to discriminate between truthful and deceptive responses were in fact useful. Our results show that, although several prosodic and lexical features were consistently perceived as trustworthy, they were not reliable cues. Also, the strategies that judges reported using in deception detection were not helpful for the task. Our work sheds light on the nature of trusted language and provides insight into the challenging problem of human deception detection.
Unsupervised Discourse Constituency Parsing Using Viterbi EM
Authors: Noriki Nishida, Hideki Nakayama
Date: 2020-01-01 00:00:00
Proceeding: TACL 2020 1
Tasks: Constituency Parsing, Discourse Parsing

In this paper, we introduce an unsupervised discourse constituency parsing algorithm. We use Viterbi EM with a margin-based criterion to train a span-based discourse parser in an unsupervised manner. We also propose initialization methods for Viterbi training of discourse constituents based on our prior knowledge of text structures. Experimental results demonstrate that our unsupervised parser achieves comparable or even superior performance to fully supervised parsers. We also investigate discourse constituents that are learned by our method.
Where's My Head? Definition, Data Set, and Models for Numeric Fused-Head Identification and Resolution
Authors: Yanai Elazar, Yoav Goldberg
Date: 2019-03-01 00:00:00
Proceeding: TACL 2019 3
Tasks: Sentence

We provide the first computational treatment of fused-heads constructions (FHs), focusing on the numeric fused-heads (NFHs). FHs constructions are noun phrases in which the head noun is missing and is said to be {``}fused{''} with its dependent modifier. This missing information is implicit and is important for sentence understanding. The missing references are easily filled in by humans but pose a challenge for computational models. We formulate the handling of FHs as a two stages process: Identification of the FH construction and resolution of the missing head. We explore the NFH phenomena in large corpora of English text and create (1) a data set and a highly accurate method for NFH identification; (2) a 10k examples (1 M tokens) crowd-sourced data set of NFH resolution; and (3) a neural baseline for the NFH resolution task. We release our code and data set, to foster further research into this challenging problem.
Measuring Online Debaters' Persuasive Skill from Text over Time
Authors: Kelvin Luu, Chenhao Tan, Noah A. Smith
Date: 2019-03-01 00:00:00
Proceeding: TACL 2019 3

Online debates allow people to express their persuasive abilities and provide exciting opportunities for understanding persuasion. Prior studies have focused on studying persuasion in debate content, but without accounting for each debater{'}s history or exploring the progression of a debater{'}s persuasive ability. We study debater skill by modeling how participants progress over time in a collection of debates from Debate.org. We build on a widely used model of skill in two-player games and augment it with linguistic features of a debater{'}s content. We show that online debaters{'} skill levels do tend to improve over time. Incorporating linguistic profiles leads to more robust skill estimation than winning records alone. Notably, we find that an interaction feature combining uncertainty cues (hedging) with terms strongly associated with either side of a particular debate (fightin{'} words) is more predictive than either feature on its own, indicating the importance of fine- grained linguistic features.
Enabling Robust Grammatical Error Correction in New Domains: Data Sets, Metrics, and Analyses
Authors: Courtney Napoles, Maria N{\u{a}}dejde, Joel Tetreault
Date: 2019-03-01 00:00:00
Proceeding: TACL 2019 3
Tasks: Grammatical Error Correction

Until now, grammatical error correction (GEC) has been primarily evaluated on text written by non-native English speakers, with a focus on student essays. This paper enables GEC development on text written by native speakers by providing a new data set and metric. We present a multiple-reference test corpus for GEC that includes 4,000 sentences in two new domains (formal and informal writing by native English speakers) and 2,000 sentences from a diverse set of non-native student writing. We also collect human judgments of several GEC systems on this new test set and perform a meta-evaluation, assessing how reliable automatic metrics are across these domains. We find that commonly used GEC metrics have inconsistent performance across domains, and therefore we propose a new ensemble metric that is robust on all three domains of text.