pwc-archive/papers-with-abstracts huggingface dataset:
A Transformer-based Framework for Multivariate Time Series Representation Learning
Authors: George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, Carsten Eickhoff
Date: 2020-10-06 00:00:00
Tasks: Computational Efficiency, General Classification, Imputation, regression, Representation Learning, Time Series, Time Series Analysis, Time Series Classification, Time Series Regression, Unsupervised Pre-training

In this work we propose for the first time a transformer-based framework for unsupervised representation learning of multivariate time series. Pre-trained models can be potentially used for downstream tasks such as regression and classification, forecasting and missing value imputation. By evaluating our models on several benchmark datasets for multivariate time series regression and classification, we show that not only does our modeling approach represent the most successful method employing unsupervised learning of multivariate time series presented to date, but also that it exceeds the current state-of-the-art performance of supervised methods; it does so even when the number of training samples is very limited, while offering computational efficiency. Finally, we demonstrate that unsupervised pre-training of our transformer models offers a substantial performance benefit over fully supervised learning, even without leveraging additional unlabeled data, i.e., by reusing the same data samples through the unsupervised objective.
COSMIC: COmmonSense knowledge for eMotion Identification in Conversations
Authors: Deepanway Ghosal, Navonil Majumder, Alexander Gelbukh, Rada Mihalcea, Soujanya Poria
Date: 2020-10-06 00:00:00
Proceeding: Findings of the Association for Computational Linguistics 2020
Tasks: Emotion Recognition, Emotion Recognition in Conversation

In this paper, we address the task of utterance level emotion recognition in conversations using commonsense knowledge. We propose COSMIC, a new framework that incorporates different elements of commonsense such as mental states, events, and causal relations, and build upon them to learn interactions between interlocutors participating in a conversation. Current state-of-the-art methods often encounter difficulties in context propagation, emotion shift detection, and differentiating between related emotion classes. By learning distinct commonsense representations, COSMIC addresses these challenges and achieves new state-of-the-art results for emotion recognition on four different benchmark conversational datasets. Our code is available at https://github.com/declare-lab/conv-emotion.
An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks
Authors: Lifu Tu, Tianyu Liu, Kevin Gimpel
Date: 2020-10-06 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Machine Translation, Representation Learning, Semantic Role Labeling, Translation

Many tasks in natural language processing involve predicting structured outputs, e.g., sequence labeling, semantic role labeling, parsing, and machine translation. Researchers are increasingly applying deep representation learning to these problems, but the structured component of these approaches is usually quite simplistic. In this work, we propose several high-order energy terms to capture complex dependencies among labels in sequence labeling, including several that consider the entire label sequence. We use neural parameterizations for these energy terms, drawing from convolutional, recurrent, and self-attention networks. We use the framework of learning energy-based inference networks (Tu and Gimpel, 2018) for dealing with the difficulties of training and inference with such models. We empirically demonstrate that this approach achieves substantial improvement using a variety of high-order energy terms on four sequence labeling tasks, while having the same decoding speed as simple, local classifiers. We also find high-order energies to help in noisy data conditions.
A Multi-Task Incremental Learning Framework with Category Name Embedding for Aspect-Category Sentiment Analysis
Authors: Zehui Dai, Cheng Peng, Huajie Chen, Yadong Ding
Date: 2020-10-06 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Aspect Category Sentiment Analysis, Decoder, Incremental Learning, Multi-Task Learning, Sentence, Sentiment Analysis

(T)ACSA tasks, including aspect-category sentiment analysis (ACSA) and targeted aspect-category sentiment analysis (TACSA), aims at identifying sentiment polarity on predefined categories. Incremental learning on new categories is necessary for (T)ACSA real applications. Though current multi-task learning models achieve good performance in (T)ACSA tasks, they suffer from catastrophic forgetting problems in (T)ACSA incremental learning tasks. In this paper, to make multi-task learning feasible for incremental learning, we proposed Category Name Embedding network (CNE-net). We set both encoder and decoder shared among all categories to weaken the catastrophic forgetting problem. Besides the origin input sentence, we applied another input feature, i.e., category name, for task discrimination. Our model achieved state-of-the-art on two (T)ACSA benchmark datasets. Furthermore, we proposed a dataset for (T)ACSA incremental learning and achieved the best performance compared with other strong baselines.
Friendship is All we Need: A Multi-graph Embedding Approach for Modeling Customer Behavior
Authors: Amir Jalilifard, Dehua Chen, Lucas Pereira Lopes, Isaac Ben-Akiva, Pedro Henrique Gonçalves Inazawa
Date: 2020-10-06 00:00:00
Tasks: All, Graph Embedding, Recommendation Systems

Understanding customer behavior is fundamental for many use-cases in industry, especially in accelerated growth areas such as fin-tech and e-commerce. Structured data are often expensive, time-consuming and inadequate to analyze and study complex customer behaviors. In this paper, we propose a multi-graph embedding approach for creating a non-linear representation of customers in order to have a better knowledge of their characteristics without having any prior information about their financial status or their interests. By applying the current method we are able to predict users' future behavior with a reasonably high accuracy only by having the information of their friendship network. Potential applications include recommendation systems and credit risk forecasting.
Compressing Deep Convolutional Neural Networks by Stacking Low-dimensional Binary Convolution Filters
Authors: Weichao Lan, Liang Lan
Date: 2020-10-06 00:00:00
Tasks: Model Compression

Deep Convolutional Neural Networks (CNN) have been successfully applied to many real-life problems. However, the huge memory cost of deep CNN models poses a great challenge of deploying them on memory-constrained devices (e.g., mobile phones). One popular way to reduce the memory cost of deep CNN model is to train binary CNN where the weights in convolution filters are either 1 or -1 and therefore each weight can be efficiently stored using a single bit. However, the compression ratio of existing binary CNN models is upper bounded by around 32. To address this limitation, we propose a novel method to compress deep CNN model by stacking low-dimensional binary convolution filters. Our proposed method approximates a standard convolution filter by selecting and stacking filters from a set of low-dimensional binary convolution filters. This set of low-dimensional binary convolution filters is shared across all filters for a given convolution layer. Therefore, our method will achieve much larger compression ratio than binary CNN models. In order to train our proposed model, we have theoretically shown that our proposed model is equivalent to select and stack intermediate feature maps generated by low-dimensional binary filters. Therefore, our proposed model can be efficiently trained using the split-transform-merge strategy. We also provide detailed analysis of the memory and computation cost of our model in model inference. We compared the proposed method with other five popular model compression techniques on two benchmark datasets. Our experimental results have demonstrated that our proposed method achieves much higher compression ratio than existing methods while maintains comparable accuracy.
InstaHide: Instance-hiding Schemes for Private Distributed Learning
Authors: Yangsibo Huang, Zhao Song, Kai Li, Sanjeev Arora
Date: 2020-10-06 00:00:00

How can multiple distributed entities collaboratively train a shared deep net on their private data while preserving privacy? This paper introduces InstaHide, a simple encryption of training images, which can be plugged into existing distributed deep learning pipelines. The encryption is efficient and applying it during training has minor effect on test accuracy. InstaHide encrypts each training image with a "one-time secret key" which consists of mixing a number of randomly chosen images and applying a random pixel-wise mask. Other contributions of this paper include: (a) Using a large public dataset (e.g. ImageNet) for mixing during its encryption, which improves security. (b) Experimental results to show effectiveness in preserving privacy against known attacks with only minor effects on accuracy. (c) Theoretical analysis showing that successfully attacking privacy requires attackers to solve a difficult computational problem. (d) Demonstrating that use of the pixel-wise mask is important for security, since Mixup alone is shown to be insecure to some some efficient attacks. (e) Release of a challenge dataset https://github.com/Hazelsuko07/InstaHide_Challenge Our code is available at https://github.com/Hazelsuko07/InstaHide
High Speed Event Camera TRacking
Authors: William Chamorro, Juan Andrade-Cetto, Joan Solà
Date: 2020-10-06 00:00:00
Tasks: Vocal Bursts Intensity Prediction

Event cameras are bioinspired sensors with reaction times in the order of microseconds. This property makes them appealing for use in highly-dynamic computer vision applications. In this work,we explore the limits of this sensing technology and present an ultra-fast tracking algorithm able to estimate six-degree-of-freedom motion with dynamics over 25.8 g, at a throughput of 10 kHz,processing over a million events per second. Our method is capable of tracking either camera motion or the motion of an object in front of it, using an error-state Kalman filter formulated in a Lie-theoretic sense. The method includes a robust mechanism for the matching of events with projected line segments with very fast outlier rejection. Meticulous treatment of sparse matrices is applied to achieve real-time performance. Different motion models of varying complexity are considered for the sake of comparison and performance analysis
Averaging Atmospheric Gas Concentration Data using Wasserstein Barycenters
Authors: Mathieu Barré, Clément Giron, Matthieu Mazzolini, Alexandre d'Aspremont
Date: 2020-10-06 00:00:00

Hyperspectral satellite images report greenhouse gas concentrations worldwide on a daily basis. While taking simple averages of these images over time produces a rough estimate of relative emission rates, atmospheric transport means that simple averages fail to pinpoint the source of these emissions. We propose using Wasserstein barycenters coupled with weather data to average gas concentration data sets and better concentrate the mass around significant sources.
Inferring Microbial Biomass Yield and Cell Weight using Probabilistic Macrochemical Modeling
Authors: Antonio R. Paiva, Giovanni Pilloni
Date: 2020-10-06 00:00:00

Growth rates and biomass yields are key descriptors used in microbiology studies to understand how microbial species respond to changes in the environment. Of these, biomass yield estimates are typically obtained using cell counts and measurements of the feed substrate. These quantities are perturbed with measurement noise however. Perhaps most crucially, estimating biomass from cell counts, as needed to assess yields, relies on an assumed cell weight. Noise and discrepancies on these assumptions can lead to significant changes in conclusions regarding the microbes' response. This article proposes a methodology to address these challenges using probabilistic macrochemical models of microbial growth. It is shown that a model can be developed to fully use the experimental data, relax assumptions and greatly improve robustness to a priori estimates of the cell weight, and provides uncertainty estimates of key parameters. This methodology is demonstrated in the context of a specific case study and the estimation characteristics are validated in several scenarios using synthetically generated microbial growth data.
Learning Diverse Options via InfoMax Termination Critic
Authors: Yuji Kanagawa, Tomoyuki Kaneko
Date: 2020-10-06 00:00:00
Tasks: Continuous Control, Diversity, reinforcement-learning, Reinforcement Learning (RL), Transfer Learning

We consider the problem of autonomously learning reusable temporally extended actions, or options, in reinforcement learning. While options can speed up transfer learning by serving as reusable building blocks, learning reusable options for unknown task distribution remains challenging. Motivated by the recent success of mutual information (MI) based skill learning, we hypothesize that more diverse options are more reusable. To this end, we propose a method for learning termination conditions of options by maximizing MI between options and corresponding state transitions. We derive a scalable approximation of this MI maximization via gradient ascent, yielding the InfoMax Termination Critic (IMTC) algorithm. Our experiments demonstrate that IMTC significantly improves the diversity of learned options without extrinsic rewards combined with an intrinsic option learning method. Moreover, we test the reusability of learned options by transferring options into various tasks, confirming that IMTC helps quick adaptation, especially in complex domains where an agent needs to manipulate objects.
Stepwise Extractive Summarization and Planning with Structured Transformers
Authors: Shashi Narayan, Joshua Maynez, Jakub Adamek, Daniele Pighin, Blaž Bratanič, Ryan Mcdonald
Date: 2020-10-06 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Extractive Summarization, Sentence, Table-to-Text Generation, Text Generation

We propose encoder-centric stepwise models for extractive summarization using structured transformers -- HiBERT and Extended Transformers. We enable stepwise summarization by injecting the previously generated summary into the structured transformer as an auxiliary sub-structure. Our models are not only efficient in modeling the structure of long inputs, but they also do not rely on task-specific redundancy-aware modeling, making them a general purpose extractive content planner for different tasks. When evaluated on CNN/DailyMail extractive summarization, stepwise models achieve state-of-the-art performance in terms of Rouge without any redundancy aware modeling or sentence filtering. This also holds true for Rotowire table-to-text generation, where our models surpass previously reported metrics for content selection, planning and ordering, highlighting the strength of stepwise modeling. Amongst the two structured transformers we test, stepwise Extended Transformers provides the best performance across both datasets and sets a new standard for these challenges.
Notes on Coalgebras in Stylometry
Authors: Joël A. Doat
Date: 2020-10-06 00:00:00

The syntactic behaviour of texts can highly vary depending on their contexts (e.g. author, genre, etc.). From the standpoint of stylometry, it can be helpful to objectively measure this behaviour. In this paper, we discuss how coalgebras are used to formalise the notion of behaviour by embedding syntactic features of a given text into probabilistic transition systems. By introducing the behavioural distance, we are then able to quantitatively measure differences between points in these systems and thus, comparing features of different texts. Furthermore, the behavioural distance of points can be approximated by a polynomial-time algorithm.
Assisted Probe Positioning for Ultrasound Guided Radiotherapy Using Image Sequence Classification
Authors: Alexander Grimwood, Helen McNair, Yipeng Hu, Ester Bonmati, Dean Barratt, Emma Harris
Date: 2020-10-06 00:00:00
Tasks: Anatomy, General Classification

Effective transperineal ultrasound image guidance in prostate external beam radiotherapy requires consistent alignment between probe and prostate at each session during patient set-up. Probe placement and ultrasound image inter-pretation are manual tasks contingent upon operator skill, leading to interoperator uncertainties that degrade radiotherapy precision. We demonstrate a method for ensuring accurate probe placement through joint classification of images and probe position data. Using a multi-input multi-task algorithm, spatial coordinate data from an optically tracked ultrasound probe is combined with an image clas-sifier using a recurrent neural network to generate two sets of predictions in real-time. The first set identifies relevant prostate anatomy visible in the field of view using the classes: outside prostate, prostate periphery, prostate centre. The second set recommends a probe angular adjustment to achieve alignment between the probe and prostate centre with the classes: move left, move right, stop. The algo-rithm was trained and tested on 9,743 clinical images from 61 treatment sessions across 32 patients. We evaluated classification accuracy against class labels de-rived from three experienced observers at 2/3 and 3/3 agreement thresholds. For images with unanimous consensus between observers, anatomical classification accuracy was 97.2% and probe adjustment accuracy was 94.9%. The algorithm identified optimal probe alignment within a mean (standard deviation) range of 3.7$^{\circ}$ (1.2$^{\circ}$) from angle labels with full observer consensus, comparable to the 2.8$^{\circ}$ (2.6$^{\circ}$) mean interobserver range. We propose such an algorithm could assist ra-diotherapy practitioners with limited experience of ultrasound image interpreta-tion by providing effective real-time feedback during patient set-up.
Vec2Instance: Parameterization for Deep Instance Segmentation
Authors: N. Lakmal Deshapriya, Matthew N. Dailey, Manzul Kumar Hazarika, Hiroyuki Miyazaki
Date: 2020-10-06 00:00:00
Tasks: Instance Segmentation, Segmentation, Semantic Segmentation

Current advances in deep learning is leading to human-level accuracy in computer vision tasks such as object classification, localization, semantic segmentation, and instance segmentation. In this paper, we describe a new deep convolutional neural network architecture called Vec2Instance for instance segmentation. Vec2Instance provides a framework for parametrization of instances, allowing convolutional neural networks to efficiently estimate the complex shapes of instances around their centroids. We demonstrate the feasibility of the proposed architecture with respect to instance segmentation tasks on satellite images, which have a wide range of applications. Moreover, we demonstrate the usefulness of the new method for extracting building foot-prints from satellite images. Total pixel-wise accuracy of our approach is 89\%, near the accuracy of the state-of-the-art Mask RCNN (91\%). Vec2Instance is an alternative approach to complex instance segmentation pipelines, offering simplicity and intuitiveness. The code developed under this study is available in the Vec2Instance GitHub repository, https://github.com/lakmalnd/Vec2Instance
Learnable Uncertainty under Laplace Approximations
Authors: Agustinus Kristiadi, Matthias Hein, Philipp Hennig
Date: 2020-10-06 00:00:00
Tasks: Uncertainty Quantification

Laplace approximations are classic, computationally lightweight means for constructing Bayesian neural networks (BNNs). As in other approximate BNNs, one cannot necessarily expect the induced predictive uncertainty to be calibrated. Here we develop a formalism to explicitly "train" the uncertainty in a decoupled way to the prediction itself. To this end, we introduce uncertainty units for Laplace-approximated networks: Hidden units associated with a particular weight structure that can be added to any pre-trained, point-estimated network. Due to their weights, these units are inactive -- they do not affect the predictions. But their presence changes the geometry (in particular the Hessian) of the loss landscape, thereby affecting the network's uncertainty estimates under a Laplace approximation. We show that such units can be trained via an uncertainty-aware objective, improving standard Laplace approximations' performance in various uncertainty quantification tasks.
An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence
Authors: Agustinus Kristiadi, Matthias Hein, Philipp Hennig
Date: 2020-10-06 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Multi-class Classification

A Bayesian treatment can mitigate overconfidence in ReLU nets around the training data. But far away from them, ReLU Bayesian neural networks (BNNs) can still underestimate uncertainty and thus be asymptotically overconfident. This issue arises since the output variance of a BNN with finitely many features is quadratic in the distance from the data region. Meanwhile, Bayesian linear models with ReLU features converge, in the infinite-width limit, to a particular Gaussian process (GP) with a variance that grows cubically so that no asymptotic overconfidence can occur. While this may seem of mostly theoretical interest, in this work, we show that it can be used in practice to the benefit of BNNs. We extend finite ReLU BNNs with infinite ReLU features via the GP and show that the resulting model is asymptotically maximally uncertain far away from the data while the BNNs' predictive power is unaffected near the data. Although the resulting model approximates a full GP posterior, thanks to its structure, it can be applied \emph{post-hoc} to any pre-trained ReLU BNN at a low cost.
Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation
Authors: Minki Kang, Moonsu Han, Sung Ju Hwang
Date: 2020-10-06 00:00:00
Proceeding: EMNLP 2020 11
Tasks: Language Modeling, Language Modelling, Question Answering, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), text-classification, Text Classification

We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel reinforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task performance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy network that can consider the relative importance of words in a given text. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings.