pwc-archive/papers-with-abstracts huggingface dataset:
A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Models
Authors: Chris Kedzie, Kathleen McKeown
Date: 2019-11-08 00:00:00
Proceeding: WS 2019 10
Tasks: Decoder, Text Generation

Deep neural networks (DNN) are quickly becoming the de facto standard modeling method for many natural language generation (NLG) tasks. In order for such models to truly be useful, they must be capable of correctly generating utterances for novel meaning representations (MRs) at test time. In practice, even sophisticated DNNs with various forms of semantic control frequently fail to generate utterances faithful to the input MR. In this paper, we propose an architecture agnostic self-training method to sample novel MR/text utterance pairs to augment the original training data. Remarkably, after training on the augmented data, even simple encoder-decoder models with greedy decoding are capable of generating semantically correct utterances that are as good as state-of-the-art outputs in both automatic and human evaluations of quality.
Automatic Identification of Traditional Colombian Music Genres based on Audio Content Analysis and Machine Learning Technique
Authors: Diego A. Cruz, Sergio S. Lopez, Jorge E. Camargo
Date: 2019-11-08 00:00:00
Tasks: BIG-bench Machine Learning, Cultural Vocal Bursts Intensity Prediction, Diversity

Colombia has a diversity of genres in traditional music, which allows to express the richness of the Colombian culture according to the region. This musical diversity is the result of a mixture of African, native Indigenous, and European influences. Organizing large collections of songs is a time consuming task that requires that a human listens to fragments of audio to identify genre, singer, year, instruments and other relevant characteristics that allow to index the song dataset. This paper presents a method to automatically identify the genre of a Colombian song by means of its audio content. The method extracts audio features that are used to train a machine learning model that learns to classify the genre. The method was evaluated in a dataset of 180 musical pieces belonging to six folkloric Colombian music genres: Bambuco, Carranga, Cumbia, Joropo, Pasillo, and Vallenato. Results show that it is possible to automatically identify the music genre in spite of the complexity of Colombian rhythms reaching an average accuracy of 69\%.
SEPT: Improving Scientific Named Entity Recognition with Span Representation
Authors: Tan Yan, He-Yan Huang, Xian-Ling Mao
Date: 2019-11-08 00:00:00
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER)

We introduce a new scientific named entity recognizer called SEPT, which stands for Span Extractor with Pre-trained Transformers. In recent papers, span extractors have been demonstrated to be a powerful model compared with sequence labeling models. However, we discover that with the development of pre-trained language models, the performance of span extractors appears to become similar to sequence labeling models. To keep the advantages of span representation, we modified the model by under-sampling to balance the positive and negative samples and reduce the search space. Furthermore, we simplify the origin network architecture to combine the span extractor with BERT. Experiments demonstrate that even simplified architecture achieves the same performance and SEPT achieves a new state of the art result in scientific named entity recognition even without relation information involved.
Ask to Learn: A Study on Curiosity-driven Question Generation
Authors: Thomas Scialom, Jacopo Staiano
Date: 2019-11-08 00:00:00
Proceeding: COLING 2020 8
Tasks: Conversational Question Answering, Machine Reading Comprehension, Question Answering, Question Generation, Question-Generation, Reading Comprehension, Reinforcement Learning, Text Generation

We propose a novel text generation task, namely Curiosity-driven Question Generation. We start from the observation that the Question Generation task has traditionally been considered as the dual problem of Question Answering, hence tackling the problem of generating a question given the text that contains its answer. Such questions can be used to evaluate machine reading comprehension. However, in real life, and especially in conversational settings, humans tend to ask questions with the goal of enriching their knowledge and/or clarifying aspects of previously gathered information. We refer to these inquisitive questions as Curiosity-driven: these questions are generated with the goal of obtaining new information (the answer) which is not present in the input text. In this work, we experiment on this new task using a conversational Question Answering (QA) dataset; further, since the majority of QA dataset are not built in a conversational manner, we describe a methodology to derive data for this novel task from non-conversational QA data. We investigate several automated metrics to measure the different properties of Curious Questions, and experiment different approaches on the Curiosity-driven Question Generation task, including model pre-training and reinforcement learning. Finally, we report a qualitative evaluation of the generated outputs.
Macro F1 and Macro F1
Authors: Juri Opitz, Sebastian Burst
Date: 2019-11-08 00:00:00
Tasks: Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION

The 'macro F1' metric is frequently used to evaluate binary, multi-class and multi-label classification problems. Yet, we find that there exist two different formulas to calculate this quantity. In this note, we show that only under rare circumstances the two computations can be considered equivalent. More specifically, one formula well 'rewards' classifiers which produce a skewed error type distribution. In fact, the difference in outcome of the two computations can be as high as 0.5. The two computations may not only diverge in their scalar result but can also lead to different classifier rankings.
Content-Consistent Generation of Realistic Eyes with Style
Authors: Marcel Bühler, Seonwook Park, Shalini De Mello, Xucong Zhang, Otmar Hilliges
Date: 2019-11-08 00:00:00
Tasks: Semantic Segmentation

Accurately labeled real-world training data can be scarce, and hence recent works adapt, modify or generate images to boost target datasets. However, retaining relevant details from input data in the generated images is challenging and failure could be critical to the performance on the final task. In this work, we synthesize person-specific eye images that satisfy a given semantic segmentation mask (content), while following the style of a specified person from only a few reference images. We introduce two approaches, (a) one used to win the OpenEDS Synthetic Eye Generation Challenge at ICCV 2019, and (b) a principled approach to solving the problem involving simultaneous injection of style and content information at multiple scales. Our implementation is available at https://github.com/mcbuehler/Seg2Eye.
Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
Authors: Nora Kassner, Hinrich Schütze
Date: 2019-11-08 00:00:00
Proceeding: ACL 2020 6
Tasks: Language Modelling, Negation, Question Answering

Building on Petroni et al. (2019), we propose two new probing tasks analyzing factual knowledge stored in Pretrained Language Models (PLMs). (1) Negation. We find that PLMs do not distinguish between negated ("Birds cannot [MASK]") and non-negated ("Birds can [MASK]") cloze questions. (2) Mispriming. Inspired by priming methods in human psychology, we add "misprimes" to cloze questions ("Talk? Birds can [MASK]"). We find that PLMs are easily distracted by misprimes. These results suggest that PLMs still have a long way to go to adequately learn human-like factual knowledge.
Dynamic Deep Multi-task Learning for Caricature-Visual Face Recognition
Authors: Zuheng Ming, Jean-Christophe Burie, Muhammad Muzzamil Luqman
Date: 2019-11-08 00:00:00
Tasks: Caricature, Face Recognition, Multi-Task Learning

Rather than the visual images, the face recognition of the caricatures is far from the performance of the visual images. The challenge is the extreme non-rigid distortions of the caricatures introduced by exaggerating the facial features to strengthen the characters. In this paper, we propose dynamic multi-task learning based on deep CNNs for cross-modal caricature-visual face recognition. Instead of the conventional multi-task learning with fixed weights of the tasks, the proposed dynamic multi-task learning dynamically updates the weights of tasks according to the importance of the tasks, which enables the training of the networks focus on the hard task instead of being stuck in the overtraining of the easy task. The experimental results demonstrate the effectiveness of the dynamic multi-task learning for caricature-visual face recognition. The performance evaluated on the datasets CaVI and WebCaricature show the superiority over the state-of-art methods. The implementation code is available here.
Comparison of D-Wave Quantum Annealing and Classical Simulated Annealing for Local Minima Determination
Authors: Yaroslav Koshka, M. A. Novotny
Date: 2019-11-08 00:00:00

Restricted Boltzmann Machines trained with different numbers of iterations were used to provide a diverse set of energy functions each containing many local valleys (LVs) with different energies, widths, escape barrier heights, etc. They were used to verify the previously reported possibility of using the D-Wave quantum annealer (QA) to find potentially important LVs in the energy functions of Ising spin glasses that may be missed by classical searches. For classical search, extensive simulated annealing (SA) was conducted to find as many LVs as possible regardless of the computational cost. SA was conducted long enough to ensure that the number of SA-found LVs approaches that and eventually significantly exceeds the number of the LVs found by a single call submitted to the D-Wave. Even after a prohibitively long SA search, as many as 30-50% of the D-Wave-found LVs remained not found by the SA. In order to establish if LVs found only by the D-Wave represent potentially important regions of the configuration space, they were compared to those that were found by both techniques. While the LVs found by the D-Wave but missed by SA predominantly had higher energies and lower escape barriers, there was a significant fraction having intermediate values of the energy and barrier height. With respect to most other important LV parameters, the LVs found only by the D-Wave were distributed in a wide range of the parameters' values. It was established that for large or small, shallow or deep, wide or narrow LVs, the LVs found only by the D-Wave are distinguished by a few-times smaller size of the LV basin of attraction (BoA). Apparently, the size of the BoA is not or at least is less important for QA search compared to the classical search, allowing QA to easily find many potentially important (e.g., wide and deep) LVs missed by even prohibitively lengthy classical searches.
Hierarchical Clustering for Smart Meter Electricity Loads based on Quantile Autocovariances
Authors: Andrés M. Alonso, F. Javier Nogales, Carlos Ruiz
Date: 2019-11-08 00:00:00
Tasks: Clustering, Time Series, Time Series Analysis

In order to improve the efficiency and sustainability of electricity systems, most countries worldwide are deploying advanced metering infrastructures, and in particular household smart meters, in the residential sector. This technology is able to record electricity load time series at a very high frequency rates, information that can be exploited to develop new clustering models to group individual households by similar consumptions patterns. To this end, in this work we propose three hierarchical clustering methodologies that allow capturing different characteristics of the time series. These are based on a set of "dissimilarity" measures computed over different features: quantile auto-covariances, and simple and partial autocorrelations. The main advantage is that they allow summarizing each time series in a few representative features so that they are computationally efficient, robust against outliers, easy to automatize, and scalable to hundreds of thousands of smart meters series. We evaluate the performance of each clustering model in a real-world smart meter dataset with thousands of half-hourly time series. The results show how the obtained clusters identify relevant consumption behaviors of households and capture part of their geo-demographic segmentation. Moreover, we apply a supervised classification procedure to explore which features are more relevant to define each cluster.
Feature discriminativity estimation in CNNs for transfer learning
Authors: Victor Gimenez-Abalos, Armand Vilalta, Dario Garcia-Gasulla, Jesus Labarta, Eduard Ayguadé
Date: 2019-11-08 00:00:00
Tasks: Transfer Learning

The purpose of feature extraction on convolutional neural networks is to reuse deep representations learnt for a pre-trained model to solve a new, potentially unrelated problem. However, raw feature extraction from all layers is unfeasible given the massive size of these networks. Recently, a supervised method using complexity reduction was proposed, resulting in significant improvements in performance for transfer learning tasks. This approach first computes the discriminative power of features, and then discretises them using thresholds computed for the task. In this paper, we analyse the behaviour of these thresholds, with the purpose of finding a methodology for their estimation. After a comprehensive study, we find a very strong correlation between problem size and threshold value, with coefficient of determination above 90%. These results allow us to propose a unified model for threshold estimation, with potential application to transfer learning tasks.
Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages
Authors: Mirac Suzgun, Sebastian Gehrmann, Yonatan Belinkov, Stuart M. Shieber
Date: 2019-11-08 00:00:00
Tasks: Language Modeling, Language Modelling

We introduce three memory-augmented Recurrent Neural Networks (MARNNs) and explore their capabilities on a series of simple language modeling tasks whose solutions require stack-based mechanisms. We provide the first demonstration of neural networks recognizing the generalized Dyck languages, which express the core of what it means to be a language with hierarchical structure. Our memory-augmented architectures are easy to train in an end-to-end fashion and can learn the Dyck languages over as many as six parenthesis-pairs, in addition to two deterministic palindrome languages and the string-reversal transduction task, by emulating pushdown automata. Our experiments highlight the increased modeling capacity of memory-augmented models over simple RNNs, while inflecting our understanding of the limitations of these models.
Transforming Wikipedia into Augmented Data for Query-Focused Summarization
Authors: Haichao Zhu, Li Dong, Furu Wei, Bing Qin, Ting Liu
Date: 2019-11-08 00:00:00
Tasks: Data Augmentation, Query-focused Summarization

The limited size of existing query-focused summarization datasets renders training data-driven summarization models challenging. Meanwhile, the manual construction of a query-focused summarization corpus is costly and time-consuming. In this paper, we use Wikipedia to automatically collect a large query-focused summarization dataset (named WIKIREF) of more than 280, 000 examples, which can serve as a means of data augmentation. We also develop a BERT-based query-focused summarization model (Q-BERT) to extract sentences from the documents as summaries. To better adapt a huge model containing millions of parameters to tiny benchmarks, we identify and fine-tune only a sparse subnetwork, which corresponds to a small fraction of the whole model parameters. Experimental results on three DUC benchmarks show that the model pre-trained on WIKIREF has already achieved reasonable performance. After fine-tuning on the specific benchmark datasets, the model with data augmentation outperforms strong comparison systems. Moreover, both our proposed Q-BERT model and subnetwork fine-tuning further improve the model performance. The dataset is publicly available at https://aka.ms/wikiref.
Deep Transfer Learning for Thermal Dynamics Modeling in Smart Buildings
Authors: Zhanhong Jiang, Young M. Lee
Date: 2019-11-08 00:00:00
Tasks: Domain Adaptation, Transfer Learning

Thermal dynamics modeling has been a critical issue in building heating, ventilation, and air-conditioning (HVAC) systems, which can significantly affect the control and maintenance strategies. Due to the uniqueness of each specific building, traditional thermal dynamics modeling approaches heavily depending on physics knowledge cannot generalize well. This study proposes a deep supervised domain adaptation (DSDA) method for thermal dynamics modeling of building indoor temperature evolution and energy consumption. A long short term memory network based Sequence to Sequence scheme is pre-trained based on a large amount of data collected from a building and then adapted to another building which has a limited amount of data by applying the model fine-tuning. We use four publicly available datasets: SML and AHU for temperature evolution, long-term datasets from two different commercial buildings, termed as Building 1 and Building 2 for energy consumption. We show that the deep supervised domain adaptation is effective to adapt the pre-trained model from one building to another building and has better predictive performance than learning from scratch with only a limited amount of data.
Online learning-based Model Predictive Control with Gaussian Process Models and Stability Guarantees
Authors: Michael Maiworm, Daniel Limon, Rolf Findeisen
Date: 2019-11-08 00:00:00
Tasks: Gaussian Processes, Model Predictive Control, Prediction

Model predictive control allows to provide high performance and safety guarantees in the form of constraint satisfaction. These properties, however, can be satisfied only if the underlying model, used for prediction, of the controlled process is sufficiently accurate. One way to address this challenge is by data-driven and machine learning approaches, such as Gaussian processes, that allow to refine the model online during operation. We present a combination of an output feedback model predictive control scheme and a Gaussian process-based prediction model that is capable of efficient online learning. To this end, the concept of evolving Gaussian processes is combined with recursive posterior prediction updates. The presented approach guarantees recursive constraint satisfaction and input-to-state stability with respect to the model-plant mismatch. Simulation studies underline that the Gaussian process prediction model can be successfully and efficiently learned online. The resulting computational load is significantly reduced via the combination of the recursive update procedure and by limiting the number of training data points while maintaining good performance.
FANN-on-MCU: An Open-Source Toolkit for Energy-Efficient Neural Network Inference at the Edge of the Internet of Things
Authors: Xiaying Wang, Michele Magno, Lukas Cavigelli, Luca Benini
Date: 2019-11-08 00:00:00
Tasks: BIG-bench Machine Learning, Edge-computing, Efficient Neural Network

The growing number of low-power smart devices in the Internet of Things is coupled with the concept of "Edge Computing", that is moving some of the intelligence, especially machine learning, towards the edge of the network. Enabling machine learning algorithms to run on resource-constrained hardware, typically on low-power smart devices, is challenging in terms of hardware (optimized and energy-efficient integrated circuits), algorithmic and firmware implementations. This paper presents FANN-on-MCU, an open-source toolkit built upon the Fast Artificial Neural Network (FANN) library to run lightweight and energy-efficient neural networks on microcontrollers based on both the ARM Cortex-M series and the novel RISC-V-based Parallel Ultra-Low-Power (PULP) platform. The toolkit takes multi-layer perceptrons trained with FANN and generates code targeted at execution on low-power microcontrollers either with a floating-point unit (i.e., ARM Cortex-M4F and M7F) or without (i.e., ARM Cortex M0-M3 or PULP-based processors). This paper also provides an architectural performance evaluation of neural networks on the most popular ARM Cortex-M family and the parallel RISC-V processor called Mr. Wolf. The evaluation includes experimental results for three different applications using a self-sustainable wearable multi-sensor bracelet. Experimental results show a measured latency in the order of only a few microseconds and a power consumption of few milliwatts while keeping the memory requirements below the limitations of the targeted microcontrollers. In particular, the parallel implementation on the octa-core RISC-V platform reaches a speedup of 22x and a 69% reduction in energy consumption with respect to a single-core implementation on Cortex-M4 for continuous real-time classification.
How Language-Neutral is Multilingual BERT?
Authors: Jindřich Libovický, Rudolf Rosa, Alexander Fraser
Date: 2019-11-08 00:00:00
Tasks: Retrieval, Sentence, Sentence Retrieval, Transfer Learning, Word Alignment

Multilingual BERT (mBERT) provides sentence representations for 104 languages, which are useful for many multi-lingual tasks. Previous work probed the cross-linguality of mBERT using zero-shot transfer learning on morphological and syntactic tasks. We instead focus on the semantic properties of mBERT. We show that mBERT representations can be split into a language-specific component and a language-neutral component, and that the language-neutral component is sufficiently general in terms of modeling semantics to allow high-accuracy word-alignment and sentence retrieval but is not yet good enough for the more difficult task of MT quality estimation. Our work presents interesting challenges which must be solved to build better language-neutral representations, particularly for tasks requiring linguistic transfer of semantics.
Fully Bayesian Recurrent Neural Networks for Safe Reinforcement Learning
Authors: Matt Benatan, Edward O. Pyzer-Knapp
Date: 2019-11-08 00:00:00
Tasks: Collision Avoidance, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Safe Reinforcement Learning

Reinforcement Learning (RL) has demonstrated state-of-the-art results in a number of autonomous system applications, however many of the underlying algorithms rely on black-box predictions. This results in poor explainability of the behaviour of these systems, raising concerns as to their use in safety-critical applications. Recent work has demonstrated that uncertainty-aware models exhibit more cautious behaviours through the incorporation of model uncertainty estimates. In this work, we build on Probabilistic Backpropagation to introduce a fully Bayesian Recurrent Neural Network architecture. We apply this within a Safe RL scenario, and demonstrate that the proposed method significantly outperforms a popular approach for obtaining model uncertainties in collision avoidance tasks. Furthermore, we demonstrate that the proposed approach requires less training and is far more efficient than the current leading method, both in terms of compute resource and memory footprint.