pwc-archive/papers-with-abstracts huggingface dataset:
Generalization bounds for deep learning
Authors: Guillermo Valle-Pérez, Ard A. Louis
Date: 2020-12-07 00:00:00
Tasks: Deep Learning, Generalization Bounds

Generalization in deep learning has been the topic of much recent theoretical and empirical research. Here we introduce desiderata for techniques that predict generalization errors for deep learning models in supervised learning. Such predictions should 1) scale correctly with data complexity; 2) scale correctly with training set size; 3) capture differences between architectures; 4) capture differences between optimization algorithms; 5) be quantitatively not too far from the true error (in particular, be non-vacuous); 6) be efficiently computable; and 7) be rigorous. We focus on generalization error upper bounds, and introduce a categorisation of bounds depending on assumptions on the algorithm and data. We review a wide range of existing approaches, from classical VC dimension to recent PAC-Bayesian bounds, commenting on how well they perform against the desiderata. We next use a function-based picture to derive a marginal-likelihood PAC-Bayesian bound. This bound is, by one definition, optimal up to a multiplicative constant in the asymptotic limit of large training sets, as long as the learning curve follows a power law, which is typically found in practice for deep learning problems. Extensive empirical analysis demonstrates that our marginal-likelihood PAC-Bayes bound fulfills desiderata 1-3 and 5. The results for 6 and 7 are promising, but not yet fully conclusive, while only desideratum 4 is currently beyond the scope of our bound. Finally, we comment on why this function-based bound performs significantly better than current parameter-based PAC-Bayes bounds.
Plane Wave Elastography: A Frequency-Domain Ultrasound Shear Wave Elastography Approach
Authors: Reza Khodayi-mehr, Matthew W. Urban, Michael M. Zavlanos, Wilkins Aquino
Date: 2020-12-08 00:00:00

In this paper, we propose Plane Wave Elastography (PWE), a novel ultrasound shear wave elastography (SWE) approach. Currently, commercial methods for SWE rely on directional filtering based on the prior knowledge of the wave propagation direction, to remove complicated wave patterns formed due to reflection and refraction. The result is a set of decomposed directional waves that are separately analyzed to construct shear modulus fields that are then combined through compounding. Instead, PWE relies on a rigorous representation of the wave propagation using the frequency-domain scalar wave equation to automatically select appropriate propagation directions and simultaneously reconstruct shear modulus fields. Specifically, assuming a homogeneous, isotropic, incompressible, linear-elastic medium, we represent the solution of the wave equation using a linear combination of plane waves propagating in arbitrary directions. Given this closed-form solution, we formulate the SWE problem as a nonlinear least-squares optimization problem which can be solved very efficiently. Through numerous phantom studies, we show that PWE can handle complicated waveforms without prior filtering and is competitive with state-of-the-art that requires prior filtering based on the knowledge of propagation directions.
Adaptive Sampling for Estimating Distributions: A Bayesian Upper Confidence Bound Approach
Authors: Dhruva Kartik, Neeraj Sood, Urbashi Mitra, Tara Javidi
Date: 2020-12-08 00:00:00

The problem of adaptive sampling for estimating probability mass functions (pmf) uniformly well is considered. Performance of the sampling strategy is measured in terms of the worst-case mean squared error. A Bayesian variant of the existing upper confidence bound (UCB) based approaches is proposed. It is shown analytically that the performance of this Bayesian variant is no worse than the existing approaches. The posterior distribution on the pmfs in the Bayesian setting allows for a tighter computation of upper confidence bounds which leads to significant performance gains in practice. Using this approach, adaptive sampling protocols are proposed for estimating SARS-CoV-2 seroprevalence in various groups such as location and ethnicity. The effectiveness of this strategy is discussed using data obtained from a seroprevalence survey in Los Angeles county.
Neural fidelity warping for efficient robot morphology design
Authors: Sha Hu, Zeshi Yang, Greg Mori
Date: 2020-12-08 00:00:00
Tasks: Bayesian Optimization

We consider the problem of optimizing a robot morphology to achieve the best performance for a target task, under computational resource limitations. The evaluation process for each morphological design involves learning a controller for the design, which can consume substantial time and computational resources. To address the challenge of expensive robot morphology evaluation, we present a continuous multi-fidelity Bayesian Optimization framework that efficiently utilizes computational resources via low-fidelity evaluations. We identify the problem of non-stationarity over fidelity space. Our proposed fidelity warping mechanism can learn representations of learning epochs and tasks to model non-stationary covariances between continuous fidelity evaluations which prove challenging for off-the-shelf stationary kernels. Various experiments demonstrate that our method can utilize the low-fidelity evaluations to efficiently search for the optimal robot morphology, outperforming state-of-the-art methods.
Channel Gain Cartography via Mixture of Experts
Authors: Luis M. Lopez-Ramos, Yves Teganya, Baltasar Beferull-Lozano, Seung-Jun Kim
Date: 2020-12-08 00:00:00
Tasks: Mixture-of-Experts

In order to estimate the channel gain (CG) between the locations of an arbitrary transceiver pair across a geographic area of interest, CG maps can be constructed from spatially distributed sensor measurements. Most approaches to build such spectrum maps are location-based, meaning that the input variable to the estimating function is a pair of spatial locations. The performance of such maps depends critically on the ability of the sensors to determine their positions, which may be drastically impaired if the positioning pilot signals are affected by multi-path channels. An alternative location-free approach was recently proposed for spectrum power maps, where the input variable to the maps consists of features extracted from the positioning signals, instead of location estimates. The location-based and the location-free approaches have complementary merits. In this work, apart from adapting the location-free features for the CG maps, a method that can combine both approaches is proposed in a mixture-of-experts framework.
Cross-lingual Transfer of Abstractive Summarizer to Less-resource Language
Authors: Aleš Žagar, Marko Robnik-Šikonja
Date: 2020-12-08 00:00:00
Tasks: Abstractive Text Summarization, Articles, Cross-Lingual Transfer, Decoder, Language Modelling, Text Generation, Text Summarization

Automatic text summarization extracts important information from texts and presents the information in the form of a summary. Abstractive summarization approaches progressed significantly by switching to deep neural networks, but results are not yet satisfactory, especially for languages where large training sets do not exist. In several natural language processing tasks, a cross-lingual model transfer is successfully applied in less-resource languages. For summarization, the cross-lingual model transfer was not attempted due to a non-reusable decoder side of neural models that cannot correct target language generation. In our work, we use a pre-trained English summarization model based on deep neural networks and sequence-to-sequence architecture to summarize Slovene news articles. We address the problem of inadequate decoder by using an additional language model for the evaluation of the generated text in target language. We test several cross-lingual summarization models with different amounts of target data for fine-tuning. We assess the models with automatic evaluation measures and conduct a small-scale human evaluation. Automatic evaluation shows that the summaries of our best cross-lingual model are useful and of quality similar to the model trained only in the target language. Human evaluation shows that our best model generates summaries with high accuracy and acceptable readability. However, similar to other abstractive models, our models are not perfect and may occasionally produce misleading or absurd content.
Quality-Diversity Optimization: a novel branch of stochastic optimization
Authors: Konstantinos Chatzilygeroudis, Antoine Cully, Vassilis Vassiliades, Jean-Baptiste Mouret
Date: 2020-12-08 00:00:00
Tasks: Diversity, Stochastic Optimization

Traditional optimization algorithms search for a single global optimum that maximizes (or minimizes) the objective function. Multimodal optimization algorithms search for the highest peaks in the search space that can be more than one. Quality-Diversity algorithms are a recent addition to the evolutionary computation toolbox that do not only search for a single set of local optima, but instead try to illuminate the search space. In effect, they provide a holistic view of how high-performing solutions are distributed throughout a search space. The main differences with multimodal optimization algorithms are that (1) Quality-Diversity typically works in the behavioral space (or feature space), and not in the genotypic (or parameter) space, and (2) Quality-Diversity attempts to fill the whole behavior space, even if the niche is not a peak in the fitness landscape. In this chapter, we provide a gentle introduction to Quality-Diversity optimization, discuss the main representative algorithms, and the main current topics under consideration in the community. Throughout the chapter, we also discuss several successful applications of Quality-Diversity algorithms, including deep learning, robotics, and reinforcement learning.
Deep learning for clustering of continuous gravitational wave candidates II: identification of low-SNR candidates
Authors: Banafsheh Beheshtipour, Maria Alessandra Papa
Date: 2020-12-08 00:00:00
Tasks: Clustering, Deep Learning

Broad searches for continuous gravitational wave signals rely on hierarchies of follow-up stages for candidates above a given significance threshold. An important step to simplify these follow-ups and reduce the computational cost is to bundle together in a single follow-up nearby candidates. This step is called clustering and we investigate carrying it out with a deep learning network. In our first paper [1], we implemented a deep learning clustering network capable of correctly identifying clusters due to large signals. In this paper, a network is implemented that can detect clusters due to much fainter signals. These two networks are complementary and we show that a cascade of the two networks achieves an excellent detection efficiency across a wide range of signal strengths, with a false alarm rate comparable/lower than that of methods currently in use.
NavRep: Unsupervised Representations for Reinforcement Learning of Robot Navigation in Dynamic Human Environments
Authors: Daniel Dugas, Juan Nieto, Roland Siegwart, Jen Jen Chung
Date: 2020-12-08 00:00:00
Tasks: OpenAI Gym, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Robot Navigation

Robot navigation is a task where reinforcement learning approaches are still unable to compete with traditional path planning. State-of-the-art methods differ in small ways, and do not all provide reproducible, openly available implementations. This makes comparing methods a challenge. Recent research has shown that unsupervised learning methods can scale impressively, and be leveraged to solve difficult problems. In this work, we design ways in which unsupervised learning can be used to assist reinforcement learning for robot navigation. We train two end-to-end, and 18 unsupervised-learning-based architectures, and compare them, along with existing approaches, in unseen test cases. We demonstrate our approach working on a real life robot. Our results show that unsupervised learning methods are competitive with end-to-end methods. We also highlight the importance of various components such as input representation, predictive unsupervised learning, and latent features. We make all our models publicly available, as well as training and testing environments, and tools. This release also includes OpenAI-gym-compatible environments designed to emulate the training conditions described by other papers, with as much fidelity as possible. Our hope is that this helps in bringing together the field of RL for robot navigation, and allows meaningful comparisons across state-of-the-art methods.
Comparison of Atom Representations in Graph Neural Networks for Molecular Property Prediction
Authors: Agnieszka Pocha, Tomasz Danel, Łukasz Maziarka
Date: 2020-11-23 00:00:00
Tasks: Molecular Property Prediction, Property Prediction

Graph neural networks have recently become a standard method for analysing chemical compounds. In the field of molecular property prediction, the emphasis is now put on designing new model architectures, and the importance of atom featurisation is oftentimes belittled. When contrasting two graph neural networks, the use of different atom features possibly leads to the incorrect attribution of the results to the network architecture. To provide a better understanding of this issue, we compare multiple atom representations for graph models and evaluate them on the prediction of free energy, solubility, and metabolic stability. To the best of our knowledge, this is the first methodological study that focuses on the relevance of atom representation to the predictive performance of graph neural networks.
Combining Reinforcement Learning with Lin-Kernighan-Helsgaun Algorithm for the Traveling Salesman Problem
Authors: Jiongzhi Zheng, Kun He, Jianrong Zhou, Yan Jin, Chu-min Li
Date: 2020-12-08 00:00:00
Tasks: Combinatorial Optimization, Q-Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Traveling Salesman Problem

We address the Traveling Salesman Problem (TSP), a famous NP-hard combinatorial optimization problem. And we propose a variable strategy reinforced approach, denoted as VSR-LKH, which combines three reinforcement learning methods (Q-learning, Sarsa and Monte Carlo) with the well-known TSP algorithm, called Lin-Kernighan-Helsgaun (LKH). VSR-LKH replaces the inflexible traversal operation in LKH, and lets the program learn to make choice at each search step by reinforcement learning. Experimental results on 111 TSP benchmarks from the TSPLIB with up to 85,900 cities demonstrate the excellent performance of the proposed method.
Privacy-Preserving Synthetic Smart Meters Data
Authors: Ganesh Del Grosso, Georg Pichler, Pablo Piantanida
Date: 2020-12-06 00:00:00
Tasks: Privacy Preserving

Power consumption data is very useful as it allows to optimize power grids, detect anomalies and prevent failures, on top of being useful for diverse research purposes. However, the use of power consumption data raises significant privacy concerns, as this data usually belongs to clients of a power company. As a solution, we propose a method to generate synthetic power consumption samples that faithfully imitate the originals, but are detached from the clients and their identities. Our method is based on Generative Adversarial Networks (GANs). Our contribution is twofold. First, we focus on the quality of the generated data, which is not a trivial task as no standard evaluation methods are available. Then, we study the privacy guarantees provided to members of the training set of our neural network. As a minimum requirement for privacy, we demand our neural network to be robust to membership inference attacks, as these provide a gateway for further attacks in addition to presenting a privacy threat on their own. We find that there is a compromise to be made between the privacy and the performance provided by the algorithm.
Transfer Learning for Human Activity Recognition using Representational Analysis of Neural Networks
Authors: Sizhe An, Ganapati Bhat, Suat Gumussoy, Umit Ogras
Date: 2020-12-05 00:00:00
Tasks: Activity Recognition, Human Activity Recognition, Transfer Learning

Human activity recognition (HAR) research has increased in recent years due to its applications in mobile health monitoring, activity recognition, and patient rehabilitation. The typical approach is training a HAR classifier offline with known users and then using the same classifier for new users. However, the accuracy for new users can be low with this approach if their activity patterns are different than those in the training data. At the same time, training from scratch for new users is not feasible for mobile applications due to the high computational cost and training time. To address this issue, we propose a HAR transfer learning framework with two components. First, a representational analysis reveals common features that can transfer across users and user-specific features that need to be customized. Using this insight, we transfer the reusable portion of the offline classifier to new users and fine-tune only the rest. Our experiments with five datasets show up to 43% accuracy improvement and 66% training time reduction when compared to the baseline without using transfer learning. Furthermore, measurements on the Nvidia Jetson Xavier-NX hardware platform reveal that the power and energy consumption decrease by 43% and 68%, respectively, while achieving the same or higher accuracy as training from scratch.
Retrieval of Case 2 Water Quality Parameters with Machine Learning
Authors: Ana B. Ruescas, Gonzalo Mateo-Garcia, Gustau Camps-Valls, Martin Hieronymi
Date: 2020-12-08 00:00:00
Tasks: BIG-bench Machine Learning, regression, Retrieval

Water quality parameters are derived applying several machine learning regression methods on the Case2eXtreme dataset (C2X). The used data are based on Hydrolight in-water radiative transfer simulations at Sentinel-3 OLCI wavebands, and the application is done exclusively for absorbing waters with high concentrations of coloured dissolved organic matter (CDOM). The regression approaches are: regularized linear, random forest, Kernel ridge, Gaussian process and support vector regressors. The validation is made with and an independent simulation dataset. A comparison with the OLCI Neural Network Swarm (ONSS) is made as well. The best approached is applied to a sample scene and compared with the standard OLCI product delivered by EUMETSAT/ESA
Dartmouth CS at WNUT-2020 Task 2: Informative COVID-19 Tweet Classification Using BERT
Authors: Dylan Whang, Soroush Vosoughi
Date: 2020-12-07 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: General Classification, Task 2

We describe the systems developed for the WNUT-2020 shared task 2, identification of informative COVID-19 English Tweets. BERT is a highly performant model for Natural Language Processing tasks. We increased BERT's performance in this classification task by fine-tuning BERT and concatenating its embeddings with Tweet-specific features and training a Support Vector Machine (SVM) for classification (henceforth called BERT+). We compared its performance to a suite of machine learning models. We used a Twitter specific data cleaning pipeline and word-level TF-IDF to extract features for the non-BERT models. BERT+ was the top performing model with an F1-score of 0.8713.
Improvements and Extensions on Metaphor Detection
Authors: Weicheng Ma, Ruibo Liu, Lili Wang, Soroush Vosoughi
Date: 2020-12-07 00:00:00
Proceeding: ACL (unimplicit) 2021 8
Tasks: Natural Language Understanding

Metaphors are ubiquitous in human language. The metaphor detection task (MD) aims at detecting and interpreting metaphors from written language, which is crucial in natural language understanding (NLU) research. In this paper, we introduce a pre-trained Transformer-based model into MD. Our model outperforms the previous state-of-the-art models by large margins in our evaluations, with relative improvements on the F-1 score from 5.33% to 28.39%. Second, we extend MD to a classification task about the metaphoricity of an entire piece of text to make MD applicable in more general NLU scenes. Finally, we clean up the improper or outdated annotations in one of the MD benchmark datasets and re-benchmark it with our Transformer-based model. This approach could be applied to other existing MD datasets as well, since the metaphoricity annotations in these benchmark datasets may be outdated. Future research efforts are also necessary to build an up-to-date and well-annotated dataset consisting of longer and more complex texts.
DeepNVM++: Cross-Layer Modeling and Optimization Framework of Non-Volatile Memories for Deep Learning
Authors: Ahmet Inci, Mehmet Meric Isgenc, Diana Marculescu
Date: 2020-12-08 00:00:00
Tasks: GPU

Non-volatile memory (NVM) technologies such as spin-transfer torque magnetic random access memory (STT-MRAM) and spin-orbit torque magnetic random access memory (SOT-MRAM) have significant advantages compared to conventional SRAM due to their non-volatility, higher cell density, and scalability features. While previous work has investigated several architectural implications of NVM for generic applications, in this work we present DeepNVM++, a framework to characterize, model, and analyze NVM-based caches in GPU architectures for deep learning (DL) applications by combining technology-specific circuit-level models and the actual memory behavior of various DL workloads. We present both iso-capacity and iso-area performance and energy analysis for systems whose last-level caches rely on conventional SRAM and emerging STT-MRAM and SOT-MRAM technologies. In the iso-capacity case, STT-MRAM and SOT-MRAM provide up to 3.8x and 4.7x energy-delay product (EDP) reduction and 2.4x and 2.8x area reduction compared to conventional SRAM, respectively. Under iso-area assumptions, STT-MRAM and SOT-MRAM provide up to 2x and 2.3x EDP reduction and accommodate 2.3x and 3.3x cache capacity when compared to SRAM, respectively. We also perform a scalability analysis and show that STT-MRAM and SOT-MRAM achieve orders of magnitude EDP reduction when compared to SRAM for large cache capacities. Our comprehensive cross-layer framework is demonstrated on STT-/SOT-MRAM technologies and can be used for the characterization, modeling, and analysis of any NVM technology for last-level caches in GPUs for DL applications.
I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch
Authors: Joseph Turian, Max Henry
Date: 2020-12-08 00:00:00

Growing research demonstrates that synthetic failure modes imply poor generalization. We compare commonly used audio-to-audio losses on a synthetic benchmark, measuring the pitch distance between two stationary sinusoids. The results are surprising: many have poor sense of pitch direction. These shortcomings are exposed using simple rank assumptions. Our task is trivial for humans but difficult for these audio distances, suggesting significant progress can be made in self-supervised audio learning by improving current losses.
Estimation of the Mean Function of Functional Data via Deep Neural Networks
Authors: Shuoyang Wang, Guanqun Cao, Zuofeng Shang
Date: 2020-12-08 00:00:00
Tasks: regression

In this work, we propose a deep neural network method to perform nonparametric regression for functional data. The proposed estimators are based on sparsely connected deep neural networks with ReLU activation function. By properly choosing network architecture, our estimator achieves the optimal nonparametric convergence rate in empirical norm. Under certain circumstances such as trigonometric polynomial kernel and a sufficiently large sampling frequency, the convergence rate is even faster than root-$n$ rate. Through Monte Carlo simulation studies we examine the finite-sample performance of the proposed method. Finally, the proposed method is applied to analyze positron emission tomography images of patients with Alzheimer disease obtained from the Alzheimer Disease Neuroimaging Initiative database.