pwc-archive/papers-with-abstracts huggingface dataset:
Deep Neural Networks Are Effective At Learning High-Dimensional Hilbert-Valued Functions From Limited Data
Authors: Ben Adcock, Simone Brugiapaglia, Nick Dexter, Sebastian Moraga
Date: 2020-12-11 00:00:00

Accurate approximation of scalar-valued functions from sample points is a key task in computational science. Recently, machine learning with Deep Neural Networks (DNNs) has emerged as a promising tool for scientific computing, with impressive results achieved on problems where the dimension of the data or problem domain is large. This work broadens this perspective, focusing on approximating functions that are Hilbert-valued, i.e. take values in a separable, but typically infinite-dimensional, Hilbert space. This arises in science and engineering problems, in particular those involving solution of parametric Partial Differential Equations (PDEs). Such problems are challenging: 1) pointwise samples are expensive to acquire, 2) the function domain is high dimensional, and 3) the range lies in a Hilbert space. Our contributions are twofold. First, we present a novel result on DNN training for holomorphic functions with so-called hidden anisotropy. This result introduces a DNN training procedure and full theoretical analysis with explicit guarantees on error and sample complexity. The error bound is explicit in three key errors occurring in the approximation procedure: the best approximation, measurement, and physical discretization errors. Our result shows that there exists a procedure (albeit non-standard) for learning Hilbert-valued functions via DNNs that performs as well as, but no better than current best-in-class schemes. It gives a benchmark lower bound for how well DNNs can perform on such problems. Second, we examine whether better performance can be achieved in practice through different types of architectures and training. We provide preliminary numerical results illustrating practical performance of DNNs on parametric PDEs. We consider different parameters, modifying the DNN architecture to achieve better and competitive results, comparing these to current best-in-class schemes.
Generative Learning With Euler Particle Transport
Authors: Yuan Gao, Jian Huang, Yuling Jiao, Jin Liu, Xiliang Lu, Zhijian Yang
Date: 2020-12-11 00:00:00

We propose an Euler particle transport (EPT) approach for generative learning. The proposed approach is motivated by the problem of finding an optimal transport map from a reference distribution to a target distribution characterized by the Monge-Ampere equation. Interpreting the infinitesimal linearization of the Monge-Ampere equation from the perspective of gradient flows in measure spaces leads to a stochastic McKean-Vlasov equation. We use the forward Euler method to solve this equation. The resulting forward Euler map pushes forward a reference distribution to the target. This map is the composition of a sequence of simple residual maps, which are computationally stable and easy to train. The key task in training is the estimation of the density ratios or differences that determine the residual maps. We estimate the density ratios (differences) based on the Bregman divergence with a gradient penalty using deep density-ratio (difference) fitting. We show that the proposed density-ratio (difference) estimators do not suffer from the "curse of dimensionality" if data is supported on a lower-dimensional manifold. Numerical experiments with multi-mode synthetic datasets and comparisons with the existing methods on real benchmark datasets support our theoretical results and demonstrate the effectiveness of the proposed method.
A Review of Hidden Markov Models and Recurrent Neural Networks for Event Detection and Localization in Biomedical Signals
Authors: Yassin Khalifa, Danilo Mandic, Ervin Sejdić
Date: 2020-12-11 00:00:00
Tasks: Event Detection, Rhythm, Time Series, Time Series Analysis

Biomedical signals carry signature rhythms of complex physiological processes that control our daily bodily activity. The properties of these rhythms indicate the nature of interaction dynamics among physiological processes that maintain a homeostasis. Abnormalities associated with diseases or disorders usually appear as disruptions in the structure of the rhythms which makes isolating these rhythms and the ability to differentiate between them, indispensable. Computer aided diagnosis systems are ubiquitous nowadays in almost every medical facility and more closely in wearable technology, and rhythm or event detection is the first of many intelligent steps that they perform. How these rhythms are isolated? How to develop a model that can describe the transition between processes in time? Many methods exist in the literature that address these questions and perform the decoding of biomedical signals into separate rhythms. In here, we demystify the most effective methods that are used for detection and isolation of rhythms or events in time series and highlight the way in which they were applied to different biomedical signals and how they contribute to information fusion. The key strengths and limitations of these methods are also discussed as well as the challenges encountered with application in biomedical signals.
Pair-view Unsupervised Graph Representation Learning
Authors: You Li, Binli Luo, Ning Gui
Date: 2020-12-11 00:00:00
Tasks: Graph Representation Learning, Link Prediction, Node Classification, Representation Learning

Low-dimension graph embeddings have proved extremely useful in various downstream tasks in large graphs, e.g., link-related content recommendation and node classification tasks, etc. Most existing embedding approaches take nodes as the basic unit for information aggregation, e.g., node perception fields in GNN or con-textual nodes in random walks. The main drawback raised by such node-view is its lack of support for expressing the compound relationships between nodes, which results in the loss of a certain degree of graph information during embedding. To this end, this paper pro-poses PairE(Pair Embedding), a solution to use "pair", a higher level unit than a "node" as the core for graph embeddings. Accordingly, a multi-self-supervised auto-encoder is designed to fulfill two pretext tasks, to reconstruct the feature distribution for respective pairs and their surrounding context. PairE has three major advantages: 1) Informative, embedding beyond node-view are capable to preserve richer information of the graph; 2) Simple, the solutions provided by PairE are time-saving, storage-efficient, and require the fewer hyper-parameters; 3) High adaptability, with the introduced translator operator to map pair embeddings to the node embeddings, PairE can be effectively used in both the link-based and the node-based graph analysis. Experiment results show that PairE consistently outperforms the state of baselines in all four downstream tasks, especially with significant edges in the link-prediction and multi-label node classification tasks.
$π$-ROAD: a Learn-as-You-Go Framework for On-Demand Emergency Slices in V2X Scenarios
Authors: Armin Okic, Lanfranco Zanzi, Vincenzo Sciancalepore, Alessandro Redondi, Xavier Costa-Perez
Date: 2020-12-11 00:00:00
Tasks: Autonomous Driving

Vehicle-to-everything (V2X) is expected to become one of the main drivers of 5G business in the near future. Dedicated \emph{network slices} are envisioned to satisfy the stringent requirements of advanced V2X services, such as autonomous driving, aimed at drastically reducing road casualties. However, as V2X services become more mission-critical, new solutions need to be devised to guarantee their successful service delivery even in exceptional situations, e.g. road accidents, congestion, etc. In this context, we propose $\pi$-ROAD, a \emph{deep learning} framework to automatically learn regular mobile traffic patterns along roads, detect non-recurring events and classify them by severity level. $\pi$-ROAD enables operators to \emph{proactively} instantiate dedicated \emph{Emergency Network Slices (ENS)} as needed while re-dimensioning the existing slices according to their service criticality level. Our framework is validated by means of real mobile network traces collected within $400~km$ of a highway in Europe and augmented with publicly available information on related road events. Our results show that $\pi$-ROAD successfully detects and classifies non-recurring road events and reduces up to $30\%$ the impact of ENS on already running services.
Query Understanding for Natural Language Enterprise Search
Authors: Francisco Borges, Georgios Balikas, Marc Brette, Guillaume Kempf, Arvind Srikantan, Matthieu Landos, Darya Brazouskaya, Qianqian Shi
Date: 2020-12-11 00:00:00

Natural Language Search (NLS) extends the capabilities of search engines that perform keyword search allowing users to issue queries in a more "natural" language. The engine tries to understand the meaning of the queries and to map the query words to the symbols it supports like Persons, Organizations, Time Expressions etc.. It, then, retrieves the information that satisfies the user's need in different forms like an answer, a record or a list of records. We present an NLS system we implemented as part of the Search service of a major CRM platform. The system is currently in production serving thousands of customers. Our user studies showed that creating dynamic reports with NLS saved more than 50% of our user's time compared to achieving the same result with navigational search. We describe the architecture of the system, the particularities of the CRM domain as well as how they have influenced our design decisions. Among several submodules of the system we detail the role of a Deep Learning Named Entity Recognizer. The paper concludes with discussion over the lessons learned while developing this product.
The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks
Authors: Bohan Wang, Qi Meng, Wei Chen, Tie-Yan Liu
Date: 2020-12-11 00:00:00

Despite their overwhelming capacity to overfit, deep neural networks trained by specific optimization algorithms tend to generalize well to unseen data. Recently, researchers explained it by investigating the implicit regularization effect of optimization algorithms. A remarkable progress is the work (Lyu&Li, 2019), which proves gradient descent (GD) maximizes the margin of homogeneous deep neural networks. Except GD, adaptive algorithms such as AdaGrad, RMSProp and Adam are popular owing to their rapid training process. However, theoretical guarantee for the generalization of adaptive optimization algorithms is still lacking. In this paper, we study the implicit regularization of adaptive optimization algorithms when they are optimizing the logistic loss on homogeneous deep neural networks. We prove that adaptive algorithms that adopt exponential moving average strategy in conditioner (such as Adam and RMSProp) can maximize the margin of the neural network, while AdaGrad that directly sums historical squared gradients in conditioner can not. It indicates superiority on generalization of exponential moving average strategy in the design of the conditioner. Technically, we provide a unified framework to analyze convergent direction of adaptive optimization algorithms by constructing novel adaptive gradient flow and surrogate margin. Our experiments can well support the theoretical findings on convergent direction of adaptive optimization algorithms.
Structured learning of rigid-body dynamics: A survey and unified view from a robotics perspective
Authors: A. René Geist, Sebastian Trimpe
Date: 2020-12-11 00:00:00
Tasks: Gaussian Processes, regression

Accurate models of mechanical system dynamics are often critical for model-based control and reinforcement learning. Fully data-driven dynamics models promise to ease the process of modeling and analysis, but require considerable amounts of data for training and often do not generalize well to unseen parts of the state space. Combining data-driven modelling with prior analytical knowledge is an attractive alternative as the inclusion of structural knowledge into a regression model improves the model's data efficiency and physical integrity. In this article, we survey supervised regression models that combine rigid-body mechanics with data-driven modelling techniques. We analyze the different latent functions (such as kinetic energy or dissipative forces) and operators (such as differential operators and projection matrices) underlying common descriptions of rigid-body mechanics. Based on this analysis, we provide a unified view on the combination of data-driven regression models, such as neural networks and Gaussian processes, with analytical model priors. Further, we review and discuss key techniques for designing structured models such as automatic differentiation.
Comparison of Anomaly Detectors: Context Matters
Authors: Vít Škvára, Jan Franců, Matěj Zorek, Tomáš Pevný, Václav Šmídl
Date: 2020-12-11 00:00:00
Tasks: Anomaly Detection, Model Selection

Deep generative models are challenging the classical methods in the field of anomaly detection nowadays. Every new method provides evidence of outperforming its predecessors, often with contradictory results. The objective of this comparison is twofold: to compare anomaly detection methods of various paradigms with focus on deep generative models, and identification of sources of variability that can yield different results. The methods were compared on popular tabular and image datasets. We identified the main sources of variability to be experimental conditions: i) the type data set (tabular or image) and the nature of anomalies (statistical or semantic), and ii) strategy of selection of hyperparameters, especially the number of available anomalies in the validation set. Different methods perform the best in different contexts, i.e. combination of experimental conditions together with computational time. This explains the variability of the previous results and highlights the importance of careful specification of the context in the publication of a new method. All our code and results are available for download.
Hard-ODT: Hardware-Friendly Online Decision Tree Learning Algorithm and System
Authors: Zhe Lin, Sharad Sinha, Wei zhang
Date: 2020-12-11 00:00:00

Decision trees are machine learning models commonly used in various application scenarios. In the era of big data, traditional decision tree induction algorithms are not suitable for learning large-scale datasets due to their stringent data storage requirement. Online decision tree learning algorithms have been devised to tackle this problem by concurrently training with incoming samples and providing inference results. However, even the most up-to-date online tree learning algorithms still suffer from either high memory usage or high computational intensity with dependency and long latency, making them challenging to implement in hardware. To overcome these difficulties, we introduce a new quantile-based algorithm to improve the induction of the Hoeffding tree, one of the state-of-the-art online learning models. The proposed algorithm is light-weight in terms of both memory and computational demand, while still maintaining high generalization ability. A series of optimization techniques dedicated to the proposed algorithm have been investigated from the hardware perspective, including coarse-grained and fine-grained parallelism, dynamic and memory-based resource sharing, pipelining with data forwarding. Following this, we present Hard-ODT, a high-performance, hardware-efficient and scalable online decision tree learning system on a field-programmable gate array (FPGA) with system-level optimization techniques. Performance and resource utilization are modeled for the complete learning system for early and fast analysis of the trade-off between various design metrics. Finally, we propose a design flow in which the proposed learning system is applied to FPGA run-time power monitoring as a case study.
Autoencoding Slow Representations for Semi-supervised Data Efficient Regression
Authors: Oliver Struckmeier, Kshitij Tiwari, Ville Kyrki
Date: 2020-12-11 00:00:00
Tasks: regression, Representation Learning

The slowness principle is a concept inspired by the visual cortex of the brain. It postulates that the underlying generative factors of a quickly varying sensory signal change on a slower time scale. Unsupervised learning of intermediate representations utilizing abundant unlabeled sensory data can be leveraged to perform data-efficient supervised downstream regression. In this paper, we propose a general formulation of slowness for unsupervised representation learning adding a slowness regularization term to the estimate lower bound of the beta-VAE to encourage temporal similarity in observation and latent space. Within this framework we compare existing slowness regularization terms such as the L1 and L2 loss used in existing end-to-end methods, the SlowVAE and propose a new term based on Brownian motion. We empirically evaluate these slowness regularization terms with respect to their downstream task performance and data efficiency. We find that slow representations lead to equal or better downstream task performance and data efficiency in different experiment domains when compared to representations without slowness regularization. Finally, we discuss how the Frechet Inception Distance (FID), traditionally used to determine the generative capabilities of GANs, can serve as a measure to predict the performance of pre-trained Autoencoder model in a supervised downstream task and accelerate hyperparameter search.
Acoustic Leak Detection in Water Networks
Authors: Robert Müller, Steffen Illium, Fabian Ritz, Tobias Schröder, Christian Platschek, Jörg Ochs, Claudia Linnhoff-Popien
Date: 2020-12-11 00:00:00
Tasks: Anomaly Detection

In this work, we present a general procedure for acoustic leak detection in water networks that satisfies multiple real-world constraints such as energy efficiency and ease of deployment. Based on recordings from seven contact microphones attached to the water supply network of a municipal suburb, we trained several shallow and deep anomaly detection models. Inspired by how human experts detect leaks using electronic sounding-sticks, we use these models to repeatedly listen for leaks over a predefined decision horizon. This way we avoid constant monitoring of the system. While we found the detection of leaks in close proximity to be a trivial task for almost all models, neural network based approaches achieve better results at the detection of distant leaks.
Sheaf Neural Networks
Authors: Jakob Hansen, Thomas Gebhart
Date: 2020-12-08 00:00:00
Proceeding: NeurIPS Workshop TDA_and_Beyond 2020 12

We present a generalization of graph convolutional networks by generalizing the diffusion operation underlying this class of graph neural networks. These sheaf neural networks are based on the sheaf Laplacian, a generalization of the graph Laplacian that encodes additional relational structure parameterized by the underlying graph. The sheaf Laplacian and associated matrices provide an extended version of the diffusion operation in graph convolutional networks, providing a proper generalization for domains where relations between nodes are non-constant, asymmetric, and varying in dimension. We show that the resulting sheaf neural networks can outperform graph convolutional networks in domains where relations between nodes are asymmetric and signed.
Nonlinear Distribution Regression for Remote Sensing Applications
Authors: Jose E. Adsuara, Adrián Pérez-Suay, Jordi Muñoz-Marí, Anna Mateo-Sanchis, Maria Piles, Gustau Camps-Valls
Date: 2020-12-07 00:00:00
Tasks: Gaussian Processes, Multiple Instance Learning, regression

In many remote sensing applications one wants to estimate variables or parameters of interest from observations. When the target variable is available at a resolution that matches the remote sensing observations, standard algorithms such as neural networks, random forests or Gaussian processes are readily available to relate the two. However, we often encounter situations where the target variable is only available at the group level, i.e. collectively associated to a number of remotely sensed observations. This problem setting is known in statistics and machine learning as {\em multiple instance learning} or {\em distribution regression}. This paper introduces a nonlinear (kernel-based) method for distribution regression that solves the previous problems without making any assumption on the statistics of the grouped data. The presented formulation considers distribution embeddings in reproducing kernel Hilbert spaces, and performs standard least squares regression with the empirical means therein. A flexible version to deal with multisource data of different dimensionality and sample sizes is also presented and evaluated. It allows working with the native spatial resolution of each sensor, avoiding the need of match-up procedures. Noting the large computational cost of the approach, we introduce an efficient version via random Fourier features to cope with millions of points and groups.
An AI-Assisted Design Method for Topology Optimization Without Pre-Optimized Training Data
Authors: Alex Halle, L. Flavio Campanile, Alexander Hasse
Date: 2020-12-11 00:00:00

Topology optimization is widely used by engineers during the initial product development process to get a first possible geometry design. The state-of-the-art is the iterative calculation, which requires both time and computational power. Some newly developed methods use artificial intelligence to accelerate the topology optimization. These require conventionally pre-optimized data and therefore are dependent on the quality and number of available data. This paper proposes an AI-assisted design method for topology optimization, which does not require pre-optimized data. The designs are provided by an artificial neural network, the predictor, on the basis of boundary conditions and degree of filling (the volume percentage filled by material) as input data. In the training phase, geometries generated on the basis of random input data are evaluated with respect to given criteria. The results of those evaluations flow into an objective function which is minimized by adapting the predictor's parameters. After the training is completed, the presented AI-assisted design procedure supplies geometries which are similar to the ones generated by conventional topology optimizers, but requires a small fraction of the computational effort required by those algorithms. We anticipate our paper to be a starting point for AI-based methods that requires data, that is hard to compute or not available.
Learning physically consistent mathematical models from data using group sparsity
Authors: Suryanarayana Maddu, Bevan L. Cheeseman, Christian L. Müller, Ivo F. Sbalzarini
Date: 2020-12-11 00:00:00

We propose a statistical learning framework based on group-sparse regression that can be used to 1) enforce conservation laws, 2) ensure model equivalence, and 3) guarantee symmetries when learning or inferring differential-equation models from measurement data. Directly learning $\textit{interpretable}$ mathematical models from data has emerged as a valuable modeling approach. However, in areas like biology, high noise levels, sensor-induced correlations, and strong inter-system variability can render data-driven models nonsensical or physically inconsistent without additional constraints on the model structure. Hence, it is important to leverage $\textit{prior}$ knowledge from physical principles to learn "biologically plausible and physically consistent" models rather than models that simply fit the data best. We present a novel group Iterative Hard Thresholding (gIHT) algorithm and use stability selection to infer physically consistent models with minimal parameter tuning. We show several applications from systems biology that demonstrate the benefits of enforcing $\textit{priors}$ in data-driven modeling.