pwc-archive/papers-with-abstracts huggingface dataset:
PDE-constrained Models with Neural Network Terms: Optimization and Global Convergence
Authors: Justin Sirignano, Jonathan MacArt, Konstantinos Spiliopoulos
Date: 2021-05-18 00:00:00

Recent research has used deep learning to develop partial differential equation (PDE) models in science and engineering. The functional form of the PDE is determined by a neural network, and the neural network parameters are calibrated to available data. Calibration of the embedded neural network can be performed by optimizing over the PDE. Motivated by these applications, we rigorously study the optimization of a class of linear elliptic PDEs with neural network terms. The neural network parameters in the PDE are optimized using gradient descent, where the gradient is evaluated using an adjoint PDE. As the number of parameters become large, the PDE and adjoint PDE converge to a non-local PDE system. Using this limit PDE system, we are able to prove convergence of the neural network-PDE to a global minimum during the optimization. Finally, we use this adjoint method to train a neural network model for an application in fluid mechanics, in which the neural network functions as a closure model for the Reynolds-averaged Navier--Stokes (RANS) equations. The RANS neural network model is trained on several datasets for turbulent channel flow and is evaluated out-of-sample at different Reynolds numbers.
WOVe: Incorporating Word Order in GloVe Word Embeddings
Authors: Mohammed Ibrahim, Susan Gauch, Tyler Gerth, Brandon Cox
Date: 2021-05-18 00:00:00
Tasks: Word Embeddings, Word Similarity

Word vector representations open up new opportunities to extract useful information from unstructured text. Defining a word as a vector made it easy for the machine learning algorithms to understand a text and extract information from. Word vector representations have been used in many applications such word synonyms, word analogy, syntactic parsing, and many others. GloVe, based on word contexts and matrix vectorization, is an ef-fective vector-learning algorithm. It improves on previous vector-learning algorithms. However, the GloVe model fails to explicitly consider the order in which words appear within their contexts. In this paper, multiple methods of incorporating word order in GloVe word embeddings are proposed. Experimental results show that our Word Order Vector (WOVe) word embeddings approach outperforms unmodified GloVe on the natural lan-guage tasks of analogy completion and word similarity. WOVe with direct concatenation slightly outperformed GloVe on the word similarity task, increasing average rank by 2%. However, it greatly improved on the GloVe baseline on a word analogy task, achieving an average 36.34% improvement in accuracy.
Multi-Aspect Temporal Network Embedding: A Mixture of Hawkes Process View
Authors: Yutian Chang, Guannan Liu, Yuan Zuo, Junjie Wu
Date: 2021-05-18 00:00:00
Tasks: Graph Attention, Network Embedding

Recent years have witnessed the tremendous research interests in network embedding. Extant works have taken the neighborhood formation as the critical information to reveal the inherent dynamics of network structures, and suggested encoding temporal edge formation sequences to capture the historical influences of neighbors. In this paper, however, we argue that the edge formation can be attributed to a variety of driving factors including the temporal influence, which is better referred to as multiple aspects. As a matter of fact, different node aspects can drive the formation of distinctive neighbors, giving birth to the multi-aspect embedding that relates to but goes beyond a temporal scope. Along this vein, we propose a Mixture of Hawkes-based Temporal Network Embeddings (MHNE) model to capture the aspect-driven neighborhood formation of networks. In MHNE, we encode the multi-aspect embeddings into the mixture of Hawkes processes to gain the advantages in modeling the excitation effects and the latent aspects. Specifically, a graph attention mechanism is used to assign different weights to account for the excitation effects of history events, while a Gumbel-Softmax is plugged in to derive the distribution over the aspects. Extensive experiments on 8 different temporal networks have demonstrated the great performance of the multi-aspect embeddings obtained by MHNE in comparison with the state-of-the-art methods.
Robust Learning in Heterogeneous Contexts
Authors: Muhammad Osama, Dave Zachariah, Petre Stoica
Date: 2021-05-18 00:00:00

We consider the problem of learning from training data obtained in different contexts, where the underlying context distribution is unknown and is estimated empirically. We develop a robust method that takes into account the uncertainty of the context distribution. Unlike the conventional and overly conservative minimax approach, we focus on excess risks and construct distribution sets with statistical coverage to achieve an appropriate trade-off between performance and robustness. The proposed method is computationally scalable and shown to interpolate between empirical risk minimization and minimax regret objectives. Using both real and synthetic data, we demonstrate its ability to provide robustness in worst-case scenarios without harming performance in the nominal scenario.
Transformers à Grande Vitesse
Authors: Farid Arthaud, Guillaume Lecoeur, Alban Pierre
Date: 2021-05-18 00:00:00

Robust travel time predictions are of prime importance in managing any transportation infrastructure, and particularly in rail networks where they have major impacts both on traffic regulation and passenger satisfaction. We aim at predicting the travel time of trains on rail sections at the scale of an entire rail network in real-time, by estimating trains' delays relative to a theoretical circulation plan. Predicting the evolution of a given train's delay is a uniquely hard problem, distinct from mainstream road traffic forecasting problems, since it involves several hard-to-model phenomena: train spacing, station congestion and heterogeneous rolling stock among others. We first offer empirical evidence of the previously unexplored phenomenon of delay propagation at the scale of a railway network, leading to delays being amplified by interactions between trains and the network's physical limitations. We then contribute a novel technique using the transformer architecture and pre-trained embeddings to make real-time massively parallel predictions for train delays at the scale of the whole rail network (over 3000 trains at peak hours, making predictions at an average horizon of 70 minutes). Our approach yields very positive results on real-world data when compared to currently-used and experimental prediction techniques.
Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation
Authors: Mathias Müller, Rico Sennrich
Date: 2021-05-18 00:00:00
Proceeding: ACL 2021 5
Tasks: Machine Translation, NMT, Translation

Neural Machine Translation (NMT) currently exhibits biases such as producing translations that are too short and overgenerating frequent words, and shows poor robustness to copy noise in training data or domain shift. Recent work has tied these shortcomings to beam search -- the de facto standard inference algorithm in NMT -- and Eikema & Aziz (2020) propose to use Minimum Bayes Risk (MBR) decoding on unbiased samples instead. In this paper, we empirically investigate the properties of MBR decoding on a number of previously reported biases and failure cases of beam search. We find that MBR still exhibits a length and token frequency bias, owing to the MT metrics used as utility functions, but that MBR also increases robustness against copy noise in the training data and domain shift.
Univariate Long-Term Municipal Water Demand Forecasting
Authors: Blake VanBerlo, Matthew A. S. Ross, Daniel Hsia
Date: 2021-05-18 00:00:00
Tasks: Demand Forecasting, Management, regression, Time Series, Time Series Analysis, Time Series Forecasting, Univariate Time Series Forecasting

This study describes an investigation into the modelling of citywide water consumption in London, Canada. Multiple modelling techniques were evaluated for the task of univariate time series forecasting with water consumption, including linear regression, Facebook's Prophet method, recurrent neural networks, and convolutional neural networks. Prophet was identified as the model of choice, having achieved a mean absolute percentage error of 2.51%, averaged across a 5-fold cross validation. Prophet was also found to have other advantages deemed valuable to water demand management stakeholders, including inherent interpretability and graceful handling of missing data. The implementation for the methods described in this paper has been open sourced, as they may be adaptable by other municipalities.
Learning stochastic dynamical systems with neural networks mimicking the Euler-Maruyama scheme
Authors: Noura Dridi, Lucas Drumetz, Ronan Fablet
Date: 2021-05-18 00:00:00

Stochastic differential equations (SDEs) are one of the most important representations of dynamical systems. They are notable for the ability to include a deterministic component of the system and a stochastic one to represent random unknown factors. However, this makes learning SDEs much more challenging than ordinary differential equations (ODEs). In this paper, we propose a data driven approach where parameters of the SDE are represented by a neural network with a built-in SDE integration scheme. The loss function is based on a maximum likelihood criterion, under order one Markov Gaussian assumptions. The algorithm is applied to the geometric brownian motion and a stochastic version of the Lorenz-63 model. The latter is particularly hard to handle due to the presence of a stochastic component that depends on the state. The algorithm performance is attested using different simulations results. Besides, comparisons are performed with the reference gradient matching method used for non linear drift estimation, and a neural networks-based method, that does not consider the stochastic term.
Automatic Assessment of Alzheimer's Disease Diagnosis Based on Deep Learning Techniques
Authors: Alejandro Puente-Castro, Enrique Fernandez-Blanco, Alejandro Pazos, Cristian R. Munteanu
Date: 2021-05-18 00:00:00
Tasks: Transfer Learning

Early detection is crucial to prevent the progression of Alzheimer's disease (AD). Thus, specialists can begin preventive treatment as soon as possible. They demand fast and precise assessment in the diagnosis of AD in the earliest and hardest to detect stages. The main objective of this work is to develop a system that automatically detects the presence of the disease in sagittal magnetic resonance images (MRI), which are not generally used. Sagittal MRIs from ADNI and OASIS data sets were employed. Experiments were conducted using Transfer Learning (TL) techniques in order to achieve more accurate results. There are two main conclusions to be drawn from this work: first, the damages related to AD and its stages can be distinguished in sagittal MRI and, second, the results obtained using DL models with sagittal MRIs are similar to the state-of-the-art, which uses the horizontal-plane MRI. Although sagittal-plane MRIs are not commonly used, this work proved that they were, at least, as effective as MRI from other planes at identifying AD in early stages. This could pave the way for further research. Finally, one should bear in mind that in certain fields, obtaining the examples for a data set can be very expensive. This study proved that DL models could be built in these fields, whereas TL is an essential tool for completing the task with fewer examples.
StackVAE-G: An efficient and interpretable model for time series anomaly detection
Authors: Wenkai Li, WenBo Hu, Ting Chen, Ning Chen, Cheng Feng
Date: 2021-05-18 00:00:00
Tasks: Anomaly Detection, Graph Learning, Graph Neural Network, Time Series, Time Series Analysis, Time Series Anomaly Detection

Recent studies have shown that autoencoder-based models can achieve superior performance on anomaly detection tasks due to their excellent ability to fit complex data in an unsupervised manner. In this work, we propose a novel autoencoder-based model, named StackVAE-G that can significantly bring the efficiency and interpretability to multivariate time series anomaly detection. Specifically, we utilize the similarities across the time series channels by the stacking block-wise reconstruction with a weight-sharing scheme to reduce the size of learned models and also relieve the overfitting to unknown noises in the training data. We also leverage a graph learning module to learn a sparse adjacency matrix to explicitly capture the stable interrelation structure among multiple time series channels for the interpretable pattern reconstruction of interrelated channels. Combining these two modules, we introduce the stacking block-wise VAE (variational autoencoder) with GNN (graph neural network) model for multivariate time series anomaly detection. We conduct extensive experiments on three commonly used public datasets, showing that our model achieves comparable (even better) performance with the state-of-the-art modelsand meanwhile requires much less computation and memory cost. Furthermore, we demonstrate that the adjacency matrix learned by our model accurately captures the interrelation among multiple channels, and can provide valuable information for failure diagnosis applications.
Online Multimodal Transportation Planning using Deep Reinforcement Learning
Authors: Amirreza Farahani, Laura Genga, Remco Dijkman
Date: 2021-05-18 00:00:00
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In this paper we propose a Deep Reinforcement Learning approach to solve a multimodal transportation planning problem, in which containers must be assigned to a truck or to trains that will transport them to their destination. While traditional planning methods work "offline" (i.e., they take decisions for a batch of containers before the transportation starts), the proposed approach is "online", in that it can take decisions for individual containers, while transportation is being executed. Planning transportation online helps to effectively respond to unforeseen events that may affect the original transportation plan, thus supporting companies in lowering transportation costs. We implemented different container selection heuristics within the proposed Deep Reinforcement Learning algorithm and we evaluated its performance for each heuristic using data that simulate a realistic scenario, designed on the basis of a real case study at a logistics company. The experimental results revealed that the proposed method was able to learn effective patterns of container assignment. It outperformed tested competitors in terms of total transportation costs and utilization of train capacity by 20.48% to 55.32% for the cost and by 7.51% to 20.54% for the capacity. Furthermore, it obtained results within 2.7% for the cost and 0.72% for the capacity of the optimal solution generated by an Integer Linear Programming solver in an offline setting.
A Millimeter-Wave Self-Mixing Array with Large Gain and Wide Angular Receiving Range
Authors: Jonas Kornprobst, Thomas J. Mittermaier, Thomas F. Eibert
Date: 2021-05-18 00:00:00

The concept of self-mixing antenna arrays is presented and analyzed with respect to its beneficial behavior of large gain over a wide angular range. The large gain is attained by an antenna array with large element spacing, where all array element signals are combined approximately coherently over the entire angular receiving range. This functionality is achieved by the self-mixing principle, where an exact description via an intermediate frequency (IF) array factor is derived. For verification purposes, a 4 x 2 self-mixing array is fabricated and measured in the frequency range from 34 GHz to 39 GHz. A multiple-resonances millimeter-wave microstrip patch antenna has been especially developed to achieve large bandwidth and a wide angular receiving range. The broad beamwidth is achieved by two parasitic patches and suitable radiation characteristics of the resonant modes. The self-mixing of the receive signal is realized at each antenna element by a Schottky diode with an optimized operating point. The down-converted array element signals are then combined and measured at the IF. The receive power is increased significantly over a large angular range as compared to conventional array feeding techniques. The simulation results are verified by measurements, which show very good agreement.
Solving the electronic Schrödinger equation for multiple nuclear geometries with weight-sharing deep neural networks
Authors: Michael Scherbela, Rafael Reisenhofer, Leon Gerard, Philipp Marquetand, Philipp Grohs
Date: 2021-05-18 00:00:00
Tasks: Machine Translation, Transfer Learning, Translation

Accurate numerical solutions for the Schr\"odinger equation are of utmost importance in quantum chemistry. However, the computational cost of current high-accuracy methods scales poorly with the number of interacting particles. Combining Monte Carlo methods with unsupervised training of neural networks has recently been proposed as a promising approach to overcome the curse of dimensionality in this setting and to obtain accurate wavefunctions for individual molecules at a moderately scaling computational cost. These methods currently do not exploit the regularity exhibited by wavefunctions with respect to their molecular geometries. Inspired by recent successful applications of deep transfer learning in machine translation and computer vision tasks, we attempt to leverage this regularity by introducing a weight-sharing constraint when optimizing neural network-based models for different molecular geometries. That is, we restrict the optimization process such that up to 95 percent of weights in a neural network model are in fact equal across varying molecular geometries. We find that this technique can accelerate optimization when considering sets of nuclear geometries of the same molecule by an order of magnitude and that it opens a promising route towards pre-trained neural network wavefunctions that yield high accuracy even across different molecules.
A mm-Wave Patch Antenna with Broad Bandwidth and a Wide Angular Range
Authors: Jonas Kornprobst, Kun Wang, Gerhard Hamberger, Thomas F. Eibert
Date: 2021-05-17 00:00:00

A novel mm-wave microstrip-fed patch antenna with broad bandwidth and wide angular coverage suitable for integration in planar arrays is designed, analyzed and verified by measurements. The antenna provides a bandwidth of 13.1% between 34.1 GHz and 38.9 GHz, which is achieved by a slotted multiple resonances microstrip patch and a matching circuit in microstrip technology. The antenna is built on RO3003 substrate with top and ground layers, which is low cost compared to other techniques. For simple integration with microstrip and frontend circuits, the feeding happens in the top layer with a microstrip coupling gap feed. The wide half power beamwidth is achieved by suitably designed parasitic patches for the first resonant mode. The second resonant mode has a wide half power beamwidth by default. The half power beamwidth is between 100{\deg} and 125{\deg} within the matched bandwidth, which is a very good value for a microstrip patch antenna radiating over a ground plane. The measured input impedance and radiation characteristic show very good agreement with simulation results.
On Convex Clustering Solutions
Authors: Canh Hao Nguyen, Hiroshi Mamitsuka
Date: 2021-05-18 00:00:00
Tasks: Clustering

Convex clustering is an attractive clustering algorithm with favorable properties such as efficiency and optimality owing to its convex formulation. It is thought to generalize both k-means clustering and agglomerative clustering. However, it is not known whether convex clustering preserves desirable properties of these algorithms. A common expectation is that convex clustering may learn difficult cluster types such as non-convex ones. Current understanding of convex clustering is limited to only consistency results on well-separated clusters. We show new understanding of its solutions. We prove that convex clustering can only learn convex clusters. We then show that the clusters have disjoint bounding balls with significant gaps. We further characterize the solutions, regularization hyperparameters, inclusterable cases and consistency.
Label Inference Attacks from Log-loss Scores
Authors: Abhinav Aggarwal, Shiva Prasad Kasiviswanathan, Zekun Xu, Oluwaseyi Feyisetan, Nathanael Teissier
Date: 2021-05-18 00:00:00

Log-loss (also known as cross-entropy loss) metric is ubiquitously used across machine learning applications to assess the performance of classification algorithms. In this paper, we investigate the problem of inferring the labels of a dataset from single (or multiple) log-loss score(s), without any other access to the dataset. Surprisingly, we show that for any finite number of label classes, it is possible to accurately infer the labels of the dataset from the reported log-loss score of a single carefully constructed prediction vector if we allow arbitrary precision arithmetic. Additionally, we present label inference algorithms (attacks) that succeed even under addition of noise to the log-loss scores and under limited precision arithmetic. All our algorithms rely on ideas from number theory and combinatorics and require no model training. We run experimental simulations on some real datasets to demonstrate the ease of running these attacks in practice.
Sharp Restricted Isometry Property Bounds for Low-rank Matrix Recovery Problems with Corrupted Measurements
Authors: Ziye Ma, Yingjie Bi, Javad Lavaei, Somayeh Sojoudi
Date: 2021-05-18 00:00:00
Tasks: Matrix Completion, Retrieval

In this paper, we study a general low-rank matrix recovery problem with linear measurements corrupted by some noise. The objective is to understand under what conditions on the restricted isometry property (RIP) of the problem local search methods can find the ground truth with a small error. By analyzing the landscape of the non-convex problem, we first propose a global guarantee on the maximum distance between an arbitrary local minimizer and the ground truth under the assumption that the RIP constant is smaller than $1/2$. We show that this distance shrinks to zero as the intensity of the noise reduces. Our new guarantee is sharp in terms of the RIP constant and is much stronger than the existing results. We then present a local guarantee for problems with an arbitrary RIP constant, which states that any local minimizer is either considerably close to the ground truth or far away from it. Next, we prove the strict saddle property, which guarantees the global convergence of the perturbed gradient descent method in polynomial time. The developed results demonstrate how the noise intensity and the RIP constant of the problem affect the landscape of the problem.