pwc-archive/papers-with-abstracts huggingface dataset:
Neural Network Solver for Coherent Synchrotron Radiation Wakefield Calculations in Accelerator-based Charged Particle Beams
Authors: Auralee Edelen, Christopher Mayes
Date: 2022-03-14 00:00:00

Particle accelerators support a wide array of scientific, industrial, and medical applications. To meet the needs of these applications, accelerator physicists rely heavily on detailed simulations of the complicated particle beam dynamics through the accelerator. One of the most computationally expensive and difficult-to-model effects is the impact of Coherent Synchrotron Radiation (CSR). As a beam travels through a curved trajectory (e.g. due to a bending magnet), it emits radiation that in turn interacts with the rest of the beam. At each step through the trajectory, the electromagnetic field introduced by CSR (called the CSR wakefield) needs to computed and used when calculating the updates to the positions and momenta of every particle in the beam. CSR is one of the major drivers of growth in the beam emittance, which is a key metric of beam quality that is critical in many applications. The CSR wakefield is very computationally intensive to compute with traditional electromagnetic solvers, and this is a major limitation in accurately simulating accelerators. Here, we demonstrate a new approach for the CSR wakefield computation using a neural network solver structured in a way that is readily generalizable to new setups. We validate its performance by adding it to a standard beam tracking test problem and show a ten-fold speedup along with high accuracy.
Denoising and feature extraction in photoemission spectra with variational auto-encoder neural networks
Authors: Francisco Restrepo, Junjing Zhao, Utpal Chatterjee
Date: 2022-03-14 00:00:00
Tasks: BIG-bench Machine Learning, Denoising

In recent years, distinct machine learning (ML) models have been separately used for feature extraction and noise reduction from energy-momentum dispersion intensity maps obtained from raw angle-resolved photoemission spectroscopy (ARPES) data. In this work, we employ a shallow variational auto-encoder (VAE) neural network to demonstrate the prospect of using ML for both denoising of as well as feature extraction from ARPES dispersion maps.
Convolutional-Recurrent Neural Network Proxy for Robust Optimization and Closed-Loop Reservoir Management
Authors: Yong Do Kim, Louis J. Durlofsky
Date: 2022-03-14 00:00:00
Tasks: Management

Production optimization under geological uncertainty is computationally expensive, as a large number of well control schedules must be evaluated over multiple geological realizations. In this work, a convolutional-recurrent neural network (CNN-RNN) proxy model is developed to predict well-by-well oil and water rates, for given time-varying well bottom-hole pressure (BHP) schedules, for each realization in an ensemble. This capability enables the estimation of the objective function and nonlinear constraint values required for robust optimization. The proxy model represents an extension of a recently developed long short-term memory (LSTM) RNN proxy designed to predict well rates for a single geomodel. A CNN is introduced here to processes permeability realizations, and this provides the initial states for the RNN. The CNN-RNN proxy is trained using simulation results for 300 different sets of BHP schedules and permeability realizations. We demonstrate proxy accuracy for oil-water flow through multiple realizations of 3D multi-Gaussian permeability models. The proxy is then incorporated into a closed-loop reservoir management (CLRM) workflow, where it is used with particle swarm optimization and a filter-based method for nonlinear constraint satisfaction. History matching is achieved using an adjoint-gradient-based procedure. The proxy model is shown to perform well in this setting for five different (synthetic) `true' models. Improved net present value along with constraint satisfaction and uncertainty reduction are observed with CLRM. For the robust production optimization steps, the proxy provides O(100) runtime speedup over simulation-based optimization.
Closing the Loop: A Framework for Trustworthy Machine Learning in Power Systems
Authors: Jochen Stiasny, Samuel Chevalier, Rahul Nellikkath, Brynjar Sævarsson, Spyros Chatzivasileiadis
Date: 2022-03-14 00:00:00
Tasks: BIG-bench Machine Learning, Dataset Generation

Deep decarbonization of the energy sector will require massive penetration of stochastic renewable energy resources and an enormous amount of grid asset coordination; this represents a challenging paradigm for the power system operators who are tasked with maintaining grid stability and security in the face of such changes. With its ability to learn from complex datasets and provide predictive solutions on fast timescales, machine learning (ML) is well-posed to help overcome these challenges as power systems transform in the coming decades. In this work, we outline five key challenges (dataset generation, data pre-processing, model training, model assessment, and model embedding) associated with building trustworthy ML models which learn from physics-based simulation data. We then demonstrate how linking together individual modules, each of which overcomes a respective challenge, at sequential stages in the machine learning pipeline can help enhance the overall performance of the training process. In particular, we implement methods that connect different elements of the learning pipeline through feedback, thus "closing the loop" between model training, performance assessments, and re-training. We demonstrate the effectiveness of this framework, its constituent modules, and its feedback connections by learning the N-1 small-signal stability margin associated with a detailed model of a proposed North Sea Wind Power Hub system.
Repairing Regressors for Fair Binary Classification at Any Decision Threshold
Authors: Kweku Kwegyir-Aggrey, A. Feder Cooper, Jessica Dai, John Dickerson, Keegan Hines, Suresh Venkatasubramanian
Date: 2022-03-14 00:00:00
Tasks: All, Binary Classification, Classification, Fairness

We study the problem of post-processing a supervised machine-learned regressor to maximize fair binary classification at all decision thresholds. By decreasing the statistical distance between each group's score distributions, we show that we can increase fair performance across all thresholds at once, and that we can do so without a large decrease in accuracy. To this end, we introduce a formal measure of Distributional Parity, which captures the degree of similarity in the distributions of classifications for different protected groups. Our main result is to put forward a novel post-processing algorithm based on optimal transport, which provably maximizes Distributional Parity, thereby attaining common notions of group fairness like Equalized Odds or Equal Opportunity at all thresholds. We demonstrate on two fairness benchmarks that our technique works well empirically, while also outperforming and generalizing similar techniques from related work.
Simplicial Attention Neural Networks
Authors: L. Giusti, C. Battiloro, P. Di Lorenzo, S. Sardellitti, S. Barbarossa
Date: 2022-03-14 00:00:00
Tasks: Imputation, Trajectory Prediction

The aim of this work is to introduce simplicial attention networks (SANs), i.e., novel neural architectures that operate on data defined on simplicial complexes leveraging masked self-attentional layers. Hinging on formal arguments from topological signal processing, we introduce a proper self-attention mechanism able to process data components at different layers (e.g., nodes, edges, triangles, and so on), while learning how to weight both upper and lower neighborhoods of the given topological domain in a totally task-oriented fashion. The proposed SANs generalize most of the current architectures available for processing data defined on simplicial complexes. The proposed approach compares favorably with other methods when applied to different (inductive and transductive) tasks such as trajectory prediction and missing data imputations in citation complexes.
Distributed On-Sensor Compute System for AR/VR Devices: A Semi-Analytical Simulation Framework for Power Estimation
Authors: Jorge Gomez, Saavan Patel, Syed Shakib Sarwar, Ziyun Li, Raffaele Capoccia, Zhao Wang, Reid Pinkham, Andrew Berkovich, Tsung-Hsun Tsai, Barbara De Salvo, Chiao Liu
Date: 2022-03-14 00:00:00

Augmented Reality/Virtual Reality (AR/VR) glasses are widely foreseen as the next generation computing platform. AR/VR glasses are a complex "system of systems" which must satisfy stringent form factor, computing-, power- and thermal- requirements. In this paper, we will show that a novel distributed on-sensor compute architecture, coupled with new semiconductor technologies (such as dense 3D-IC interconnects and Spin-Transfer Torque Magneto Random Access Memory, STT-MRAM) and, most importantly, a full hardware-software co-optimization are the solutions to achieve attractive and socially acceptable AR/VR glasses. To this end, we developed a semi-analytical simulation framework to estimate the power consumption of novel AR/VR distributed on-sensor computing architectures. The model allows the optimization of the main technological features of the system modules, as well as the computer-vision algorithm partition strategy across the distributed compute architecture. We show that, in the case of the compute-intensive machine learning based Hand Tracking algorithm, the distributed on-sensor compute architecture can reduce the system power consumption compared to a centralized system, with the additional benefits in terms of latency and privacy.
Simultaneous Learning of the Inputs and Parameters in Neural Collaborative Filtering
Authors: Ramin Raziperchikolaei, Young-joo Chung
Date: 2022-03-14 00:00:00
Tasks: Collaborative Filtering

Neural network-based collaborative filtering systems focus on designing network architectures to learn better representations while fixing the input to the user/item interaction vectors and/or ID. In this paper, we first show that the non-zero elements of the inputs are learnable parameters that determine the weights in combining the user/item embeddings, and fixing them limits the power of the models in learning the representations. Then, we propose to learn the value of the non-zero elements of the inputs jointly with the neural network parameters. We analyze the model complexity and the empirical risk of our approach and prove that learning the input leads to a better generalization bound. Our experiments on several real-world datasets show that our method outperforms the state-of-the-art methods, even using shallow network structures with a smaller number of layers and parameters.
On Connecting Deep Trigonometric Networks with Deep Gaussian Processes: Covariance, Expressivity, and Neural Tangent Kernel
Authors: Chi-Ken Lu, Patrick Shafto
Date: 2022-03-14 00:00:00
Tasks: Gaussian Processes

Deep Gaussian Process (DGP) as a model prior in Bayesian learning intuitively exploits the expressive power in function composition. DGPs also offer diverse modeling capabilities, but inference is challenging because marginalization in latent function space is not tractable. With Bochner's theorem, DGP with squared exponential kernel can be viewed as a deep trigonometric network consisting of the random feature layers, sine and cosine activation units, and random weight layers. In the wide limit with a bottleneck, we show that the weight space view yields the same effective covariance functions which were obtained previously in function space. Also, varying the prior distributions over network parameters is equivalent to employing different kernels. As such, DGPs can be translated into the deep bottlenecked trig networks, with which the exact maximum a posteriori estimation can be obtained. Interestingly, the network representation enables the study of DGP's neural tangent kernel, which may also reveal the mean of the intractable predictive distribution. Statistically, unlike the shallow networks, deep networks of finite width have covariance deviating from the limiting kernel, and the inner and outer widths may play different roles in feature learning. Numerical simulations are present to support our findings.
Respecting causality is all you need for training physics-informed neural networks
Authors: Sifan Wang, Shyam Sankaran, Paris Perdikaris
Date: 2022-03-14 00:00:00
Tasks: All, Attribute

While the popularity of physics-informed neural networks (PINNs) is steadily rising, to this date PINNs have not been successful in simulating dynamical systems whose solution exhibits multi-scale, chaotic or turbulent behavior. In this work we attribute this shortcoming to the inability of existing PINNs formulations to respect the spatio-temporal causal structure that is inherent to the evolution of physical systems. We argue that this is a fundamental limitation and a key source of error that can ultimately steer PINN models to converge towards erroneous solutions. We address this pathology by proposing a simple re-formulation of PINNs loss functions that can explicitly account for physical causality during model training. We demonstrate that this simple modification alone is enough to introduce significant accuracy improvements, as well as a practical quantitative mechanism for assessing the convergence of a PINNs model. We provide state-of-the-art numerical results across a series of benchmarks for which existing PINNs formulations fail, including the chaotic Lorenz system, the Kuramoto-Sivashinsky equation in the chaotic regime, and the Navier-Stokes equations in the turbulent regime. To the best of our knowledge, this is the first time that PINNs have been successful in simulating such systems, introducing new opportunities for their applicability to problems of industrial complexity.
Simulating Liquids with Graph Networks
Authors: Jonathan Klimesch, Philipp Holl, Nils Thuerey
Date: 2022-03-14 00:00:00
Tasks: Liquid Simulation, Physical Simulations, Rolling Shutter Correction

Simulating complex dynamics like fluids with traditional simulators is computationally challenging. Deep learning models have been proposed as an efficient alternative, extending or replacing parts of traditional simulators. We investigate graph neural networks (GNNs) for learning fluid dynamics and find that their generalization capability is more limited than previous works would suggest. We also challenge the current practice of adding random noise to the network inputs in order to improve its generalization capability and simulation stability. We find that inserting the real data distribution, e.g. by unrolling multiple simulation steps, improves accuracy and that hiding all domain-specific features from the learning model improves generalization. Our results indicate that learning models, such as GNNs, fail to learn the exact underlying dynamics unless the training set is devoid of any other problem-specific correlations that could be used as shortcuts.
Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networks
Authors: Andrea Basteri, Dario Trevisan
Date: 2022-03-14 00:00:00

Given any deep fully connected neural network, initialized with random Gaussian parameters, we bound from above the quadratic Wasserstein distance between its output distribution and a suitable Gaussian process. Our explicit inequalities indicate how the hidden and output layers sizes affect the Gaussian behaviour of the network and quantitatively recover the distributional convergence results in the wide limit, i.e., if all the hidden layers sizes become large.
Dawn of the transformer era in speech emotion recognition: closing the valence gap
Authors: Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, Björn W. Schuller
Date: 2022-03-14 00:00:00
Tasks: Cross-corpus, Emotion Recognition, Fairness, Speech Emotion Recognition

Recent advances in transformer-based architectures which are pre-trained in self-supervised manner have shown great promise in several machine learning tasks. In the audio domain, such architectures have also been successfully utilised in the field of speech emotion recognition (SER). However, existing works have not evaluated the influence of model size and pre-training data on downstream performance, and have shown limited attention to generalisation, robustness, fairness, and efficiency. The present contribution conducts a thorough analysis of these aspects on several pre-trained variants of wav2vec 2.0 and HuBERT that we fine-tuned on the dimensions arousal, dominance, and valence of MSP-Podcast, while additionally using IEMOCAP and MOSI to test cross-corpus generalisation. To the best of our knowledge, we obtain the top performance for valence prediction without use of explicit linguistic information, with a concordance correlation coefficient (CCC) of .638 on MSP-Podcast. Furthermore, our investigations reveal that transformer-based architectures are more robust to small perturbations compared to a CNN-based baseline and fair with respect to biological sex groups, but not towards individual speakers. Finally, we are the first to show that their extraordinary success on valence is based on implicit linguistic information learnt during fine-tuning of the transformer layers, which explains why they perform on-par with recent multimodal approaches that explicitly utilise textual information. Our findings collectively paint the following picture: transformer-based architectures constitute the new state-of-the-art in SER, but further advances are needed to mitigate remaining robustness and individual speaker issues. To make our findings reproducible, we release the best performing model to the community.
Magnetic Field Prediction Using Generative Adversarial Networks
Authors: Stefan Pollok, Nataniel Olden-Jørgensen, Peter Stanley Jørgensen, Rasmus Bjørk
Date: 2022-03-14 00:00:00
Tasks: Generative Adversarial Network, Prediction

Plenty of scientific and real-world applications are built on magnetic fields and their characteristics. To retrieve the valuable magnetic field information in high resolution, extensive field measurements are required, which are either time-consuming to conduct or even not feasible due to physical constraints. To alleviate this problem, we predict magnetic field values at a random point in space from a few point measurements by using a generative adversarial network (GAN) structure. The deep learning (DL) architecture consists of two neural networks: a generator, which predicts missing field values of a given magnetic field, and a critic, which is trained to calculate the statistical distance between real and generated magnetic field distributions. By minimizing this statistical distance, a reconstruction loss as well as physical losses, our trained generator has learned to predict the missing field values with a median reconstruction test error of 5.14%, when a single coherent region of field points is missing, and 5.86%, when only a few point measurements in space are available and the field measurements around are predicted. We verify the results on an experimentally validated field.
Geometric reconstructions of density based clusterings
Authors: A. L. Garcia-Pulido, K. P. Samardzhiev
Date: 2022-03-14 00:00:00
Tasks: Clustering

DBSCAN* and HDBSCAN* are well established density based clustering algorithms. However, obtaining the clusters of very large datasets is infeasible, limiting their use in real world applications. By exploiting the geometry of Euclidean space, we prove that it is possible to systematically construct the DBSCAN* and HDBSCAN* clusters of a finite $X\subset \mathbb{R}^n$ from specific subsets of $X$. We are able to control the size of these subsets and therefore our results make it possible to cluster very large datasets. To illustrate our theory, we cluster the Microsoft Building Footprint Database of the US, which is not possible using the standard implementations.
Combining AI/ML and PHY Layer Rule Based Inference -- Some First Results
Authors: Brenda Vilas Boas, Wolfgang Zirwas, Martin Haardt
Date: 2022-03-14 00:00:00

In 3GPP New Radio (NR) Release 18 we see the first study item starting in May 2022, which will evaluate the potential of AI/ML methods for Radio Access Network (RAN) 1, i.e., for mobile radio PHY and MAC layer applications. We use the profiling method for accurate iterative estimation of multipath component parameters for PHY layer reference, as it promises a large channel prediction horizon. We investigate options to partly or fully replace some functionalities of this rule based PHY layer method by AI/ML inferences, with the goal to achieve either a higher performance, lower latency, or, reduced processing complexity. We provide first results for noise reduction, then a combined scheme for model order selection, compare options to infer multipath component start parameters, and, provide an outlook on a possible channel prediction framework.
Can A Neural Network Hear the Shape of A Drum?
Authors: Yueqi Zhao, Michael M. Fogler
Date: 2022-03-14 00:00:00
Tasks: Decoder

We have developed a deep neural network that reconstructs the shape of a polygonal domain given the first hundred of its Laplacian eigenvalues. Having an encoder-decoder structure, the network maps input spectra to a latent space and then predicts the discretized image of the domain on a square grid. We tested this network on randomly generated pentagons. The prediction accuracy is high and the predictions obey the Laplacian scaling rule. The network recovers the continuous rotational degree of freedom beyond the symmetry of the grid. The variation of the latent variables under the scaling transformation shows they are strongly correlated with Weyl' s parameters (area, perimeter, and a certain function of the angles) of the test polygons.
From Big to Small: Adaptive Learning to Partial-Set Domains
Authors: Zhangjie Cao, Kaichao You, Ziyang Zhang, Jianmin Wang, Mingsheng Long
Date: 2022-03-14 00:00:00
Tasks: Domain Adaptation, Partial Domain Adaptation

Domain adaptation targets at knowledge acquisition and dissemination from a labeled source domain to an unlabeled target domain under distribution shift. Still, the common requirement of identical class space shared across domains hinders applications of domain adaptation to partial-set domains. Recent advances show that deep pre-trained models of large scale endow rich knowledge to tackle diverse downstream tasks of small scale. Thus, there is a strong incentive to adapt models from large-scale domains to small-scale domains. This paper introduces Partial Domain Adaptation (PDA), a learning paradigm that relaxes the identical class space assumption to that the source class space subsumes the target class space. First, we present a theoretical analysis of partial domain adaptation, which uncovers the importance of estimating the transferable probability of each class and each instance across domains. Then, we propose Selective Adversarial Network (SAN and SAN++) with a bi-level selection strategy and an adversarial adaptation mechanism. The bi-level selection strategy up-weighs each class and each instance simultaneously for source supervised training, target self-training, and source-target adversarial adaptation through the transferable probability estimated alternately by the model. Experiments on standard partial-set datasets and more challenging tasks with superclasses show that SAN++ outperforms several domain adaptation methods.