pwc-archive/papers-with-abstracts huggingface dataset:
Environment Invariant Linear Least Squares
Authors: Jianqing Fan, Cong Fang, Yihong Gu, Tong Zhang
Date: 2023-03-06 00:00:00
Tasks: Causal Inference, regression, Transfer Learning, Variable Selection

This paper considers a multi-environment linear regression model in which data from multiple experimental settings are collected. The joint distribution of the response variable and covariates may vary across different environments, yet the conditional expectations of $y$ given the unknown set of important variables are invariant. Such a statistical model is related to the problem of endogeneity, causal inference, and transfer learning. The motivation behind it is illustrated by how the goals of prediction and attribution are inherent in estimating the true parameter and the important variable set. We construct a novel environment invariant linear least squares (EILLS) objective function, a multi-environment version of linear least-squares regression that leverages the above conditional expectation invariance structure and heterogeneity among different environments to determine the true parameter. Our proposed method is applicable without any additional structural knowledge and can identify the true parameter under a near-minimal identification condition. We establish non-asymptotic $\ell_2$ error bounds on the estimation error for the EILLS estimator in the presence of spurious variables. Moreover, we further show that the $\ell_0$ penalized EILLS estimator can achieve variable selection consistency in high-dimensional regimes. These non-asymptotic results demonstrate the sample efficiency of the EILLS estimator and its capability to circumvent the curse of endogeneity in an algorithmic manner without any prior structural knowledge. To the best of our knowledge, this paper is the first to realize statistically efficient invariance learning in the general linear model.
On Regression in Extreme Regions
Authors: Nathan Huet, Stephan Clémençon, Anne Sabourin
Date: 2023-03-06 00:00:00
Tasks: regression

The statistical learning problem consists in building a predictive function $\hat{f}$ based on independent copies of $(X,Y)$ so that $Y$ is approximated by $\hat{f}(X)$ with minimum (squared) error. Motivated by various applications, special attention is paid here to the case of extreme (i.e. very large) observations $X$. Because of their rarity, the contributions of such observations to the (empirical) error is negligible, and the predictive performance of empirical risk minimizers can be consequently very poor in extreme regions. In this paper, we develop a general framework for regression on extremes. Under appropriate regular variation assumptions regarding the pair $(X,Y)$, we show that an asymptotic notion of risk can be tailored to summarize appropriately predictive performance in extreme regions. It is also proved that minimization of an empirical and nonasymptotic version of this 'extreme risk', based on a fraction of the largest observations solely, yields good generalization capacity. In addition, numerical results providing strong empirical evidence of the relevance of the approach proposed are displayed.
A neural network based model for multi-dimensional nonlinear Hawkes processes
Authors: Sobin Joseph, Shashi Jain
Date: 2023-03-06 00:00:00

This paper introduces the Neural Network for Nonlinear Hawkes processes (NNNH), a non-parametric method based on neural networks to fit nonlinear Hawkes processes. Our method is suitable for analyzing large datasets in which events exhibit both mutually-exciting and inhibitive patterns. The NNNH approach models the individual kernels and the base intensity of the nonlinear Hawkes process using feed forward neural networks and jointly calibrates the parameters of the networks by maximizing the log-likelihood function. We utilize Stochastic Gradient Descent to search for the optimal parameters and propose an unbiased estimator for the gradient, as well as an efficient computation method. We demonstrate the flexibility and accuracy of our method through numerical experiments on both simulated and real-world data, and compare it with state-of-the-art methods. Our results highlight the effectiveness of the NNNH method in accurately capturing the complexities of nonlinear Hawkes processes.
Reinforcement Learning Based Self-play and State Stacking Techniques for Noisy Air Combat Environment
Authors: Ahmet Semih Tasbas, Safa Onur Sahin, Nazim Kemal Ure
Date: 2023-03-06 00:00:00
Tasks: Reinforcement Learning (RL)

Reinforcement learning (RL) has recently proven itself as a powerful instrument for solving complex problems and even surpassed human performance in several challenging applications. This signifies that RL algorithms can be used in the autonomous air combat problem, which has been studied for many years. The complexity of air combat arises from aggressive close-range maneuvers and agile enemy behaviors. In addition to these complexities, there may be uncertainties in real-life scenarios due to sensor errors, which prevent estimation of the actual position of the enemy. In this case, autonomous aircraft should be successful even in the noisy environments. In this study, we developed an air combat simulation, which provides noisy observations to the agents, therefore, make the air combat problem even more challenging. Thus, we present a state stacking method for noisy RL environments as a noise reduction technique. In our extensive set of experiments, the proposed method significantly outperforms the baseline algorithms in terms of the winning ratio, where the performance improvement is even more pronounced in the high noise levels. In addition, we incorporate a self-play scheme to our training process by periodically updating the enemy with a frozen copy of the training agent. By this way, the training agent performs air combat simulations to an enemy with smarter strategies, which improves the performance and robustness of the agents. In our simulations, we demonstrate that the self-play scheme provides important performance gains compared to the classical RL training.
Convolutional Neural Networks as 2-D systems
Authors: Dennis Gramlich, Patricia Pauli, Carsten W. Scherer, Frank Allgöwer, Christian Ebenbauer
Date: 2023-03-06 00:00:00

This paper introduces a novel representation of convolutional Neural Networks (CNNs) in terms of 2-D dynamical systems. To this end, the usual description of convolutional layers with convolution kernels, i.e., the impulse responses of linear filters, is realized in state space as a linear time-invariant 2-D system. The overall convolutional Neural Network composed of convolutional layers and nonlinear activation functions is then viewed as a 2-D version of a Lur'e system, i.e., a linear dynamical system interconnected with static nonlinear components. One benefit of this 2-D Lur'e system perspective on CNNs is that we can use robust control theory much more efficiently for Lipschitz constant estimation than previously possible.
Deep Clustering with a Constraint for Topological Invariance based on Symmetric InfoNCE
Authors: Yuhui Zhang, Yuichiro Wada, Hiroki Waida, Kaito Goto, Yusaku Hino, Takafumi Kanamori
Date: 2023-03-06 00:00:00
Tasks: Clustering, Deep Clustering

We consider the scenario of deep clustering, in which the available prior knowledge is limited. In this scenario, few existing state-of-the-art deep clustering methods can perform well for both non-complex topology and complex topology datasets. To address the problem, we propose a constraint utilizing symmetric InfoNCE, which helps an objective of deep clustering method in the scenario train the model so as to be efficient for not only non-complex topology but also complex topology datasets. Additionally, we provide several theoretical explanations of the reason why the constraint can enhances performance of deep clustering methods. To confirm the effectiveness of the proposed constraint, we introduce a deep clustering method named MIST, which is a combination of an existing deep clustering method and our constraint. Our numerical experiments via MIST demonstrate that the constraint is effective. In addition, MIST outperforms other state-of-the-art deep clustering methods for most of the commonly used ten benchmark datasets.
Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Loss
Authors: Pierre Bréchet, Katerina Papagiannouli, Jing An, Guido Montúfar
Date: 2023-03-06 00:00:00

We consider a deep matrix factorization model of covariance matrices trained with the Bures-Wasserstein distance. While recent works have made advances in the study of the optimization problem for overparametrized low-rank matrix approximation, much emphasis has been placed on discriminative settings and the square loss. In contrast, our model considers another type of loss and connects with the generative setting. We characterize the critical points and minimizers of the Bures-Wasserstein distance over the space of rank-bounded matrices. The Hessian of this loss at low-rank matrices can theoretically blow up, which creates challenges to analyze convergence of gradient optimization methods. We establish convergence results for gradient flow using a smooth perturbative version of the loss as well as convergence results for finite step size gradient descent under certain assumptions on the initial weights.
Scaling strategies for on-device low-complexity source separation with Conv-Tasnet
Authors: Mohamed Nabih Ali, Francesco Paissan, Daniele Falavigna, Alessio Brutti
Date: 2023-03-06 00:00:00
Tasks: Speech Separation

Recently, several very effective neural approaches for single-channel speech separation have been presented in the literature. However, due to the size and complexity of these models, their use on low-resource devices, e.g. for hearing aids, and earphones, is still a challenge and established solutions are not available yet. Although approaches based on either pruning or compressing neural models have been proposed, the design of a model architecture suitable for a certain application domain often requires heuristic procedures not easily portable to different low-resource platforms. Given the modular nature of the well-known Conv-Tasnet speech separation architecture, in this paper we consider three parameters that directly control the overall size of the model, namely: the number of residual blocks, the number of repetitions of the separation blocks and the number of channels in the depth-wise convolutions, and experimentally evaluate how they affect the speech separation performance. In particular, experiments carried out on the Libri2Mix show that the number of dilated 1D-Conv blocks is the most critical parameter and that the usage of extra-dilation in the residual blocks allows reducing the performance drop.
Searching for Effective Neural Network Architectures for Heart Murmur Detection from Phonocardiogram
Authors: Hao Wen, Jingsu Kang
Date: 2023-03-06 00:00:00
Tasks: Model Selection, Multi-Task Learning

Aim: The George B. Moody PhysioNet Challenge 2022 raised problems of heart murmur detection and related abnormal cardiac function identification from phonocardiograms (PCGs). This work describes the novel approaches developed by our team, Revenger, to solve these problems. Methods: PCGs were resampled to 1000 Hz, then filtered with a Butterworth band-pass filter of order 3, cutoff frequencies 25 - 400 Hz, and z-score normalized. We used the multi-task learning (MTL) method via hard parameter sharing to train one neural network (NN) model for all the Challenge tasks. We performed neural architecture searching among a set of network backbones, including multi-branch convolutional neural networks (CNNs), SE-ResNets, TResNets, simplified wav2vec2, etc. Based on a stratified splitting of the subjects, 20% of the public data was left out as a validation set for model selection. The AdamW optimizer was adopted, along with the OneCycle scheduler, to optimize the model weights. Results: Our murmur detection classifier received a weighted accuracy score of 0.736 (ranked 14th out of 40 teams) and a Challenge cost score of 12944 (ranked 19th out of 39 teams) on the hidden validation set. Conclusion: We provided a practical solution to the problems of detecting heart murmurs and providing clinical diagnosis suggestions from PCGs.
Primal and Dual Analysis of Entropic Fictitious Play for Finite-sum Problems
Authors: Atsushi Nitanda, Kazusato Oko, Denny Wu, Nobuhito Takenouchi, Taiji Suzuki
Date: 2023-03-06 00:00:00
Tasks: Image Generation

The entropic fictitious play (EFP) is a recently proposed algorithm that minimizes the sum of a convex functional and entropy in the space of measures -- such an objective naturally arises in the optimization of a two-layer neural network in the mean-field regime. In this work, we provide a concise primal-dual analysis of EFP in the setting where the learning problem exhibits a finite-sum structure. We establish quantitative global convergence guarantees for both the continuous-time and discrete-time dynamics based on properties of a proximal Gibbs measure introduced in Nitanda et al. (2022). Furthermore, our primal-dual framework entails a memory-efficient particle-based implementation of the EFP update, and also suggests a connection to gradient boosting methods. We illustrate the efficiency of our novel implementation in experiments including neural network optimization and image synthesis.
Graph Positional Encoding via Random Feature Propagation
Authors: Moshe Eliasof, Fabrizio Frasca, Beatrice Bevilacqua, Eran Treister, Gal Chechik, Haggai Maron
Date: 2023-03-06 00:00:00
Tasks: Graph Classification, Node Classification

Two main families of node feature augmentation schemes have been explored for enhancing GNNs: random features and spectral positional encoding. Surprisingly, however, there is still no clear understanding of the relation between these two augmentation schemes. Here we propose a novel family of positional encoding schemes which draws a link between the above two approaches and improves over both. The new approach, named Random Feature Propagation (RFP), is inspired by the power iteration method and its generalizations. It concatenates several intermediate steps of an iterative algorithm for computing the dominant eigenvectors of a propagation matrix, starting from random node features. Notably, these propagation steps are based on graph-dependent propagation operators that can be either predefined or learned. We explore the theoretical and empirical benefits of RFP. First, we provide theoretical justifications for using random features, for incorporating early propagation steps, and for using multiple random initializations. Then, we empirically demonstrate that RFP significantly outperforms both spectral PE and random features in multiple node classification and graph classification benchmarks.
A Topological Distance Measure between Multi-Fields for Classification and Analysis of Shapes and Data
Authors: Yashwanth Ramamurthi, Amit Chattopadhyay
Date: 2023-03-06 00:00:00
Tasks: Computational chemistry

Distance measures play an important role in shape classification and data analysis problems. Topological distances based on Reeb graphs and persistence diagrams have been employed to obtain effective algorithms in shape matching and scalar data analysis. In the current paper, we propose an improved distance measure between two multi-fields by computing a multi-dimensional Reeb graph (MDRG) each of which captures the topology of a multi-field through a hierarchy of Reeb graphs in different dimensions. A hierarchy of persistence diagrams is then constructed by computing a persistence diagram corresponding to each Reeb graph of the MDRG. Based on this representation, we propose a novel distance measure between two MDRGs by extending the bottleneck distance between two Reeb graphs. We show that the proposed measure satisfies the pseudo-metric and stability properties. We examine the effectiveness of the proposed multi-field topology-based measure on two different applications: (1) shape classification and (2) detection of topological features in a time-varying multi-field data. In the shape classification problem, the performance of the proposed measure is compared with the well-known topology-based measures in shape matching. In the second application, we consider a time-varying volumetric multi-field data from the field of computational chemistry where the goal is to detect the site of stable bond formation between Pt and CO molecules. We demonstrate the ability of the proposed distance in classifying each of the sites as occurring before and after the bond stabilization.
$α$-divergence Improves the Entropy Production Estimation via Machine Learning
Authors: Euijoon Kwon, Yongjoo Baek
Date: 2023-03-06 00:00:00

Recent years have seen a surge of interest in the algorithmic estimation of stochastic entropy production (EP) from trajectory data via machine learning. A crucial element of such algorithms is the identification of a loss function whose minimization guarantees the accurate EP estimation. In this study, we show that there exists a host of loss functions, namely those implementing a variational representation of the $\alpha$-divergence, which can be used for the EP estimation. By fixing $\alpha$ to a value between $-1$ and $0$, the $\alpha$-NEEP (Neural Estimator for Entropy Production) exhibits a much more robust performance against strong nonequilibrium driving or slow dynamics, which adversely affects the existing method based on the Kullback-Leibler divergence ($\alpha = 0$). In particular, the choice of $\alpha = -0.5$ tends to yield the optimal results. To corroborate our findings, we present an exactly solvable simplification of the EP estimation problem, whose loss function landscape and stochastic properties give deeper intuition into the robustness of the $\alpha$-NEEP.
Stabilizing the Maximal Entropy Moment Method for Rarefied Gas Dynamics at Single-Precision
Authors: Candi Zheng, Wang Yang, Shiyi Chen
Date: 2023-03-06 00:00:00
Tasks: GPU, valid

The maximal entropy moment method (MEM) is systematic solution of the challenging problem: generating extended hydrodynamic equations valid for both dense and rarefied gases. However, simulating MEM suffers from a computational expensive and ill-conditioned maximal entropy problem. It causes numerical overflow and breakdown when the numerical precision is insufficient, especially for flows like high-speed shock waves. It also prevents modern GPUs from accelerating MEM with their enormous single floating-point precision computation power. This paper aims to stabilize MEM, making it possible to simulating very strong normal shock waves on modern GPUs at single precision. We improve the condition number of the maximal entropy problem by proposing gauge transformations, which moves not only flow fields but also hydrodynamic equations into a more optimal coordinate system. We addressed numerical overflow and breakdown in the maximal entropy problem by employing the canonical form of distribution and a modified Newton optimization method. Moreover, we discovered a counter-intuitive phenomenon that over-refined spatial mesh beyond mean free path degrades the stability of MEM. With these techniques, we accomplished single-precision GPU simulations of high speed shock wave up to Mach 10 utilizing 35 moments MEM, while previous methods only achieved Mach 4 on double-precision.
An Analysis of Physics-Informed Neural Networks
Authors: Edward Small
Date: 2023-03-06 00:00:00
Tasks: Seismic Imaging

Whilst the partial differential equations that govern the dynamics of our world have been studied in great depth for centuries, solving them for complex, high-dimensional conditions and domains still presents an incredibly large mathematical and computational challenge. Analytical methods can be cumbersome to utilise, and numerical methods can lead to errors and inaccuracies. On top of this, sometimes we lack the information or knowledge to pose the problem well enough to apply these kinds of methods. Here, we present a new approach to approximating the solution to physical systems - physics-informed neural networks. The concept of artificial neural networks is introduced, the objective function is defined, and optimisation strategies are discussed. The partial differential equation is then included as a constraint in the loss function for the optimisation problem, giving the network access to knowledge of the dynamics of the physical system it is modelling. Some intuitive examples are displayed, and more complex applications are considered to showcase the power of physics informed neural networks, such as in seismic imaging. Solution error is analysed, and suggestions are made to improve convergence and/or solution precision. Problems and limitations are also touched upon in the conclusions, as well as some thoughts as to where physics informed neural networks are most useful, and where they could go next.
Very fast, approximate counterfactual explanations for decision forests
Authors: Miguel Á. Carreira-Perpiñán, Suryabhan Singh Hada
Date: 2023-03-06 00:00:00
Tasks: counterfactual, Counterfactual Explanation

We consider finding a counterfactual explanation for a classification or regression forest, such as a random forest. This requires solving an optimization problem to find the closest input instance to a given instance for which the forest outputs a desired value. Finding an exact solution has a cost that is exponential on the number of leaves in the forest. We propose a simple but very effective approach: we constrain the optimization to only those input space regions defined by the forest that are populated by actual data points. The problem reduces to a form of nearest-neighbor search using a certain distance on a certain dataset. This has two advantages: first, the solution can be found very quickly, scaling to large forests and high-dimensional data, and enabling interactive use. Second, the solution found is more likely to be realistic in that it is guided towards high-density areas of input space.
DR-Label: Improving GNN Models for Catalysis Systems by Label Deconstruction and Reconstruction
Authors: Bowen Wang, Chen Liang, Jiaze Wang, Furui Liu, Shaogang Hao, Dong Li, Jianye Hao, Guangyong Chen, Xiaolong Zou, Pheng-Ann Heng
Date: 2023-03-06 00:00:00
Tasks: Graph Neural Network, Initial Structure to Relaxed Energy (IS2RE), Direct, Property Prediction

Attaining the equilibrium state of a catalyst-adsorbate system is key to fundamentally assessing its effective properties, such as adsorption energy. Machine learning methods with finer supervision strategies have been applied to boost and guide the relaxation process of an atomic system and better predict its properties at the equilibrium state. In this paper, we present a novel graph neural network (GNN) supervision and prediction strategy DR-Label. The method enhances the supervision signal, reduces the multiplicity of solutions in edge representation, and encourages the model to provide node predictions that are graph structural variation robust. DR-Label first Deconstructs finer-grained equilibrium state information to the model by projecting the node-level supervision signal to each edge. Reversely, the model Reconstructs a more robust equilibrium state prediction by transforming edge-level predictions to node-level with a sphere-fitting algorithm. The DR-Label strategy was applied to three radically distinct models, each of which displayed consistent performance enhancements. Based on the DR-Label strategy, we further proposed DRFormer, which achieved a new state-of-the-art performance on the Open Catalyst 2020 (OC20) dataset and the Cu-based single-atom-alloyed CO adsorption (SAA) dataset. We expect that our work will highlight crucial steps for the development of a more accurate model in equilibrium state property prediction of a catalysis system.