pwc-archive/papers-with-abstracts huggingface dataset:
Improving the performance of weak supervision searches using transfer and meta-learning
Authors: Hugues Beauchesne, Zong-En Chen, Cheng-Wei Chiang
Date: 2023-12-11 00:00:00
Tasks: Meta-Learning

Weak supervision searches have in principle the advantages of both being able to train on experimental data and being able to learn distinctive signal properties. However, the practical applicability of such searches is limited by the fact that successfully training a neural network via weak supervision can require a large amount of signal. In this work, we seek to create neural networks that can learn from less experimental signal by using transfer and meta-learning. The general idea is to first train a neural network on simulations, thereby learning concepts that can be reused or becoming a more efficient learner. The neural network would then be trained on experimental data and should require less signal because of its previous training. We find that transfer and meta-learning can substantially improve the performance of weak supervision searches.
Compute-in-Memory based Neural Network Accelerators for Safety-Critical Systems: Worst-Case Scenarios and Protections
Authors: Zheyu Yan, Xiaobo Sharon Hu, Yiyu Shi
Date: 2023-12-11 00:00:00

Emerging non-volatile memory (NVM)-based Computing-in-Memory (CiM) architectures show substantial promise in accelerating deep neural networks (DNNs) due to their exceptional energy efficiency. However, NVM devices are prone to device variations. Consequently, the actual DNN weights mapped to NVM devices can differ considerably from their targeted values, inducing significant performance degradation. Many existing solutions aim to optimize average performance amidst device variations, which is a suitable strategy for general-purpose conditions. However, the worst-case performance that is crucial for safety-critical applications is largely overlooked in current research. In this study, we define the problem of pinpointing the worst-case performance of CiM DNN accelerators affected by device variations. Additionally, we introduce a strategy to identify a specific pattern of the device value deviations in the complex, high-dimensional value deviation space, responsible for this worst-case outcome. Our findings reveal that even subtle device variations can precipitate a dramatic decline in DNN accuracy, posing risks for CiM-based platforms in supporting safety-critical applications. Notably, we observe that prevailing techniques to bolster average DNN performance in CiM accelerators fall short in enhancing worst-case scenarios. In light of this issue, we propose a novel worst-case-aware training technique named A-TRICE that efficiently combines adversarial training and noise-injection training with right-censored Gaussian noise to improve the DNN accuracy in the worst-case scenarios. Our experimental results demonstrate that A-TRICE improves the worst-case accuracy under device variations by up to 33%.
Order Matters in the Presence of Dataset Imbalance for Multilingual Learning
Authors: Dami Choi, Derrick Xin, Hamid Dadkhahi, Justin Gilmer, Ankush Garg, Orhan Firat, Chih-Kuan Yeh, Andrew M. Dai, Behrooz Ghorbani
Date: 2023-12-11 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Language Modeling, Language Modelling, Machine Translation, Multi-Task Learning, NMT, Translation

In this paper, we empirically study the optimization dynamics of multi-task learning, particularly focusing on those that govern a collection of tasks with significant data imbalance. We present a simple yet effective method of pre-training on high-resource tasks, followed by fine-tuning on a mixture of high/low-resource tasks. We provide a thorough empirical study and analysis of this method's benefits showing that it achieves consistent improvements relative to the performance trade-off profile of standard static weighting. We analyze under what data regimes this method is applicable and show its improvements empirically in neural machine translation (NMT) and multi-lingual language modeling.
GTA: Gated Toxicity Avoidance for LM Performance Preservation
Authors: Heegyu Kim, Hyunsouk Cho
Date: 2023-12-11 00:00:00
Tasks: Language Modeling, Language Modelling, Text Generation

Caution: This paper includes offensive words that could potentially cause unpleasantness. The fast-paced evolution of generative language models such as GPT-4 has demonstrated outstanding results in various NLP generation tasks. However, due to the potential generation of offensive words related to race or gender, various Controllable Text Generation (CTG) methods have been proposed to mitigate the occurrence of harmful words. However, existing CTG methods not only reduce toxicity but also negatively impact several aspects of the language model's generation performance, including topic consistency, grammar, and perplexity. This paper explores the limitations of previous methods and introduces a novel solution in the form of a simple Gated Toxicity Avoidance (GTA) that can be applied to any CTG method. We also evaluate the effectiveness of the proposed GTA by comparing it with state-of-the-art CTG methods across various datasets. Our findings reveal that gated toxicity avoidance efficiently achieves comparable levels of toxicity reduction to the original CTG methods while preserving the generation performance of the language model.
Complex-valued Neural Networks -- Theory and Analysis
Authors: Rayyan Abdalla
Date: 2023-12-11 00:00:00

Complex-valued neural networks (CVNNs) have recently been successful in various pioneering areas which involve wave-typed information and frequency-domain processing. This work addresses different structures and classification of CVNNs. The theory behind complex activation functions, implications related to complex differentiability and special activations for CVNN output layers are presented. The work also discusses CVNN learning and optimization using gradient and non-gradient based algorithms. Complex Backpropagation utilizing complex chain rule is also explained in terms of Wirtinger calculus. Moreover, special modules for building CVNN models, such as complex batch normalization and complex random initialization are also discussed. The work also highlights libraries and software blocks proposed for CVNN implementations and discusses future directions. The objective of this work is to understand the dynamics and most recent developments of CVNNs.
IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions
Authors: Ziheng Zeng, Kellen Tan Cheng, Srihari Venkat Nanniyur, Jianing Zhou, Suma Bhat
Date: 2023-12-11 00:00:00
Tasks: Natural Language Understanding

Idiomatic expression (IE) processing and comprehension have challenged pre-trained language models (PTLMs) because their meanings are non-compositional. Unlike prior works that enable IE comprehension through fine-tuning PTLMs with sentences containing IEs, in this work, we construct IEKG, a commonsense knowledge graph for figurative interpretations of IEs. This extends the established ATOMIC2020 graph, converting PTLMs into knowledge models (KMs) that encode and infer commonsense knowledge related to IE use. Experiments show that various PTLMs can be converted into KMs with IEKG. We verify the quality of IEKG and the ability of the trained KMs with automatic and human evaluation. Through applications in natural language understanding, we show that a PTLM injected with knowledge from IEKG exhibits improved IE comprehension ability and can generalize to IEs unseen during training.
Federated Multilinear Principal Component Analysis with Applications in Prognostics
Authors: Chengyu Zhou, Yuqi Su, Tangbin Xia, Xiaolei Fang
Date: 2023-12-11 00:00:00
Tasks: Dimensionality Reduction, Federated Learning

Multilinear Principal Component Analysis (MPCA) is a widely utilized method for the dimension reduction of tensor data. However, the integration of MPCA into federated learning remains unexplored in existing research. To tackle this gap, this article proposes a Federated Multilinear Principal Component Analysis (FMPCA) method, which enables multiple users to collaboratively reduce the dimension of their tensor data while keeping each user's data local and confidential. The proposed FMPCA method is guaranteed to have the same performance as traditional MPCA. An application of the proposed FMPCA in industrial prognostics is also demonstrated. Simulated data and a real-world data set are used to validate the performance of the proposed method.
Fast Classification of Large Time Series Datasets
Authors: Muhammad Marwan Muhammad Fuad
Date: 2023-12-10 00:00:00
Tasks: Classification, Time Series, Time Series Classification

Time series classification (TSC) is the most import task in time series mining as it has several applications in medicine, meteorology, finance cyber security, and many others. With the ever increasing size of time series datasets, several traditional TSC methods are no longer efficient enough to perform this task on such very large datasets. Yet, most recent papers on TSC focus mainly on accuracy by using methods that apply deep learning, for instance, which require extensive computational resources that cannot be applied efficiently to very large datasets. The method we introduce in this paper focuses on these very large time series datasets with the main objective being efficiency. We achieve this through a simplified representation of the time series. This in turn is enhanced by a distance measure that considers only some of the values of the represented time series. The result of this combination is a very efficient representation method for TSC. This has been tested experimentally against another time series method that is particularly popular for its efficiency. The experiments show that our method is not only 4 times faster, on average, but it is also superior in terms of classification accuracy, as it gives better results on 24 out of the 29 tested time series datasets. .
Stress Management Using Virtual Reality-Based Attention Training
Authors: Rojaina Mahmoud, Mona Mamdouh, Omneya Attallah, Ahmad Al-Kabbany
Date: 2023-12-10 00:00:00
Tasks: EEG, Management

In this research, we are concerned with the applicability of virtual reality-based attention training as a tool for stress management. Mental stress is a worldwide challenge that is still far from being fully managed. This has maintained a remarkable research attention on developing and validating tools for detecting and managing stress. Technology-based tools have been at the heart of these endeavors, including virtual reality (VR) technology. Nevertheless, the potential of VR lies, to a large part, in the nature of the content being consumed through such technology. In this study, we investigate the impact of a special type of content, namely, attention training, on the feasibility of using VR for stress management. On a group of fourteen undergraduate engineering students, we conducted a study in which the participants got exposed twice to a stress inducer while their EEG signals were being recorded. The first iteration involved VR-based attention training before starting the stress task while the second time did not. Using multiple features and various machine learning models, we show that VR-based attention training has consistently resulted in reducing the number of recognized stress instances in the recorded EEG signals. This research gives preliminary insights on adopting VR-based attention training for managing stress, and future studies are required to replicate the results in larger samples.
Speeding up astrochemical reaction networks with autoencoders and neural ODEs
Authors: Immanuel Sulzer, Tobias Buck
Date: 2023-12-10 00:00:00
Tasks: Dimensionality Reduction

In astrophysics, solving complex chemical reaction networks is essential but computationally demanding due to the high dimensionality and stiffness of the ODE systems. Traditional approaches for reducing computational load are often specialized to specific chemical networks and require expert knowledge. This paper introduces a machine learning-based solution employing autoencoders for dimensionality reduction and a latent space neural ODE solver to accelerate astrochemical reaction network computations. Additionally, we propose a cost-effective latent space linear function solver as an alternative to neural ODEs. These methods are assessed on a dataset comprising 29 chemical species and 224 reactions. Our findings demonstrate that the neural ODE achieves a 55x speedup over the baseline model while maintaining significantly higher accuracy by up to two orders of magnitude reduction in relative error. Furthermore, the linear latent model enhances accuracy and achieves a speedup of up to 4000x compared to standard methods.
A Note on the Convergence of Denoising Diffusion Probabilistic Models
Authors: Sokhna Diarra Mbacke, Omar Rivasplata
Date: 2023-12-10 00:00:00
Tasks: Denoising

Diffusion models are one of the most important families of deep generative models. In this note, we derive a quantitative upper bound on the Wasserstein distance between the data-generating distribution and the distribution learned by a diffusion model. Unlike previous works in this field, our result does not make assumptions on the learned score function. Moreover, our bound holds for arbitrary data-generating distributions on bounded instance spaces, even those without a density w.r.t. the Lebesgue measure, and the upper bound does not suffer from exponential dependencies. Our main result builds upon the recent work of Mbacke et al. (2023) and our proofs are elementary.
Neural Architecture Codesign for Fast Bragg Peak Analysis
Authors: Luke McDermott, Jason Weitz, Dmitri Demler, Daniel Cummings, Nhan Tran, Javier Duarte
Date: 2023-12-10 00:00:00
Tasks: AutoML, Model Compression, Network Pruning, Neural Architecture Search, Quantization

We develop an automated pipeline to streamline neural architecture codesign for fast, real-time Bragg peak analysis in high-energy diffraction microscopy. Traditional approaches, notably pseudo-Voigt fitting, demand significant computational resources, prompting interest in deep learning models for more efficient solutions. Our method employs neural architecture search and AutoML to enhance these models, including hardware costs, leading to the discovery of more hardware-efficient neural architectures. Our results match the performance, while achieving a 13$\times$ reduction in bit operations compared to the previous state-of-the-art. We show further speedup through model compression techniques such as quantization-aware-training and neural network pruning. Additionally, our hierarchical search space provides greater flexibility in optimization, which can easily extend to other tasks and domains.
Learning the Causal Structure of Networked Dynamical Systems under Latent Nodes and Structured Noise
Authors: Augusto Santos, Diogo Rente, Rui Seabra, José M. F. Moura
Date: 2023-12-10 00:00:00
Tasks: Causal Inference, Time Series

This paper considers learning the hidden causal network of a linear networked dynamical system (NDS) from the time series data at some of its nodes -- partial observability. The dynamics of the NDS are driven by colored noise that generates spurious associations across pairs of nodes, rendering the problem much harder. To address the challenge of noise correlation and partial observability, we assign to each pair of nodes a feature vector computed from the time series data of observed nodes. The feature embedding is engineered to yield structural consistency: there exists an affine hyperplane that consistently partitions the set of features, separating the feature vectors corresponding to connected pairs of nodes from those corresponding to disconnected pairs. The causal inference problem is thus addressed via clustering the designed features. We demonstrate with simple baseline supervised methods the competitive performance of the proposed causal inference mechanism under broad connectivity regimes and noise correlation levels, including a real world network. Further, we devise novel technical guarantees of structural consistency for linear NDS under the considered regime.
TransGlow: Attention-augmented Transduction model based on Graph Neural Networks for Water Flow Forecasting
Authors: Naghmeh Shafiee Roudbari, Charalambos Poullis, Zachary Patterson, Ursula Eicker
Date: 2023-12-10 00:00:00
Tasks: Decoder, Graph Learning

The hydrometric prediction of water quantity is useful for a variety of applications, including water management, flood forecasting, and flood control. However, the task is difficult due to the dynamic nature and limited data of water systems. Highly interconnected water systems can significantly affect hydrometric forecasting. Consequently, it is crucial to develop models that represent the relationships between other system components. In recent years, numerous hydrological applications have been studied, including streamflow prediction, flood forecasting, and water quality prediction. Existing methods are unable to model the influence of adjacent regions between pairs of variables. In this paper, we propose a spatiotemporal forecasting model that augments the hidden state in Graph Convolution Recurrent Neural Network (GCRN) encoder-decoder using an efficient version of the attention mechanism. The attention layer allows the decoder to access different parts of the input sequence selectively. Since water systems are interconnected and the connectivity information between the stations is implicit, the proposed model leverages a graph learning module to extract a sparse graph adjacency matrix adaptively based on the data. Spatiotemporal forecasting relies on historical data. In some regions, however, historical data may be limited or incomplete, making it difficult to accurately predict future water conditions. Further, we present a new benchmark dataset of water flow from a network of Canadian stations on rivers, streams, and lakes. Experimental results demonstrate that our proposed model TransGlow significantly outperforms baseline methods by a wide margin.
VAE-IF: Deep feature extraction with averaging for fully unsupervised artifact detection in routinely acquired ICU time-series
Authors: Hollan Haule, Ian Piper, Patricia Jones, Chen Qin, Tsz-Yan Milly Lo, Javier Escudero
Date: 2023-12-10 00:00:00
Tasks: Artifact Detection, Time Series

Artifacts are a common problem in physiological time series collected from intensive care units (ICU) and other settings. They affect the quality and reliability of clinical research and patient care. Manual annotation of artifacts is costly and time-consuming, rendering it impractical. Automated methods are desired. Here, we propose a novel fully unsupervised approach to detect artifacts in clinical-standard, minute-by-minute resolution ICU data without any prior labeling or signal-specific knowledge. Our approach combines a variational autoencoder (VAE) and an isolation forest (IF) into a hybrid model to learn features and identify anomalies in different types of vital signs, such as blood pressure, heart rate, and intracranial pressure. We evaluate our approach on a real-world ICU dataset and compare it with supervised benchmark models based on long short-term memory (LSTM) and XGBoost and statistical methods such as ARIMA. We show that our unsupervised approach achieves comparable sensitivity to fully supervised methods and generalizes well to an external dataset. We also visualize the latent space learned by the VAE and demonstrate its ability to disentangle clean and noisy samples. Our approach offers a promising solution for cleaning ICU data in clinical research and practice without the need for any labels whatsoever.
Learning Differentiable Particle Filter on the Fly
Authors: Jiaxi Li, Xiongjie Chen, Yunpeng Li
Date: 2023-12-10 00:00:00
Tasks: Bayesian Inference, Object Tracking, parameter estimation, Sequential Bayesian Inference, State Space Models

Differentiable particle filters are an emerging class of sequential Bayesian inference techniques that use neural networks to construct components in state space models. Existing approaches are mostly based on offline supervised training strategies. This leads to the delay of the model deployment and the obtained filters are susceptible to distribution shift of test-time data. In this paper, we propose an online learning framework for differentiable particle filters so that model parameters can be updated as data arrive. The technical constraint is that there is no known ground truth state information in the online inference setting. We address this by adopting an unsupervised loss to construct the online model updating procedure, which involves a sequence of filtering operations for online maximum likelihood-based parameter estimation. We empirically evaluate the effectiveness of the proposed method, and compare it with supervised learning methods in simulation settings including a multivariate linear Gaussian state-space model and a simulated object tracking experiment.
Ensemble Kalman Filtering Meets Gaussian Process SSM for Non-Mean-Field and Online Inference
Authors: Zhidi Lin, Yiyong Sun, Feng Yin, Alexandre Hoang Thiéry
Date: 2023-12-10 00:00:00
Tasks: State Space Models, Variational Inference

The Gaussian process state-space models (GPSSMs) represent a versatile class of data-driven nonlinear dynamical system models. However, the presence of numerous latent variables in GPSSM incurs unresolved issues for existing variational inference approaches, particularly under the more realistic non-mean-field (NMF) assumption, including extensive training effort, compromised inference accuracy, and infeasibility for online applications, among others. In this paper, we tackle these challenges by incorporating the ensemble Kalman filter (EnKF), a well-established model-based filtering technique, into the NMF variational inference framework to approximate the posterior distribution of the latent states. This novel marriage between EnKF and GPSSM not only eliminates the need for extensive parameterization in learning variational distributions, but also enables an interpretable, closed-form approximation of the evidence lower bound (ELBO). Moreover, owing to the streamlined parameterization via the EnKF, the new GPSSM model can be easily accommodated in online learning applications. We demonstrate that the resulting EnKF-aided online algorithm embodies a principled objective function by ensuring data-fitting accuracy while incorporating model regularizations to mitigate overfitting. We also provide detailed analysis and fresh insights for the proposed algorithms. Comprehensive evaluation across diverse real and synthetic datasets corroborates the superior learning and inference performance of our EnKF-aided variational inference algorithms compared to existing methods.
A conservative hybrid physics-informed neural network method for Maxwell-Ampère-Nernst-Planck equations
Authors: Cheng Chang, Zhouping Xin, Tieyong Zeng
Date: 2023-12-10 00:00:00

Maxwell-Amp\`{e}re-Nernst-Planck (MANP) equations were recently proposed to model the dynamics of charged particles. In this study, we enhance a numerical algorithm of this system with deep learning tools. The proposed hybrid algorithm provides an automated means to determine a proper approximation for the dummy variables, which can otherwise only be obtained through massive numerical tests. In addition, the original method is validated for 2-dimensional problems. However, when the spatial dimension is one, the original curl-free relaxation component is inapplicable, and the approximation formula for dummy variables, which works well in a 2-dimensional scenario, fails to provide a reasonable output in the 1-dimensional case. The proposed method can be readily generalised to cases with one spatial dimension. Experiments show numerical stability and good convergence to the steady-state solution obtained from Poisson-Boltzmann type equations in the 1-dimensional case. The experiments conducted in the 2-dimensional case indicate that the proposed method preserves the conservation properties.