pwc-archive/papers-with-abstracts huggingface dataset:
Parameter-Efficient Fine-Tuning via Circular Convolution
Authors: Aochuan Chen, Jiashun Cheng, Zijing Liu, Ziqi Gao, Fugee Tsung, Yu Li, Jia Li
Date: 2024-07-27 00:00:00
Tasks: parameter-efficient fine-tuning

Low-Rank Adaptation (LoRA) has gained popularity for fine-tuning large foundation models, leveraging low-rank matrices $\mathbf{A}$ and $\mathbf{B}$ to represent weight changes (i.e., $\Delta \mathbf{W} = \mathbf{B} \mathbf{A}$). This method reduces trainable parameters and mitigates heavy memory consumption associated with full delta matrices by sequentially multiplying $\mathbf{A}$ and $\mathbf{B}$ with the activation. Despite its success, the intrinsic low-rank characteristic may limit its performance. Although several variants have been proposed to address this issue, they often overlook the crucial computational and memory efficiency brought by LoRA. In this paper, we propose Circular Convolution Adaptation (C$^3$A), which not only achieves high-rank adaptation with enhanced performance but also excels in both computational power and memory utilization. Extensive experiments demonstrate that C$^3$A consistently outperforms LoRA and its variants across various fine-tuning tasks.
An image segmentation algorithm for measurement of flotation froth bubble size distributions
Authors: A. Jahedsaravani, M. Massinaei, M.H. Marhaban
Date: 2017-07-23 00:00:00
Proceeding: Elsevier 2017 7
Tasks: image-classification, Image Classification, Image Segmentation, Semantic Segmentation

The bubble size distribution at the froth surface of a flotation cell is closely related to the process condi-
tion and performance. The flotation performance can be reasonably predicted through continuous mea-
suring the bubble size distribution by a machine vision system. In this work a new watershed algorithm
based on whole and sub-image classification techniques is introduced and successfully validated by sev-
eral laboratory and industrial scale froth images taken under different process conditions. The results
indicate that the developed algorithms, in particular the sub-image classification based segmentation
algorithm, can accurately and reliably identify the individual small and large bubbles in the actual froth
images, which is often problematic.
Friends in Unexpected Places: Enhancing Local Fairness in Federated Learning through Clustering
Authors: Yifan Yang, Ali Payani, Parinaz Naghizadeh
Date: 2024-07-27 00:00:00
Tasks: Clustering, Fairness, Federated Learning, Personalized Federated Learning

Federated Learning (FL) has been a pivotal paradigm for collaborative training of machine learning models across distributed datasets. In heterogeneous settings, it has been observed that a single shared FL model can lead to low local accuracy, motivating personalized FL algorithms. In parallel, fair FL algorithms have been proposed to enforce group fairness on the global models. Again, in heterogeneous settings, global and local fairness do not necessarily align, motivating the recent literature on locally fair FL. In this paper, we propose new FL algorithms for heterogeneous settings, spanning the space between personalized and locally fair FL. Building on existing clustering-based personalized FL methods, we incorporate a new fairness metric into cluster assignment, enabling a tunable balance between local accuracy and fairness. Our methods match or exceed the performance of existing locally fair FL approaches, without explicit fairness intervention. We further demonstrate (numerically and analytically) that personalization alone can improve local fairness and that our methods exploit this alignment when present.
Accounting for plasticity: An extension of inelastic Constitutive Artificial Neural Networks
Authors: Birte Boes, Jaan-Willem Simon, Hagen Holthusen
Date: 2024-07-27 00:00:00

The class of Constitutive Artificial Neural Networks (CANNs) represents a new approach of neural networks in the field of constitutive modeling. So far, CANNs have proven to be a powerful tool in predicting elastic and inelastic material behavior. However, the specification of inelastic constitutive artificial neural networks (iCANNs) to capture plasticity remains to be discussed. We present the extension and application of an iCANN to the inelastic phenomena of plasticity. This includes the prediction of a formulation for the elastic and plastic Helmholtz free energies, the inelastic flow rule, and the yield condition that defines the onset of plasticity. Thus, we learn four feed-forward networks in combination with a recurrent neural network and use the second Piola-Kirchhoff stress measure for training. The presented formulation captures both, associative and non-associative plasticity. In addition, the formulation includes kinematic hardening effects by introducing the plastic Helmholtz free energy. This opens the range of application to a wider class of materials. The capabilities of the presented framework are demonstrated by training on artificially generated data of models for perfect plasticity of von-Mises type, tension-compression asymmetry, and kinematic hardening. We observe already satisfactory results for training on one load case only while extremely precise agreement is found for an increase in load cases. In addition, the performance of the specified iCANN was validated using experimental data of X10CrMoVNb9-1 steel. Training has been performed on both, uniaxial tension and cyclic loading, separately and the predicted results are then validated on the opposing set. The results underline that the autonomously discovered material model is capable to describe and predict the underlying experimental data.
Deep Learning Based Crime Prediction Models: Experiments and Analysis
Authors: Rittik Basak Utsha, Muhtasim Noor Alif, Yeasir Rayhan, Tanzima Hashem, Mohammad Eunus Ali
Date: 2024-07-27 00:00:00
Tasks: Crime Prediction, Deep Learning, Prediction

Crime prediction is a widely studied research problem due to its importance in ensuring safety of city dwellers. Starting from statistical and classical machine learning based crime prediction methods, in recent years researchers have focused on exploiting deep learning based models for crime prediction. Deep learning based crime prediction models use complex architectures to capture the latent features in the crime data, and outperform the statistical and classical machine learning based crime prediction methods. However, there is a significant research gap in existing research on the applicability of different models in different real-life scenarios as no longitudinal study exists comparing all these approaches in a unified setting. In this paper, we conduct a comprehensive experimental evaluation of all major state-of-the-art deep learning based crime prediction models. Our evaluation provides several key insights on the pros and cons of these models, which enables us to select the most suitable models for different application scenarios. Based on the findings, we further recommend certain design practices that should be taken into account while building future deep learning based crime prediction models.
WindsorML: High-Fidelity Computational Fluid Dynamics Dataset For Automotive Aerodynamics
Authors: Neil Ashton, Jordan B. Angel, Aditya S. Ghate, Gaetan K. W. Kenway, Man Long Wong, Cetin Kiris, Astrid Walle, Danielle C. Maddix, Gary Page
Date: 2024-07-27 00:00:00
Tasks: GPU

This paper presents a new open-source high-fidelity dataset for Machine Learning (ML) containing 355 geometric variants of the Windsor body, to help the development and testing of ML surrogate models for external automotive aerodynamics. Each Computational Fluid Dynamics (CFD) simulation was run with a GPU-native high-fidelity Wall-Modeled Large-Eddy Simulations (WMLES) using a Cartesian immersed-boundary method using more than 280M cells to ensure the greatest possible accuracy. The dataset contains geometry variants that exhibits a wide range of flow characteristics that are representative of those observed on road-cars. The dataset itself contains the 3D time-averaged volume & boundary data as well as the geometry and force & moment coefficients. This paper discusses the validation of the underlying CFD methods as well as contents and structure of the dataset. To the authors knowledge, this represents the first, large-scale high-fidelity CFD dataset for the Windsor body with a permissive open-source license (CC-BY-SA).
Can Modifying Data Address Graph Domain Adaptation?
Authors: Renhong Huang, Jiarong Xu, Xin Jiang, Ruichuan An, Yang Yang
Date: 2024-07-27 00:00:00
Tasks: Domain Adaptation, GRAPH DOMAIN ADAPTATION, Transfer Learning

Graph neural networks (GNNs) have demonstrated remarkable success in numerous graph analytical tasks. Yet, their effectiveness is often compromised in real-world scenarios due to distribution shifts, limiting their capacity for knowledge transfer across changing environments or domains. Recently, Unsupervised Graph Domain Adaptation (UGDA) has been introduced to resolve this issue. UGDA aims to facilitate knowledge transfer from a labeled source graph to an unlabeled target graph. Current UGDA efforts primarily focus on model-centric methods, such as employing domain invariant learning strategies and designing model architectures. However, our critical examination reveals the limitations inherent to these model-centric methods, while a data-centric method allowed to modify the source graph provably demonstrates considerable potential. This insight motivates us to explore UGDA from a data-centric perspective. By revisiting the theoretical generalization bound for UGDA, we identify two data-centric principles for UGDA: alignment principle and rescaling principle. Guided by these principles, we propose GraphAlign, a novel UGDA method that generates a small yet transferable graph. By exclusively training a GNN on this new graph with classic Empirical Risk Minimization (ERM), GraphAlign attains exceptional performance on the target graph. Extensive experiments under various transfer scenarios demonstrate the GraphAlign outperforms the best baselines by an average of 2.16%, training on the generated graph as small as 0.25~1% of the original training graph.
Bayesian meta learning for trustworthy uncertainty quantification
Authors: Zhenyuan Yuan, Thinh T. Doan
Date: 2024-07-27 00:00:00
Tasks: Meta-Learning, regression, Uncertainty Quantification

We consider the problem of Bayesian regression with trustworthy uncertainty quantification. We define that the uncertainty quantification is trustworthy if the ground truth can be captured by intervals dependent on the predictive distributions with a pre-specified probability. Furthermore, we propose, Trust-Bayes, a novel optimization framework for Bayesian meta learning which is cognizant of trustworthy uncertainty quantification without explicit assumptions on the prior model/distribution of the functions. We characterize the lower bounds of the probabilities of the ground truth being captured by the specified intervals and analyze the sample complexity with respect to the feasible probability for trustworthy uncertainty quantification. Monte Carlo simulation of a case study using Gaussian process regression is conducted for verification and comparison with the Meta-prior algorithm.
On Using Secure Aggregation in Differentially Private Federated Learning with Multiple Local Steps
Authors: Mikko A. Heikkilä
Date: 2024-07-27 00:00:00
Tasks: Federated Learning

Federated learning is a distributed learning setting where the main aim is to train machine learning models without having to share raw data but only what is required for learning. To guarantee training data privacy and high-utility models, differential privacy and secure aggregation techniques are often combined with federated learning. However, with fine-grained protection granularities, e.g., with the common sample-level protection, the currently existing techniques generally require the parties to communicate for each local optimization step, if they want to fully benefit from the secure aggregation in terms of the resulting formal privacy guarantees. In this paper, we show how a simple new analysis allows the parties to perform multiple local optimization steps while still benefiting from using secure aggregation. We show that our analysis enables higher utility models with guaranteed privacy protection under limited number of communication rounds.
Towards Robust Few-shot Class Incremental Learning in Audio Classification using Contrastive Representation
Authors: Riyansha Singh, Parinita Nema, Vinod K Kurmi
Date: 2024-07-27 00:00:00
Tasks: Audio Classification, class-incremental learning, Class Incremental Learning, Contrastive Learning, Few-Shot Class-Incremental Learning, Incremental Learning

In machine learning applications, gradual data ingress is common, especially in audio processing where incremental learning is vital for real-time analytics. Few-shot class-incremental learning addresses challenges arising from limited incoming data. Existing methods often integrate additional trainable components or rely on a fixed embedding extractor post-training on base sessions to mitigate concerns related to catastrophic forgetting and the dangers of model overfitting. However, using cross-entropy loss alone during base session training is suboptimal for audio data. To address this, we propose incorporating supervised contrastive learning to refine the representation space, enhancing discriminative power and leading to better generalization since it facilitates seamless integration of incremental classes, upon arrival. Experimental results on NSynth and LibriSpeech datasets with 100 classes, as well as ESC dataset with 50 and 10 classes, demonstrate state-of-the-art performance.
Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications
Authors: Till Speicher, Mohammad Aflah Khan, Qinyuan Wu, Vedant Nanda, Soumi Das, Bishwamittra Ghosh, Krishna P. Gummadi, Evimaria Terzi
Date: 2024-07-27 00:00:00
Tasks: In-Context Learning

Understanding whether and to what extent large language models (LLMs) have memorised training data has important implications for the reliability of their output and the privacy of their training data. In order to cleanly measure and disentangle memorisation from other phenomena (e.g. in-context learning), we create an experimental framework that is based on repeatedly exposing LLMs to random strings. Our framework allows us to better understand the dynamics, i.e., the behaviour of the model, when repeatedly exposing it to random strings. Using our framework, we make several striking observations: (a) we find consistent phases of the dynamics across families of models (Pythia, Phi and Llama2), (b) we identify factors that make some strings easier to memorise than others, and (c) we identify the role of local prefixes and global context in memorisation. We also show that sequential exposition to different random strings has a significant effect on memorisation. Our results, often surprising, have significant downstream implications in the study and usage of LLMs.
Comprehensive Survey of Complex-Valued Neural Networks: Insights into Backpropagation and Activation Functions
Authors: M. M. Hammad
Date: 2024-07-27 00:00:00
Tasks: Survey

Artificial neural networks (ANNs), particularly those employing deep learning models, have found widespread application in fields such as computer vision, signal processing, and wireless communications, where complex numbers are crucial. Despite the prevailing use of real-number implementations in current ANN frameworks, there is a growing interest in developing ANNs that utilize complex numbers. This paper presents a comprehensive survey of recent advancements in complex-valued neural networks (CVNNs), focusing on their activation functions (AFs) and learning algorithms. We delve into the extension of the backpropagation algorithm to the complex domain, which enables the training of neural networks with complex-valued inputs, weights, AFs, and outputs. This survey considers three complex backpropagation algorithms: the complex derivative approach, the partial derivatives approach, and algorithms incorporating the Cauchy-Riemann equations. A significant challenge in CVNN design is the identification of suitable nonlinear Complex Valued Activation Functions (CVAFs), due to the conflict between boundedness and differentiability over the entire complex plane as stated by Liouville theorem. We examine both fully complex AFs, which strive for boundedness and differentiability, and split AFs, which offer a practical compromise despite not preserving analyticity. This review provides an in-depth analysis of various CVAFs essential for constructing effective CVNNs. Moreover, this survey not only offers a comprehensive overview of the current state of CVNNs but also contributes to ongoing research and development by introducing a new set of CVAFs (fully complex, split and complex amplitude-phase AFs).
Nonlinear spectral analysis extracts harmonics from land-atmosphere fluxes
Authors: Leonard Schulz, Jürgen Vollmer, Miguel D. Mahecha, Karin Mora
Date: 2024-07-27 00:00:00

Understanding the dynamics of the land-atmosphere exchange of CO$_2$ is key to advance our predictive capacities of the coupled climate-carbon feedback system. In essence, the net vegetation flux is the difference of the uptake of CO$_2$ via photosynthesis and the release of CO$_2$ via respiration, while the system is driven by periodic processes at different time-scales. The complexity of the underlying dynamics poses challenges to classical decomposition methods focused on maximizing data variance, such as singular spectrum analysis. Here, we explore whether nonlinear data-driven methods can better separate periodic patterns and their harmonics from noise and stochastic variability. We find that Nonlinear Laplacian Spectral Analysis (NLSA) outperforms the linear method and detects multiple relevant harmonics. However, these harmonics are not detected in the presence of substantial measurement irregularities. In summary, the NLSA approach can be used to both extract the seasonal cycle more accurately than linear methods, but likewise detect irregular signals resulting from irregular land-atmosphere interactions or measurement failures. Improving the detection capabilities of time-series decomposition is essential for improving land-atmosphere interactions models that should operate accurately on any time scale.
Ordered Momentum for Asynchronous SGD
Authors: Chang-Wei Shi, Yi-Rui Yang, Wu-Jun Li
Date: 2024-07-27 00:00:00

Distributed learning is essential for training large-scale deep models. Asynchronous SGD (ASGD) and its variants are commonly used distributed learning methods, particularly in scenarios where the computing capabilities of workers in the cluster are heterogeneous. Momentum has been acknowledged for its benefits in both optimization and generalization in deep model training. However, existing works have found that naively incorporating momentum into ASGD can impede the convergence. In this paper, we propose a novel method called ordered momentum (OrMo) for ASGD. In OrMo, momentum is incorporated into ASGD by organizing the gradients in order based on their iteration indexes. We theoretically prove the convergence of OrMo with both constant and delay-adaptive learning rates for non-convex problems. To the best of our knowledge, this is the first work to establish the convergence analysis of ASGD with momentum without dependence on the maximum delay. Empirical results demonstrate that OrMo can achieve better convergence performance compared with ASGD and other asynchronous methods with momentum.
Alleviating Over-Smoothing via Aggregation over Compact Manifolds
Authors: Dongzhuoran Zhou, Hui Yang, Bo Xiong, Yue Ma, Evgeny Kharlamov
Date: 2024-07-27 00:00:00

Graph neural networks (GNNs) have achieved significant success in various applications. Most GNNs learn the node features with information aggregation of its neighbors and feature transformation in each layer. However, the node features become indistinguishable after many layers, leading to performance deterioration: a significant limitation known as over-smoothing. Past work adopted various techniques for addressing this issue, such as normalization and skip-connection of layer-wise output. After the study, we found that the information aggregations in existing work are all contracted aggregations, with the intrinsic property that features will inevitably converge to the same single point after many layers. To this end, we propose the aggregation over compacted manifolds method (ACM) that replaces the existing information aggregation with aggregation over compact manifolds, a special type of manifold, which avoids contracted aggregations. In this work, we theoretically analyze contracted aggregation and its properties. We also provide an extensive empirical evaluation that shows ACM can effectively alleviate over-smoothing and outperforms the state-of-the-art. The code can be found in https://github.com/DongzhuoranZhou/ACM.git.
Long Range Switching Time Series Prediction via State Space Model
Authors: Jiaming Zhang, Yang Ding, Yunfeng Gao
Date: 2024-07-27 00:00:00
Tasks: Change Point Detection, Time Series, Time Series Prediction

In this study, we delve into the Structured State Space Model (S4), Change Point Detection methodologies, and the Switching Non-linear Dynamics System (SNLDS). Our central proposition is an enhanced inference technique and long-range dependency method for SNLDS. The cornerstone of our approach is the fusion of S4 and SNLDS, leveraging the strengths of both models to effectively address the intricacies of long-range dependencies in switching time series. Through rigorous testing, we demonstrate that our proposed methodology adeptly segments and reproduces long-range dependencies in both the 1-D Lorenz dataset and the 2-D bouncing ball dataset. Notably, our integrated approach outperforms the standalone SNLDS in these tasks.
A simulation study of cluster search algorithms in data set generated by Gaussian mixture models
Authors: Ryosuke Motegi, Yoichi Seki
Date: 2024-07-27 00:00:00

Determining the number of clusters is a fundamental issue in data clustering. Several algorithms have been proposed, including centroid-based algorithms using the Euclidean distance and model-based algorithms using a mixture of probability distributions. Among these, greedy algorithms for searching the number of clusters by repeatedly splitting or merging clusters have advantages in terms of computation time for problems with large sample sizes. However, studies comparing these methods in systematic evaluation experiments still need to be included. This study examines centroid- and model-based cluster search algorithms in various cases that Gaussian mixture models (GMMs) can generate. The cases are generated by combining five factors: dimensionality, sample size, the number of clusters, cluster overlap, and covariance type. The results show that some cluster-splitting criteria based on Euclidean distance make unreasonable decisions when clusters overlap. The results also show that model-based algorithms are insensitive to covariance type and cluster overlap compared to the centroid-based method if the sample size is sufficient. Our cluster search implementation codes are available at https://github.com/lipryou/searchClustK