pwc-archive/papers-with-abstracts huggingface dataset:
Weight for Robustness: A Comprehensive Approach towards Optimal Fault-Tolerant Asynchronous ML
Authors: Tehila Dahan, Kfir Y. Levy
Date: 2025-01-16 00:00:00

We address the challenges of Byzantine-robust training in asynchronous distributed machine learning systems, aiming to enhance efficiency amid massive parallelization and heterogeneous computing resources. Asynchronous systems, marked by independently operating workers and intermittent updates, uniquely struggle with maintaining integrity against Byzantine failures, which encompass malicious or erroneous actions that disrupt learning. The inherent delays in such settings not only introduce additional bias to the system but also obscure the disruptions caused by Byzantine faults. To tackle these issues, we adapt the Byzantine framework to asynchronous dynamics by introducing a novel weighted robust aggregation framework. This allows for the extension of robust aggregators and a recent meta-aggregator to their weighted versions, mitigating the effects of delayed updates. By further incorporating a recent variance-reduction technique, we achieve an optimal convergence rate for the first time in an asynchronous Byzantine environment. Our methodology is rigorously validated through empirical and theoretical analysis, demonstrating its effectiveness in enhancing fault tolerance and optimizing performance in asynchronous ML systems.
ARMAX identification of low rank graphical models
Authors: Wenqi Cao, Aming Li
Date: 2025-01-16 00:00:00
Tasks: parameter estimation

In large-scale systems, complex internal relationships are often present. Such interconnected systems can be effectively described by low rank stochastic processes. When identifying a predictive model of low rank processes from sampling data, the rank-deficient property of spectral densities is often obscured by the inevitable measurement noise in practice. However, existing low rank identification approaches often did not take noise into explicit consideration, leading to non-negligible inaccuracies even under weak noise. In this paper, we address the identification issue of low rank processes under measurement noise. We find that the noisy measurement model admits a sparse plus low rank structure in latent-variable graphical models. Specifically, we first decompose the problem into a maximum entropy covariance extension problem, and a low rank graphical estimation problem based on an autoregressive moving-average with exogenous input (ARMAX) model. To identify the ARMAX low rank graphical models, we propose an estimation approach based on maximum likelihood. The identifiability and consistency of this approach are proven under certain conditions. Simulation results confirm the reliable performance of the entire algorithm in both the parameter estimation and noisy data filtering.
EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning
Authors: Siddharth Aravindan, Dixant Mittal, Wee Sun Lee
Date: 2025-01-16 00:00:00
Tasks: Model-based Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Thompson Sampling

Posterior Sampling for Reinforcement Learning (PSRL) is a well-known algorithm that augments model-based reinforcement learning (MBRL) algorithms with Thompson sampling. PSRL maintains posterior distributions of the environment transition dynamics and the reward function, which are intractable for tasks with high-dimensional state and action spaces. Recent works show that dropout, used in conjunction with neural networks, induces variational distributions that can approximate these posteriors. In this paper, we propose Event-based Variational Distributions for Exploration (EVaDE), which are variational distributions that are useful for MBRL, especially when the underlying domain is object-based. We leverage the general domain knowledge of object-based domains to design three types of event-based convolutional layers to direct exploration. These layers rely on Gaussian dropouts and are inserted between the layers of the deep neural network model to help facilitate variational Thompson sampling. We empirically show the effectiveness of EVaDE-equipped Simulated Policy Learning (EVaDE-SimPLe) on the 100K Atari game suite.
Adversarial-Ensemble Kolmogorov Arnold Networks for Enhancing Indoor Wi-Fi Positioning: A Defensive Approach Against Spoofing and Signal Manipulation Attacks
Authors: Mitul Goswami, Romit Chatterjee, Somnath Mahato, Prasant Kumar Pattnaik
Date: 2025-01-16 00:00:00
Tasks: Kolmogorov-Arnold Networks

The research presents a study on enhancing the robustness of Wi-Fi-based indoor positioning systems against adversarial attacks. The goal is to improve the positioning accuracy and resilience of these systems under two attack scenarios: Wi-Fi Spoofing and Signal Strength Manipulation. Three models are developed and evaluated: a baseline model (M_Base), an adversarially trained robust model (M_Rob), and an ensemble model (M_Ens). All models utilize a Kolmogorov-Arnold Network (KAN) architecture. The robust model is trained with adversarially perturbed data, while the ensemble model combines predictions from both the base and robust models. Experimental results show that the robust model reduces positioning error by approximately 10% compared to the baseline, achieving 2.03 meters error under Wi-Fi spoofing and 2.00 meters under signal strength manipulation. The ensemble model further outperforms with errors of 2.01 meters and 1.975 meters for the respective attack types. This analysis highlights the effectiveness of adversarial training techniques in mitigating attack impacts. The findings underscore the importance of considering adversarial scenarios in developing indoor positioning systems, as improved resilience can significantly enhance the accuracy and reliability of such systems in mission-critical environments.
Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation -- Extended Version
Authors: Muhammad Rajabinasab, Anton D. Lautrup, Arthur Zimek
Date: 2025-01-16 00:00:00
Tasks: feature selection

Measuring inter-dataset similarity is an important task in machine learning and data mining with various use cases and applications. Existing methods for measuring inter-dataset similarity are computationally expensive, limited, or sensitive to different entities and non-trivial choices for parameters. They also lack a holistic perspective on the entire dataset. In this paper, we propose two novel metrics for measuring inter-dataset similarity. We discuss the mathematical foundation and the theoretical basis of our proposed metrics. We demonstrate the effectiveness of the proposed metrics by investigating two applications in the evaluation of synthetic data and in the evaluation of feature selection methods. The theoretical and empirical studies conducted in this paper illustrate the effectiveness of the proposed metrics.
Atleus: Accelerating Transformers on the Edge Enabled by 3D Heterogeneous Manycore Architectures
Authors: Pratyush Dhingra, Janardhan Rao Doppa, Partha Pratim Pande
Date: 2025-01-16 00:00:00
Tasks: Model Compression, Quantization

Transformer architectures have become the standard neural network model for various machine learning applications including natural language processing and computer vision. However, the compute and memory requirements introduced by transformer models make them challenging to adopt for edge applications. Furthermore, fine-tuning pre-trained transformers (e.g., foundation models) is a common task to enhance the model's predictive performance on specific tasks/applications. Existing transformer accelerators are oblivious to complexities introduced by fine-tuning. In this paper, we propose the design of a three-dimensional (3D) heterogeneous architecture referred to as Atleus that incorporates heterogeneous computing resources specifically optimized to accelerate transformer models for the dual purposes of fine-tuning and inference. Specifically, Atleus utilizes non-volatile memory and systolic array for accelerating transformer computational kernels using an integrated 3D platform. Moreover, we design a suitable NoC to achieve high performance and energy efficiency. Finally, Atleus adopts an effective quantization scheme to support model compression. Experimental results demonstrate that Atleus outperforms existing state-of-the-art by up to 56x and 64.5x in terms of performance and energy efficiency respectively
Towards Spectral Convergence of Locally Linear Embedding on Manifolds with Boundary
Authors: Andrew Lyons
Date: 2025-01-16 00:00:00

We study the eigenvalues and eigenfunctions of a differential operator that governs the asymptotic behavior of the unsupervised learning algorithm known as Locally Linear Embedding when a large data set is sampled from an interval or disc. In particular, the differential operator is of second order, mixed-type, and degenerates near the boundary. We show that a natural regularity condition on the eigenfunctions imposes a consistent boundary condition and use the Frobenius method to estimate pointwise behavior. We then determine the limiting sequence of eigenvalues analytically and compare them to numerical predictions. Finally, we propose a variational framework for determining eigenvalues on other compact manifolds.
Overshoot: Taking advantage of future gradients in momentum-based stochastic optimization
Authors: Jakub Kopal, Michal Gregor, Santiago de Leon-Martinez, Jakub Simko
Date: 2025-01-16 00:00:00
Tasks: Stochastic Optimization

Overshoot is a novel, momentum-based stochastic gradient descent optimization method designed to enhance performance beyond standard and Nesterov's momentum. In conventional momentum methods, gradients from previous steps are aggregated with the gradient at current model weights before taking a step and updating the model. Rather than calculating gradient at the current model weights, Overshoot calculates the gradient at model weights shifted in the direction of the current momentum. This sacrifices the immediate benefit of using the gradient w.r.t. the exact model weights now, in favor of evaluating at a point, which will likely be more relevant for future updates. We show that incorporating this principle into momentum-based optimizers (SGD with momentum and Adam) results in faster convergence (saving on average at least 15% of steps). Overshoot consistently outperforms both standard and Nesterov's momentum across a wide range of tasks and integrates into popular momentum-based optimizers with zero memory and small computational overhead.
Intra-day Solar and Power Forecast for Optimization of Intraday Market Participation
Authors: Nelson Salazar-Pena, Adolfo Palma-Vergara, Mateo Montes-Vera, Maria Alejandra Vargas-Torres, Rodrigo Hernandez-Vanegas, Maria Amador, Boris Rojas, Adriana Salinas, Andres Velasco, Alejandra Tabares, Andres Gonzalez-Mancera
Date: 2025-01-16 00:00:00

The prediction of solar irradiance enhances reliability in photovoltaic (PV) solar plant generation and grid integration. In Colombia, PV plants face penalties if energy production deviates beyond governmental thresholds from intraday market offers. This research employs Long Short-Term Memory (LSTM) and Bidirectional-LSTM (Bi-LSTM) models, utilizing meteorological data from a PV plant in El Paso, Cesar, Colombia, to predict solar irradiance with a 6-hour horizon and 10-minute resolution. While Bi-LSTM showed superior performance, the LSTM model achieved comparable results with significantly reduced training time (6 hours versus 18 hours), making it computationally advantageous. The LSTM predictions were averaged to create an hourly resolution model, evaluated using Mean Absolute Error, Root-Mean-Square Error, Normalized Root-Mean-Square Error, and Mean Absolute Percentage Error metrics. Comparison with the Global Forecast System (GFS) revealed similar performance, with both models effectively capturing daily solar irradiance patterns. The forecast model integrates with an Object-Oriented power production model, enabling accurate energy offers in the intraday market while minimizing penalty costs.
MOGNET: A Mux-residual quantized Network leveraging Online-Generated weights
Authors: Van Thien Nguyen, William Guicquero, Gilles Sicard
Date: 2025-01-16 00:00:00

This paper presents a compact model architecture called MOGNET, compatible with a resource-limited hardware. MOGNET uses a streamlined Convolutional factorization block based on a combination of 2 point-wise (1x1) convolutions with a group-wise convolution in-between. To further limit the overall model size and reduce the on-chip required memory, the second point-wise convolution's parameters are on-line generated by a Cellular Automaton structure. In addition, MOGNET enables the use of low-precision weights and activations, by taking advantage of a Multiplexer mechanism with a proper Bitshift rescaling for integrating residual paths without increasing the hardware-related complexity. To efficiently train this model we also introduce a novel weight ternarization method favoring the balance between quantized levels. Experimental results show that given tiny memory budget (sub-2Mb), MOGNET can achieve higher accuracy with a clear gap up to 1% at a similar or even lower model size compared to recent state-of-the-art methods.
Confidence Estimation for Error Detection in Text-to-SQL Systems
Authors: Oleg Somov, Elena Tutubalina
Date: 2025-01-16 00:00:00
Tasks: Decoder, In-Context Learning, Text to SQL, Text-To-SQL

Text-to-SQL enables users to interact with databases through natural language, simplifying the retrieval and synthesis of information. Despite the success of large language models (LLMs) in converting natural language questions into SQL queries, their broader adoption is limited by two main challenges: achieving robust generalization across diverse queries and ensuring interpretative confidence in their predictions. To tackle these issues, our research investigates the integration of selective classifiers into Text-to-SQL systems. We analyse the trade-off between coverage and risk using entropy based confidence estimation with selective classifiers and assess its impact on the overall performance of Text-to-SQL models. Additionally, we explore the models' initial calibration and improve it with calibration techniques for better model alignment between confidence and accuracy. Our experimental results show that encoder-decoder T5 is better calibrated than in-context-learning GPT 4 and decoder-only Llama 3, thus the designated external entropy-based selective classifier has better performance. The study also reveal that, in terms of error detection, selective classifier with a higher probability detects errors associated with irrelevant questions rather than incorrect query generations.
Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging
Authors: Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, DaCheng Tao
Date: 2025-01-16 00:00:00

Deep model merging represents an emerging research direction that combines multiple fine-tuned models to harness their specialized capabilities across different tasks and domains. Current model merging techniques focus on merging all available models simultaneously, with weight interpolation-based methods being the predominant approaches. However, these conventional approaches are not well-suited for scenarios where models become available sequentially, and they often suffer from high memory requirements and potential interference between tasks. In this study, we propose a training-free projection-based continual merging method that processes models sequentially through orthogonal projections of weight matrices and adaptive scaling mechanisms. Our method operates by projecting new parameter updates onto subspaces orthogonal to existing merged parameter updates while using an adaptive scaling mechanism to maintain stable parameter distances, enabling efficient sequential integration of task-specific knowledge. Our approach maintains constant memory complexity to the number of models, minimizes interference between tasks through orthogonal projections, and retains the performance of previously merged models through adaptive task vector scaling. Extensive experiments on CLIP-ViT models demonstrate that our method achieves a 5-8% average accuracy improvement while maintaining robust performance in different task orderings.
Multi-task deep-learning for sleep event detection and stage classification
Authors: Adriana Anido-Alonso, Diego Alvarez-Estevez
Date: 2025-01-16 00:00:00
Tasks: EEG, Event Detection, object-detection, Object Detection

Polysomnographic sleep analysis is the standard clinical method to accurately diagnose and treat sleep disorders. It is an intricate process which involves the manual identification, classification, and location of multiple sleep event patterns. This is complex, for which identification of different types of events involves focusing on different subsets of signals, resulting on an iterative time-consuming process entailing several visual analysis passes. In this paper we propose a multi-task deep-learning approach for the simultaneous detection of sleep events and hypnogram construction in one single pass. Taking as reference state-of-the-art methodology for object-detection in the field of Computer Vision, we reformulate the problem for the analysis of multi-variate time sequences, and more specifically for pattern detection in the sleep analysis scenario. We investigate the performance of the resulting method in identifying different assembly combinations of EEG arousals, respiratory events (apneas and hypopneas) and sleep stages, also considering different input signal montage configurations. Furthermore, we evaluate our approach using two independent datasets, assessing true-generalization effects involving local and external validation scenarios. Based on our results, we analyze and discuss our method's capabilities and its potential wide-range applicability across different settings and datasets.
PIER: A Novel Metric for Evaluating What Matters in Code-Switching
Authors: Enes Yavuz Ugan, Ngoc-Quan Pham, Leonard Bärmann, Alex Waibel
Date: 2025-01-16 00:00:00
Tasks: Automatic Speech Recognition, Decoder, speech-recognition, Speech Recognition

Code-switching, the alternation of languages within a single discourse, presents a significant challenge for Automatic Speech Recognition. Despite the unique nature of the task, performance is commonly measured with established metrics such as Word-Error-Rate (WER). However, in this paper, we question whether these general metrics accurately assess performance on code-switching. Specifically, using both Connectionist-Temporal-Classification and Encoder-Decoder models, we show fine-tuning on non-code-switched data from both matrix and embedded language improves classical metrics on code-switching test sets, although actual code-switched words worsen (as expected). Therefore, we propose Point-of-Interest Error Rate (PIER), a variant of WER that focuses only on specific words of interest. We instantiate PIER on code-switched utterances and show that this more accurately describes the code-switching performance, showing huge room for improvement in future work. This focused evaluation allows for a more precise assessment of model performance, particularly in challenging aspects such as inter-word and intra-word code-switching.
Multimodal Marvels of Deep Learning in Medical Diagnosis: A Comprehensive Review of COVID-19 Detection
Authors: Md Shofiqul Islam, Khondokar Fida Hasan, Hasibul Hossain Shajeeb, Humayan Kabir Rana, Md Saifur Rahmand, Md Munirul Hasan, AKM Azad, Ibrahim Abdullah, Mohammad Ali Moni
Date: 2025-01-16 00:00:00
Tasks: Deep Learning, Diagnostic, Medical Diagnosis, Multimodal Deep Learning, text-classification, Text Classification

This study presents a comprehensive review of the potential of multimodal deep learning (DL) in medical diagnosis, using COVID-19 as a case example. Motivated by the success of artificial intelligence applications during the COVID-19 pandemic, this research aims to uncover the capabilities of DL in disease screening, prediction, and classification, and to derive insights that enhance the resilience, sustainability, and inclusiveness of science, technology, and innovation systems. Adopting a systematic approach, we investigate the fundamental methodologies, data sources, preprocessing steps, and challenges encountered in various studies and implementations. We explore the architecture of deep learning models, emphasising their data-specific structures and underlying algorithms. Subsequently, we compare different deep learning strategies utilised in COVID-19 analysis, evaluating them based on methodology, data, performance, and prerequisites for future research. By examining diverse data types and diagnostic modalities, this research contributes to scientific understanding and knowledge of the multimodal application of DL and its effectiveness in diagnosis. We have implemented and analysed 11 deep learning models using COVID-19 image, text, and speech (ie, cough) data. Our analysis revealed that the MobileNet model achieved the highest accuracy of 99.97% for COVID-19 image data and 93.73% for speech data (i.e., cough). However, the BiGRU model demonstrated superior performance in COVID-19 text classification with an accuracy of 99.89%. The broader implications of this research suggest potential benefits for other domains and disciplines that could leverage deep learning techniques for image, text, and speech analysis.
Utilizing AI Language Models to Identify Prognostic Factors for Coronary Artery Disease: A Study in Mashhad Residents
Authors: Bami Zahra, Behnampour Nasser, Doosti Hassan, Ghayour Mobarhan Majid
Date: 2025-01-16 00:00:00
Tasks: Model Selection, Sensitivity, Specificity

Abstract: Background: Understanding cardiovascular artery disease risk factors, the leading global cause of mortality, is crucial for influencing its etiology, prevalence, and treatment. This study aims to evaluate prognostic markers for coronary artery disease in Mashhad using Naive Bayes, REP Tree, J48, CART, and CHAID algorithms. Methods: Using data from the 2009 MASHAD STUDY, prognostic factors for coronary artery disease were determined with Naive Bayes, REP Tree, J48, CART, CHAID, and Random Forest algorithms using R 3.5.3 and WEKA 3.9.4. Model efficiency was compared by sensitivity, specificity, and accuracy. Cases were patients with coronary artery disease; each had three controls (totally 940). Results: Prognostic factors for coronary artery disease in Mashhad residents varied by algorithm. CHAID identified age, myocardial infarction history, and hypertension. CART included depression score and physical activity. REP added education level and anxiety score. NB included diabetes and family history. J48 highlighted father's heart disease and weight loss. CHAID had the highest accuracy (0.80). Conclusion: Key prognostic factors for coronary artery disease in CART and CHAID models include age, myocardial infarction history, hypertension, depression score, physical activity, and BMI. NB, REP Tree, and J48 identified numerous factors. CHAID had the highest accuracy, sensitivity, and specificity. CART offers simpler interpretation, aiding physician and paramedic model selection based on specific. Keywords: RF, Na\"ive Bayes, REP, J48 algorithms, Coronary Artery Disease (CAD).