pwc-archive/papers-with-abstracts huggingface dataset:
The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms
Authors: Elizabeth Collins-Woodfin, Inbar Seroussi, Begoña García Malaxechebarría, Andrew W. Mackenzie, Elliot Paquette, Courtney Paquette
Date: 2024-05-30 00:00:00

We develop a framework for analyzing the training and learning rate dynamics on a large class of high-dimensional optimization problems, which we call the high line, trained using one-pass stochastic gradient descent (SGD) with adaptive learning rates. We give exact expressions for the risk and learning rate curves in terms of a deterministic solution to a system of ODEs. We then investigate in detail two adaptive learning rates -- an idealized exact line search and AdaGrad-Norm -- on the least squares problem. When the data covariance matrix has strictly positive eigenvalues, this idealized exact line search strategy can exhibit arbitrarily slower convergence when compared to the optimal fixed learning rate with SGD. Moreover we exactly characterize the limiting learning rate (as time goes to infinity) for line search in the setting where the data covariance has only two distinct eigenvalues. For noiseless targets, we further demonstrate that the AdaGrad-Norm learning rate converges to a deterministic constant inversely proportional to the average eigenvalue of the data covariance matrix, and identify a phase transition when the covariance density of eigenvalues follows a power law distribution. We provide our code for evaluation at https://github.com/amackenzie1/highline2024.
Comparison of Point Process Learning and its special case Takacs-Fiksel estimation
Authors: Julia Jansson, Ottmar Cronie
Date: 2024-05-29 00:00:00
Tasks: Point Processes

Recently, Cronie et al. (2024) introduced the notion of cross-validation for point processes and a new statistical methodology called Point Process Learning (PPL). In PPL one splits a point process/pattern into a training and a validation set, and then predicts the latter from the former through a parametrised Papangelou conditional intensity. The model parameters are estimated by minimizing a point process prediction error; this notion was introduced as the second building block of PPL. It was shown that PPL outperforms the state-of-the-art in both kernel intensity estimation and estimation of the parameters of the Gibbs hard-core process. In the latter case, the state-of-the-art was represented by pseudolikelihood estimation. In this paper we study PPL in relation to Takacs-Fiksel estimation, of which pseudolikelihood is a special case. We show that Takacs-Fiksel estimation is a special case of PPL in the sense that PPL with a specific loss function asymptotically reduces to Takacs-Fiksel estimation if we let the cross-validation regime tend to leave-one-out cross-validation. Moreover, PPL involves a certain type of hyperparameter given by a weight function which ensures that the prediction errors have expectation zero if and only if we have the correct parametrisation. We show that the weight function takes an explicit but intractable form for general Gibbs models. Consequently, we propose different approaches to estimate the weight function in practice. In order to assess how the general PPL setup performs in relation to its special case Takacs-Fiksel estimation, we conduct a simulation study where we find that for common Gibbs models we can find loss functions and hyperparameters so that PPL typically outperforms Takacs-Fiksel estimation significantly in terms of mean square error. Here, the hyperparameters are the cross-validation parameters and the weight function estimate.
Optimal Multiclass U-Calibration Error and Beyond
Authors: Haipeng Luo, Spandan Senapati, Vatsal Sharan
Date: 2024-05-28 00:00:00

We consider the problem of online multiclass U-calibration, where a forecaster aims to make sequential distributional predictions over $K$ classes with low U-calibration error, that is, low regret with respect to all bounded proper losses simultaneously. Kleinberg et al. (2023) developed an algorithm with U-calibration error $O(K\sqrt{T})$ after $T$ rounds and raised the open question of what the optimal bound is. We resolve this question by showing that the optimal U-calibration error is $\Theta(\sqrt{KT})$ -- we start with a simple observation that the Follow-the-Perturbed-Leader algorithm of Daskalakis and Syrgkanis (2016) achieves this upper bound, followed by a matching lower bound constructed with a specific proper loss (which, as a side result, also proves the optimality of the algorithm of Daskalakis and Syrgkanis (2016) in the context of online learning against an adversary with finite choices). We also strengthen our results under natural assumptions on the loss functions, including $\Theta(\log T)$ U-calibration error for Lipschitz proper losses, $O(\log T)$ U-calibration error for a certain class of decomposable proper losses, U-calibration error bounds for proper losses with a low covering number, and others.
Generating Query Recommendations via LLMs
Authors: Andrea Bacciu, Enrico Palumbo, Andreas Damianou, Nicola Tonellotto, Fabrizio Silvestri
Date: 2024-05-30 00:00:00
Tasks: Recommendation Systems

Query recommendation systems are ubiquitous in modern search engines, assisting users in producing effective queries to meet their information needs. However, these systems require a large amount of data to produce good recommendations, such as a large collection of documents to index and query logs. In particular, query logs and user data are not available in cold start scenarios. Query logs are expensive to collect and maintain and require complex and time-consuming cascading pipelines for creating, combining, and ranking recommendations. To address these issues, we frame the query recommendation problem as a generative task, proposing a novel approach called Generative Query Recommendation (GQR). GQR uses an LLM as its foundation and does not require to be trained or fine-tuned to tackle the query recommendation problem. We design a prompt that enables the LLM to understand the specific recommendation task, even using a single example. We then improved our system by proposing a version that exploits query logs called Retriever-Augmented GQR (RA-GQR). RA-GQr dynamically composes its prompt by retrieving similar queries from query logs. GQR approaches reuses a pre-existing neural architecture resulting in a simpler and more ready-to-market approach, even in a cold start scenario. Our proposed GQR obtains state-of-the-art performance in terms of NDCG@10 and clarity score against two commercial search engines and the previous state-of-the-art approach on the Robust04 and ClueWeb09B collections, improving on average the NDCG@10 performance up to ~4% on Robust04 and ClueWeb09B w.r.t the previous best competitor. RA-GQR further improve the NDCG@10 obtaining an increase of ~11%, ~6\% on Robust04 and ClueWeb09B w.r.t the best competitor. Furthermore, our system obtained ~59% of user preferences in a blind user study, proving that our method produces the most engaging queries.
Keyword-driven Retrieval-Augmented Large Language Models for Cold-start User Recommendations
Authors: Hai-Dang Kieu, Minh Duc Nguyen, Thanh-Son Nguyen, Dung D. Le
Date: 2024-05-30 00:00:00
Tasks: Recommendation Systems, Re-Ranking, Retrieval

Recent advancements in Large Language Models (LLMs) have shown significant potential in enhancing recommender systems. However, addressing the cold-start recommendation problem, where users lack historical data, remains a considerable challenge. In this paper, we introduce KALM4Rec (Keyword-driven Retrieval-Augmented Large Language Models for Cold-start User Recommendations), a novel framework specifically designed to tackle this problem by requiring only a few input keywords from users in a practical scenario of cold-start user restaurant recommendations. KALM4Rec operates in two main stages: candidates retrieval and LLM-based candidates re-ranking. In the first stage, keyword-driven retrieval models are used to identify potential candidates, addressing LLMs' limitations in processing extensive tokens and reducing the risk of generating misleading information. In the second stage, we employ LLMs with various prompting strategies, including zero-shot and few-shot techniques, to re-rank these candidates by integrating multiple examples directly into the LLM prompts. Our evaluation, using a Yelp restaurant dataset with user reviews from three English-speaking cities, shows that our proposed framework significantly improves recommendation quality. Specifically, the integration of in-context instructions with LLMs for re-ranking markedly enhances the performance of the cold-start user recommender system.
MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encodings
Authors: Laxman Dhulipala, Majid Hadian, Rajesh Jayaram, Jason Lee, Vahab Mirrokni
Date: 2024-05-29 00:00:00
Tasks: Information Retrieval, Retrieval

Neural embedding models have become a fundamental component of modern information retrieval (IR) pipelines. These models produce a single embedding $x \in \mathbb{R}^d$ per data-point, allowing for fast retrieval via highly optimized maximum inner product search (MIPS) algorithms. Recently, beginning with the landmark ColBERT paper, multi-vector models, which produce a set of embedding per data point, have achieved markedly superior performance for IR tasks. Unfortunately, using these models for IR is computationally expensive due to the increased complexity of multi-vector retrieval and scoring. In this paper, we introduce MUVERA (MUlti-VEctor Retrieval Algorithm), a retrieval mechanism which reduces multi-vector similarity search to single-vector similarity search. This enables the usage of off-the-shelf MIPS solvers for multi-vector retrieval. MUVERA asymmetrically generates Fixed Dimensional Encodings (FDEs) of queries and documents, which are vectors whose inner product approximates multi-vector similarity. We prove that FDEs give high-quality $\epsilon$-approximations, thus providing the first single-vector proxy for multi-vector similarity with theoretical guarantees. Empirically, we find that FDEs achieve the same recall as prior state-of-the-art heuristics while retrieving 2-5$\times$ fewer candidates. Compared to prior state of the art implementations, MUVERA achieves consistently good end-to-end recall and latency across a diverse set of the BEIR retrieval datasets, achieving an average of 10$\%$ improved recall with $90\%$ lower latency.
A light-weight rectangular decomposition large kernel convolution network for deformable medical image registration.
Authors: Yuzhu Cao, Weiwei Cao, Ziyu Wang, Gang Yuan, Zeyi Li, Xinye Ni, Jian Zheng
Date: 2024-05-27 00:00:00
Proceeding: Biomedical Signal Processing and Control 2024 5
Tasks: Deformable Medical Image Registration, Image Registration, Medical Image Registration

The performance and speed of medical image registration have been greatly boosted by advanced deep-learning based methods. However, most current methods are challenged by large deformations between input images, which necessitate a compromise in computational cost to enhance the model’s receptive field and its ability to model long-range spatial relationships for improving registration performance. In order to enhance the performance of registration for images with large deformations at a lower computational cost, in this paper, we propose a light-weight registration model with the ability to model large receptive fields and long-range spatial relationships, named LL-Net. The core components of LL-Net consist of a Rectangular Decomposition Large Kernel Attention (RD-LKA) layer and a Spatial and Channel Fusion Attention (SC-Fusion) layer. The RD-LKA layer utilizes anisotropic depth-wise large kernel convolutions to capture large receptive fields with an extremely low parameter count while modeling long-range spatial relationships. Moreover, the SC-Fusion layer enhances the model’s feature fusion capability and strengthens feature representations at critical locations. Our LL-Net exhibits state-of-the-art performance across multiple datasets. Specifically, it achieves a Dice score of 76.7% and an HD95 of 2.983 mm on the IXI dataset, and a Dice score of 87.8% and an HD95 of 1.042 mm on the OASIS dataset. Experimental results substantiate the efficacy of LL-Net in capturing large receptive fields and modeling long-range spatial relationships. The code for LL-Net is available at https://github.com/BoyOfChu/LL_Net.
Center-aware Adversarial Augmentation for Single Domain Generalization
Authors: Tianle Chen, Mahsa Baktashmotlagh, Zijian Wang, Mathieu Salzmann
Date: 2023-01-01 00:00:00
Proceeding: WACV 2023 1
Tasks: Data Augmentation, Domain Generalization, Photo to Rest Generalization, Single-Source Domain Generalization

Domain generalization (DG) aims to learn a model from multiple training (i.e., source) domains that can generalize well to the unseen test (i.e., target) data coming from a different distribution. Single domain generalization (SingleDG) has recently emerged to tackle a more challenging, yet realistic setting, where only one source domain is available at training time. The existing Single-DG approaches typically are based on data augmentation strategies and aim to expand the span of source data by augmenting out-of domain samples. Generally speaking, they aim to generate hard examples to confuse the classifier. While this may make the classifier robust to small perturbation, the generated samples are typically not diverse enough to mimic a large domain shift, resulting in sub-optimal generalization performance. To alleviate this, we propose a centeraware adversarial augmentation technique that expands the source distribution by altering the source samples so as to push them away from the class centers via a novel angular center loss. We conduct extensive experiments to demonstrate the effectiveness of our approach on several benchmark datasets for Single-DG and show that our method outperforms the state-of-the-art in most cases.
NERULA: A Dual-Pathway Self-Supervised Learning Framework for Electrocardiogram Signal Analysis
Authors: Gouthamaan Manimaran, Sadasivan Puthusserypady, Helena Domínguez, Adrian Atienza, Jakob E. Bardram
Date: 2024-05-21 00:00:00
Tasks: Activity Recognition, Contrastive Learning, Gender Classification, Human Activity Recognition, Self-Supervised Learning

Electrocardiogram (ECG) signals are critical for diagnosing heart conditions and capturing detailed cardiac patterns. As wearable single-lead ECG devices become more common, efficient analysis methods are essential. We present NERULA (Non-contrastive ECG and Reconstruction Unsupervised Learning Algorithm), a self-supervised framework designed for single-lead ECG signals. NERULA's dual-pathway architecture combines ECG reconstruction and non-contrastive learning to extract detailed cardiac features. Our 50% masking strategy, using both masked and inverse-masked signals, enhances model robustness against real-world incomplete or corrupted data. The non-contrastive pathway aligns representations of masked and inverse-masked signals, while the reconstruction pathway comprehends and reconstructs missing features. We show that combining generative and discriminative paths into the training spectrum leads to better results by outperforming state-of-the-art self-supervised learning benchmarks in various tasks, demonstrating superior performance in ECG analysis, including arrhythmia classification, gender classification, age regression, and human activity recognition. NERULA's dual-pathway design offers a robust, efficient solution for comprehensive ECG signal interpretation.
Machine Learning 5G Attack Detection in Programmable Logic
Authors: Cooper Coldwell, Denver Conger, Edward Goodell, Brendan Jacobson, Bryton Petersen, Damon Spencer, Matthew Anderson, Matthew Sgambati
Date: 2022-12-04 00:00:00
Proceeding: IEEE Globecom 2022 12
Tasks: Low-latency processing

Machine learning-assisted network security may significantly contribute to securing 5G components. However, machine learning network security inference generally requires tens to hundreds of milliseconds, thereby introducing significant latency in 5G operations. The inference latency can be reduced by deploying the machine learning model to programmable logic in a field programmable gate array at the cost of a small loss in accuracy. In order to quantify this loss, as well as to establish baseline performance inference latency for programmable logic implementations, this work explores an autoencoder and a β-variational autoencoder deployed on two different field programmable gate array evaluation boards and compares accuracy and performance against an NVIDIA A100 graphics processing unit implementation. A publicly available 5G dataset containing 10 types of attacks along with normal traffic is introduced as part of the evaluation.
Students Engagement Level Detection in Online e-Learning Using Hybrid EfficientNetB7 Together With TCN, LSTM, and Bi-LSTM
Authors: Tasneem Selim, Islam Elkabani, Mohamed A. Abdou
Date: 2022-09-26 00:00:00
Proceeding: IEEE Access 2022 9
Tasks: Student Engagement Level Detection (Four Class Video Classification)

Students engagement level detection in online e-learning has become a crucial problem due to the rapid advance of digitalization in education. In this paper, a novel Videos Recorded for Egyptian Students Engagement in E-learning (VRESEE) dataset is introduced for students engagement level detection in online e-learning. This dataset is based on an experiment conducted on a group of Egyptian college students by video recording them during online e-learning sessions. Each recorded video is labeled with a value from 0 to 3 representing the level of engagement of each student during the online session. Moreover, three new hybrid end-to-end deep learning models have been proposed for detecting student’s engagement level in an online e-learning video. These models are evaluated using the VRESEE dataset and also using a public Dataset for the Affective States in E-Environment (DAiSEE). The first proposed hybrid model uses EfficientNet B7 together with Temporal Convolution Network (TCN) and achieved an accuracy of 64.67% on DAiSEE and 81.14% on VRESEE. The second model uses a hybrid EfficientNet B7 along with Long Short Term Memory (LSTM) and reached an accuracy of 67.48% on DAiSEE and 93.99% on VRESEE. Finally, the third hybrid model uses EfficientNet B7 along with a Bidirectional LSTM and achieved an accuracy of 66.39% on DAiSEE and 94.47% on VRESEE. The results of the first, second and third proposed models outperform the results of currently existing models by 1.08%, 3.89%, and 2.8% respectively in students engagement level detection.
Discrete-Time I&I Adaptive Interconnection and Damping Passivity-Based Control for Nonlinearly Parameterized Port-Controlled Hamiltonian Systems
Authors: Mohammed Alkrunz, Yaprak Yalcin
Date: 2024-05-30 00:00:00

In this paper, a discrete-time I&I-based adaptive IDA-PBC controller for uncertain nonlinearly parameterized port-controlled Hamiltonian systems (PCH), where the parameter uncertainties are assumed in the energy function, is constructed. A proper formulation for the uncertain system dynamics is established where the uncertainties appear in nonlinearly parameterized form in the gradient of the Hamiltonian function. The adaptive IDA-PBC controller is constructed considering this formulation. For the adaptation mechanism of the IDA-PBC controller, a discrete-time parameter estimator is derived based on the immersion and invariance (I&I) approach. A structure for a free design function in the I&I-based estimator is proposed including some other free design functions. If these free design functions are selected to satisfy some conditions, derived in this paper, the Lyapunov asymptotic stability of the estimator dynamics is guaranteed. Besides, assuming these conditions are satisfied, local asymptotic stability of the closed-loop system, in a sufficiently large set is shown. The proposed method is applied to the two physical system examples and the performance of the adaptive controller is tested by simulation. It is demonstrated that the performance of the certain IDA-PBC controller is maintained by the adaptive IDA-PBC controller successfully.
A Triumvirate of AI Driven Theoretical Discovery
Authors: Yang-Hui He
Date: 2024-05-30 00:00:00

Recent years have seen the dramatic rise of the usage of AI algorithms in pure mathematics and fundamental sciences such as theoretical physics. This is perhaps counter-intuitive since mathematical sciences require the rigorous definitions, derivations, and proofs, in contrast to the experimental sciences which rely on the modelling of data with error-bars. In this Perspective, we categorize the approaches to mathematical discovery as "top-down", "bottom-up" and "meta-mathematics", as inspired by historical examples. We review some of the progress over the last few years, comparing and contrasting both the advances and the short-comings in each approach. We argue that while the theorist is in no way in danger of being replaced by AI in the near future, the hybrid of human expertise and AI algorithms will become an integral part of theoretical discovery.
Learning the Language of Protein Structure
Authors: Benoit Gaujac, Jérémie Donà, Liviu Copoiu, Timothy Atkinson, Thomas Pierrot, Thomas D. Barrett
Date: 2024-05-24 00:00:00
Tasks: Protein Design, Representation Learning

Representation learning and \emph{de novo} generation of proteins are pivotal computational biology tasks. Whilst natural language processing (NLP) techniques have proven highly effective for protein sequence modelling, structure modelling presents a complex challenge, primarily due to its continuous and three-dimensional nature. Motivated by this discrepancy, we introduce an approach using a vector-quantized autoencoder that effectively tokenizes protein structures into discrete representations. This method transforms the continuous, complex space of protein structures into a manageable, discrete format with a codebook ranging from 4096 to 64000 tokens, achieving high-fidelity reconstructions with backbone root mean square deviations (RMSD) of approximately 1-5 \AA. To demonstrate the efficacy of our learned representations, we show that a simple GPT model trained on our codebooks can generate novel, diverse, and designable protein structures. Our approach not only provides representations of protein structure, but also mitigates the challenges of disparate modal representations and sets a foundation for seamless, multi-modal integration, enhancing the capabilities of computational methods in protein design.
Network analytics for insurance fraud detection: a critical case study
Authors: Bruno Deprez, Félix Vandervorst, Wouter Verbeke, Bart Baesens, Tim Verdonck
Date: 2024-05-14 00:00:00
Proceeding: European Actuarial Journal 2024 5
Tasks: Fraud Detection

There has been an increasing interest in fraud detection methods, driven by new regulations and by the financial losses linked to fraud. One of the state-of-the-art methods to fight fraud is network analytics. Network analytics leverages the interactions between different entities to detect complex patterns that are indicative of fraud. However, network analytics has only recently been applied to fraud detection in the actuarial literature. Although it shows much potential, many network methods are not yet applied. This paper extends the literature in two main ways. First, we review and apply multiple methods in the context of insurance fraud and assess their predictive power against each other. Second, we analyse the added value of network features over intrinsic features to detect fraud. We conclude that (1) complex methods do not necessarily outperform basic network features, and that (2) network analytics helps to detect different fraud patterns, compared to models trained on claim-specific features alone.