pwc-archive/papers-with-abstracts huggingface dataset:
Do the Right Thing, Just Debias! Multi-Category Bias Mitigation Using LLMs
Authors: Amartya Roy, Danush Khanna, Devanshu Mahapatra, Vasanthakumar, Avirup Das, Kripabandhu Ghosh
Date: 2024-09-24 00:00:00
Tasks: In-Context Learning, Sentence

This paper tackles the challenge of building robust and generalizable bias mitigation models for language. Recognizing the limitations of existing datasets, we introduce ANUBIS, a novel dataset with 1507 carefully curated sentence pairs encompassing nine social bias categories. We evaluate state-of-the-art models like T5, utilizing Supervised Fine-Tuning (SFT), Reinforcement Learning (PPO, DPO), and In-Context Learning (ICL) for effective bias mitigation. Our analysis focuses on multi-class social bias reduction, cross-dataset generalizability, and environmental impact of the trained models. ANUBIS and our findings offer valuable resources for building more equitable AI systems and contribute to the development of responsible and unbiased technologies with broad societal impact.
Metaheuristic Method for Solving Systems of Equations
Authors: Samson Odan
Date: 2024-09-25 00:00:00
Tasks: valid

This study investigates the effectiveness of Genetic Algorithms (GAs) in solving both linear and nonlinear systems of equations, comparing their performance to traditional methods such as Gaussian Elimination, Newton's Method, and Levenberg-Marquardt. The GA consistently delivered accurate solutions across various test cases, demonstrating its robustness and flexibility. A key advantage of the GA is its ability to explore the solution space broadly, uncovering multiple sets of solutions -- a feat that traditional methods, which typically converge to a single solution, cannot achieve. This feature proved especially beneficial in complex nonlinear systems, where multiple valid solutions exist, highlighting the GA's superiority in navigating intricate solution landscapes.
Evolutionary Greedy Algorithm for Optimal Sensor Placement Problem in Urban Sewage Surveillance
Authors: Sunyu Wang, Yutong Xia, Huanfa Chen, Xinyi Tong, Yulun Zhou
Date: 2024-09-25 00:00:00

Designing a cost-effective sensor placement plan for sewage surveillance is a crucial task because it allows cost-effective early pandemic outbreak detection as supplementation for individual testing. However, this problem is computationally challenging to solve, especially for massive sewage networks having complicated topologies. In this paper, we formulate this problem as a multi-objective optimization problem to consider the conflicting objectives and put forward a novel evolutionary greedy algorithm (EG) to enable efficient and effective optimization for large-scale directed networks. The proposed model is evaluated on both small-scale synthetic networks and a large-scale, real-world sewage network in Hong Kong. The experiments on small-scale synthetic networks demonstrate a consistent efficiency improvement with reasonable optimization performance and the real-world application shows that our method is effective in generating optimal sensor placement plans to guide policy-making.
An Adaptive Re-evaluation Method for Evolution Strategy under Additive Noise
Authors: Catalin-Viorel Dinu, Yash J. Patel, Xavier Bonet-Monroig, Hao Wang
Date: 2024-09-25 00:00:00

The Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) is one of the most advanced algorithms in numerical black-box optimization. For noisy objective functions, several approaches were proposed to mitigate the noise, e.g., re-evaluations of the same solution or adapting the population size. In this paper, we devise a novel method to adaptively choose the optimal re-evaluation number for function values corrupted by additive Gaussian white noise. We derive a theoretical lower bound of the expected improvement achieved in one iteration of CMA-ES, given an estimation of the noise level and the Lipschitz constant of the function's gradient. Solving for the maximum of the lower bound, we obtain a simple expression of the optimal re-evaluation number. We experimentally compare our method to the state-of-the-art noise-handling methods for CMA-ES on a set of artificial test functions across various noise levels, optimization budgets, and dimensionality. Our method demonstrates significant advantages in terms of the probability of hitting near-optimal function values.
On the Complexity of Neural Computation in Superposition
Authors: Micah Adler, Nir Shavit
Date: 2024-09-05 00:00:00
Tasks: Computational Efficiency, LEMMA

Superposition, the ability of neural networks to represent more features than neurons, is increasingly seen as key to the efficiency of large models. This paper investigates the theoretical foundations of computing in superposition, establishing complexity bounds for explicit, provably correct algorithms. We present the first lower bounds for a neural network computing in superposition, showing that for a broad class of problems, including permutations and pairwise logical operations, computing $m'$ features in superposition requires at least $\Omega(\sqrt{m' \log m'})$ neurons and $\Omega(m' \log m')$ parameters. This implies the first subexponential upper bound on superposition capacity: a network with $n$ neurons can compute at most $O(n^2 / \log n)$ features. Conversely, we provide a nearly tight constructive upper bound: logical operations like pairwise AND can be computed using $O(\sqrt{m'} \log m')$ neurons and $O(m' \log^2 m')$ parameters. There is thus an exponential gap between the complexity of computing in superposition (the subject of this work) versus merely representing features, which can require as little as $O(\log m')$ neurons based on the Johnson-Lindenstrauss Lemma. Our hope is that our results open a path for using complexity theoretic techniques in neural network interpretability research.
Stochastic neighborhood embedding and the gradient flow of relative entropy
Authors: Ben Weinkove
Date: 2024-09-25 00:00:00
Tasks: Dimensionality Reduction

Dimension reduction, widely used in science, maps high-dimensional data into low-dimensional space. We investigate a basic mathematical model underlying the techniques of stochastic neighborhood embedding (SNE) and its popular variant t-SNE. Distances between points in high dimensions are used to define a probability distribution on pairs of points, measuring how similar the points are. The aim is to map these points to low dimensions in an optimal way so that similar points are closer together. This is carried out by minimizing the relative entropy between two probability distributions. We consider the gradient flow of the relative entropy and analyze its long-time behavior. This is a self-contained mathematical problem about the behavior of a system of nonlinear ordinary differential equations. We find optimal bounds for the diameter of the evolving sets as time tends to infinity. In particular, the diameter may blow up for the t-SNE version, but remains bounded for SNE.
Revisiting Extragradient-Type Methods -- Part 1: Generalizations and Sublinear Convergence Rates
Authors: Quoc Tran-Dinh, Nghia Nguyen-Trung
Date: 2024-09-25 00:00:00

This paper presents a comprehensive analysis of the well-known extragradient (EG) method for solving both equations and inclusions. First, we unify and generalize EG for [non]linear equations to a wider class of algorithms, encompassing various existing schemes and potentially new variants. Next, we analyze both sublinear ``best-iterate'' and ``last-iterate'' convergence rates for the entire class of algorithms, and derive new convergence results for two well-known instances. Second, we extend our EG framework above to ``monotone'' inclusions, introducing a new class of algorithms and its corresponding convergence results. Third, we also unify and generalize Tseng's forward-backward-forward splitting (FBFS) method to a broader class of algorithms to solve [non]linear inclusions when a weak-Minty solution exists, and establish its ``best-iterate'' convergence rate. Fourth, to complete our picture, we also investigate sublinear rates of two other common variants of EG using our EG analysis framework developed here: the reflected forward-backward splitting and the golden ratio methods. Finally, we conduct an extensive numerical experiment to validate our theoretical findings. Our results demonstrate that several new variants of our proposed algorithms outperform existing schemes in the majority of examples.
Conditional Testing based on Localized Conformal p-values
Authors: Xiaoyang Wu, Lin Lu, Zhaojun Wang, Changliang Zou
Date: 2024-09-25 00:00:00
Tasks: Outlier Detection, Prediction Intervals

In this paper, we address conditional testing problems through the conformal inference framework. We define the localized conformal p-values by inverting prediction intervals and prove their theoretical properties. These defined p-values are then applied to several conditional testing problems to illustrate their practicality. Firstly, we propose a conditional outlier detection procedure to test for outliers in the conditional distribution with finite-sample false discovery rate (FDR) control. We also introduce a novel conditional label screening problem with the goal of screening multivariate response variables and propose a screening procedure to control the family-wise error rate (FWER). Finally, we consider the two-sample conditional distribution test and define a weighted U-statistic through the aggregation of localized p-values. Numerical simulations and real-data examples validate the superior performance of our proposed strategies.
A Novel Framework for Analyzing Structural Transformation in Data-Constrained Economies Using Bayesian Modeling and Machine Learning
Authors: Ronald Katende
Date: 2024-09-25 00:00:00
Tasks: Imputation

Structural transformation, the shift from agrarian economies to more diversified industrial and service-based systems, is a key driver of economic development. However, in low- and middle-income countries (LMICs), data scarcity and unreliability hinder accurate assessments of this process. This paper presents a novel statistical framework designed to address these challenges by integrating Bayesian hierarchical modeling, machine learning-based data imputation, and factor analysis. The framework is specifically tailored for conditions of data sparsity and is capable of providing robust insights into sectoral productivity and employment shifts across diverse economies. By utilizing Bayesian models, uncertainties in data are effectively managed, while machine learning techniques impute missing data points, ensuring the integrity of the analysis. Factor analysis reduces the dimensionality of complex datasets, distilling them into core economic structures. The proposed framework has been validated through extensive simulations, demonstrating its ability to predict structural changes even when up to 60\% of data is missing. This approach offers policymakers and researchers a valuable tool for making informed decisions in environments where data quality is limited, contributing to the broader understanding of economic development in LMICs.
Exact gradients for linear optics with single photons
Authors: Giorgio Facelli, David D. Roberts, Hugo Wallner, Alexander Makarovskiy, Zoë Holmes, William R. Clements
Date: 2024-09-24 00:00:00

Though parameter shift rules have drastically improved gradient estimation methods for several types of quantum circuits, leading to improved performance in downstream tasks, so far they have not been transferable to linear optics with single photons. In this work, we derive an analytical formula for the gradients in these circuits with respect to phaseshifters via a generalized parameter shift rule, where the number of parameter shifts depends linearly on the total number of photons. Experimentally, this enables access to derivatives in photonic systems without the need for finite difference approximations. Building on this, we propose two strategies through which one can reduce the number of shifts in the expression, and hence reduce the overall sample complexity. Numerically, we show that this generalized parameter-shift rule can converge to the minimum of a cost function with fewer parameter update steps than alternative techniques. We anticipate that this method will open up new avenues to solving optimization problems with photonic systems, as well as provide new techniques for the experimental characterization and control of linear optical systems.
Enhancing Automatic Keyphrase Labelling with Text-to-Text Transfer Transformer (T5) Architecture: A Framework for Keyphrase Generation and Filtering
Authors: Jorge Gabín, M. Eduardo Ares, Javier Parapar
Date: 2024-09-25 00:00:00
Tasks: Keyphrase Generation

Automatic keyphrase labelling stands for the ability of models to retrieve words or short phrases that adequately describe documents' content. Previous work has put much effort into exploring extractive techniques to address this task; however, these methods cannot produce keyphrases not found in the text. Given this limitation, keyphrase generation approaches have arisen lately. This paper presents a keyphrase generation model based on the Text-to-Text Transfer Transformer (T5) architecture. Having a document's title and abstract as input, we learn a T5 model to generate keyphrases which adequately define its content. We name this model docT5keywords. We not only perform the classic inference approach, where the output sequence is directly selected as the predicted values, but we also report results from a majority voting approach. In this approach, multiple sequences are generated, and the keyphrases are ranked based on their frequency of occurrence across these sequences. Along with this model, we present a novel keyphrase filtering technique based on the T5 architecture. We train a T5 model to learn whether a given keyphrase is relevant to a document. We devise two evaluation methodologies to prove our model's capability to filter inadequate keyphrases. First, we perform a binary evaluation where our model has to predict if a keyphrase is relevant for a given document. Second, we filter the predicted keyphrases by several AKG models and check if the evaluation scores are improved. Experimental results demonstrate that our keyphrase generation model significantly outperforms all the baselines, with gains exceeding 100\% in some cases. The proposed filtering technique also achieves near-perfect accuracy in eliminating false positives across all datasets.
A Prompting-Based Representation Learning Method for Recommendation with Large Language Models
Authors: Junyi Chen, Toyotaro Suzumura
Date: 2024-09-25 00:00:00
Tasks: Collaborative Filtering, Profile Generation, Recommendation Systems, Representation Learning

In recent years, Recommender Systems (RS) have witnessed a transformative shift with the advent of Large Language Models (LLMs) in the field of Natural Language Processing (NLP). Models such as GPT-3.5/4, Llama, have demonstrated unprecedented capabilities in understanding and generating human-like text. The extensive information pre-trained by these LLMs allows for the potential to capture a more profound semantic representation from different contextual information of users and items. While the great potential lies behind the thriving of LLMs, the challenge of leveraging user-item preferences from contextual information and its alignment with the improvement of Recommender Systems needs to be addressed. Believing that a better understanding of the user or item itself can be the key factor in improving recommendation performance, we conduct research on generating informative profiles using state-of-the-art LLMs. To boost the linguistic abilities of LLMs in Recommender Systems, we introduce the Prompting-Based Representation Learning Method for Recommendation (P4R). In our P4R framework, we utilize the LLM prompting strategy to create personalized item profiles. These profiles are then transformed into semantic representation spaces using a pre-trained BERT model for text embedding. Furthermore, we incorporate a Graph Convolution Network (GCN) for collaborative filtering representation. The P4R framework aligns these two embedding spaces in order to address the general recommendation tasks. In our evaluation, we compare P4R with state-of-the-art Recommender models and assess the quality of prompt-based profile generation.
Train Once, Deploy Anywhere: Matryoshka Representation Learning for Multimodal Recommendation
Authors: Yueqi Wang, Zhenrui Yue, Huimin Zeng, Dong Wang, Julian McAuley
Date: 2024-09-25 00:00:00
Tasks: Multimodal Recommendation, Recommendation Systems, Representation Learning, Sequential Recommendation

Despite recent advancements in language and vision modeling, integrating rich multimodal knowledge into recommender systems continues to pose significant challenges. This is primarily due to the need for efficient recommendation, which requires adaptive and interactive responses. In this study, we focus on sequential recommendation and introduce a lightweight framework called full-scale Matryoshka representation learning for multimodal recommendation (fMRLRec). Our fMRLRec captures item features at different granularities, learning informative representations for efficient recommendation across multiple dimensions. To integrate item features from diverse modalities, fMRLRec employs a simple mapping to project multimodal item features into an aligned feature space. Additionally, we design an efficient linear transformation that embeds smaller features into larger ones, substantially reducing memory requirements for large-scale training on recommendation data. Combined with improved state space modeling techniques, fMRLRec scales to different dimensions and only requires one-time training to produce multiple models tailored to various granularities. We demonstrate the effectiveness and efficiency of fMRLRec on multiple benchmark datasets, which consistently achieves superior performance over state-of-the-art baseline methods. We make our code and data publicly available at https://github.com/yueqirex/fMRLRec.
FusionANNS: An Efficient CPU/GPU Cooperative Processing Architecture for Billion-scale Approximate Nearest Neighbor Search
Authors: Bing Tian, Haikun Liu, YuHang Tang, Shihai Xiao, Zhuohui Duan, Xiaofei Liao, Xuecang Zhang, Junhua Zhu, Yu Zhang
Date: 2024-09-25 00:00:00
Tasks: Collaborative Filtering, CPU, GPU, Re-Ranking

Approximate nearest neighbor search (ANNS) has emerged as a crucial component of database and AI infrastructure. Ever-increasing vector datasets pose significant challenges in terms of performance, cost, and accuracy for ANNS services. None of modern ANNS systems can address these issues simultaneously. We present FusionANNS, a high-throughput, low-latency, cost-efficient, and high-accuracy ANNS system for billion-scale datasets using SSDs and only one entry-level GPU. The key idea of FusionANNS lies in CPU/GPU collaborative filtering and re-ranking mechanisms, which significantly reduce I/O operations across CPUs, GPU, and SSDs to break through the I/O performance bottleneck. Specifically, we propose three novel designs: (1) multi-tiered indexing to avoid data swapping between CPUs and GPU, (2) heuristic re-ranking to eliminate unnecessary I/Os and computations while guaranteeing high accuracy, and (3) redundant-aware I/O deduplication to further improve I/O efficiency. We implement FusionANNS and compare it with the state-of-the-art SSD-based ANNS system--SPANN and GPU-accelerated in-memory ANNS system--RUMMY. Experimental results show that FusionANNS achieves 1) 9.4-13.1X higher query per second (QPS) and 5.7-8.8X higher cost efficiency compared with SPANN; 2) and 2-4.9X higher QPS and 2.3-6.8X higher cost efficiency compared with RUMMY, while guaranteeing low latency and high accuracy.
NIF: A Fast Implicit Image Compression with Bottleneck Layers and Modulated Sinusoidal Activations
Authors: Lorenzo Catania, Dario Allegra
Date: 2023-10-27 00:00:00
Proceeding: ACM Multimedia 2023 10
Tasks: Image Compression, Quantization

In Implicit Neural Representations (INRs) a discrete signal is parameterized by a neural network that maps coordinates to the signal samples. INRs were successfully employed for encoding and compression, but such approaches are in their early stage and are still overcome by traditional codecs and autoencoders. Despite this, they have recently gained the attention of the research community due to their promising results as novel representation strategies for encoding visual content. In this paper, we propose Neural Imaging Format (NIF), an open-source INR-based image compression codec which takes advantage of a novel neural architecture which consists of two modules: a Genesis network, for mapping coordinates to pixels through bottleneck layers with sinusoidal activation units, and a Modulation network, for varying the period of the sinusoidal activations. Additionally, a final weights quantization step leads to an improvement in the compression ratio. Our proposal (NIF) consistently outperforms state-of-art INR-based compressors in terms of PSNR, by achieving comparable or better results with an outstanding up to X26 encoding speed. We also show that NIF reduces the gap between INR-based methods with respect to traditional approaches. Interestingly, our approach outperforms established codecs such as JPEG and WebP when one encodes high-resolution images at low-bitrate regimes. Extensive experiments on different datasets, a visual comparison, and an ablation study, prove the validity of the proposed approach.
Integrating Physics-Informed Vectors for Improved Wind Speed Forecasting with Neural Networks
Authors: Laeeq Aslam, Runmin Zou, Ebrahim Awan, Sharjeel Abid Butt
Date: 2024-07-05 00:00:00
Proceeding: Asian Control Conference 2024 7

This paper introduces an approach to enhance wind speed prediction by integrating Physics-Informed Vectors with neural network architectures, specifically Long Short-Term Memory and Temporal Convolution Networks. It also proposes a hybrid decaying loss function aimed at improving the efficiency of model training and its predictive performance. The methodology utilizes physical knowledge and atmospheric data from various global locations to develop predictive models. Evaluations conducted using datasets from Canada, Chile, Kazakhstan, and Mongolia illustrate the advantages of including Physics-Informed Vectors. The incorporation of these vectors leads to improvements in Mean Squared Error, Mean Absolute Error, and R2 Score across different volumes of data. The analysis reveals improvements of up to 8.43% in Mean Absolute Error, 16.39% in Mean Squared Error, and 0.82% in R2 Score for models based on Long Short-Term Memory. For models based on Temporal Convolution Networks, improvements of up to 17.27% in Mean Absolute Error, 29.24% in Mean Squared Error, and 1.55% in R2 Score were observed. The introduction of a custom loss function, which merges mean squared error with Physics-Informed Vector estimates, aids in hastening the convergence of models by modulating the influence of these vectors during the training phase. The study underscores the effectiveness of incorporating physics-informed techniques into machine learning for predicting renewable energy sources, thereby opening paths for further research and application in this domain.