pwc-archive/papers-with-abstracts huggingface dataset:
Risk forecasting using Long Short-Term Memory Mixture Density Networks
Authors: Nico Herrig
Date: 2025-01-02 00:00:00
Tasks: Clustering

This work aims to implement Long Short-Term Memory mixture density networks (LSTM-MDNs) for Value-at-Risk forecasting and compare their performance with established models (historical simulation, CMM, and GARCH) using a defined backtesting procedure. The focus was on the neural network's ability to capture volatility clustering and its real-world applicability. Three architectures were tested: a 2-component mixture density network, a regularized 2-component model (Arimond et al., 2020), and a 3-component mixture model, the latter being tested for the first time in Value-at-Risk forecasting. Backtesting was performed on three stock indices (FTSE 100, S&P 500, EURO STOXX 50) over two distinct two-year periods (2017-2018 as a calm period, 2021-2022 as turbulent). Model performance was assessed through unconditional coverage and independence assumption tests. The neural network's ability to handle volatility clustering was validated via correlation analysis and graphical evaluation. Results show limited success for the neural network approach. LSTM-MDNs performed poorly for 2017/2018 but outperformed benchmark models in 2021/2022. The LSTM mechanism allowed the neural network to capture volatility clustering similarly to GARCH models. However, several issues were identified: the need for proper model initialization and reliance on large datasets for effective learning. The findings suggest that while LSTM-MDNs provide adequate risk forecasts, further research and adjustments are necessary for stable performance.
Conditional Consistency Guided Image Translation and Enhancement
Authors: Amil Bhagat, Milind Jain, A. V. Subramanyam
Date: 2025-01-02 00:00:00
Tasks: Denoising, Image Enhancement, Low-Light Image Enhancement, Translation

Consistency models have emerged as a promising alternative to diffusion models, offering high-quality generative capabilities through single-step sample generation. However, their application to multi-domain image translation tasks, such as cross-modal translation and low-light image enhancement remains largely unexplored. In this paper, we introduce Conditional Consistency Models (CCMs) for multi-domain image translation by incorporating additional conditional inputs. We implement these modifications by introducing task-specific conditional inputs that guide the denoising process, ensuring that the generated outputs retain structural and contextual information from the corresponding input domain. We evaluate CCMs on 10 different datasets demonstrating their effectiveness in producing high-quality translated images across multiple domains. Code is available at https://github.com/amilbhagat/Conditional-Consistency-Models.
An Efficient Attention Mechanism for Sequential Recommendation Tasks: HydraRec
Authors: Uzma Mushtaque
Date: 2025-01-02 00:00:00
Tasks: Language Modelling, Recommendation Systems, Sequential Recommendation

Transformer based models are increasingly being used in various domains including recommender systems (RS). Pretrained transformer models such as BERT have shown good performance at language modelling. With the greater ability to model sequential tasks, variants of Encoder-only models (like BERT4Rec, SASRec etc.) have found success in sequential RS problems. Computing dot-product attention in traditional transformer models has quadratic complexity in sequence length. This is a bigger problem with RS because unlike language models, new items are added to the catalogue every day. User buying history is a dynamic sequence which depends on multiple factors. Recently, various linear attention models have tried to solve this problem by making the model linear in sequence length (token dimensions). Hydra attention is one such linear complexity model proposed for vision transformers which reduces the complexity of attention for both the number of tokens as well as model embedding dimensions. Building on the idea of Hydra attention, we introduce an efficient Transformer based Sequential RS (HydraRec) which significantly improves theoretical complexity of computing attention for longer sequences and bigger datasets while preserving the temporal context. Extensive experiments are conducted to evaluate other linear transformer-based RS models and compared with HydraRec across various evaluation metrics. HydraRec outperforms other linear attention-based models as well as dot-product based attention models when used with causal masking for sequential recommendation next item prediction tasks. For bi-directional models its performance is comparable to the BERT4Rec model with an improvement in running time.
Superposition in Transformers: A Novel Way of Building Mixture of Experts
Authors: Ayoub Ben Chaliah, Hela Dellagi
Date: 2024-12-31 00:00:00
Tasks: Mixture-of-Experts

Catastrophic forgetting remains a major challenge when adapting large language models (LLMs) to new tasks or domains. Conventional fine-tuning often overwrites existing knowledge, causing performance degradation on original tasks. We introduce Superposition in Transformers, a novel architecture that leverages autoencoders to superimpose the hidden representations of a base model and a fine-tuned model within a shared parameter space. By using B-spline-based blending coefficients and autoencoders that adaptively reconstruct hidden states based on the input data distribution, our method effectively mitigates catastrophic forgetting and enables a new paradigm of "in-model" superposition. This approach preserves original model capabilities while allowing compact domain-specific expertise to be added, and it supports dynamic switching between model states during inference.
Cached Adaptive Token Merging: Dynamic Token Reduction and Redundant Computation Elimination in Diffusion Model
Authors: Omid Saghatchian, Atiyeh Gh. Moghadam, Ahmad Nickabadi
Date: 2025-01-01 00:00:00
Tasks: Denoising, Token Reduction

Diffusion models have emerged as a promising approach for generating high-quality, high-dimensional images. Nevertheless, these models are hindered by their high computational cost and slow inference, partly due to the quadratic computational complexity of the self-attention mechanisms with respect to input size. Various approaches have been proposed to address this drawback. One such approach focuses on reducing the number of tokens fed into the self-attention, known as token merging (ToMe). In our method, which is called cached adaptive token merging(CA-ToMe), we calculate the similarity between tokens and then merge the r proportion of the most similar tokens. However, due to the repetitive patterns observed in adjacent steps and the variation in the frequency of similarities, we aim to enhance this approach by implementing an adaptive threshold for merging tokens and adding a caching mechanism that stores similar pairs across several adjacent steps. Empirical results demonstrate that our method operates as a training-free acceleration method, achieving a speedup factor of 1.24 in the denoising process while maintaining the same FID scores compared to existing approaches.
Research on regional carbon emission scenario simulation based on GA-BP-KDE under uncertain conditions
Authors: Ke Pan a, Bin Liu a, B, *, Zuli He a, Jian Liu c, Xinyu Xia a, Xueqi Zhang a, Yang Wei d, Siying Wang b, Huangyuying Zheng b
Date: 2025-02-01 00:00:00
Proceeding: 大气污染研究 2025 2

The continuous advancement of the “dual carbon” goal presents a persistent challenge for regions striving to
balance rapid economic growth with sustainable development. Consequently, investigating scientifically efficient
regional carbon emission reduction pathways has become increasingly significant. This study utilizes carbon
emission data and related impact factors from 21 cities in Sichuan province spanning 2011 to 2022 to construct a
GA-BP-KDE carbon emission interval prediction model. This model, based on the GA-BP point prediction model,
accounts for the inherent uncertainty in carbon emission forecasting. Eight scenarios were employed to predict
carbon emission intervals and potential “carbon peak” timeframes for Sichuan province from 2021 to 2040.
These scenarios were categorized into three groups: “peak,” “possible peak,” and “unable to peak.” Subsequently,
a tailored cascade peak pathway was developed for the 21 cities in Sichuan province. The findings reveal: (1)
Pearson coefficient correlation measure results indicate that the overall performance of the genetic algorithm-
Back propagation network (GA-BP) model (0.985) surpassed that of the Extreme Learning Machine (ELM)
(0.900), Random Forest (RF) (0.981), and support vector machine (SVR) (0.982). The validity of the GA-BP-KDE
interval prediction model was evaluated using PICP and PINAW metrics. (2) Under baseline, ideal, and low-
carbon scenarios, Sichuan province is projected to definitively reach its carbon peak by 2030. In the weak
emission reduction scenario, a 2030 peak remains possible. (3) The 21 cities in Sichuan province were classified
into three echelons based on their average peak times, with Chengdu and Mianyang positioned in the first
echelon. This research introduces a novel approach to studying “carbon peak” pathways, potentially providing
decision-makers with more reliable and comprehensive information for formulating rational and effective carbon
emission reduction measures.
Sustainability of underground infrastructure – Part 1: Digitalisation-based carbon assessment and baseline for TBM tunnelling
Authors: Xilin Chen a, Mengqi Huang b, Yu Bai a, Qian-Bing Zhang a
Date: 2024-06-01 00:00:00
Proceeding: Tunnelling and Underground Space Technology 2024 6

Alongside the transition towards sustainable and renewable infrastructure and energy production, tunnelling projects that adopt tunnel boring machines (TBM) continue to increase in size and complexity. Life Cycle Assessment (LCA) methodologies have a significant impact on the Architecture, Engineering and Construction (AEC) industry, with a shift in focus to cost and carbon management across the infrastructure lifecycle through the use of digitalisation techniques and sustainability rating systems. This paper aims to develop a systematic framework to enhance the sustainability of underground infrastructure in the early design and construction stages, aiming for “build clever and efficiently”. This framework integrates digitalisation, carbon assessment standards, numerical modelling, and optimisation techniques to assess carbon emissions and establish benchmarks for construction products and processes. The approach uses parametric models to define tunnelling-related entities such as geological formations, TBM, and segments, designed with dimensional constraints that allow for adaptive modelling and interactions. Enabled by building information modelling (BIM) through parametric models enriched with domain properties, module-based carbon accounting is adopted to evaluate the footprint of material- and asset-level key emitters with functional units. Leveraging established databases, baselines for segmental lining designs and carbon reduction strategies are established. Through project-driven case studies, the correlations among geotechnical conditions, tunnel dimensions, and TBM operating parameters are developed to quantify TBM carbon emissions. A prototype programme demonstrates the framework's application on typical metro and road tunnels at different levels of details (LoD). This paper serves as a guideline for conducting embodied carbon assessments specifically tailored to TBM tunnelling, acknowledging the substantial contribution of the TBM construction process, which is controlled by geotechnical conditions.
Momentum-based minimization of the Ginzburg-Landau functional on Euclidean spaces and graphs
Authors: Oluwatosin Akande, Patrick Dondl, Kanan Gupta, Akwum Onwunta, Stephan Wojtowytsch
Date: 2024-12-31 00:00:00

We study the momentum-based minimization of a diffuse perimeter functional on Euclidean spaces and on graphs with applications to semi-supervised classification tasks in machine learning. While the gradient flow in the task at hand is a parabolic partial differential equation, the momentum method corresponds to a damped hyperbolic PDE, leading to qualitatively and quantitatively different trajectories. Using a convex-concave splitting-based FISTA-type time discretization, we demonstrate empirically that momentum can lead to faster convergence if the time step size is large but not too large. With large time steps, the PDE analysis offers only limited insight into the geometric behavior of solutions and typical hyperbolic phenomena like loss of regularity are not be observed in sample simulations. We obtain the singular limit of the evolution equations as the length parameter of the phase fields tends to zero by formal expansions and numerically confirm its validity for circles in two dimensions. Our analysis is complemented by numerical experiments for planar curves, surfaces in three-dimensional space, and semi-supervised learning tasks on graphs.
Dementia Detection using Multi-modal Methods on Audio Data
Authors: Saugat Kannojia, Anirudh Praveen, Danish Vasdev, Saket Nandedkar, Divyansh Mittal, Sarthak Kalankar, Shaurya Johari, Vipul Arora
Date: 2024-12-31 00:00:00

Dementia is a neurodegenerative disease that causes gradual cognitive impairment, which is very common in the world and undergoes a lot of research every year to prevent and cure it. It severely impacts the patient's ability to remember events and communicate clearly, where most variations of it have no known cure, but early detection can help alleviate symptoms before they become worse. One of the main symptoms of dementia is difficulty in expressing ideas through speech. This paper attempts to talk about a model developed to predict the onset of the disease using audio recordings from patients. An ASR-based model was developed that generates transcripts from the audio files using Whisper model and then applies RoBERTa regression model to generate an MMSE score for the patient. This score can be used to predict the extent to which the cognitive ability of a patient has been affected. We use the PROCESS_V1 dataset for this task, which is introduced through the PROCESS Grand Challenge 2025. The model achieved an RMSE score of 2.6911 which is around 10 percent lower than the described baseline.
A3: Android Agent Arena for Mobile GUI Agents
Authors: Yuxiang Chai, Hanhao Li, Jiayu Zhang, Liang Liu, Guozhi Wang, Shuai Ren, Siyuan Huang, Hongsheng Li
Date: 2025-01-02 00:00:00
Tasks: Information Retrieval

AI agents have become increasingly prevalent in recent years, driven by significant advancements in the field of large language models (LLMs). Mobile GUI agents, a subset of AI agents, are designed to autonomously perform tasks on mobile devices. While numerous studies have introduced agents, datasets, and benchmarks to advance mobile GUI agent research, many existing datasets focus on static frame evaluations and fail to provide a comprehensive platform for assessing performance on real-world, in-the-wild tasks. To address this gap, we present Android Agent Arena (A3), a novel evaluation platform. Unlike existing in-the-wild systems, A3 offers: (1) meaningful and practical tasks, such as real-time online information retrieval and operational instructions; (2) a larger, more flexible action space, enabling compatibility with agents trained on any dataset; and (3) automated business-level LLM-based evaluation process. A3 includes 21 widely used general third-party apps and 201 tasks representative of common user scenarios, providing a robust foundation for evaluating mobile GUI agents in real-world situations and a new autonomous evaluation process for less human labor and coding expertise. The project is available at \url{https://yuxiangchai.github.io/Android-Agent-Arena/}.
Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs
Authors: Dibakar Gope, David Mansell, Danny Loh, Ian Bratt
Date: 2024-12-23 00:00:00
Tasks: Quantization

Large language models (LLMs) have transformed the way we think about language understanding and generation, enthralling both researchers and developers. However, deploying LLMs for inference has been a significant challenge due to their unprecedented size and resource requirements. While quantizing model weights to sub-byte precision has emerged as a promising solution to ease memory pressure, the group quantization formats commonly used for LLM quantization have significant compute overheads and a resource-intensive dequantization process. As a result, a higher proportion of compute instructions do not perform multiplies, i.e., real work, rendering them unsuitable for meeting the required latency requirements for LLMs deployed on commodity CPUs. In this work, we propose a set of highly optimized kernels to accelerate LLM inference and unleash the full potential of CPUs, particularly Arm CPUs. These kernels amortize the cost of loading the operands and the cost of weight unpacking across multiple output rows. This, along with the introduction of an optimized interleaved group data layout for weights and decompression path optimizations to reduce unnecessary operations and dequantization overhead while maximizing the use of vector and matrix multiply operations, significantly improves the efficiency of MAC operations. Furthermore, we present a groupwise non-uniform codebook-based quantization method for ultra-low-precision quantization of LLMs to better match non-uniform patterns in their weight distributions, demonstrating better throughput during token generation while ensuring better quality than the state-of-the-art. Applying these improvements to 4-bit LLMs results in a 3-3.2x improvement in prompt processing and a 2x improvement in autoregressive decoding on Arm CPUs, compared to LLaMA.cpp-based solution. The optimized kernels are available at https://github.com/ggerganov/llama.cpp.
Numerical solutions of fixed points in two-dimensional Kuramoto-Sivashinsky equation expedited by reinforcement learning
Authors: Juncheng Jiang, Dongdong Wan, Mengqi Zhang
Date: 2024-12-27 00:00:00
Tasks: Deep Reinforcement Learning, Navigate, reinforcement-learning, Reinforcement Learning

This paper presents a combined approach to enhancing the effectiveness of Jacobian-Free Newton-Krylov (JFNK) method by deep reinforcement learning (DRL) in identifying fixed points within the 2D Kuramoto-Sivashinsky Equation (KSE). JFNK approach entails a good initial guess for improved convergence when searching for fixed points. With a properly defined reward function, we utilise DRL as a preliminary step to enhance the initial guess in the converging process. We report new results of fixed points in the 2D KSE which have not been reported in the literature. Additionally, we explored control optimization for the 2D KSE to navigate the system trajectories between known fixed points, based on parallel reinforcement learning techniques. This combined method underscores the improved JFNK approach to finding new fixed-point solutions within the context of 2D KSE, which may be instructive for other high-dimensional dynamical systems.
Stroke Prediction using Clinical and Social Features in Machine Learning
Authors: Aidan Chadha
Date: 2024-12-27 00:00:00
Tasks: Prediction, regression

Every year in the United States, 800,000 individuals suffer a stroke - one person every 40 seconds, with a death occurring every four minutes. While individual factors vary, certain predictors are more prevalent in determining stroke risk. As strokes are the second leading cause of death and disability worldwide, predicting stroke likelihood based on lifestyle factors is crucial. Showing individuals their stroke risk could motivate lifestyle changes, and machine learning offers solutions to this prediction challenge. Neural networks excel at predicting outcomes based on training features like lifestyle factors, however, they're not the only option. Logistic regression models can also effectively compute the likelihood of binary outcomes based on independent variables, making them well-suited for stroke prediction. This analysis will compare both neural networks (dense and convolutional) and logistic regression models for stroke prediction, examining their pros, cons, and differences to develop the most effective predictor that minimizes false negatives.
DDD-GenDT: Dynamic Data-driven Generative Digital Twin Framework
Authors: Yu-Zheng Lin, Qinxuan Shi, Zhanglong Yang, Banafsheh Saber Latibari, Sicong Shao, Soheil Salehi, Pratik Satam
Date: 2024-12-28 00:00:00

Digital twin (DT) technology has emerged as a transformative approach to simulate, predict, and optimize the behavior of physical systems, with applications that span manufacturing, healthcare, climate science, and more. However, the development of DT models often faces challenges such as high data requirements, integration complexity, and limited adaptability to dynamic changes in physical systems. This paper presents a new method inspired by dynamic data-driven applications systems (DDDAS), called the dynamic data-driven generative of digital twins framework (DDD-GenDT), which combines the physical system with LLM, allowing LLM to act as DT to interact with the physical system operating status and generate the corresponding physical behaviors. We apply DDD-GenDT to the computer numerical control (CNC) machining process, and we use the spindle current measurement data in the NASA milling wear data set as an example to enable LLMs to forecast the physical behavior from historical data and interact with current observations. Experimental results show that in the zero-shot prediction setting, the LLM-based DT can adapt to the change in the system, and the average RMSE of the GPT-4 prediction is 0.479A, which is 4.79% of the maximum spindle motor current measurement of 10A, with little training data and instructions required. Furthermore, we analyze the performance of DDD-GenDT in this specific application and their potential to construct digital twins. We also discuss the limitations and challenges that may arise in practical implementations.