pwc-archive/papers-with-abstracts huggingface dataset:
Skill Learning Using Process Mining for Large Language Model Plan Generation
Authors: Andrei Cosmin Redis, Mohammadreza Fani Sani, Bahram Zarrin, Andrea Burattin
Date: 2024-10-14 00:00:00
Tasks: Decision Making, Language Modeling, Language Modelling, Large Language Model, Retrieval

Large language models (LLMs) hold promise for generating plans for complex tasks, but their effectiveness is limited by sequential execution, lack of control flow models, and difficulties in skill retrieval. Addressing these issues is crucial for improving the efficiency and interpretability of plan generation as LLMs become more central to automation and decision-making. We introduce a novel approach to skill learning in LLMs by integrating process mining techniques, leveraging process discovery for skill acquisition, process models for skill storage, and conformance checking for skill retrieval. Our methods enhance text-based plan generation by enabling flexible skill discovery, parallel execution, and improved interpretability. Experimental results suggest the effectiveness of our approach, with our skill retrieval method surpassing state-of-the-art accuracy baselines under specific conditions.
AI-Driven Autonomous Control of Proton-Boron Fusion Reactors Using Backpropagation Neural Networks
Authors: Michele Laurelli
Date: 2024-10-14 00:00:00

Proton-boron (p-11B) fusion presents a promising path towards sustainable, neutron-free energy generation. However, its implementation is hindered by extreme operational conditions, such as plasma temperatures exceeding billions of degrees and the complexity of controlling high-energy particles. Traditional control systems face significant challenges in managing the highly dynamic and non-linear behavior of the plasma. In this paper, we propose a novel approach utilizing backpropagation-based neural networks to autonomously control key parameters in a proton-boron fusion reactor. Our method leverages real-time feedback and learning from physical data to adapt to changing plasma conditions, offering a potential breakthrough in stable and efficient p-11B fusion. Furthermore, we expand on the scalability and generalization of our approach to other fusion systems and future AI technologies.
Not All Options Are Created Equal: Textual Option Weighting for Token-Efficient LLM-Based Knowledge Tracing
Authors: JongWoo Kim, SeongYeub Chu, Bryan Wong, Mun Yi
Date: 2024-10-14 00:00:00
Tasks: All, Binary Classification, In-Context Learning, Knowledge Tracing, Language Modeling, Language Modelling, Large Language Model, Multiple-choice

Large Language Models (LLMs) have recently emerged as promising tools for knowledge tracing (KT) due to their strong reasoning and generalization abilities. While recent LLM-based KT methods have proposed new prompt formats, they struggle to represent the full interaction histories of example learners within a single prompt during in-context learning (ICL), resulting in limited scalability and high computational cost under token constraints. In this work, we present \textit{LLM-based Option-weighted Knowledge Tracing (LOKT)}, a simple yet effective framework that encodes the interaction histories of example learners in context as \textit{textual categorical option weights (TCOW)}. TCOW are semantic labels (e.g., ``inadequate'') assigned to the options selected by learners when answering questions, enhancing the interpretability of LLMs. Experiments on multiple-choice datasets show that LOKT outperforms existing non-LLM and LLM-based KT models in both cold-start and warm-start settings. Moreover, LOKT enables scalable and cost-efficient inference, achieving strong performance even under strict token constraints. Our code is available at \href{https://anonymous.4open.science/r/LOKT_model-3233}{https://anonymous.4open.science/r/LOKT\_model-3233}.
Improving Instruction-Following in Language Models through Activation Steering
Authors: Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, Besmira Nushi
Date: 2024-10-15 00:00:00
Tasks: Instruction Following, Text Generation

The ability to follow instructions is crucial for numerous real-world applications of language models. In pursuit of deeper insights and more powerful capabilities, we derive instruction-specific vector representations from language models and use them to steer models accordingly. These vectors are computed as the difference in activations between inputs with and without instructions, enabling a modular approach to activation steering. We demonstrate how this method can enhance model adherence to constraints such as output format, length, and word inclusion, providing inference-time control over instruction following. Our experiments across four models demonstrate how we can use the activation vectors to guide models to follow constraints even without explicit instructions and to enhance performance when instructions are present. Additionally, we explore the compositionality of activation steering, successfully applying multiple instructions simultaneously. Finally, we demonstrate that steering vectors computed on instruction-tuned models can transfer to improve base models. Our findings demonstrate that activation steering offers a practical and scalable approach for fine-grained control in language generation.
Towards More Effective Table-to-Text Generation: Assessing In-Context Learning and Self-Evaluation with Open-Source Models
Authors: Sahar Iravani, Tim . O . F Conrad
Date: 2024-10-15 00:00:00
Tasks: In-Context Learning, Language Modeling, Language Modelling, Large Language Model, Table-to-Text Generation, Text Generation

Table processing, a key task in natural language processing, has significantly benefited from recent advancements in language models (LMs). However, the capabilities of LMs in table-to-text generation, which transforms structured data into coherent narrative text, require an in-depth investigation, especially with current open-source models. This study explores the effectiveness of various in-context learning strategies in LMs across benchmark datasets, focusing on the impact of providing examples to the model. More importantly, we examine a real-world use case, offering valuable insights into practical applications. To complement traditional evaluation metrics, we employ a large language model (LLM) self-evaluation approach using chain-of-thought reasoning and assess its correlation with human-aligned metrics like BERTScore. Our findings highlight the significant impact of examples in improving table-to-text generation and suggest that, while LLM self-evaluation has potential, its current alignment with human judgment could be enhanced. This points to the need for more reliable evaluation methods.
TextLap: Customizing Language Models for Text-to-Layout Planning
Authors: Jian Chen, Ruiyi Zhang, Yufan Zhou, Jennifer Healey, Jiuxiang Gu, Zhiqiang Xu, Changyou Chen
Date: 2024-10-09 00:00:00
Tasks: Image Generation, Natural Language Understanding

Automatic generation of graphical layouts is crucial for many real-world applications, including designing posters, flyers, advertisements, and graphical user interfaces. Given the incredible ability of Large language models (LLMs) in both natural language understanding and generation, we believe that we could customize an LLM to help people create compelling graphical layouts starting with only text instructions from the user. We call our method TextLap (text-based layout planning). It uses a curated instruction-based layout planning dataset (InsLap) to customize LLMs as a graphic designer. We demonstrate the effectiveness of TextLap and show that it outperforms strong baselines, including GPT-4 based methods, for image generation and graphical design benchmarks.
The Large Language Model GreekLegalRoBERTa
Authors: Vasileios Saketos, Despina-Athanasia Pantazi, Manolis Koubarakis
Date: 2024-10-10 00:00:00
Tasks: Language Modeling, Language Modelling, Large Language Model, model, named-entity-recognition, Named Entity Recognition, Topic Classification

We develop four versions of GreekLegalRoBERTa, which are four large language models trained on Greek legal and nonlegal text. We show that our models surpass the performance of GreekLegalBERT, Greek- LegalBERT-v2, and GreekBERT in two tasks involving Greek legal documents: named entity recognition and multi-class legal topic classification. We view our work as a contribution to the study of domain-specific NLP tasks in low-resource languages, like Greek, using modern NLP techniques and methodologies.
In-context KV-Cache Eviction for LLMs via Attention-Gate
Authors: Zihao Zeng, Bokai Lin, Tianqi Hou, Hao Zhang, Zhijie Deng
Date: 2024-10-15 00:00:00
Tasks: RTE

The KV-Cache technique has become the standard for the inference of large language models (LLMs). Yet, it is widely criticized that KV-Cache can become a bottleneck of the LLM inference system. This paper enables a novel dynamic KV-Cache eviction policy by injecting a lightweight module called Attention-Gate to the model. It accepts the global context as input and yields eviction flags for each token. The self-attention modules in the model proceed according to the flags and cache only a subset of the KV states for next token prediction. The Attention-Gates can yield various flags for different heads and layers and be easily tuned on top of a pre-trained LLM via continual pre-training or supervised fine-tuning. The computational and memory overhead introduced by Attention-Gates can be minimal. We empirically evaluate the proposed approach across multiple scenarios, showing that effective eviction of redundant tokens can not only improve efficiency but also enhance performance.
Generative Reward Models
Authors: Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, Alon Albalak
Date: 2024-10-02 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning

Reinforcement Learning from Human Feedback (RLHF) has greatly improved the performance of modern Large Language Models (LLMs). The RLHF process is resource-intensive and technically challenging, generally requiring a large collection of human preference labels over model-generated outputs. Reinforcement Learning from AI Feedback (RLAIF) addresses this data collection challenge by leveraging synthetic preferences generated by an LLM. However, recent work has shown that synthetic preferences labels may not align well with human preference judgments. To address this, we propose a hybrid approach that unifies RLHF and RLAIF methodologies. We introduce GenRM, an iterative algorithm that trains an LLM on self-generated reasoning traces, leading to synthetic preference labels matching human preference judgments. Empirically, we show that zero-shot LLM-based judgments under-perform compared to Bradley-Terry reward models on in-distribution tasks (between 9-36%). In contrast, GenRM achieves in-distribution accuracy comparable to Bradley-Terry models, while significantly outperforming them on out-of-distribution tasks (between 10-45%). Moreover, GenRM surpasses the performance of using LLMs as judges on both in-distribution (by 9-31%) and out-of-distribution tasks (by 2- 6%). Our results show that combining the strengths of RLHF and RLAIF offers a promising approach for improving the quality of synthetic preference labels.
Developing Guidelines for Functionally-Grounded Evaluation of Explainable Artificial Intelligence using Tabular Data
Authors: Mythreyi Velmurugan, Chun Ouyang, Yue Xu, Renuka Sindhgatta, Bemali Wickramanayake, Catarina Moreira
Date: 2024-09-30 00:00:00
Tasks: Explainable artificial intelligence, Explainable Artificial Intelligence (XAI)

Explainable Artificial Intelligence (XAI) techniques are used to provide transparency to complex, opaque predictive models. However, these techniques are often designed for image and text data, and it is unclear how fit-for-purpose they are when applied to tabular data. As XAI techniques are rarely evaluated in settings with tabular data, the applicability of existing evaluation criteria and methods are also unclear and needs (re-)examination. For example, some works suggest that evaluation methods may unduly influence the evaluation results when using tabular data. This lack of clarity on evaluation procedures can lead to reduced transparency and ineffective use of XAI techniques in real world settings. In this study, we examine literature on XAI evaluation to derive guidelines on functionally-grounded assessment of local, post hoc XAI techniques. We identify 20 evaluation criteria and associated evaluation methods, and derive guidelines on when and how each criterion should be evaluated. We also identify key research gaps to be addressed by future work. Our study contributes to the body of knowledge on XAI evaluation through in-depth examination of functionally-grounded XAI evaluation protocols, and has laid the groundwork for future research on XAI evaluation.
Hip Fracture Patient Pathways and Agent-based Modelling
Authors: Alison N. O'Connor, Stephen E. Ryan, Gauri Vaidya, Paul Harford, Meghana Kshirsagar
Date: 2024-09-30 00:00:00

Increased healthcare demand is significantly straining European services. Digital solutions including advanced modelling techniques offer a promising solution to optimising patient flow without impacting day-to-day healthcare provision. In this work we outline an ongoing project that aims to optimise healthcare resources using agent-based simulations.
Cerebral microbleeds: Association with cognitive decline and pathology build-up
Authors: Saima Rathore, Jatin Chaudhary, Boning Tong, Selen Bozkurt
Date: 2024-09-30 00:00:00
Tasks: Diagnostic

Cerebral microbleeds, markers of brain damage from vascular and amyloid pathologies, are linked to cognitive decline in aging, but their role in Alzheimer's disease (AD) onset and progression remains unclear. This study aimed to explore whether the presence and location of lobar microbleeds are associated with amyloid-$\beta$ (A$\beta$)-PET, tau tangle formation (tau-PET), and longitudinal cognitive decline. We analyzed 1,573 ADNI participants with MR imaging data and information on the number and location of microbleeds. Associations between lobar microbleeds and pathology, cerebrospinal fluid (CSF), genetics, and cognition were examined, focusing on regional microbleeds and domain-specific cognitive decline using ordinary least-squares regression while adjusting for covariates. Cognitive decline was assessed with ADAS-Cog11 and its domain-specific sub-scores. Participants underwent neuropsychological testing at least twice, with a minimum two-year interval between assessments. Among the 1,573 participants (692 women, mean age 71.23 years), 373 participants had microbleeds. The presence of microbleeds was linked to cognitive decline, particularly in the semantic, language, and praxis domains for those with temporal lobe microbleeds. Microbleeds in the overall cortex were associated with language decline. Pathologically, temporal lobe microbleeds were associated with increased tau in the overall cortex, while cortical microbleeds were linked to elevated A$\beta$ in the temporal, parietal, and frontal regions. In this mixed population, microbleeds were connected to longitudinal cognitive decline, especially in semantic and language domains, and were associated with higher baseline A$\beta$ and tau pathology. These findings suggest that lobar microbleeds should be included in AD diagnostic and prognostic evaluations.
Restoring Super-High Resolution GPS Mobility Data
Authors: Haruki Yonekura, Ren Ozeki, Hamada Rizk, Hirozumi Yamaguchi
Date: 2024-10-01 00:00:00
Tasks: Decoder, Synthetic Data Generation

This paper presents a novel system for reconstructing high-resolution GPS trajectory data from truncated or synthetic low-resolution inputs, addressing the critical challenge of balancing data utility with privacy preservation in mobility applications. The system integrates transformer-based encoder-decoder models with graph convolutional networks (GCNs) to effectively capture both the temporal dependencies of trajectory data and the spatial relationships in road networks. By combining these techniques, the system is able to recover fine-grained trajectory details that are lost through data truncation or rounding, a common practice to protect user privacy. We evaluate the system on the Beijing trajectory dataset, demonstrating its superior performance over traditional map-matching algorithms and LSTM-based synthetic data generation methods. The proposed model achieves an average Fr\'echet distance of 0.198 km, significantly outperforming map-matching algorithms (0.632 km) and synthetic trajectory models (0.498 km). The results show that the system is not only capable of accurately reconstructing real-world trajectories but also generalizes effectively to synthetic data. These findings suggest that the system can be deployed in urban mobility applications, providing both high accuracy and robust privacy protection.
Advancing Spatio-temporal Storm Surge Prediction with Hierarchical Deep Neural Networks
Authors: Saeed Saviz Naeini, Reda Snaiki, Teng Wu
Date: 2024-10-01 00:00:00
Tasks: Time Series

Coastal regions in North America face major threats from storm surges caused by hurricanes and nor'easters. Traditional numerical models, while accurate, are computationally expensive, limiting their practicality for real-time predictions. Recently, deep learning techniques have been developed for efficient simulation of time-dependent storm surge. To resolve the small scales of storm surge in both time and space over a long duration and a large area, these simulations typically need to employ oversized neural networks that struggle with the accumulation of prediction errors over successive time steps. To address these challenges, this study introduces a hierarchical deep neural network (HDNN) combined with a convolutional autoencoder (CAE) to accurately and efficiently predict storm surge time series. The CAE reduces the dimensionality of storm surge data, streamlining the learning process. HDNNs then map storm parameters to the low-dimensional representation of storm surge, allowing for sequential predictions across different time scales. Specifically, the current-level neural network is utilized to predict future states with a relatively large time step, which are passed as inputs to the next-level neural network for smaller time-step predictions. This process continues sequentially for all time steps. The results from different-level neural networks across various time steps are then stacked to acquire the entire time series of storm surge. The simulated low-dimensional representations are finally decoded back into storm surge time series. The proposed model was trained and tested using synthetic data from the North Atlantic Comprehensive Coastal Study. Results demonstrate its excellent performance to effectively handle high-dimensional surge data while mitigating the accumulation of prediction errors over time, making it a promising tool for advancing storm surge prediction.
Optimization of Actuarial Neural Networks with Response Surface Methodology
Authors: Belguutei Ariuntugs, Kehelwala Dewage Gayan Madurang
Date: 2024-10-01 00:00:00

In the data-driven world of actuarial science, machine learning (ML) plays a crucial role in predictive modeling, enhancing risk assessment and pricing strategies. Neural networks, specifically combined actuarial neural networks (CANN), are vital for tasks such as mortality forecasting and pricing. However, optimizing hyperparameters (e.g., learning rates, layers) is essential for resource efficiency. This study utilizes a factorial design and response surface methodology (RSM) to optimize CANN performance. RSM effectively explores the hyperparameter space and captures potential curvature, outperforming traditional grid search. Our results show accurate performance predictions, identifying critical hyperparameters. By dropping statistically insignificant hyperparameters, we reduced runs from 288 to 188, with negligible loss in accuracy, achieving near-optimal out-of-sample Poisson deviance loss.
TIMeSynC: Temporal Intent Modelling with Synchronized Context Encodings for Financial Service Applications
Authors: Dwipam Katariya, Juan Manuel Origgi, Yage Wang, Thomas Caputo
Date: 2024-10-01 00:00:00
Tasks: Decoder, NMT, Prediction, Representation Learning

Users engage with financial services companies through multiple channels, often interacting with mobile applications, web platforms, call centers, and physical locations to service their accounts. The resulting interactions are recorded at heterogeneous temporal resolutions across these domains. This multi-channel data can be combined and encoded to create a comprehensive representation of the customer's journey for accurate intent prediction. This demands sequential learning solutions. NMT transformers achieve state-of-the-art sequential representation learning by encoding context and decoding for the next best action to represent long-range dependencies. However, three major challenges exist while combining multi-domain sequences within an encoder-decoder transformers architecture for intent prediction applications: a) aligning sequences with different sampling rates b) learning temporal dynamics across multi-variate, multi-domain sequences c) combining dynamic and static sequences. We propose an encoder-decoder transformer model to address these challenges for contextual and sequential intent prediction in financial servicing applications. Our experiments show significant improvement over the existing tabular method.
Information Importance-Aware Defense against Adversarial Attack for Automatic Modulation Classification:An XAI-Based Approach
Authors: Jingchun Wang, Peihao Dong, Fuhui Zhou, Qihui Wu
Date: 2024-10-15 00:00:00
Tasks: Adversarial Attack, Classification, Explainable artificial intelligence, Explainable Artificial Intelligence (XAI)

Deep learning (DL) has significantly improved automatic modulation classification (AMC) by leveraging neural networks as the feature extractor.However, as the DL-based AMC becomes increasingly widespread, it is faced with the severe secure issue from various adversarial attacks. Existing defense methods often suffer from the high computational cost, intractable parameter tuning, and insufficient robustness.This paper proposes an eXplainable artificial intelligence (XAI) defense approach, which uncovers the negative information caused by the adversarial attack through measuring the importance of input features based on the SHapley Additive exPlanations (SHAP).By properly removing the negative information in adversarial samples and then fine-tuning(FT) the model, the impact of the attacks on the classification result can be mitigated.Experimental results demonstrate that the proposed SHAP-FT improves the classification performance of the model by 15%-20% under different attack levels,which not only enhances model robustness against various attack levels but also reduces the resource consumption, validating its effectiveness in safeguarding communication networks.
A Control Theoretic Study on Omnidirectional MAVs with Minimum Number of Actuators and No Internal Forces at Any Orientation
Authors: Ahmed Ali, Chiara Gabellieri, Antonio Franchi
Date: 2024-10-16 00:00:00

We propose a new multirotor aerial vehicle class of designs composed of a multi-body structure in which a main body is connected by passive joints to links equipped with propellers. We have investigated some instances of such class, some of which are shown to achieve omnidirectionality while having a minimum number of inputs equal to the main body Degrees of Freedom DoF's, only uni-directional positive thrust propellers, and no internal forces generated at steady state. After dynamics are derived following the Euler-Lagrange approach, an I/O dynamic feedback linearization strategy is then used to show the controllability of any desired pose with stable zero dynamics. We finally verify the developed controller with closed-loop simulations.