pwc-archive/papers-with-abstracts huggingface dataset:
Multimodal Quantum Natural Language Processing: A Novel Framework for using Quantum Methods to Analyse Real Data
Authors: Hala Hawashin
Date: 2024-10-29 00:00:00
Tasks: Data Integration, Image-text Classification, Language Modeling, Language Modelling, text-classification, Text Classification

Despite significant advances in quantum computing across various domains, research on applying quantum approaches to language compositionality - such as modeling linguistic structures and interactions - remains limited. This gap extends to the integration of quantum language data with real-world data from sources like images, video, and audio. This thesis explores how quantum computational methods can enhance the compositional modeling of language through multimodal data integration. Specifically, it advances Multimodal Quantum Natural Language Processing (MQNLP) by applying the Lambeq toolkit to conduct a comparative analysis of four compositional models and evaluate their influence on image-text classification tasks. Results indicate that syntax-based models, particularly DisCoCat and TreeReader, excel in effectively capturing grammatical structures, while bag-of-words and sequential models struggle due to limited syntactic awareness. These findings underscore the potential of quantum methods to enhance language modeling and drive breakthroughs as quantum technology evolves.
Deep Learning and Machine Learning -- Natural Language Processing: From Theory to Application
Authors: Keyu Chen, Cheng Fei, Ziqian Bi, Junyu Liu, Benji Peng, Sen Zhang, Xuanhe Pan, Jiawei Xu, Jinlang Wang, Caitlyn Heqi Yin, Yichao Zhang, Pohsun Feng, Yizhu Wen, Tianyang Wang, Ming Li, Jintao Ren, Qian Niu, Silin Chen, Weiche Hsieh, Lawrence K. Q. Yan, Chia Xin Liang, Han Xu, Hong-Ming Tseng, Xinyuan Song, Ming Liu
Date: 2024-10-30 00:00:00
Tasks: text-classification, Text Classification

With a focus on natural language processing (NLP) and the role of large language models (LLMs), we explore the intersection of machine learning, deep learning, and artificial intelligence. As artificial intelligence continues to revolutionize fields from healthcare to finance, NLP techniques such as tokenization, text classification, and entity recognition are essential for processing and understanding human language. This paper discusses advanced data preprocessing techniques and the use of frameworks like Hugging Face for implementing transformer-based models. Additionally, it highlights challenges such as handling multilingual data, reducing bias, and ensuring model robustness. By addressing key aspects of data processing and model fine-tuning, this work aims to provide insights into deploying effective and ethically sound AI solutions.
Towards Probabilistic Planning of Explanations for Robot Navigation
Authors: Amar Halilovic, Senka Krivic
Date: 2024-10-26 00:00:00
Tasks: Robot Navigation

In robotics, ensuring that autonomous systems are comprehensible and accountable to users is essential for effective human-robot interaction. This paper introduces a novel approach that integrates user-centered design principles directly into the core of robot path planning processes. We propose a probabilistic framework for automated planning of explanations for robot navigation, where the preferences of different users regarding explanations are probabilistically modeled to tailor the stochasticity of the real-world human-robot interaction and the communication of decisions of the robot and its actions towards humans. This approach aims to enhance the transparency of robot path planning and adapt to diverse user explanation needs by anticipating the types of explanations that will satisfy individual users.
Scattered Forest Search: Smarter Code Space Exploration with LLMs
Authors: Jonathan Light, Yue Wu, Yiyou Sun, Wenchao Yu, Yanchi Liu, Xujiang Zhao, Ziniu Hu, Haifeng Chen, Wei Cheng
Date: 2024-10-22 00:00:00
Tasks: Code Generation, Diversity, Efficient Exploration, HumanEval, mbpp

We frame code generation as a black-box optimization problem within the code space and demonstrate how optimization-inspired techniques can enhance inference scaling. Based on this perspective, we propose SCATTERED FOREST SEARCH (SFS), a novel approach that improves solution diversity and better exploits feedback during evolutionary search. Our theoretical analysis illustrates how these methods help avoid local optima during optimization, leading to more efficient exploration. Extensive experiments on HumanEval, MBPP, APPS, CodeContests, and Leetcode reveal significant performance gains. For instance, our method achieves a pass@1 rate of 67.1% on HumanEval+ and 87.2% on HumanEval with GPT-3.5, marking improvements of 8.6% and 4.3% over the state-of-the-art, while also halving the iterations needed to find the correct solution. Furthermore, our approach scales more efficiently than existing search techniques, including tree search, line search, and repeated sampling.
Reservoir computing for system identification and predictive control with limited data
Authors: Jan P. Williams, J. Nathan Kutz, Krithika Manohar
Date: 2024-10-23 00:00:00
Tasks: Model Predictive Control, valid

Model predictive control (MPC) is an industry standard control technique that iteratively solves an open-loop optimization problem to guide a system towards a desired state or trajectory. Consequently, an accurate forward model of system dynamics is critical for the efficacy of MPC and much recent work has been aimed at the use of neural networks to act as data-driven surrogate models to enable MPC. Perhaps the most common network architecture applied to this task is the recurrent neural network (RNN) due to its natural interpretation as a dynamical system. In this work, we assess the ability of RNN variants to both learn the dynamics of benchmark control systems and serve as surrogate models for MPC. We find that echo state networks (ESNs) have a variety of benefits over competing architectures, namely reductions in computational complexity, longer valid prediction times, and reductions in cost of the MPC objective function.
Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks
Authors: Pranav Guruprasad, Harshvardhan Sikka, Jaewoo Song, Yangyue Wang, Paul Pu Liang
Date: 2024-11-04 00:00:00
Tasks: Action Generation, Benchmarking, Prompt Engineering, Vision-Language-Action

Vision-language-action (VLA) models represent a promising direction for developing general-purpose robotic systems, demonstrating the ability to combine visual understanding, language comprehension, and action generation. However, systematic evaluation of these models across diverse robotic tasks remains limited. In this work, we present a comprehensive evaluation framework and benchmark suite for assessing VLA models. We profile three state-of-the-art VLM and VLAs - GPT-4o, OpenVLA, and JAT - across 20 diverse datasets from the Open-X-Embodiment collection, evaluating their performance on various manipulation tasks. Our analysis reveals several key insights: 1. current VLA models show significant variation in performance across different tasks and robot platforms, with GPT-4o demonstrating the most consistent performance through sophisticated prompt engineering, 2. all models struggle with complex manipulation tasks requiring multi-step planning, and 3. model performance is notably sensitive to action space characteristics and environmental factors. We release our evaluation framework and findings to facilitate systematic assessment of future VLA models and identify critical areas for improvement in the development of general purpose robotic systems.
GlocalCLIP: Object-agnostic Global-Local Prompt Learning for Zero-shot Anomaly Detection
Authors: Jiyul Ham, Yonggon Jung, Jun-Geol Baek
Date: 2024-11-09 00:00:00
Tasks: Anomaly Detection, Contrastive Learning, Prompt Learning, zero-shot anomaly detection

Zero-shot anomaly detection (ZSAD) is crucial for detecting anomalous patterns in target datasets without using training samples, specifically in scenarios where there are distributional differences between the target domain and training data or where data scarcity arises because of restricted access. Although recently pretrained vision-language models demonstrate strong zero-shot performance across various visual tasks, they focus on learning class semantics, which makes their direct application to ZSAD challenging. To address this scenario, we propose GlocalCLIP, which uniquely separates global and local prompts and jointly optimizes them. This approach enables the object-agnostic glocal semantic prompt to effectively capture general normal and anomalous patterns without dependency on specific objects in the image. We refine the text prompts for more precise adjustments by utilizing deep-text prompt tuning in the text encoder. In the vision encoder, we apply V-V attention layers to capture detailed local image features. Finally, we introduce glocal contrastive learning to improve the complementary learning of global and local prompts, effectively detecting anomalous patterns across various domains. The generalization performance of GlocalCLIP in ZSAD was demonstrated on 15 real-world datasets from both the industrial and medical domains, achieving superior performance compared to existing methods. Code will be made available at https://github.com/YUL-git/GlocalCLIP.
A capacity renting framework for shared energy storage considering peer-to-peer energy trading of prosumers with privacy protection
Authors: Yingcong Sun, Laijun Chen, Yue Chen, Mingrui Tang, Shengwei Mei
Date: 2024-11-09 00:00:00
Tasks: energy trading, Privacy Preserving

Shared energy storage systems (ESS) present a promising solution to the temporal imbalance between energy generation from renewable distributed generators (DGs) and the power demands of prosumers. However, as DG penetration rates rise, spatial energy imbalances become increasingly significant, necessitating the integration of peer-to-peer (P2P) energy trading within the shared ESS framework. Two key challenges emerge in this context: the absence of effective mechanisms and the greater difficulty for privacy protection due to increased data communication. This research proposes a capacity renting framework for shared ESS considering P2P energy trading of prosumers. In the proposed framework, prosumers can participate in P2P energy trading and rent capacities from shared ESS. A generalized Nash game is formulated to model the trading process and the competitive interactions among prosumers, and the variational equilibrium of the game is proved to be equivalent to the optimal solution of a quadratic programming (QP) problem. To address the privacy protection concern, the problem is solved using the alternating direction method of multipliers (ADMM) with the Paillier cryptosystem. Finally, numerical simulations demonstrate the impact of P2P energy trading on the shared ESS framework and validate the effectiveness of the proposed privacy-preserving algorithm.
One Small and One Large for Document-level Event Argument Extraction
Authors: Jiaren Peng, Hongda Sun, Wenzhong Yang, Fuyuan Wei, Liang He, Liejun Wang
Date: 2024-11-08 00:00:00
Tasks: Event Argument Extraction

Document-level Event Argument Extraction (EAE) faces two challenges due to increased input length: 1) difficulty in distinguishing semantic boundaries between events, and 2) interference from redundant information. To address these issues, we propose two methods. The first method introduces the Co and Structure Event Argument Extraction model (CsEAE) based on Small Language Models (SLMs). CsEAE includes a co-occurrences-aware module, which integrates information about all events present in the current input through context labeling and co-occurrences event prompts extraction. Additionally, CsEAE includes a structure-aware module that reduces interference from redundant information by establishing structural relationships between the sentence containing the trigger and other sentences in the document. The second method introduces new prompts to transform the extraction task into a generative task suitable for Large Language Models (LLMs), addressing gaps in EAE performance using LLMs under Supervised Fine-Tuning (SFT) conditions. We also fine-tuned multiple datasets to develop an LLM that performs better across most datasets. Finally, we applied insights from CsEAE to LLMs, achieving further performance improvements. This suggests that reliable insights validated on SLMs are also applicable to LLMs. We tested our models on the Rams, WikiEvents, and MLEE datasets. The CsEAE model achieved improvements of 2.1\%, 2.3\%, and 3.2\% in the Arg-C F1 metric compared to the baseline, PAIE~\cite{PAIE}. For LLMs, we demonstrated that their performance on document-level datasets is comparable to that of SLMs~\footnote{All code is available at https://github.com/simon-p-j-r/CsEAE}.
Structure-Aware Single-Source Generalization with Pixel-Level Disentanglement for Joint Optic Disc and Cup Segmentation
Authors: Jia-Xuan Jiang, Yuee Li, Zhong Wang
Date: 2024-09-10 00:00:00
Proceeding: Biomedical Signal Processing and Control 2024 9
Tasks: Disentanglement, Domain Generalization, Medical Image Analysis, Medical Image Segmentation, Single-Source Domain Generalization

Deploying deep segmentation models in new medical centers poses a significant challenge due to statistical disparities between source and unknown domains. Recent advancements in domain generalization (DG) have shown improved generalization performance by leveraging disentanglement techniques on domain-specific and domain-invariant features. However, existing DG methods face challenges in achieving optimal feature segregation. To address this, we introduce a pixel-level contrastive single domain generalization (PCSDG) framework and a structure-aware brightness augmentation (SABA) technique for joint optic disc and cup segmentation. First, a disentanglement module captures content and style-related maps, pixel-wise multiplied with the original image to generate saliency-based attention maps, resulting in distinct structure and style representations. Second, Using a contrastive loss in the latent space enhances segregation. Finally, SABA introduces random brightness variations, preserving anatomical information and diversifying sample styles. Experimental validation on two public fundus image datasets with two source domains and five target domains demonstrates the superior performance of PCSDG and SABA across diverse domains when compared to state-of-the-art methods. Our code and models are made public at: https://github.com/HopkinsKwong/PCSDG
Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models
Authors: XiaoJun Wu, Junxi Liu, Huanyi Su, Zhouchi Lin, Yiyan Qi, Chengjin Xu, Jiajun Su, Jiajie Zhong, Fuwei Wang, Saizhuo Wang, Fengrui Hua, Jia Li, Jian Guo
Date: 2024-11-09 00:00:00

As large language models become increasingly prevalent in the financial sector, there is a pressing need for a standardized method to comprehensively assess their performance. However, existing finance benchmarks often suffer from limited language and task coverage, as well as challenges such as low-quality datasets and inadequate adaptability for LLM evaluation. To address these limitations, we propose "Golden Touchstone", the first comprehensive bilingual benchmark for financial LLMs, which incorporates representative datasets from both Chinese and English across eight core financial NLP tasks. Developed from extensive open source data collection and industry-specific demands, this benchmark includes a variety of financial tasks aimed at thoroughly assessing models' language understanding and generation capabilities. Through comparative analysis of major models on the benchmark, such as GPT-4o Llama3, FinGPT and FinMA, we reveal their strengths and limitations in processing complex financial information. Additionally, we open-sourced Touchstone-GPT, a financial LLM trained through continual pre-training and financial instruction tuning, which demonstrates strong performance on the bilingual benchmark but still has limitations in specific tasks.This research not only provides the financial large language models with a practical evaluation tool but also guides the development and optimization of future research. The source code for Golden Touchstone and model weight of Touchstone-GPT have been made publicly available at \url{https://github.com/IDEA-FinAI/Golden-Touchstone}, contributing to the ongoing evolution of FinLLMs and fostering further research in this critical area.
Foundation Model for Composite Microstructures: Reconstruction, Stiffness, and Nonlinear Behavior Prediction
Authors: Ting-Ju Wei, Chuin-Shan Chen
Date: 2024-11-10 00:00:00
Tasks: Transfer Learning

The rapid advancement of machine learning has unlocked numerous opportunities for materials science, particularly in accelerating the design and analysis of materials. However, a significant challenge lies in the scarcity and high cost of obtaining high-quality materials datasets. While foundation models pre-trained on large datasets have excelled in fields like natural language processing by leveraging latent features through transfer learning, their application in materials science remains limited. Here, we present a foundation model specifically designed for composite materials. Pre-trained on a dataset of short-fiber composites to learn robust latent features, the model accurately predicts homogenized stiffness during transfer learning, even with limited training data. Additionally, our model effectively predicts the material's nonlinear behavior by transferring these learned features to an Interaction-based Material Network, which is a constitutive surrogate model. These results demonstrate the potential of our foundation model to capture complex material behaviors. Our findings validate the feasibility and effectiveness of foundation models in composite materials. We anticipate extending this approach to more complex three-dimensional composite materials, polycrystalline materials, and beyond. Moreover, this framework enables high-accuracy predictions even when experimental data are scarce, paving the way for more efficient and cost-effective materials design and analysis.
Large-scale moral machine experiment on large language models
Authors: Muhammad Shahrul Zaim bin Ahmad, Kazuhiro Takemoto
Date: 2024-11-11 00:00:00
Tasks: Autonomous Driving, Computational Efficiency, Decision Making

The rapid advancement of Large Language Models (LLMs) and their potential integration into autonomous driving systems necessitates understanding their moral decision-making capabilities. While our previous study examined four prominent LLMs using the Moral Machine experimental framework, the dynamic landscape of LLM development demands a more comprehensive analysis. Here, we evaluate moral judgments across 52 different LLMs, including multiple versions of proprietary models (GPT, Claude, Gemini) and open-source alternatives (Llama, Gemma), to assess their alignment with human moral preferences in autonomous driving scenarios. Using a conjoint analysis framework, we evaluated how closely LLM responses aligned with human preferences in ethical dilemmas and examined the effects of model size, updates, and architecture. Results showed that proprietary models and open-source models exceeding 10 billion parameters demonstrated relatively close alignment with human judgments, with a significant negative correlation between model size and distance from human judgments in open-source models. However, model updates did not consistently improve alignment with human preferences, and many LLMs showed excessive emphasis on specific ethical principles. These findings suggest that while increasing model size may naturally lead to more human-like moral judgments, practical implementation in autonomous driving systems requires careful consideration of the trade-off between judgment quality and computational efficiency. Our comprehensive analysis provides crucial insights for the ethical design of autonomous systems and highlights the importance of considering cultural contexts in AI moral decision-making.
StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification
Authors: Yichen He, Yuan Lin, Jianchao Wu, Hanchong Zhang, Yuchen Zhang, Ruicheng Le
Date: 2024-11-11 00:00:00
Tasks: Large Language Model, Multimodal Large Language Model, Multiple-choice, Video Description

Existing large vision-language models (LVLMs) are largely limited to processing short, seconds-long videos and struggle with generating coherent descriptions for extended video spanning minutes or more. Long video description introduces new challenges, such as plot-level consistency across descriptions. To address these, we figure out audio-visual character identification, matching character names to each dialogue, as a key factor. We propose StoryTeller, a system for generating dense descriptions of long videos, incorporating both low-level visual concepts and high-level plot information. StoryTeller uses a multimodal large language model that integrates visual, audio, and text modalities to perform audio-visual character identification on minute-long video clips. The results are then fed into a LVLM to enhance consistency of video description. We validate our approach on movie description tasks and introduce MovieStory101, a dataset with dense descriptions for three-minute movie clips. To evaluate long video descriptions, we create MovieQA, a large set of multiple-choice questions for the MovieStory101 test set. We assess descriptions by inputting them into GPT-4 to answer these questions, using accuracy as an automatic evaluation metric. Experiments show that StoryTeller outperforms all open and closed-source baselines on MovieQA, achieving 9.5% higher accuracy than the strongest baseline, Gemini-1.5-pro, and demonstrating a +15.56% advantage in human side-by-side evaluations. Additionally, incorporating audio-visual character identification from StoryTeller improves the performance of all video description models, with Gemini-1.5-pro and GPT-4o showing relative improvement of 5.5% and 13.0%, respectively, in accuracy on MovieQA.
UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts
Authors: Bo Yang, Qingping Yang, Yingwei Ma, Runtao Liu
Date: 2024-11-11 00:00:00
Tasks: Code Generation, GSM8K, Math, Mathematical Reasoning

The evaluation of mathematical reasoning capabilities is essential for advancing Artificial General Intelligence (AGI). While Large Language Models (LLMs) have shown impressive performance in solving mathematical problems, existing benchmarks such as GSM8K and MATH present limitations, including narrow problem definitions with specific numbers and reliance on predetermined rules that hinder accurate assessments of reasoning and generality. This paper introduces the UTMath Benchmark, a robust evaluation framework designed to assess LLMs through extensive unit tests, with a focus on both the accuracy and generality of model responses. It comprises 1,053 cutting-edge problems spanning nine mathematical domains, with an average of 68 test cases per problem. UTMath is highly challenging, with the best-performing model, o1-mini, solving only 32.57\% of the problems, followed by o1-preview at 27.16\%, and GPT-4o at 26.93\%. Furthermore, we present the Reasoning-to-Coding of Thoughts (RCoT) approach, which encourages LLMs to engage in explicit reasoning prior to code generation, thereby facilitating the production of more sophisticated solutions and enhancing overall performance and efficiency. Additionally, we also release the UTMath-Train training dataset (more than 70k samples), to support the community in further exploring mathematical reasoning. Our benchmark can be accessed via the following link: https://github.com/UTMathGroup/UTMath