pwc-archive/papers-with-abstracts huggingface dataset:
A Structure-aware Generative Model for Biomedical Event Extraction
Authors: Haohan Yuan, Siu Cheung Hui, Haopeng Zhang
Date: 2024-08-13 00:00:00
Tasks: Event Extraction

Biomedical Event Extraction (BEE) is a challenging task that involves modeling complex relationships between fine-grained entities in biomedical text. BEE has traditionally been formulated as a classification problem. With the recent technological advancements in large language models (LLMs), generation-based models that cast event extraction as a sequence generation problem have attracted much attention from the NLP research communities. However, current generative models often overlook the importance of cross-instance information from complex event structures such as nested events and overlapping events, which contribute to over 20% of the events in the benchmark datasets. In this paper, we propose an event structure-aware generative model named GenBEE, which can capture complex event structures in biomedical text for biomedical event extraction. In particular, GenBEE constructs event prompts that distill knowledge from LLMs for incorporating both label semantics and argument dependency relationships into the proposed model. In addition, GenBEE also generates prefixes with event structural prompts to incorporate structural features for improving the model's overall performance. We have evaluated the proposed GenBEE model on three widely used biomedical event extraction benchmark datasets, namely MLEE, GE11, and PHEE. Experimental results show that GenBEE has achieved state-of-the-art performance on the MLEE and GE11 datasets, and achieved competitive results when compared to the state-of-the-art classification-based models on the PHEE dataset.
OpenEP: Open-Ended Future Event Prediction
Authors: Yong Guan, Hao Peng, Xiaozhi Wang, Lei Hou, Juanzi Li
Date: 2024-08-13 00:00:00
Tasks: Prediction

Future event prediction (FEP) is a long-standing and crucial task in the world, as understanding the evolution of events enables early risk identification, informed decision-making, and strategic planning. Existing work typically treats event prediction as classification tasks and confines the outcomes of future events to a fixed scope, such as yes/no questions, candidate set, and taxonomy, which is difficult to include all possible outcomes of future events. In this paper, we introduce OpenEP (an Open-Ended Future Event Prediction task), which generates flexible and diverse predictions aligned with real-world scenarios. This is mainly reflected in two aspects: firstly, the predictive questions are diverse, covering different stages of event development and perspectives; secondly, the outcomes are flexible, without constraints on scope or format. To facilitate the study of this task, we construct OpenEPBench, an open-ended future event prediction dataset. For question construction, we pose questions from seven perspectives, including location, time, event development, event outcome, event impact, event response, and other, to facilitate an in-depth analysis and understanding of the comprehensive evolution of events. For outcome construction, we collect free-form text containing the outcomes as ground truth to provide semantically complete and detail-enriched outcomes. Furthermore, we propose StkFEP, a stakeholder-enhanced future event prediction framework, that incorporates event characteristics for open-ended settings. Our method extracts stakeholders involved in events to extend questions to gather diverse information. We also collect historically events that are relevant and similar to the question to reveal potential evolutionary patterns. Experiment results indicate that accurately predicting future events in open-ended settings is challenging for existing LLMs.
CTISum: A New Benchmark Dataset For Cyber Threat Intelligence Summarization
Authors: Wei Peng, Junmei Ding, Wei Wang, Lei Cui, Wei Cai, Zhiyu Hao, Xiaochun Yun
Date: 2024-08-13 00:00:00
Tasks: Abstractive Text Summarization

Cyber Threat Intelligence (CTI) summarization task requires the system to generate concise and accurate highlights from raw intelligence data, which plays an important role in providing decision-makers with crucial information to quickly detect and respond to cyber threats in the cybersecurity domain. However, efficient techniques for summarizing CTI reports, including facts, analytical insights, attack processes, etc., have largely been unexplored, primarily due to the lack of available dataset. To this end, we present CTISum, a new benchmark for CTI summarization task. Considering the importance of attack process, a novel fine-grained subtask of attack process summarization is proposed to enable defenders to assess risk, identify security gaps, vulnerabilities, and so on. Specifically, we first design a multi-stage annotation pipeline to gather and annotate the CTI data, and then benchmark the CTISum with a collection of extractive and abstractive summarization methods. Experimental results show that current state-of-the-art models exhibit limitations when applied to CTISum, underscoring the fact that automatically producing concise summaries of CTI reports remains an open research challenge.
SparkRA: A Retrieval-Augmented Knowledge Service System Based on Spark Large Language Model
Authors: Dayong Wu, Jiaqi Li, Baoxin Wang, Honghong Zhao, Siyuan Xue, Yanjie Yang, Zhijun Chang, Rui Zhang, Li Qian, Bo wang, Shijin Wang, Zhixiong Zhang, Guoping Hu
Date: 2024-08-13 00:00:00
Tasks: Language Modeling, Language Modelling, Large Language Model, Retrieval

Large language models (LLMs) have shown remarkable achievements across various language tasks.To enhance the performance of LLMs in scientific literature services, we developed the scientific literature LLM (SciLit-LLM) through pre-training and supervised fine-tuning on scientific literature, building upon the iFLYTEK Spark LLM. Furthermore, we present a knowledge service system Spark Research Assistant (SparkRA) based on our SciLit-LLM. SparkRA is accessible online and provides three primary functions: literature investigation, paper reading, and academic writing. As of July 30, 2024, SparkRA has garnered over 50,000 registered users, with a total usage count exceeding 1.3 million.
Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Data
Authors: Mara Finkelstein, David Vilar, Markus Freitag
Date: 2024-08-13 00:00:00
Tasks: Machine Translation, NMT, Sentence

Recent research in neural machine translation (NMT) has shown that training on high-quality machine-generated data can outperform training on human-generated data. This work accompanies the first-ever release of a LLM-generated, MBR-decoded and QE-reranked dataset with both sentence-level and multi-sentence examples. We perform extensive experiments to demonstrate the quality of our dataset in terms of its downstream impact on NMT model performance. We find that training from scratch on our (machine-generated) dataset outperforms training on the (web-crawled) WMT'23 training dataset (which is 300 times larger), and also outperforms training on the top-quality subset of the WMT'23 training dataset. We also find that performing self-distillation by finetuning the LLM which generated this dataset outperforms the LLM's strong few-shot baseline. These findings corroborate the quality of our dataset, and demonstrate the value of high-quality machine-generated data in improving performance of NMT models.
Retrieval-Augmented Hierarchical in-Context Reinforcement Learning and Hindsight Modular Reflections for Task Planning with LLMs
Authors: Chuanneng Sun, Songjun Huang, Dario Pompili
Date: 2024-08-12 00:00:00
Tasks: Decision Making, Hierarchical Reinforcement Learning, In-Context Learning, In-Context Reinforcement Learning, Navigate, reinforcement-learning, Reinforcement Learning, Task Planning

Large Language Models (LLMs) have demonstrated remarkable abilities in various language tasks, making them promising candidates for decision-making in robotics. Inspired by Hierarchical Reinforcement Learning (HRL), we propose Retrieval-Augmented in-context reinforcement Learning (RAHL), a novel framework that decomposes complex tasks into sub-tasks using an LLM-based high-level policy, in which a complex task is decomposed into sub-tasks by a high-level policy on-the-fly. The sub-tasks, defined by goals, are assigned to the low-level policy to complete. To improve the agent's performance in multi-episode execution, we propose Hindsight Modular Reflection (HMR), where, instead of reflecting on the full trajectory, we let the agent reflect on shorter sub-trajectories to improve reflection efficiency. We evaluated the decision-making ability of the proposed RAHL in three benchmark environments--ALFWorld, Webshop, and HotpotQA. The results show that RAHL can achieve an improvement in performance in 9%, 42%, and 10% in 5 episodes of execution in strong baselines. Furthermore, we also implemented RAHL on the Boston Dynamics SPOT robot. The experiment shows that the robot can scan the environment, find entrances, and navigate to new rooms controlled by the LLM policy.
Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models
Authors: Hila Gonen, Terra Blevins, Alisa Liu, Luke Zettlemoyer, Noah A. Smith
Date: 2024-08-12 00:00:00

Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood. In this paper, we identify and characterize a phenomenon never discussed before, which we call semantic leakage, where models leak irrelevant information from the prompt into the generation in unexpected ways. We propose an evaluation setting to detect semantic leakage both by humans and automatically, curate a diverse test suite for diagnosing this behavior, and measure significant semantic leakage in 13 flagship models. We also show that models exhibit semantic leakage in languages besides English and across different settings and generation scenarios. This discovery highlights yet another type of bias in language models that affects their generation patterns and behavior.
TOGGL: Transcribing Overlapping Speech with Staggered Labeling
Authors: Chak-Fai Li, William Hartmann, Matthew Snover
Date: 2024-08-12 00:00:00
Tasks: Attribute, Decoder

Transcribing the speech of multiple overlapping speakers typically requires separating the audio into multiple streams and recognizing each one independently. More recent work jointly separates and transcribes, but requires a separate decoding component for each speaker. We propose the TOGGL model to simultaneously transcribe the speech of multiple speakers. The TOGGL model uses special output tokens to attribute the speech to each speaker with only a single decoder. Our approach generalizes beyond two speakers, even when trained only on two-speaker data. We demonstrate superior performance compared to competing approaches on a conversational speech dataset. Our approach also improves performance on single-speaker audio.
Evaluating Language Models for Efficient Code Generation
Authors: Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, Lingming Zhang
Date: 2024-08-12 00:00:00
Tasks: Code Generation

We introduce Differential Performance Evaluation (DPE), a framework designed to reliably evaluate Large Language Models (LLMs) for efficient code generation. Traditional coding benchmarks often fail to provide reliable insights into code efficiency, due to their reliance on simplistic test inputs and the absence of effective compound metrics. DPE addresses these issues by focusing on efficiency-demanding programming tasks and establishing an insightful compound metric for performance evaluation. DPE operates in two phases: To curate efficiency datasets, it selects efficiency-demanding tasks from existing coding benchmarks and generates computationally expensive inputs to stress the efficiency of LLM solutions. To assess the code efficiency, DPE profiles the new solution and compares it globally against a set of reference solutions that exhibit distinct efficiency levels, where the matched level defines its efficiency score. As a proof of concept, we use DPE to create EvalPerf, a benchmark with 121 performance-challenging coding tasks. Our comprehensive evaluation draws interesting findings on the efficiency impact of model sizes, instruction tuning, and prompting. For example, while the scaling law fails to account for code efficiency, general instruction tuning benefits both code correctness and efficiency. We also evaluate the evaluation by examining the effectiveness of DPE, showing that EvalPerf is reliable and convenient to use even across platforms.
Flow-Lenia.png: Evolving Multi-Scale Complexity by Means of Compression
Authors: Tadashi Adachi, Solvi Arnold, Takafumi Mochizuki, Kimitoshi Yamazaki
Date: 2024-08-08 00:00:00
Tasks: Image Compression

We propose a fitness measure quantifying multi-scale complexity for cellular automaton states, using compressibility as a proxy for complexity. The use of compressibility is grounded in the concept of Kolmogorov complexity, which defines the complexity of an object by the size of its smallest representation. With this fitness function, we explore the complexity range accessible to the well-known Flow Lenia cellular automaton, using image compression algorithms to assess state compressibility. Using a Genetic Algorithm to evolve Flow Lenia patterns, we conduct experiments with two primary objectives: 1) generating patterns of specific complexity levels, and 2) exploring the extrema of Flow Lenia's complexity domain. Evolved patterns reflect the complexity targets, with higher complexity targets yielding more intricate patterns, consistent with human perceptions of complexity. This demonstrates that our fitness function can effectively evolve patterns that match specific complexity objectives within the bounds of the complexity range accessible to Flow Lenia under a given hyperparameter configuration.
Diffusion Model for Slate Recommendation
Authors: Federico Tomasi, Francesco Fabbri, Mounia Lalmas, Zhenwen Dai
Date: 2024-08-13 00:00:00
Tasks: Diversity, model

Slate recommendation is a technique commonly used on streaming platforms and e-commerce sites to present multiple items together. A significant challenge with slate recommendation is managing the complex combinatorial choice space. Traditional methods often simplify this problem by assuming users engage with only one item at a time. However, this simplification does not reflect the reality, as users often interact with multiple items simultaneously. In this paper, we address the general slate recommendation problem, which accounts for simultaneous engagement with multiple items. We propose a generative approach using Diffusion Models, leveraging their ability to learn structures in high-dimensional data. Our model generates high-quality slates that maximize user satisfaction by overcoming the challenges of the combinatorial choice space. Furthermore, our approach enhances the diversity of recommendations. Extensive offline evaluations on applications such as music playlist generation and e-commerce bundle recommendations show that our model outperforms state-of-the-art baselines in both relevance and diversity.
Physics Informed Deep Learning for Strain Gradient Continuum Plasticity
Authors: Ankit Tyagi, Uttam Suman, Mariya Mamajiwala, Debasish Roy
Date: 2024-08-13 00:00:00
Tasks: Deep Learning

We use a space-time discretization based on physics informed deep learning (PIDL) to approximate solutions of a class of rate-dependent strain gradient plasticity models. The differential equation governing the plastic flow, the so-called microforce balance for this class of yield-free plasticity models, is very stiff, often leading to numerical corruption and a consequent lack of accuracy or convergence by finite element (FE) methods. Indeed, setting up the discretized framework, especially with an elaborate meshing around the propagating plastic bands whose locations are often unknown a-priori, also scales up the computational effort significantly. Taking inspiration from physics informed neural networks, we modify the loss function of a PIDL model in several novel ways to account for the balance laws, either through energetics or via the resulting PDEs once a variational scheme is applied, and the constitutive equations. The initial and the boundary conditions may either be imposed strictly by encoding them within the PIDL architecture, or enforced weakly as a part of the loss function. The flexibility in the implementation of a PIDL technique often makes for its ready interface with powerful optimization schemes, and this in turn provides for many possibilities in posing the problem. We have used freely available open-source libraries that perform fast, parallel computations on GPUs. Using numerical illustrations, we demonstrate how PIDL methods could address the computational challenges posed by strain gradient plasticity models. Also, PIDL methods offer abundant potentialities, vis-\'a-vis a somewhat straitjacketed and poorer approximant of FE methods, in customizing the formulation as per the problem objective.
Quantifying uncertainty in climate projections with conformal ensembles
Authors: Trevor Harris, Ryan Sriver
Date: 2024-08-13 00:00:00
Tasks: Uncertainty Quantification

Large climate model ensembles are the primary tool for robustly projecting future climate states and quantifying projection uncertainty. Despite significant advancements in climate modeling over the past few decades, overall projection certainty has not commensurately decreased with steadily improving model skill. We introduce conformal ensembling, a new approach to uncertainty quantification in climate projections based on conformal inference to reduce projection uncertainty. Unlike traditional methods, conformal ensembling seamlessly integrates climate model ensembles and observational data across a range of scales to generate statistically rigorous, easy-to-interpret uncertainty estimates. It can be applied to any climatic variable using any ensemble analysis method and outperforms existing inter-model variability methods in uncertainty quantification across all time horizons and most spatial locations under SSP2-4.5. Conformal ensembling is also computationally efficient, requires minimal assumptions, and is highly robust to the conformity measure. Experiments show that it is effective when conditioning future projections on historical reanalysis data compared with standard ensemble averaging approaches, yielding more physically consistent projections.
Learning Noise-Robust Stable Koopman Operator for Control with Hankel DMD
Authors: Shahriar Akbar Sakib, Shaowu Pan
Date: 2024-08-13 00:00:00
Tasks: Data Augmentation

We propose a noise-robust learning framework for the Koopman operator of nonlinear dynamical systems, with guaranteed long-term stability and improved model performance for better model-based predictive control tasks. Unlike some existing approaches that rely on ad hoc observables or black-box neural networks in extended dynamic mode decomposition (EDMD), our framework leverages observables generated by the system dynamics, when the system dynamics is known, through a Hankel matrix, which shares similarities with discrete Polyflow. When system dynamics is unknown, we approximate them with a neural network while maintaining structural similarities to discrete Polyflow. To enhance noise robustness and ensure long-term stability, we developed a stable parameterization of the Koopman operator, along with a progressive learning strategy for rollout loss. To further improve the performance of the model in the phase space, a simple iterative data augmentation strategy was developed. Numerical experiments of prediction and control of classic nonlinear systems with ablation study showed the effectiveness of the proposed techniques over several state-of-the-art practices.
Variance-Reduced Cascade Q-learning: Algorithms and Sample Complexity
Authors: Mohammad Boveiri, Peyman Mohajerin Esfahani
Date: 2024-08-13 00:00:00
Tasks: Q-Learning

We study the problem of estimating the optimal Q-function of $\gamma$-discounted Markov decision processes (MDPs) under the synchronous setting, where independent samples for all state-action pairs are drawn from a generative model at each iteration. We introduce and analyze a novel model-free algorithm called Variance-Reduced Cascade Q-learning (VRCQ). VRCQ comprises two key building blocks: (i) the established direct variance reduction technique and (ii) our proposed variance reduction scheme, Cascade Q-learning. By leveraging these techniques, VRCQ provides superior guarantees in the $\ell_\infty$-norm compared with the existing model-free stochastic approximation-type algorithms. Specifically, we demonstrate that VRCQ is minimax optimal. Additionally, when the action set is a singleton (so that the Q-learning problem reduces to policy evaluation), it achieves non-asymptotic instance optimality while requiring the minimum number of samples theoretically possible. Our theoretical results and their practical implications are supported by numerical experiments.
Operator Learning Using Random Features: A Tool for Scientific Computing
Authors: Nicholas H. Nelsen, Andrew M. Stuart
Date: 2024-08-12 00:00:00
Tasks: Operator learning, regression

Supervised operator learning centers on the use of training data, in the form of input-output pairs, to estimate maps between infinite-dimensional spaces. It is emerging as a powerful tool to complement traditional scientific computing, which may often be framed in terms of operators mapping between spaces of functions. Building on the classical random features methodology for scalar regression, this paper introduces the function-valued random features method. This leads to a supervised operator learning architecture that is practical for nonlinear problems yet is structured enough to facilitate efficient training through the optimization of a convex, quadratic cost. Due to the quadratic structure, the trained model is equipped with convergence guarantees and error and complexity bounds, properties that are not readily available for most other operator learning architectures. At its core, the proposed approach builds a linear combination of random operators. This turns out to be a low-rank approximation of an operator-valued kernel ridge regression algorithm, and hence the method also has strong connections to Gaussian process regression. The paper designs function-valued random features that are tailored to the structure of two nonlinear operator learning benchmark problems arising from parametric partial differential equations. Numerical results demonstrate the scalability, discretization invariance, and transferability of the function-valued random features method.
The NP-hardness of the Gromov-Wasserstein distance
Authors: Natalia Kravtsova
Date: 2024-08-12 00:00:00

This note addresses the property frequently mentioned in the literature that the Gromov-Wasserstein (GW) distance is NP-hard. We provide the details on the non-convex nature of the GW optimization problem that imply NP-hardness of the GW distance between finite spaces for any instance of an input data. We further illustrate the non-convexity of the problem with several explicit examples.