pwc-archive/papers-with-abstracts huggingface dataset:
CommitBench: A Benchmark for Commit Message Generation
Authors: Maximilian Schall, Tamara Czinczoll, Gerard de Melo
Date: 2024-03-08 00:00:00

Writing commit messages is a tedious daily task for many software developers, and often remains neglected. Automating this task has the potential to save time while ensuring that messages are informative. A high-quality dataset and an objective benchmark are vital preconditions for solid research and evaluation towards this goal. We show that existing datasets exhibit various problems, such as the quality of the commit selection, small sample sizes, duplicates, privacy issues, and missing licenses for redistribution. This can lead to unusable models and skewed evaluations, where inferior models achieve higher evaluation scores due to biases in the data. We compile a new large-scale dataset, CommitBench, adopting best practices for dataset creation. We sample commits from diverse projects with licenses that permit redistribution and apply our filtering and dataset enhancements to improve the quality of generated commit messages. We use CommitBench to compare existing models and show that other approaches are outperformed by a Transformer model pretrained on source code. We hope to accelerate future research by publishing the source code( https://github.com/Maxscha/commitbench ).
ROUGE-K: Do Your Summaries Have Keywords?
Authors: Sotaro Takeshita, Simone Paolo Ponzetto, Kai Eckert
Date: 2024-03-08 00:00:00
Tasks: Extreme Summarization

Keywords, that is, content-relevant words in summaries play an important role in efficient information conveyance, making it critical to assess if system-generated summaries contain such informative words during evaluation. However, existing evaluation metrics for extreme summarization models do not pay explicit attention to keywords in summaries, leaving developers ignorant of their presence. To address this issue, we present a keyword-oriented evaluation metric, dubbed ROUGE-K, which provides a quantitative answer to the question of -- \textit{How well do summaries include keywords?} Through the lens of this keyword-aware metric, we surprisingly find that a current strong baseline model often misses essential information in their summaries. Our analysis reveals that human annotators indeed find the summaries with more keywords to be more relevant to the source documents. This is an important yet previously overlooked aspect in evaluating summarization systems. Finally, to enhance keyword inclusion, we propose four approaches for incorporating word importance into a transformer-based model and experimentally show that it enables guiding models to include more keywords while keeping the overall quality. Our code is released at https://github.com/sobamchan/rougek.
Can we obtain significant success in RST discourse parsing by using Large Language Models?
Authors: Aru Maekawa, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura
Date: 2024-03-08 00:00:00
Tasks: Decoder, Discourse Parsing

Recently, decoder-only pre-trained large language models (LLMs), with several tens of billion parameters, have significantly impacted a wide range of natural language processing (NLP) tasks. While encoder-only or encoder-decoder pre-trained language models have already proved to be effective in discourse parsing, the extent to which LLMs can perform this task remains an open research question. Therefore, this paper explores how beneficial such LLMs are for Rhetorical Structure Theory (RST) discourse parsing. Here, the parsing process for both fundamental top-down and bottom-up strategies is converted into prompts, which LLMs can work with. We employ Llama 2 and fine-tune it with QLoRA, which has fewer parameters that can be tuned. Experimental results on three benchmark datasets, RST-DT, Instr-DT, and the GUM corpus, demonstrate that Llama 2 with 70 billion parameters in the bottom-up strategy obtained state-of-the-art (SOTA) results with significant differences. Furthermore, our parsers demonstrated generalizability when evaluated on RST-DT, showing that, in spite of being trained with the GUM corpus, it obtained similar performances to those of existing parsers trained with RST-DT.
Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering
Authors: Saeel Sandeep Nachane, Ojas Gramopadhye, Prateek Chanda, Ganesh Ramakrishnan, Kshitij Sharad Jadhav, Yatin Nandwani, Dinesh Raghu, Sachindra Joshi
Date: 2024-03-07 00:00:00
Tasks: Information Retrieval, Language Modelling, Medical Question Answering, MedQA, Question Answering, Response Generation

In this paper, we propose a modified version of the MedQA-USMLE dataset, named MEDQA-OPEN, which contains open-ended medical questions without options to mimic clinical scenarios, along with clinician-approved reasoned answers. Additionally, we implement a prompt driven by Chain of Thought (CoT) reasoning, CLINICR, to mirror the prospective process of incremental reasoning, reaching a correct response to medical questions. We empirically demonstrate how CLINICR outperforms the state-of-the-art 5-shot CoT-based prompt (Li\'evin et al., 2022). We also present an approach that mirrors real-life clinical practice by first exploring multiple differential diagnoses through MCQ-CLINICR and subsequently narrowing down to a final diagnosis using MCQ-ELIMINATIVE. Finally, emphasizing the importance of response verification in medical settings, we utilize a reward model mechanism, replacing the elimination process performed by MCQ-ELIMINATIVE.
Code-Mixed Probes Show How Pre-Trained Models Generalise On Code-Switched Text
Authors: Frances A. Laureano De Leon, Harish Tayyar Madabushi, Mark Lee
Date: 2024-03-07 00:00:00

Code-switching is a prevalent linguistic phenomenon in which multilingual individuals seamlessly alternate between languages. Despite its widespread use online and recent research trends in this area, research in code-switching presents unique challenges, primarily stemming from the scarcity of labelled data and available resources. In this study we investigate how pre-trained Language Models handle code-switched text in three dimensions: a) the ability of PLMs to detect code-switched text, b) variations in the structural information that PLMs utilise to capture code-switched text, and c) the consistency of semantic information representation in code-switched text. To conduct a systematic and controlled evaluation of the language models in question, we create a novel dataset of well-formed naturalistic code-switched text along with parallel translations into the source languages. Our findings reveal that pre-trained language models are effective in generalising to code-switched text, shedding light on the abilities of these models to generalise representations to CS corpora. We release all our code and data including the novel corpus at https://github.com/francesita/code-mixed-probes.
Evaluating Biases in Context-Dependent Health Questions
Authors: Sharon Levy, Tahilin Sanchez Karver, William D. Adler, Michelle R. Kaufman, Mark Dredze
Date: 2024-03-07 00:00:00
Tasks: Language Modeling, Language Modelling, Large Language Model

Chat-based large language models have the opportunity to empower individuals lacking high-quality healthcare access to receive personalized information across a variety of topics. However, users may ask underspecified questions that require additional context for a model to correctly answer. We study how large language model biases are exhibited through these contextual questions in the healthcare domain. To accomplish this, we curate a dataset of sexual and reproductive healthcare questions that are dependent on age, sex, and location attributes. We compare models' outputs with and without demographic context to determine group alignment among our contextual questions. Our experiments reveal biases in each of these attributes, where young adult female users are favored.
Automating the Information Extraction from Semi-Structured Interview Transcripts
Authors: Angelina Parfenova
Date: 2024-03-07 00:00:00
Tasks: Topic Models

This paper explores the development and application of an automated system designed to extract information from semi-structured interview transcripts. Given the labor-intensive nature of traditional qualitative analysis methods, such as coding, there exists a significant demand for tools that can facilitate the analysis process. Our research investigates various topic modeling techniques and concludes that the best model for analyzing interview texts is a combination of BERT embeddings and HDBSCAN clustering. We present a user-friendly software prototype that enables researchers, including those without programming skills, to efficiently process and visualize the thematic structure of interview data. This tool not only facilitates the initial stages of qualitative analysis but also offers insights into the interconnectedness of topics revealed, thereby enhancing the depth of qualitative analysis.
An Adaptive Dimension Reduction Estimation Method for High-dimensional Bayesian Optimization
Authors: Shouri Hu, Jiawei Li, Zhibo Cai
Date: 2024-03-08 00:00:00
Tasks: Bayesian Optimization, Dimensionality Reduction

Bayesian optimization (BO) has shown impressive results in a variety of applications within low-to-moderate dimensional Euclidean spaces. However, extending BO to high-dimensional settings remains a significant challenge. We address this challenge by proposing a two-step optimization framework. Initially, we identify the effective dimension reduction (EDR) subspace for the objective function using the minimum average variance estimation (MAVE) method. Subsequently, we construct a Gaussian process model within this EDR subspace and optimize it using the expected improvement criterion. Our algorithm offers the flexibility to operate these steps either concurrently or in sequence. In the sequential approach, we meticulously balance the exploration-exploitation trade-off by distributing the sampling budget between subspace estimation and function optimization, and the convergence rate of our algorithm in high-dimensional contexts has been established. Numerical experiments validate the efficacy of our method in challenging scenarios.
An Efficient Quasi-Random Sampling for Copulas
Authors: Sumin Wang, Chenxian Huang, Yongdao Zhou, Min-Qian Liu
Date: 2024-03-08 00:00:00
Tasks: Management

This paper examines an efficient method for quasi-random sampling of copulas in Monte Carlo computations. Traditional methods, like conditional distribution methods (CDM), have limitations when dealing with high-dimensional or implicit copulas, which refer to those that cannot be accurately represented by existing parametric copulas. Instead, this paper proposes the use of generative models, such as Generative Adversarial Networks (GANs), to generate quasi-random samples for any copula. GANs are a type of implicit generative models used to learn the distribution of complex data, thus facilitating easy sampling. In our study, GANs are employed to learn the mapping from a uniform distribution to copulas. Once this mapping is learned, obtaining quasi-random samples from the copula only requires inputting quasi-random samples from the uniform distribution. This approach offers a more flexible method for any copula. Additionally, we provide theoretical analysis of quasi-Monte Carlo estimators based on quasi-random samples of copulas. Through simulated and practical applications, particularly in the field of risk management, we validate the proposed method and demonstrate its superiority over various existing methods.
Deep Backward and Galerkin Methods for the Finite State Master Equation
Authors: Asaf Cohen, Mathieu Laurière, Ethan Zell
Date: 2024-03-08 00:00:00

This paper proposes and analyzes two neural network methods to solve the master equation for finite-state mean field games (MFGs). Solving MFGs provides approximate Nash equilibria for stochastic, differential games with finite but large populations of agents. The master equation is a partial differential equation (PDE) whose solution characterizes MFG equilibria for any possible initial distribution. The first method we propose relies on backward induction in a time component while the second method directly tackles the PDE without discretizing time. For both approaches, we prove two types of results: there exist neural networks that make the algorithms' loss functions arbitrarily small, and conversely, if the losses are small, then the neural networks are good approximations of the master equation's solution. We conclude the paper with numerical experiments on benchmark problems from the literature up to dimension 15, and a comparison with solutions computed by a classical method for fixed initial distributions.
Pruning neural network models for gene regulatory dynamics using data and domain knowledge
Authors: Intekhab Hossain, Jonas Fischer, Rebekka Burkholz, John Quackenbush
Date: 2024-03-05 00:00:00
Tasks: General Knowledge, Network Pruning, scientific discovery

The practical utility of machine learning models in the sciences often hinges on their interpretability. It is common to assess a model's merit for scientific discovery, and thus novel insights, by how well it aligns with already available domain knowledge--a dimension that is currently largely disregarded in the comparison of neural network models. While pruning can simplify deep neural network architectures and excels in identifying sparse models, as we show in the context of gene regulatory network inference, state-of-the-art techniques struggle with biologically meaningful structure learning. To address this issue, we propose DASH, a generalizable framework that guides network pruning by using domain-specific structural information in model fitting and leads to sparser, better interpretable models that are more robust to noise. Using both synthetic data with ground truth information, as well as real-world gene expression data, we show that DASH, using knowledge about gene interaction partners within the putative regulatory network, outperforms general pruning methods by a large margin and yields deeper insights into the biological systems being studied.
ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data
Authors: Liana Patel, Peter Kraft, Carlos Guestrin, Matei Zaharia
Date: 2024-03-07 00:00:00

Applications increasingly leverage mixed-modality data, and must jointly search over vector data, such as embedded images, text and video, as well as structured data, such as attributes and keywords. Proposed methods for this hybrid search setting either suffer from poor performance or support a severely restricted set of search predicates (e.g., only small sets of equality predicates), making them impractical for many applications. To address this, we present ACORN, an approach for performant and predicate-agnostic hybrid search. ACORN builds on Hierarchical Navigable Small Worlds (HNSW), a state-of-the-art graph-based approximate nearest neighbor index, and can be implemented efficiently by extending existing HNSW libraries. ACORN introduces the idea of predicate subgraph traversal to emulate a theoretically ideal, but impractical, hybrid search strategy. ACORN's predicate-agnostic construction algorithm is designed to enable this effective search strategy, while supporting a wide array of predicate sets and query semantics. We systematically evaluate ACORN on both prior benchmark datasets, with simple, low-cardinality predicate sets, and complex multi-modal datasets not supported by prior methods. We show that ACORN achieves state-of-the-art performance on all datasets, outperforming prior methods with 2-1,000x higher throughput at a fixed recall.
LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset
Authors: Ahmed Izzidien, Holli Sargeant, Felix Steffek
Date: 2024-03-04 00:00:00
Tasks: Language Modeling, Language Modelling, Large Language Model

To undertake computational research of the law, efficiently identifying datasets of court decisions that relate to a specific legal issue is a crucial yet challenging endeavour. This study addresses the gap in the literature working with large legal corpora about how to isolate cases, in our case summary judgments, from a large corpus of UK court decisions. We introduce a comparative analysis of two computational methods: (1) a traditional natural language processing-based approach leveraging expert-generated keywords and logical operators and (2) an innovative application of the Claude 2 large language model to classify cases based on content-specific prompts. We use the Cambridge Law Corpus of 356,011 UK court decisions and determine that the large language model achieves a weighted F1 score of 0.94 versus 0.78 for keywords. Despite iterative refinement, the search logic based on keywords fails to capture nuances in legal language. We identify and extract 3,102 summary judgment cases, enabling us to map their distribution across various UK courts over a temporal span. The paper marks a pioneering step in employing advanced natural language processing to tackle core legal research tasks, demonstrating how these technologies can bridge systemic gaps and enhance the accessibility of legal information. We share the extracted dataset metrics to support further research on summary judgments.
Meta Reinforcement Learning for Multi-Task Offloading in Vehicular Edge Computing
Authors: Penglin Dai, Yaorong Huang, Kaiwen Hu, Xiao Wu, Huanlai Xing, and Zhaofei Yu, Member, IEEE
Date: 2024-01-01 00:00:00
Proceeding: TMC 2024 1
Tasks: Edge-computing, Meta-Learning, Meta Reinforcement Learning, reinforcement-learning, Reinforcement Learning

Mobile edge computing has been a promising solution to enable real-time service in vehicular networks. However, due to high dynamics of mobile environment and heterogeneous features of vehicular services, traditional expert-based or learning-based strategies has to update handcrafted parameters or retrain learning model, which leads to intolerant overhead. Therefore, this paper investigates the problem of multi-task offloading (MTO), where there exist multiple offloading scenarios with varying parameters, such as task topology, resource requirement and transmission/computation capability. The objective is to design a unified solution to minimize task execution time under different MTO scenarios. Accordingly, we develop a Seq2seq-based Meta Reinforcement Learning algorithm for MTO (SMRL-MTO). Specifically, a bidirectional gated recurrent units integrated with attention mechanism is designed to determine offloading action by encoding sequential offloading actions and showing different preferences to different parts of input sequence. Particularly, a meta reinforcement learning framework is designed based on model-agnostic meta learning, which trains a meta policy offline and fast adapts to new MTO scenario within a few training steps. Finally, we conduct performance evaluation based on task generator DAGGEN and realistic vehicular traces, which shows that the SMRL-MTO reduces task execution time by 11.36% on average compared with greedy algorithm.
Modelization and numerical approximation of piezoelectric thin shells Part II: Approximation by finite element methods and numerical experiments
Authors: Michel Bernadou, Christophe Haenel
Date: 2003-01-01 00:00:00
Proceeding: Comput. Methods Appl. Mech. Engrg. 192 (2003) 4045–4073 2003 1
Tasks: Numerical Integration

Abstract
A two-dimensional modelization of piezoelectric thin shells has been developed in the first part of this work. Three
equivalent variational formulations have been considered:
• an homogeneous one (with respect to the potential) the bilinear form of which is positive definite but not symmetric;
• the associated nonhomogeneous one which is directly related to the natural boundary conditions on the potential;
• a third one, whose bilinear form is now symmetric but no longer positive definite.
In this second part of this work, the approximation of the second formulation by a conforming finite element method
is analyzed. It takes into account the use of numerical integration techniques and gives criteria on the choice of suitable
numerical schemes. Moreover, the drawbacks linked to the unsymmetrical character of the associated square matrix of
the linear system are circumvent by using a condensation method which consists to eliminate the potential and then to
only solve a symmetrical and elliptic linear system with respect to the displacement. This method is most effective than a
full implementation of the third variational formulation.
Finally some numerical experiments on plate and thin cylindrical shells prove the effectiveness of this method.
A Simple and Model-Free Path Filtering Algorithm for Smoothing and Accuracy
Authors: Thomas Steinecker, Hans-Joachim Wuensche
Date: 2023-07-23 00:00:00
Proceeding: IEEE Intelligent Vehicles Symposium (IV) 2023 7
Tasks: Autonomous Driving, Denoising, Trajectory Planning, Trajectory Prediction

Predominantly, complex optimization techniques are used for path reconstruction given noisy measurements. However, optimization techniques often require the selection of suitable models, tedious parameter tuning and typically fail to generalize to higher-level tasks. In this paper, we present a model-free path filtering method based on the popular moving average method, namely the Curvature Corrected Moving Average (CCMA), which convinces by its simplicity and broad applicability. The moving average is characterized by its unique noise suppression property, albeit curves are bent inwards, which adversely affects its accuracy. By utilizing the relation between both curvatures, the original curvature can be inferred based on the curvature of filtered points. Extending the symmetric filtering not only succeeds in minimizing noise but retains the original shape of the path, making it a suitable algorithm for a variety of robotic applications. We demonstrate the practicality of the approach in a real-world convoy scenario: The accumulated estimates of the leader vehicle’s position, originating from an Extended Kalman filter, are smoothed using our novel approach to generate proper inputs for the Model Predictive Controller (MPC). This cascade structure of filtering provides both responsiveness and smoothness. Furthermore, we successfully applied this method in the off-road convoy scenario for the ELROB 2022, where we won first place. The source code is publicly available.
Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models
Authors: Arijit Ghosh Chowdhury, Md Mofijul Islam, Faysal Hossain Shezan, Vaibhav Kumar, Vinija Jain, Aman Chadha
Date: 2024-03-03 00:00:00
Tasks: Data Poisoning

Large Language Models (LLMs) have become a cornerstone in the field of Natural Language Processing (NLP), offering transformative capabilities in understanding and generating human-like text. However, with their rising prominence, the security and vulnerability aspects of these models have garnered significant attention. This paper presents a comprehensive survey of the various forms of attacks targeting LLMs, discussing the nature and mechanisms of these attacks, their potential impacts, and current defense strategies. We delve into topics such as adversarial attacks that aim to manipulate model outputs, data poisoning that affects model training, and privacy concerns related to training data exploitation. The paper also explores the effectiveness of different attack methodologies, the resilience of LLMs against these attacks, and the implications for model integrity and user trust. By examining the latest research, we provide insights into the current landscape of LLM vulnerabilities and defense mechanisms. Our objective is to offer a nuanced understanding of LLM attacks, foster awareness within the AI community, and inspire robust solutions to mitigate these risks in future developments.