pwc-archive/papers-with-abstracts huggingface dataset:
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
Authors: Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J. Pal, Siva Reddy
Date: 2025-04-11 00:00:00

Web agents enable users to perform tasks on web browsers through natural language interaction. Evaluating web agents trajectories is an important problem, since it helps us determine whether the agent successfully completed the tasks. Rule-based methods are widely used for this purpose, but they are challenging to extend to new tasks and may not always recognize successful trajectories. We may achieve higher accuracy through human evaluation, but the process would be substantially slower and more expensive. Automatic evaluations with LLMs may avoid the challenges of designing new rules and manually annotating trajectories, enabling faster and cost-effective evaluation. However, it is unclear how effective they are at evaluating web agents. To this end, we propose AgentRewardBench, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents. AgentRewardBench contains 1302 trajectories across 5 benchmarks and 4 LLMs. Each trajectory in AgentRewardBench is reviewed by an expert, who answers questions pertaining to the success, side effects, and repetitiveness of the agent. Using our benchmark, we evaluate 12 LLM judges and find that no single LLM excels across all benchmarks. We also find that the rule-based evaluation used by common benchmarks tends to underreport the success rate of web agents, highlighting a key weakness of rule-based evaluation and the need to develop more flexible automatic evaluations. We release the benchmark at: https://agent-reward-bench.github.io
MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?
Authors: Yunxiang Zhang, Muhammad Khalifa, Shitanshu Bhushan, Grant D Murphy, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, Lu Wang
Date: 2025-04-13 00:00:00
Tasks: Large Language Model, scientific discovery

We introduce MLRC-Bench, a benchmark designed to quantify how effectively language agents can tackle challenging Machine Learning (ML) Research Competitions, with a focus on open research problems that demand novel methodologies. Unlike prior work, e.g., AI Scientist, which evaluates the end-to-end agentic pipeline by using LLM-as-a-judge, MLRC-Bench measures the key steps of proposing and implementing novel research methods and evaluates them with rigorous protocol and objective metrics. Our curated suite of 7 competition tasks reveals significant challenges for LLM agents. Even the best-performing tested agent (gemini-exp-1206 under MLAB) closes only 9.3% of the gap between baseline and top human participant scores. Furthermore, our analysis reveals a misalignment between the LLM-judged innovation and actual performance on cutting-edge ML research problems. MLRC-Bench is a dynamic benchmark, designed to grow with new ML competitions and encourage rigorous, objective evaluations of AI research capabilities. Our leaderboard and code are available at: https://huggingface.co/spaces/launch/MLRC_Bench
Reasoning Court: Combining Reasoning, Action, and Judgment for Multi-Hop Reasoning
Authors: Jingtian Wu, Claire Cardie
Date: 2025-04-14 00:00:00
Tasks: Fact Verification, Question Answering, Retrieval

While large language models (LLMs) have demonstrated strong capabilities in tasks like question answering and fact verification, they continue to suffer from hallucinations and reasoning errors, especially in multi-hop tasks that require integration of multiple information sources. Current methods address these issues through retrieval-based techniques (grounding reasoning in external evidence), reasoning-based approaches (enhancing coherence via improved prompting), or hybrid strategies combining both elements. One prominent hybrid method, ReAct, has outperformed purely retrieval-based or reasoning-based approaches; however, it lacks internal verification of intermediate reasoning steps, allowing potential errors to propagate through complex reasoning tasks. In this paper, we introduce Reasoning Court (RC), a novel framework that extends iterative reasoning-and-retrieval methods, such as ReAct, with a dedicated LLM judge. Unlike ReAct, RC employs this judge to independently evaluate multiple candidate answers and their associated reasoning generated by separate LLM agents. The judge is asked to select the answer that it considers the most factually grounded and logically coherent based on the presented reasoning and evidence, or synthesizes a new answer using available evidence and its pre-trained knowledge if all candidates are inadequate, flawed, or invalid. Evaluations on multi-hop benchmarks (HotpotQA, MuSiQue) and fact-verification (FEVER) demonstrate that RC consistently outperforms state-of-the-art few-shot prompting methods without task-specific fine-tuning.
Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
Authors: Yuran Li, Jama Hussein Mohamud, Chongren Sun, Di wu, Benoit Boulet
Date: 2025-04-23 00:00:00

Large language models (LLMs) are being widely applied across various fields, but as tasks become more complex, evaluating their responses is increasingly challenging. Compared to human evaluators, the use of LLMs to support performance evaluation offers a more efficient alternative. However, most studies focus mainly on aligning LLMs' judgments with human preferences, overlooking the existence of biases and mistakes in human judgment. Furthermore, how to select suitable LLM judgments given multiple potential LLM responses remains underexplored. To address these two aforementioned issues, we propose a three-stage meta-judge selection pipeline: 1) developing a comprehensive rubric with GPT-4 and human experts, 2) using three advanced LLM agents to score judgments, and 3) applying a threshold to filter out low-scoring judgments. Compared to methods using a single LLM as both judge and meta-judge, our pipeline introduces multi-agent collaboration and a more comprehensive rubric. Experimental results on the JudgeBench dataset show about 15.55\% improvement compared to raw judgments and about 8.37\% improvement over the single-agent baseline. Our work demonstrates the potential of LLMs as meta-judges and lays the foundation for future research on constructing preference datasets for LLM-as-a-judge reinforcement learning.
Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models
Authors: José Pombal, Nuno M. Guerreiro, Ricardo Rei, André F. T. Martins
Date: 2025-04-01 00:00:00
Tasks: Benchmarking

As language models improve and become capable of performing more complex tasks across modalities, evaluating them automatically becomes increasingly challenging. Developing strong and robust task-specific automatic metrics gets harder, and human-annotated test sets -- which are expensive to create -- saturate more quickly. A compelling alternative is to design reliable strategies to automate the creation of test data and evaluation, but previous attempts either rely on pre-existing data, or focus solely on individual tasks. We present Zero-shot Benchmarking (ZSB), a framework for creating high-quality benchmarks for any task by leveraging language models for both synthetic test data creation and evaluation. ZSB is simple and flexible: it requires only the creation of a prompt for data generation and one for evaluation; it is scalable to tasks and languages where collecting real-world data is costly or impractical; it is model-agnostic, allowing the creation of increasingly challenging benchmarks as models improve. To assess the effectiveness of our framework, we create benchmarks for five text-only tasks and a multi-modal one: general capabilities in four languages (English, Chinese, French, and Korean), translation, and general vision-language capabilities in English. We then rank a broad range of open and closed systems on our benchmarks. ZSB rankings consistently correlate strongly with human rankings, outperforming widely-adopted standard benchmarks. Through ablations, we find that strong benchmarks can be created with open models, and that judge model size and dataset variety are crucial drivers of performance. We release all our benchmarks, and code to reproduce our experiments and to produce new benchmarks.
Climate Research Domain BERTs: Pretraining, Adaptation, and Evaluation
Authors: Andrija Poleksić, Sanda Martinčić-Ipšić
Date: 2025-05-19 00:00:00
Proceeding: Preprint 2025 5
Tasks: Text Classification

Motivated by the pressing issue of climate change and the growing volume of data, we pretrain three new language models using climate change research papers published in top-tier journals. Adaptation of existing domain-specific models is utilized for CliSciBERT and SciClimateBERT and pretraining from scratch resulted in CliReBERT (Climate Research BERT). The performance assessment is performed on the climate change NLP benchmark ClimaBench. We evaluate SciBERT, ClimateBERT, BERT, RoBERTa and DistilRoBERTa - along with our new models - CliReBERT, CliSciBERT and SciClimateBERT - using five different random seeds on all seven ClimaBench datasets.

CliReBERT achieves the highest overall performance with a macro-averaged F1 score of 65.45%, and outperforms all other models on three of the seven tasks. Additionally, CliReBERT demonstrates the most stable fine-tuning behavior, yielding the lowest average standard deviation across seeds (0.0118). The 5-fold stratified cross-validation on the SciDCC dataset showed that CliReBERT achieved the highest overall macro-average F1 score (53.75%), slightly outperforming RoBERTa and DistilRoBERTa, while the domain-adapted models underperformed their base counterparts. The superior performance of CliReBERT is accompanied by the lowest tokenizer fertility, suggesting appropriateness to model domain-specific vocabulary.
GOUHFI: a novel contrast- and resolution-agnostic segmentation tool for Ultra-High Field MRI
Authors: Marc-Antoine Fortin, Anne Louise Kristoffersen, Michael Staff Larsen, Laurent Lamalle, Ruediger Stirnberg, Paal Erik Goa
Date: 2025-05-16 00:00:00
Tasks: Brain Segmentation, Segmentation

Recently, Ultra-High Field MRI (UHF-MRI) has become more available and one of the best tools to study the brain. One common step in quantitative neuroimaging is the brain segmentation. However, the differences between UHF-MRI and 1.5-3T images are such that the automatic segmentation techniques optimized at these field strengths usually produce unsatisfactory segmentation results for UHF images. It has been particularly challenging to perform quantitative analyses as typically done with 1.5-3T data, considerably limiting the potential of UHF-MRI. Hence, we propose a novel Deep Learning (DL)-based segmentation technique called GOUHFI: Generalized and Optimized segmentation tool for Ultra-High Field Images, designed to segment UHF images of various contrasts and resolutions. For training, we used a total of 206 label maps from four datasets acquired at 3T, 7T and 9.4T. In contrast to most DL strategies, we used a previously proposed domain randomization approach, where synthetic images generated from the label maps were used for training a 3D U-Net. GOUHFI was tested on seven different datasets and compared to techniques like FastSurferVINN and CEREBRUM-7T. GOUHFI was able to the segment six contrasts and seven resolutions tested at 3T, 7T and 9.4T. Average Dice-Sorensen Similarity Coefficient (DSC) scores of 0.87, 0.84, 0.91 were computed against the ground truth segmentations at 3T, 7T and 9.4T. Moreover, GOUHFI demonstrated impressive resistance to the typical inhomogeneities observed at UHF-MRI, making it a new powerful segmentation tool that allows to apply the usual quantitative analysis pipelines also at UHF. Ultimately, GOUHFI is a promising new segmentation tool, being the first of its kind proposing a contrast- and resolution-agnostic alternative for UHF-MRI, making it the forthcoming alternative for neuroscientists working with UHF-MRI or even lower field strengths.
Stress Detection Using Frequency Spectrum Analysis of Wrist-Measured Electrodermal Activity
Authors: Žiga Stržinar, Araceli Sanchis, Agapito Ledezma, Oscar Sipele, Boštjan Pregelj, Igor Škrjanc
Date: 2023-01-14 00:00:00
Proceeding: Sensors 2023 1

The article deals with the detection of stress using the electrodermal activity (EDA) signal
 measured at the wrist. We present an approach for feature extraction from EDA. The approach
 uses frequency spectrum analysis in multiple frequency bands. We evaluate the proposed approach
 using the 4 Hz EDA signal measured at the wrist in the publicly available Wearable Stress and
 Affect Detection (WESAD) dataset. Seven existing approaches to stress detection using EDA signals
 measured by wrist-worn sensors are analysed and the reported results are compared with ours.
 The proposed approach represents an improvement in accuracy over the other techniques studied.
 Moreover, we focus on time to detection (TTD) and show that our approach is able to outperform
 competing techniques, with fewer data points. The proposed feature extraction is computationally
 inexpensive, thus the presented approach is suitable for use in real-world wearable applications
 where both short response times and high detection performance are important. We report both
 binary (stress vs. no stress) as well as three-class (baseline/stress/amusement) results.
Man-in-the-Middle Intrusion Detection Based on CNN-LSTM Model
Authors: Jie Luo
Date: 2023-07-17 00:00:00
Proceeding: IEEE 2023 7
Tasks: Intrusion Detection

With the development of the network, the means of network attacks emerge one after another. Man-in-the-middle(MITM) attack is a kind of high threat, difficult to prevent and very common network attack. Aiming at the man-in-the-middle intrusion detection problem in the network, especially in the Internet of Things(lot), this article proposes a man-in-the-middle attack intrusion detection model based on the combination of Convolution Neural Networks(CNN) and Long Short-Term Memory networks(LSTM). The model sets two Convolution layers and two LSTM layers. Firstly, convolutional neural network is used for feature analysis, then the LSTM is used to classify learning. The data set selects "BoTNeTIoT-L01". By model training, the accuracy of prediction results can reach 99.50%. Finally, the confusion matrix and AUC area are used to evaluate the model. The evaluation results show that CNN-LSTM has a very good predictive effect in man-in-the-middle attack intrusion detection.
abdulraheemalaro
Authors: abdulraheemalaro
Date: 2025-05-19 00:00:00
Proceeding: 2025 2025 5
Tasks: Marketing

I am Hakeem i was born in September 12 2012 His parent are Mr. and Mrs. AbdulRaheem who are native of kwara state,but the live in Ilorin, kwara State. state, Nigeria. he went to His Bunmi Harmony Nursery and Primary. school in ilorin, when he is in pry 5 he left HisBunmi Harmony Nursery and Primary School and went to First Baptist high school, ilorin and went to JoelBest. When he was in SS1 Student he was introduced to a great business know as Network Marketing [Neolife] as he joined the business he has became a freelancer working on Fiverr, Upwork, kwork, freelance platform with many project ( Social Media marketer, Programming projects, Digital Marketing ) but sometimes specialized on Digital Marketing service: Google knowledge panel, Tiktok promotion, Telegram promotion Google review removing and adding, IMDb page creation, wikigenius creation, wiki fandom and so on. He has been doing this for the past 4 years. Career olamilekan start his career in the years 2022 , Because he join a great and awesome business opportunity call [1]Neolife. And he is also expert in online business that he specialize on Google knowledge panel Marketing , social media marketing,
Mega-scale experimental analysis of protein folding stability in biology and design
Authors: Kotaro Tsuboyama, Justas  Dauparas, Jonathan Chen, Elodie  Laine, Yasser Mohseni Behbahani, Jonathan J.  Weinstein, Niall M.  Mangan, Sergey Ovchinnikov, Gabriel J. Rocklin
Date: 2023-07-19 00:00:00
Proceeding: Nature 2023 7
Tasks: Protein Folding, Protein Stability Prediction

Advances in DNA sequencing and machine learning are providing insights into protein sequences and structures on an enormous scale1. However, the energetics driving folding are invisible in these structures and remain largely unknown2. The hidden thermodynamics of folding can drive disease3,4, shape protein evolution5,6,7 and guide protein engineering8,9,10, and new approaches are needed to reveal these thermodynamics for every sequence and structure. Here we present cDNA display proteolysis, a method for measuring thermodynamic folding stability for up to 900,000 protein domains in a one-week experiment. From 1.8 million measurements in total, we curated a set of around 776,000 high-quality folding stabilities covering all single amino acid variants and selected double mutants of 331 natural and 148 de novo designed protein domains 40–72 amino acids in length. Using this extensive dataset, we quantified (1) environmental factors influencing amino acid fitness, (2) thermodynamic couplings (including unexpected interactions) between protein sites, and (3) the global divergence between evolutionary amino acid usage and protein folding stability. We also examined how our approach could identify stability determinants in designed proteins and evaluate design methods. The cDNA display proteolysis method is fast, accurate and uniquely scalable, and promises to reveal the quantitative rules for how amino acid sequences encode folding stability.
Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2
Authors: Xiang Fu
Date: 2025-05-16 00:00:00

We demonstrate that a developmentally ordered curriculum markedly improves reasoning transparency and sample-efficiency in small language models (SLMs). Concretely, we train Cognivolve, a 124 M-parameter GPT-2 model, on a four-stage syllabus that ascends from lexical matching to multi-step symbolic inference and then evaluate it without any task-specific fine-tuning. Cognivolve reaches target accuracy in half the optimization steps of a single-phase baseline, activates an order-of-magnitude more gradient-salient reasoning heads, and shifts those heads toward deeper layers, yielding higher-entropy attention that balances local and long-range context. The same curriculum applied out of order or with optimizer resets fails to reproduce these gains, confirming that progression--not extra compute--drives the effect. We also identify open challenges: final-answer success still lags a conventional run by about 30%, and our saliency probe under-detects verbal-knowledge heads in the hardest stage, suggesting directions for mixed-stage fine-tuning and probe expansion.
SoftPQ: Robust Instance Segmentation Evaluation via Soft Matching and Tunable Thresholds
Authors: Ranit Karmakar, Simon F. Nørrelykke
Date: 2025-05-17 00:00:00
Tasks: Benchmarking, Binary Classification, Instance Segmentation, Segmentation, Semantic Segmentation

Segmentation evaluation metrics traditionally rely on binary decision logic: predictions are either correct or incorrect, based on rigid IoU thresholds. Detection--based metrics such as F1 and mAP determine correctness at the object level using fixed overlap cutoffs, while overlap--based metrics like Intersection over Union (IoU) and Dice operate at the pixel level, often overlooking instance--level structure. Panoptic Quality (PQ) attempts to unify detection and segmentation assessment, but it remains dependent on hard-threshold matching--treating predictions below the threshold as entirely incorrect. This binary framing obscures important distinctions between qualitatively different errors and fails to reward gradual model improvements. We propose SoftPQ, a flexible and interpretable instance segmentation metric that redefines evaluation as a graded continuum rather than a binary classification. SoftPQ introduces tunable upper and lower IoU thresholds to define a partial matching region and applies a sublinear penalty function to ambiguous or fragmented predictions. These extensions allow SoftPQ to exhibit smoother score behavior, greater robustness to structural segmentation errors, and more informative feedback for model development and evaluation. Through controlled perturbation experiments, we show that SoftPQ captures meaningful differences in segmentation quality that existing metrics overlook, making it a practical and principled alternative for both benchmarking and iterative model refinement.
Attribution Projection Calculus: A Novel Framework for Causal Inference in Bayesian Networks
Authors: M Ruhul Amin
Date: 2025-05-17 00:00:00
Tasks: Causal Inference, Fairness

This paper introduces Attribution Projection Calculus (AP-Calculus), a novel mathematical framework for determining causal relationships in structured Bayesian networks. We investigate a specific network architecture with source nodes connected to destination nodes through intermediate nodes, where each input maps to a single label with maximum marginal probability. We prove that for each label, exactly one intermediate node acts as a deconfounder while others serve as confounders, enabling optimal attribution of features to their corresponding labels. The framework formalizes the dual nature of intermediate nodes as both confounders and deconfounders depending on the context, and establishes separation functions that maximize distinctions between intermediate representations. We demonstrate that the proposed network architecture is optimal for causal inference compared to alternative structures, including those based on Pearl's causal framework. AP-Calculus provides a comprehensive mathematical foundation for analyzing feature-label attributions, managing spurious correlations, quantifying information gain, ensuring fairness, and evaluating uncertainty in prediction models, including large language models. Theoretical verification shows that AP-Calculus not only extends but can also subsume traditional do-calculus for many practical applications, offering a more direct approach to causal inference in supervised learning contexts.