pwc-archive/papers-with-abstracts huggingface dataset:
Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
Authors: JaeHun Jung, Faeze Brahman, Yejin Choi
Date: 2024-07-25 00:00:00
Tasks: Chatbot

We present a principled approach to provide LLM-based evaluation with a rigorous guarantee of human agreement. We first propose that a reliable evaluation method should not uncritically rely on model preferences for pairwise evaluation, but rather assess the confidence of judge models and selectively decide when to trust its judgement. We then show that under this selective evaluation framework, human agreement can be provably guaranteed -- such that the model evaluation aligns with that of humans to a user-specified agreement level. As part of our framework, we also introduce Simulated Annotators, a novel confidence estimation method that significantly improves judge calibration and thus enables high coverage of evaluated instances. Finally, we propose Cascaded Selective Evaluation, where we use cheaper models as initial judges and escalate to stronger models only when necessary -- again, while still providing a provable guarantee of human agreement. Experimental results show that Cascaded Selective Evaluation guarantees strong alignment with humans, far beyond what LLM judges could achieve without selective evaluation. For example, on a subset of Chatbot Arena where GPT-4 almost never achieves 80% human agreement, our method, even while employing substantially cost-effective models such as Mistral-7B, guarantees over 80% human agreement with almost 80% test coverage.
Estimation of Distribution Algorithms with Matrix Transpose in Bayesian Learning
Authors: Dae-Won Kim, Song Ko, Bo-Yeong Kang
Date: 2024-07-11 00:00:00
Tasks: Diversity

Estimation of distribution algorithms (EDAs) constitute a new branch of evolutionary optimization algorithms, providing effective and efficient optimization performance in a variety of research areas. Recent studies have proposed new EDAs that employ mutation operators in standard EDAs to increase the population diversity. We present a new mutation operator, a matrix transpose, specifically designed for Bayesian structure learning, and we evaluate its performance in Bayesian structure learning. The results indicate that EDAs with transpose mutation give markedly better performance than conventional EDAs.
Conversational Dueling Bandits in Generalized Linear Models
Authors: Shuhua Yang, Hui Yuan, Xiaoying Zhang, Mengdi Wang, Hong Zhang, Huazheng Wang
Date: 2024-07-26 00:00:00
Tasks: Conversational Recommendation, Informativeness, Recommendation Systems

Conversational recommendation systems elicit user preferences by interacting with users to obtain their feedback on recommended commodities. Such systems utilize a multi-armed bandit framework to learn user preferences in an online manner and have received great success in recent years. However, existing conversational bandit methods have several limitations. First, they only enable users to provide explicit binary feedback on the recommended items or categories, leading to ambiguity in interpretation. In practice, users are usually faced with more than one choice. Relative feedback, known for its informativeness, has gained increasing popularity in recommendation system design. Moreover, current contextual bandit methods mainly work under linear reward assumptions, ignoring practical non-linear reward structures in generalized linear models. Therefore, in this paper, we introduce relative feedback-based conversations into conversational recommendation systems through the integration of dueling bandits in generalized linear models (GLM) and propose a novel conversational dueling bandit algorithm called ConDuel. Theoretical analyses of regret upper bounds and empirical validations on synthetic and real-world data underscore ConDuel's efficacy. We also demonstrate the potential to extend our algorithm to multinomial logit bandits with theoretical and experimental guarantees, which further proves the applicability of the proposed framework.
RECOVAR: Representation Covariances on Deep Latent Spaces for Seismic Event Detection
Authors: Onur Efe, Arkadas Ozakin
Date: 2024-07-25 00:00:00
Tasks: Event Detection, Representation Learning, Time Series

While modern deep learning methods have shown great promise in the problem of earthquake detection, the most successful methods so far have been based on supervised learning, which requires large datasets with ground-truth labels. The curation of such datasets is both time consuming and prone to systematic biases, which result in difficulties with cross-dataset generalization, hindering general applicability. In this paper, we develop an unsupervised method for earthquake detection that learns to detect earthquakes from raw waveforms, without access to ground truth labels. The performance is comparable to, and in some cases better than, some state-of-the-art supervised methods. Moreover, the method has strong \emph{cross-dataset generalization} performance. The algorithm utilizes deep autoencoders that learn to reproduce the waveforms after a data-compressive bottleneck and uses a simple, cross-covariance-based triggering algorithm at the bottleneck for labeling. The approach has the potential to be useful for time series datasets from other domains.
Gaussian Process Kolmogorov-Arnold Networks
Authors: Andrew Siyuan Chen
Date: 2024-07-25 00:00:00
Tasks: Kolmogorov-Arnold Networks

In this paper, we introduce a probabilistic extension to Kolmogorov Arnold Networks (KANs) by incorporating Gaussian Process (GP) as non-linear neurons, which we refer to as GP-KAN. A fully analytical approach to handling the output distribution of one GP as an input to another GP is achieved by considering the function inner product of a GP function sample with the input distribution. These GP neurons exhibit robust non-linear modelling capabilities while using few parameters and can be easily and fully integrated in a feed-forward network structure. They provide inherent uncertainty estimates to the model prediction and can be trained directly on the log-likelihood objective function, without needing variational lower bounds or approximations. In the context of MNIST classification, a model based on GP-KAN of 80 thousand parameters achieved 98.5% prediction accuracy, compared to current state-of-the-art models with 1.5 million parameters.
Majorizing Stress Formula Two
Authors: Jan de Leeuw
Date: 2024-07-25 00:00:00

Modifications of the smacof algorithm for multidimensional scaling are proposed that provide a convergent majorization algorithm for Kruskal's stress formula two.
A Flexible and Scalable Approach for Collecting Wildlife Advertisements on the Web
Authors: Juliana Barbosa, Sunandan Chakraborty, Juliana Freire
Date: 2024-07-26 00:00:00

Wildlife traffickers are increasingly carrying out their activities in cyberspace. As they advertise and sell wildlife products in online marketplaces, they leave digital traces of their activity. This creates a new opportunity: by analyzing these traces, we can obtain insights into how trafficking networks work as well as how they can be disrupted. However, collecting such information is difficult. Online marketplaces sell a very large number of products and identifying ads that actually involve wildlife is a complex task that is hard to automate. Furthermore, given that the volume of data is staggering, we need scalable mechanisms to acquire, filter, and store the ads, as well as to make them available for analysis. In this paper, we present a new approach to collect wildlife trafficking data at scale. We propose a data collection pipeline that combines scoped crawlers for data discovery and acquisition with foundational models and machine learning classifiers to identify relevant ads. We describe a dataset we created using this pipeline which is, to the best of our knowledge, the largest of its kind: it contains almost a million ads obtained from 41 marketplaces, covering 235 species and 20 languages. The source code is publicly available at \url{https://github.com/VIDA-NYU/wildlife_pipeline}.
Decoding Knowledge Claims: The Evaluation of Scientific Publication Contributions through Semantic Analysis
Authors: Luca D'Aniello, Nicolas Robinson-Garcia, Massimo Aria, Corrado Cuccurullo
Date: 2024-07-26 00:00:00
Tasks: text similarity

The surge in scientific publications challenges the use of publication counts as a measure of scientific progress, requiring alternative metrics that emphasize the quality and novelty of scientific contributions rather than sheer quantity. This paper proposes the use of Relaxed Word Mover's Distance (RWMD), a semantic text similarity measure, to evaluate the novelty of scientific papers. We hypothesize that RWMD can more effectively gauge the growth of scientific knowledge. To test such an assumption, we apply RWMD to evaluate seminal papers, with Hirsch's H-Index paper as a primary case study. We compare RWMD results across three groups: 1) H-Index-related papers, 2) scientometric studies, and 3) unrelated papers, aiming to discern redundant literature and hype from genuine innovations. Findings suggest that emphasizing knowledge claims offers a deeper insight into scientific contributions, marking RWMD as a promising alternative method to traditional citation metrics, thus better tracking significant scientific breakthroughs.
REAPER: Reasoning based Retrieval Planning for Complex RAG Systems
Authors: Ashutosh Joshi, Sheikh Muhammad Sarwar, Samarth Varshney, Sreyashi Nag, Shrivats Agrawal, Juhi Naik
Date: 2024-07-26 00:00:00
Tasks: Language Modelling, Large Language Model, RAG, Retrieval, Retrieval-augmented Generation

Complex dialog systems often use retrieved evidence to facilitate factual responses. Such RAG (Retrieval Augmented Generation) systems retrieve from massive heterogeneous data stores that are usually architected as multiple indexes or APIs instead of a single monolithic source. For a given query, relevant evidence needs to be retrieved from one or a small subset of possible retrieval sources. Complex queries can even require multi-step retrieval. For example, a conversational agent on a retail site answering customer questions about past orders will need to retrieve the appropriate customer order first and then the evidence relevant to the customer's question in the context of the ordered product. Most RAG Agents handle such Chain-of-Thought (CoT) tasks by interleaving reasoning and retrieval steps. However, each reasoning step directly adds to the latency of the system. For large models this latency cost is significant -- in the order of multiple seconds. Multi-agent systems may classify the query to a single Agent associated with a retrieval source, though this means that a (small) classification model dictates the performance of a large language model. In this work we present REAPER (REAsoning-based PlannER) - an LLM based planner to generate retrieval plans in conversational systems. We show significant gains in latency over Agent-based systems and are able to scale easily to new and unseen use cases as compared to classification-based planning. Though our method can be applied to any RAG system, we show our results in the context of a conversational shopping assistant.
FedUD: Exploiting Unaligned Data for Cross-Platform Federated Click-Through Rate Prediction
Authors: Wentao Ouyang, Rui Dong, Ri Tao, Xiangzheng Liu
Date: 2024-07-26 00:00:00
Tasks: Click-Through Rate Prediction, Federated Learning, Knowledge Distillation, Prediction, Privacy Preserving, Vertical Federated Learning

Click-through rate (CTR) prediction plays an important role in online advertising platforms. Most existing methods use data from the advertising platform itself for CTR prediction. As user behaviors also exist on many other platforms, e.g., media platforms, it is beneficial to further exploit such complementary information for better modeling user interest and for improving CTR prediction performance. However, due to privacy concerns, data from different platforms cannot be uploaded to a server for centralized model training. Vertical federated learning (VFL) provides a possible solution which is able to keep the raw data on respective participating parties and learn a collaborative model in a privacy-preserving way. However, traditional VFL methods only utilize aligned data with common keys across parties, which strongly restricts their application scope. In this paper, we propose FedUD, which is able to exploit unaligned data, in addition to aligned data, for more accurate federated CTR prediction. FedUD contains two steps. In the first step, FedUD utilizes aligned data across parties like traditional VFL, but it additionally includes a knowledge distillation module. This module distills useful knowledge from the guest party's high-level representations and guides the learning of a representation transfer network. In the second step, FedUD applies the learned knowledge to enrich the representations of the host party's unaligned data such that both aligned and unaligned data can contribute to federated model training. Experiments on two real-world datasets demonstrate the superior performance of FedUD for federated CTR prediction.
Synergizing Knowledge Graphs with Large Language Models: A Comprehensive Review and Future Prospects
Authors: Daifeng Li, Fan Xu
Date: 2024-07-26 00:00:00
Tasks: Knowledge Graphs

Recent advancements have witnessed the ascension of Large Language Models (LLMs), endowed with prodigious linguistic capabilities, albeit marred by shortcomings including factual inconsistencies and opacity. Conversely, Knowledge Graphs (KGs) harbor verifiable knowledge and symbolic reasoning prowess, thereby complementing LLMs' deficiencies. Against this backdrop, the synergy between KGs and LLMs emerges as a pivotal research direction. Our contribution in this paper is a comprehensive dissection of the latest developments in integrating KGs with LLMs. Through meticulous analysis of their confluence points and methodologies, we introduce a unifying framework designed to elucidate and stimulate further exploration among scholars engaged in cognate disciplines. This framework serves a dual purpose: it consolidates extant knowledge while simultaneously delineating novel avenues for real-world deployment, thereby amplifying the translational impact of academic research.
Supporting Evidence-Based Medicine by Finding Both Relevant and Significant Works
Authors: Sameh Frihat, Norbert Fuhr
Date: 2024-07-25 00:00:00
Tasks: Retrieval

In this paper, we present a new approach to improving the relevance and reliability of medical IR, which builds upon the concept of Level of Evidence (LoE). LoE framework categorizes medical publications into 7 distinct levels based on the underlying empirical evidence. Despite LoE framework's relevance in medical research and evidence-based practice, only few medical publications explicitly state their LoE. Therefore, we develop a classification model for automatically assigning LoE to medical publications, which successfully classifies over 26 million documents in MEDLINE database into LoE classes. The subsequent retrieval experiments on TREC PM datasets show substantial improvements in retrieval relevance, when LoE is used as a search filter.
Machine Unlearning using a Multi-GAN based Model
Authors: Amartya Hatua, Trung T. Nguyen, Andrew H. Sung
Date: 2024-07-26 00:00:00
Tasks: Generative Adversarial Network, Machine Unlearning, model

This article presents a new machine unlearning approach that utilizes multiple Generative Adversarial Network (GAN) based models. The proposed method comprises two phases: i) data reorganization in which synthetic data using the GAN model is introduced with inverted class labels of the forget datasets, and ii) fine-tuning the pre-trained model. The GAN models consist of two pairs of generators and discriminators. The generator discriminator pairs generate synthetic data for the retain and forget datasets. Then, a pre-trained model is utilized to get the class labels of the synthetic datasets. The class labels of synthetic and original forget datasets are inverted. Finally, all combined datasets are used to fine-tune the pre-trained model to get the unlearned model. We have performed the experiments on the CIFAR-10 dataset and tested the unlearned models using Membership Inference Attacks (MIA). The inverted class labels procedure and synthetically generated data help to acquire valuable information that enables the model to outperform state-of-the-art models and other standard unlearning classifiers.
Large Language Model for Verilog Generation with Code-Structure-Guided Reinforcement Learning
Authors: Ning Wang, Bingkun Yao, Jie zhou, Xi Wang, Zhe Jiang, Nan Guan
Date: 2024-07-21 00:00:00
Tasks: Code Generation, Language Modeling, Language Modelling, Large Language Model, reinforcement-learning, Reinforcement Learning

Recent advancements in large language models (LLMs) have sparked significant interest in the automatic generation of Register Transfer Level (RTL) designs, particularly using Verilog. Current research on this topic primarily focuses on pre-training and instruction tuning, but the effectiveness of these methods is constrained by the limited availability of training data, as public Verilog code is far less abundant than software code. In particular, these methods struggle to effectively capture Verilog parallel code structures, which fundamentally differ from the imperative, sequential control flow typical in most software programming languages. This paper introduces VeriSeek, an LLM enhanced by reinforcement learning using a limited amount of high-quality training data to achieve high Verilog code generation performance. Our reinforcement learning approach employs code structure information as feedback signals to refine the pre-trained model, enabling it to effectively learn important patterns from Verilog code with parallel structures. Experiments show that VeriSeek outperforms state-of-the-art methods across multiple benchmarks.
ENST-Drums: an extensive audio-visual database for drum signals processing
Authors: Olivier Gillet, Gaël Richard.
Date: 2006-10-08 00:00:00
Proceeding: ISMIR 2006 2006 10
Tasks: Audio Signal Processing, Information Retrieval, Music Information Retrieval, Retrieval

One of the main bottlenecks in the progress of the Music
Information Retrieval (MIR) research field is the limited access to common, large and annotated audio databases that
could serve for technology development and/or evaluation.The aim of this paper is to present in detail the ENST-Drums
database, emphasizing on both the content and the recording process. This audiovisual database of drum performances
by three professional drummers was recorded on 8 audio channels and 2 video channels. The drum sequences are
fully annotated and will be, for a large part, freely distributed for research purposes. The large variety in its content
should serve research in various domains of audio signal processing involving drums, ranging from single drum event
classification to complex multimodal drum track transcription and extraction from polyphonic music.
A Review of Automatic Drum Transcription
Authors: CW Wu, C Dittmar, C Southall, R Vogl, G Widmer, J Hockman, M Müller, A Lerch
Date: 2018-04-26 00:00:00
Proceeding: IEEE/ACM Transactions on Audio, Speech, and Language Processing 2018 4
Tasks: Drum Transcription, Drum Transcription in Music (DTM), Information Retrieval, Music Information Retrieval, Rhythm

In Western popular music, drums and percussion are an important means to emphasize and shape the rhythm, often defining the musical style. If computers were able to analyze the drum part in recorded music, it would enable a variety of rhythm-related music processing tasks. Especially the detection and classification of drum sound events by computational methods is considered to be an important and challenging research problem in the broader field of music information retrieval. Over the last two decades, several authors have attempted to tackle this problem under the umbrella term automatic drum transcription (ADT). This paper presents a comprehensive review of ADT research, including a thorough discussion of the task-specific challenges, categorization of existing techniques, and evaluation of several state-of-the-art systems. To provide more insights on the practice of ADT systems, we focus on two families of ADT techniques, namely methods based on non-negative matrix factorization and recurrent neural networks. We explain the methods’ technical details and drum-specific variations and evaluate these approaches on publicly available data sets with a consistent experimental setup. Finally, the open issues and underexplored areas in ADT research are identified and discussed, providing future directions in this field
Rome was Not Built in a Single Step: Hierarchical Prompting for LLM-based Chip Design
Authors: Andre Nakkab, Sai Qian Zhang, Ramesh Karri, Siddharth Garg
Date: 2024-07-23 00:00:00

Large Language Models (LLMs) are effective in computer hardware synthesis via hardware description language (HDL) generation. However, LLM-assisted approaches for HDL generation struggle when handling complex tasks. We introduce a suite of hierarchical prompting techniques which facilitate efficient stepwise design methods, and develop a generalizable automation pipeline for the process. To evaluate these techniques, we present a benchmark set of hardware designs which have solutions with or without architectural hierarchy. Using these benchmarks, we compare various open-source and proprietary LLMs, including our own fine-tuned Code Llama-Verilog model. Our hierarchical methods automatically produce successful designs for complex hardware modules that standard flat prompting methods cannot achieve, allowing smaller open-source LLMs to compete with large proprietary models. Hierarchical prompting reduces HDL generation time and yields savings on LLM costs. Our experiments detail which LLMs are capable of which applications, and how to apply hierarchical methods in various modes. We explore case studies of generating complex cores using automatic scripted hierarchical prompts, including the first-ever LLM-designed processor with no human feedback. Tools for the Recurrent Optimization via Machine Editing (ROME) method can be found at https://github.com/ajn313/ROME-LLM
AICircuit: A Multi-Level Dataset and Benchmark for AI-Driven Analog Integrated Circuit Design
Authors: Asal Mehradfar, Xuzhe Zhao, Yue Niu, Sara Babakniya, Mahdi Alesheikh, Hamidreza Aghasi, Salman Avestimehr
Date: 2024-07-22 00:00:00

Analog and radio-frequency circuit design requires extensive exploration of both circuit topology and parameters to meet specific design criteria like power consumption and bandwidth. Designers must review state-of-the-art topology configurations in the literature and sweep various circuit parameters within each configuration. This design process is highly specialized and time-intensive, particularly as the number of circuit parameters increases and the circuit becomes more complex. Prior research has explored the potential of machine learning to enhance circuit design procedures. However, these studies primarily focus on simple circuits, overlooking the more practical and complex analog and radio-frequency systems. A major obstacle for bearing the power of machine learning in circuit design is the availability of a generic and diverse dataset, along with robust metrics, which are essential for thoroughly evaluating and improving machine learning algorithms in the analog and radio-frequency circuit domain. We present AICircuit, a comprehensive multi-level dataset and benchmark for developing and evaluating ML algorithms in analog and radio-frequency circuit design. AICircuit comprises seven commonly used basic circuits and two complex wireless transceiver systems composed of multiple circuit blocks, encompassing a wide array of design scenarios encountered in real-world applications. We extensively evaluate various ML algorithms on the dataset, revealing the potential of ML algorithms in learning the mapping from the design specifications to the desired circuit parameters.