pwc-archive/papers-with-abstracts huggingface dataset:
Bootstrap Your Own Context Length
Authors: Liang Wang, Nan Yang, Xingxing Zhang, Xiaolong Huang, Furu Wei
Date: 2024-12-25 00:00:00
Tasks: Language Modeling, Language Modelling

We introduce a bootstrapping approach to train long-context language models by exploiting their short-context capabilities only. Our method utilizes a simple agent workflow to synthesize diverse long-context instruction tuning data, thereby eliminating the necessity for manual data collection and annotation. The proposed data synthesis workflow requires only a short-context language model, a text retriever, and a document collection, all of which are readily accessible within the open-source ecosystem. Subsequently, language models are fine-tuned using the synthesized data to extend their context lengths. In this manner, we effectively transfer the short-context capabilities of language models to long-context scenarios through a bootstrapping process. We conduct experiments with the open-source Llama-3 family of models and demonstrate that our method can successfully extend the context length to up to 1M tokens, achieving superior performance across various benchmarks.
RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Prompting
Authors: Yilei Jiang, Yingshui Tan, Xiangyu Yue
Date: 2024-12-25 00:00:00

While Multimodal Large Language Models (MLLMs) have made remarkable progress in vision-language reasoning, they are also more susceptible to producing harmful content compared to models that focus solely on text. Existing defensive prompting techniques rely on a static, unified safety guideline that fails to account for the specific risks inherent in different multimodal contexts. To address these limitations, we propose RapGuard, a novel framework that uses multimodal chain-of-thought reasoning to dynamically generate scenario-specific safety prompts. RapGuard enhances safety by adapting its prompts to the unique risks of each input, effectively mitigating harmful outputs while maintaining high performance on benign tasks. Our experimental results across multiple MLLM benchmarks demonstrate that RapGuard achieves state-of-the-art safety performance, significantly reducing harmful content without degrading the quality of responses.
DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search
Authors: Lei Yang, Shaoyang Xu, Deyi Xiong
Date: 2024-12-25 00:00:00

Large language models (LLMs) based on the Transformer architecture usually have their context length limited due to the high training cost. Recent advancements extend the context window by adjusting the scaling factors of RoPE and fine-tuning. However, suboptimal initialization of these factors results in increased fine-tuning costs and reduced performance at target length. To address these challenges, we propose an innovative RoPE-based fine-tuning framework that diverges from conventional scaling factors search. Specifically, we present a Divide-and-Conquer Incremental Search (DCIS) algorithm that strategically determines the better scaling factors. Further fine-tuning with the identified scaling factors effectively extends the context window of LLMs. Empirical results demonstrate that our methodology not only mitigates performance decay at extended target lengths but also allows the model to fine-tune on short contexts and generalize to long contexts, thereby reducing the cost of fine-tuning. The scaling factors obtained through DCIS can even perform effectively without fine-tuning. Further analysis of the search space reveals that DCIS achieves twice the search efficiency compared to other methods. We also examine the impact of the non-strictly increasing scaling factors utilized in DCIS and evaluate the general capabilities of LLMs across various context lengths.
Improving Generated and Retrieved Knowledge Combination Through Zero-shot Generation
Authors: Xinkai Du, Quanjie Han, Chao Lv, Yan Liu, Yalin Sun, Hao Shu, Hongbo Shan, Maosong Sun
Date: 2024-12-25 00:00:00
Tasks: Open-Domain Question Answering, Question Answering, Reranking, Re-Ranking, TriviaQA

Open-domain Question Answering (QA) has garnered substantial interest by combining the advantages of faithfully retrieved passages and relevant passages generated through Large Language Models (LLMs). However, there is a lack of definitive labels available to pair these sources of knowledge. In order to address this issue, we propose an unsupervised and simple framework called Bi-Reranking for Merging Generated and Retrieved Knowledge (BRMGR), which utilizes re-ranking methods for both retrieved passages and LLM-generated passages. We pair the two types of passages using two separate re-ranking methods and then combine them through greedy matching. We demonstrate that BRMGR is equivalent to employing a bipartite matching loss when assigning each retrieved passage with a corresponding LLM-generated passage. The application of our model yielded experimental results from three datasets, improving their performance by +1.7 and +1.6 on NQ and WebQ datasets, respectively, and obtaining comparable result on TriviaQA dataset when compared to competitive baselines.
Using Large Language Models for Automated Grading of Student Writing about Science
Authors: Chris Impey, Matthew Wenger, Nikhil Garuda, Shahriar Golchin, Sarah Stamer
Date: 2024-12-25 00:00:00
Tasks: Astronomy, Multiple-choice, Philosophy

Assessing writing in large classes for formal or informal learners presents a significant challenge. Consequently, most large classes, particularly in science, rely on objective assessment tools such as multiple-choice quizzes, which have a single correct answer. The rapid development of AI has introduced the possibility of using large language models (LLMs) to evaluate student writing. An experiment was conducted using GPT-4 to determine if machine learning methods based on LLMs can match or exceed the reliability of instructor grading in evaluating short writing assignments on topics in astronomy. The audience consisted of adult learners in three massive open online courses (MOOCs) offered through Coursera. One course was on astronomy, the second was on astrobiology, and the third was on the history and philosophy of astronomy. The results should also be applicable to non-science majors in university settings, where the content and modes of evaluation are similar. The data comprised answers from 120 students to 12 questions across the three courses. GPT-4 was provided with total grades, model answers, and rubrics from an instructor for all three courses. In addition to evaluating how reliably the LLM reproduced instructor grades, the LLM was also tasked with generating its own rubrics. Overall, the LLM was more reliable than peer grading, both in aggregate and by individual student, and approximately matched instructor grades for all three online courses. The implication is that LLMs may soon be used for automated, reliable, and scalable grading of student science writing.
Multiple References with Meaningful Variations Improve Literary Machine Translation
Authors: Si Wu, John Wieting, David A. Smith
Date: 2024-12-24 00:00:00
Tasks: Machine Translation, Semantic Similarity, Semantic Textual Similarity, Sentence, Translation

While a source sentence can be translated in many ways, most machine translation (MT) models are trained with only a single reference. Previous work has shown that using synthetic paraphrases can improve MT. This paper investigates best practices for employing multiple references by analyzing the semantic similarity among different English translations of world literature in the Par3 dataset. We classify the semantic similarity between paraphrases into three groups: low, medium, and high, and fine-tune two different LLMs (mT5-large and LLaMA-2-7B) for downstream MT tasks. Across different models, holding the total training instances constant, single-reference but more source texts only marginally outperforms multiple-reference with half of the source texts. Moreover, using paraphrases of medium and high semantic similarity outperforms an unfiltered dataset (+BLEU 0.3-0.5, +COMET 0.2-0.9, +chrF++ 0.25-0.32). Our code is publicly available on GitHub.
Simple is not Enough: Document-level Text Simplification using Readability and Coherence
Authors: Laura Vásquez-Rodríguez, Nhung T. H. Nguyen, Piotr Przybyła, Matthew Shardlow, Sophia Ananiadou
Date: 2024-12-24 00:00:00
Tasks: Sentence, Text Simplification

In this paper, we present the SimDoc system, a simplification model considering simplicity, readability, and discourse aspects, such as coherence. In the past decade, the progress of the Text Simplification (TS) field has been mostly shown at a sentence level, rather than considering paragraphs or documents, a setting from which most TS audiences would benefit. We propose a simplification system that is initially fine-tuned with professionally created corpora. Further, we include multiple objectives during training, considering simplicity, readability, and coherence altogether. Our contributions include the extension of professionally annotated simplification corpora by the association of existing annotations into (complex text, simple text, readability label) triples to benefit from readability during training. Also, we present a comparative analysis in which we evaluate our proposed models in a zero-shot, few-shot, and fine-tuning setting using document-level TS corpora, demonstrating novel methods for simplification. Finally, we show a detailed analysis of outputs, highlighting the difficulties of simplification at a document level.
Quantum memristors for neuromorphic quantum machine learning
Authors: Lucas Lamata
Date: 2024-12-25 00:00:00
Tasks: Quantum Machine Learning

Quantum machine learning may permit to realize more efficient machine learning calculations with near-term quantum devices. Among the diverse quantum machine learning paradigms which are currently being considered, quantum memristors are promising as a way of combining, in the same quantum hardware, a unitary evolution with the nonlinearity provided by the measurement and feedforward. Thus, an efficient way of deploying neuromorphic quantum computing for quantum machine learning may be enabled.
Generalized Grade-of-Membership Estimation for High-dimensional Locally Dependent Data
Authors: Ling Chen, Chengzhu Huang, Yuqi Gu
Date: 2024-12-27 00:00:00

This work focuses on the mixed membership models for multivariate categorical data widely used for analyzing survey responses and population genetics data. These grade of membership (GoM) models offer rich modeling power but present significant estimation challenges for high-dimensional polytomous data. Popular existing approaches, such as Bayesian MCMC inference, are not scalable and lack theoretical guarantees in high-dimensional settings. To address this, we first observe that data from this model can be reformulated as a three-way (quasi-)tensor, with many subjects responding to many items with varying numbers of categories. We introduce a novel and simple approach that flattens the three-way quasi-tensor into a "fat" matrix, and then perform a singular value decomposition of it to estimate parameters by exploiting the singular subspace geometry. Our fast spectral method can accommodate a broad range of data distributions with arbitrarily locally dependent noise, which we formalize as the generalized-GoM models. We establish finite-sample entrywise error bounds for the generalized-GoM model parameters. This is supported by a new sharp two-to-infinity singular subspace perturbation theory for locally dependent and flexibly distributed noise, a contribution of independent interest. Simulations and applications to data in political surveys, population genetics, and single-cell sequencing demonstrate our method's superior performance.
Constructive approximate transport maps with normalizing flows
Authors: Antonio Álvarez-López, Borjan Geshkovski, Domènec Ruiz-Balet
Date: 2024-12-26 00:00:00

We study an approximate controllability problem for the continuity equation and its application to constructing transport maps with normalizing flows. Specifically, we construct time-dependent controls $\theta=(w, a, b)$ in the vector field $x\mapsto w(a^\top x + b)_+$ to approximately transport a known base density $\rho_{\mathrm{B}}$ to a target density $\rho_*$. The approximation error is measured in relative entropy, and $\theta$ are constructed piecewise constant, with bounds on the number of switches being provided. Our main result relies on an assumption on the relative tail decay of $\rho_*$ and $\rho_{\mathrm{B}}$, and provides hints on characterizing the reachable space of the continuity equation in relative entropy.
Stationary Processes, Wiener-Granger Causality, and Matrix Spectral Factorization
Authors: Lasha Ephremidze
Date: 2024-12-25 00:00:00
Tasks: Time Series

Granger causality has become an indispensable tool for analyzing causal relationships between time series. In this paper, we provide a detailed overview of its mathematical foundations, trace its historical development, and explore how recent computational advancements can enhance its application in various fields. We will not hesitate to present the proofs in full if they are simple and transparent. For more complex theorems on which we rely, we will provide supporting citations. We also discuss potential future directions for the method, particularly in the context of largescale data analysis.
High-accuracy sampling from constrained spaces with the Metropolis-adjusted Preconditioned Langevin Algorithm
Authors: Vishwak Srinivasan, Andre Wibisono, Ashia Wilson
Date: 2024-12-24 00:00:00

In this work, we propose a first-order sampling method called the Metropolis-adjusted Preconditioned Langevin Algorithm for approximate sampling from a target distribution whose support is a proper convex subset of $\mathbb{R}^{d}$. Our proposed method is the result of applying a Metropolis-Hastings filter to the Markov chain formed by a single step of the preconditioned Langevin algorithm with a metric $\mathscr{G}$, and is motivated by the natural gradient descent algorithm for optimisation. We derive non-asymptotic upper bounds for the mixing time of this method for sampling from target distributions whose potentials are bounded relative to $\mathscr{G}$, and for exponential distributions restricted to the support. Our analysis suggests that if $\mathscr{G}$ satisfies stronger notions of self-concordance introduced in Kook and Vempala (2024), then these mixing time upper bounds have a strictly better dependence on the dimension than when is merely self-concordant. We also provide numerical experiments that demonstrates the practicality of our proposed method. Our method is a high-accuracy sampler due to the polylogarithmic dependence on the error tolerance in our mixing time upper bounds.
RecLM: Recommendation Instruction Tuning
Authors: Yangqin Jiang, Yuhao Yang, Lianghao Xia, Da Luo, Kangyi Lin, Chao Huang
Date: 2024-12-26 00:00:00
Tasks: Collaborative Filtering, Diversity, Language Modelling, Recommendation Systems

Modern recommender systems aim to deeply understand users' complex preferences through their past interactions. While deep collaborative filtering approaches using Graph Neural Networks (GNNs) excel at capturing user-item relationships, their effectiveness is limited when handling sparse data or zero-shot scenarios, primarily due to constraints in ID-based embedding functions. To address these challenges, we propose a model-agnostic recommendation instruction-tuning paradigm that seamlessly integrates large language models with collaborative filtering. Our proposed $\underline{Rec}$ommendation $\underline{L}$anguage $\underline{M}$odel (RecLM) enhances the capture of user preference diversity through a carefully designed reinforcement learning reward function that facilitates self-augmentation of language models. Comprehensive evaluations demonstrate significant advantages of our approach across various settings, and its plug-and-play compatibility with state-of-the-art recommender systems results in notable performance enhancements. The implementation of our RecLM framework is publicly available at: https://github.com/HKUDS/RecLM.
Towards Popularity-Aware Recommendation: A Multi-Behavior Enhanced Framework with Orthogonality Constraint
Authors: Yishan Han, Biao Xu, Yao Wang, Shanxing Gao
Date: 2024-12-26 00:00:00
Tasks: Recommendation Systems, Sensitivity

Top-$K$ recommendation involves inferring latent user preferences and generating personalized recommendations accordingly, which is now ubiquitous in various decision systems. Nonetheless, recommender systems usually suffer from severe \textit{popularity bias}, leading to the over-recommendation of popular items. Such a bias deviates from the central aim of reflecting user preference faithfully, compromising both customer satisfaction and retailer profits. Despite the prevalence, existing methods tackling popularity bias still have limitations due to the considerable accuracy-debias tradeoff and the sensitivity to extensive parameter selection, further exacerbated by the extreme sparsity in positive user-item interactions. In this paper, we present a \textbf{Pop}ularity-aware top-$K$ recommendation algorithm integrating multi-behavior \textbf{S}ide \textbf{I}nformation (PopSI), aiming to enhance recommendation accuracy and debias performance simultaneously. Specifically, by leveraging multiple user feedback that mirrors similar user preferences and formulating it as a three-dimensional tensor, PopSI can utilize all slices to capture the desiring user preferences effectively. Subsequently, we introduced a novel orthogonality constraint to refine the estimated item feature space, enforcing it to be invariant to item popularity features thereby addressing our model's sensitivity to popularity bias. Comprehensive experiments on real-world e-commerce datasets demonstrate the general improvements of PopSI over state-of-the-art debias methods with a marginal accuracy-debias tradeoff and scalability to practical applications. The source code for our algorithm and experiments is available at \url{https://github.com/Eason-sys/PopSI}.
Jasper and Stella: distillation of SOTA embedding models
Authors: Dun Zhang, Jiacheng Li, Ziyang Zeng, Fulong Wang
Date: 2024-12-26 00:00:00
Tasks: RAG, Representation Learning, Retrieval, Retrieval-augmented Generation

A crucial component in many deep learning applications, such as Frequently Asked Questions (FAQ) and Retrieval-Augmented Generation (RAG), is dense retrieval. In this process, embedding models transform raw text into numerical vectors. However, the embedding models that currently excel on text embedding benchmarks, like the Massive Text Embedding Benchmark (MTEB), often have numerous parameters and high vector dimensionality. This poses challenges for their application in real-world scenarios. To address this issue, we propose a novel multi-stage distillation framework that enables a smaller student embedding model to distill multiple larger teacher embedding models through three carefully designed losses. Meanwhile, we utilize Matryoshka Representation Learning (MRL) to reduce the vector dimensionality of the student embedding model effectively. Our student model named Jasper with 2 billion parameters, built upon the Stella embedding model, obtained the No.3 position on the MTEB leaderboard (as of December 24, 2024), achieving an average 71.54 score across 56 datasets. We have released the model and data on the Hugging Face Hub (https://huggingface.co/infgrad/jasper_en_vision_language_v1) (https://huggingface.co/datasets/infgrad/jasper_text_distill_dataset), and the training codes are available in this project repository (https://github.com/NLPJCL/RAG-Retrieval).
Musings About the Future of Search: A Return to the Past?
Authors: Jimmy Lin, Pankaj Gupta, Will Horn, Gilad Mishne
Date: 2024-12-25 00:00:00

When you have a question, the most effective way to have the question answered is to directly connect with experts on the topic and have a conversation with them. Prior to the invention of writing, this was the only way. Although effective, this solution exhibits scalability challenges. Writing allowed knowledge to be materialized, preserved, and replicated, enabling the development of different technologies over the centuries to connect information seekers with relevant information. This progression ultimately culminated in the ten-blue-links web search paradigm we're familiar with, just before the recent emergence of generative AI. However, we often forget that consuming static content is an imperfect solution. With the advent of large language models, it has become possible to develop a superior experience by allowing users to directly engage with experts. These interactions can of course satisfy information needs, but expert models can do so much more. This coming future requires reimagining search.
Zema Dataset: A Comprehensive Study of Yaredawi Zema with a Focus on Horologium Chants
Authors: Mequanent Argaw Muluneh, Yan-Tsung Peng, Worku Abebe Degife, Nigussie Abate Tadesse, Aknachew Mebreku Demeku, Li Su
Date: 2024-12-25 00:00:00
Tasks: Music Generation

Computational music research plays a critical role in advancing music production, distribution, and understanding across various musical styles worldwide. Despite the immense cultural and religious significance, the Ethiopian Orthodox Tewahedo Church (EOTC) chants are relatively underrepresented in computational music research. This paper contributes to this field by introducing a new dataset specifically tailored for analyzing EOTC chants, also known as Yaredawi Zema. This work provides a comprehensive overview of a 10-hour dataset, 369 instances, creation, and curation process, including rigorous quality assurance measures. Our dataset has a detailed word-level temporal boundary and reading tone annotation along with the corresponding chanting mode label of audios. Moreover, we have also identified the chanting options associated with multiple chanting notations in the manuscript by annotating them accordingly. Our goal in making this dataset available to the public 1 is to encourage more research and study of EOTC chants, including lyrics transcription, lyric-to-audio alignment, and music generation tasks. Such research work will advance knowledge and efforts to preserve this distinctive liturgical music, a priceless cultural artifact for the Ethiopian people.
Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks Against Black-box Neural Ranking Models
Authors: Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng
Date: 2024-12-25 00:00:00

Neural ranking models (NRMs) have been shown to be highly effective in terms of retrieval performance. Unfortunately, they have also displayed a higher degree of sensitivity to attacks than previous generation models. To help expose and address this lack of robustness, we introduce a novel ranking attack framework named Attack-in-the-Chain, which tracks interactions between large language models (LLMs) and NRMs based on chain-of-thought (CoT) prompting to generate adversarial examples under black-box settings. Our approach starts by identifying anchor documents with higher ranking positions than the target document as nodes in the reasoning chain. We then dynamically assign the number of perturbation words to each node and prompt LLMs to execute attacks. Finally, we verify the attack performance of all nodes at each reasoning step and proceed to generate the next reasoning step. Empirical results on two web search benchmarks show the effectiveness of our method.