pwc-archive/papers-with-abstracts huggingface dataset:
Counter Turing Test ($CT^2$): Investigating AI-Generated Text Detection for Hindi -- Ranking LLMs based on Hindi AI Detectability Index ($ADI_{hi}$)
Authors: Ishan Kavathekar, Anku Rani, Ashmit Chamoli, Ponnurangam Kumaraguru, Amit Sheth, Amitava Das
Date: 2024-07-22 00:00:00
Tasks: Text Detection

The widespread adoption of Large Language Models (LLMs) and awareness around multilingual LLMs have raised concerns regarding the potential risks and repercussions linked to the misapplication of AI-generated text, necessitating increased vigilance. While these models are primarily trained for English, their extensive training on vast datasets covering almost the entire web, equips them with capabilities to perform well in numerous other languages. AI-Generated Text Detection (AGTD) has emerged as a topic that has already received immediate attention in research, with some initial methods having been proposed, soon followed by the emergence of techniques to bypass detection. In this paper, we report our investigation on AGTD for an indic language Hindi. Our major contributions are in four folds: i) examined 26 LLMs to evaluate their proficiency in generating Hindi text, ii) introducing the AI-generated news article in Hindi ($AG_{hi}$) dataset, iii) evaluated the effectiveness of five recently proposed AGTD techniques: ConDA, J-Guard, RADAR, RAIDAR and Intrinsic Dimension Estimation for detecting AI-generated Hindi text, iv) proposed Hindi AI Detectability Index ($ADI_{hi}$) which shows a spectrum to understand the evolving landscape of eloquence of AI-generated text in Hindi. The code and dataset is available at https://github.com/ishank31/Counter_Turing_Test
Weak convergence implies convergence in mean within GGC
Authors: Hasanjan Sayit
Date: 2024-07-21 00:00:00

We prove that weak convergence within generalized gamma convolution (GGC) distributions implies convergence in the mean value. We use this fact to show the robustness of the expected utility maximizing optimal portfolio under exponential utility function when return vectors are modelled by hyperbolic distributions.
An Integral Equation Approach for the Valuation of Finite-maturity margin-call Stock Loans
Authors: Minh-Quan Nguyen, Nhat-Tan Le, Khuong Nguyen-An, Duc-Thi Luu
Date: 2024-07-20 00:00:00

This paper examines the pricing issue of margin-call stock loans with finite maturities under the Black-Scholes-Merton framework. In particular, using a Fourier Sine transform method, we reduce the partial differential equation governing the price of a margin-call stock loan into an ordinary differential equation, the solution of which can be easily found (in the Fourier Sine space) and analytically inverted into the original space. As a result, we obtain an integral representation of the value of the stock loan in terms of the unknown optimal exit prices, which are, in turn, governed by a Volterra integral equation. We thus can break the pricing problem of margin-call stock loans into two steps: 1) finding the optimal exit prices by solving numerically the governing Volterra integral equation and 2) calculating the values of margin-call stock loans based on the obtained optimal exit prices. By validating and comparing with other available numerical methods, we show that our proposed numerical scheme offers a reliable and efficient way to calculate the service fee of a margin-call stock loan contract, track the contract value over time, and compute the level of stock price above which it is optimal to exit the contract. The effects of the margin-call feature on the loan contract are also examined and quantified.
StylusAI: Stylistic Adaptation for Robust German Handwritten Text Generation
Authors: Nauman Riaz, Saifullah Saifullah, Stefan Agne, Andreas Dengel, Sheraz Ahmed
Date: 2024-07-22 00:00:00
Tasks: Benchmarking, Text Generation

In this study, we introduce StylusAI, a novel architecture leveraging diffusion models in the domain of handwriting style generation. StylusAI is specifically designed to adapt and integrate the stylistic nuances of one language's handwriting into another, particularly focusing on blending English handwriting styles into the context of the German writing system. This approach enables the generation of German text in English handwriting styles and German handwriting styles into English, enriching machine-generated handwriting diversity while ensuring that the generated text remains legible across both languages. To support the development and evaluation of StylusAI, we present the \lq{Deutscher Handschriften-Datensatz}\rq~(DHSD), a comprehensive dataset encompassing 37 distinct handwriting styles within the German language. This dataset provides a fundamental resource for training and benchmarking in the realm of handwritten text generation. Our results demonstrate that StylusAI not only introduces a new method for style adaptation in handwritten text generation but also surpasses existing models in generating handwriting samples that improve both text quality and stylistic fidelity, evidenced by its performance on the IAM database and our newly proposed DHSD. Thus, StylusAI represents a significant advancement in the field of handwriting style generation, offering promising avenues for future research and applications in cross-linguistic style adaptation for languages with similar scripts.
Global Balance and Systemic Risk in Financial Correlation Networks
Authors: Paolo Bartesaghi, Fernando Diaz-Diaz, Rosanna Grassi, Pierpaolo Uberti
Date: 2024-07-19 00:00:00

The global balance index is used in the network literature to quantify how balanced a signed network is. In this paper we show that the global balance index of financial correlation networks can be used as a systemic risk measure. We define the global balance index of a network starting from a diffusive process that describes how the information spreads across nodes in a network, providing an alternative derivation to the usual combinatorial one. The steady state of this process is the solution of a linear system governed by the exponential of the replication matrix of the process. We provide a bridge between the numerical stability of this linear system, measured by the condition number in an opportune norm, and the structural predictability of the underlying signed network. The link between the condition number and related systemic risk measures, such as the market rank indicators, allows the global balance index to be interpreted as a new systemic risk measure. A comprehensive empirical application to real financial data finally confirms that the global balance index of financial correlation networks represents a valuable and effective systemic risk indicator.
An Empirical Study of Retrieval Augmented Generation with Chain-of-Thought
Authors: Yuetong Zhao, Hongyu Cao, Xianyu Zhao, Zhijian Ou
Date: 2024-07-22 00:00:00
Tasks: Form, Logical Reasoning, RAG, Retrieval, Retrieval-augmented Generation

Since the launch of ChatGPT at the end of 2022, generative dialogue models represented by ChatGPT have quickly become essential tools in daily life. As user expectations increase, enhancing the capability of generative dialogue models to solve complex problems has become a focal point of current research. This paper delves into the effectiveness of the RAFT (Retrieval Augmented Fine-Tuning) method in improving the performance of Generative dialogue models. RAFT combines chain-of-thought with model supervised fine-tuning (SFT) and retrieval augmented generation (RAG), which significantly enhanced the model's information extraction and logical reasoning abilities. We evaluated the RAFT method across multiple datasets and analysed its performance in various reasoning tasks, including long-form QA and short-form QA tasks, tasks in both Chinese and English, and supportive and comparison reasoning tasks. Notably, it addresses the gaps in previous research regarding long-form QA tasks and Chinese datasets. Moreover, we also evaluate the benefit of the chain-of-thought (CoT) in the RAFT method. This work offers valuable insights for studies focused on enhancing the performance of generative dialogue models.
SETTP: Style Extraction and Tunable Inference via Dual-level Transferable Prompt Learning
Authors: Chunzhen Jin, Yongfeng Huang, Yaqi Wang, Peng Cao, Osmar Zaiane
Date: 2024-07-22 00:00:00
Tasks: Prompt Learning, Style Transfer, Text Style Transfer

Text style transfer, an important research direction in natural language processing, aims to adapt the text to various preferences but often faces challenges with limited resources. In this work, we introduce a novel method termed Style Extraction and Tunable Inference via Dual-level Transferable Prompt Learning (SETTP) for effective style transfer in low-resource scenarios. First, SETTP learns source style-level prompts containing fundamental style characteristics from high-resource style transfer. During training, the source style-level prompts are transferred through an attention module to derive a target style-level prompt for beneficial knowledge provision in low-resource style transfer. Additionally, we propose instance-level prompts obtained by clustering the target resources based on the semantic content to reduce semantic bias. We also propose an automated evaluation approach of style similarity based on alignment with human evaluations using ChatGPT-4. Our experiments across three resourceful styles show that SETTP requires only 1/20th of the data volume to achieve performance comparable to state-of-the-art methods. In tasks involving scarce data like writing style and role style, SETTP outperforms previous methods by 16.24\%.
Refining Corpora from a Model Calibration Perspective for Chinese Spelling Correction
Authors: Dingyao Yu, Yang An, Wei Ye, Xiongfeng Xiao, Shaoguang Mao, Tao Ge, Shikun Zhang
Date: 2024-07-22 00:00:00
Tasks: Data Augmentation, Optical Character Recognition (OCR), Spelling Correction

Chinese Spelling Correction (CSC) commonly lacks large-scale high-quality corpora, due to the labor-intensive labeling of spelling errors in real-life human writing or typing scenarios. Two data augmentation methods are widely adopted: (1) \textit{Random Replacement} with the guidance of confusion sets and (2) \textit{OCR/ASR-based Generation} that simulates character misusing. However, both methods inevitably introduce noisy data (e.g., false spelling errors), potentially leading to over-correction. By carefully analyzing the two types of corpora, we find that though the latter achieves more robust generalization performance, the former yields better-calibrated CSC models. We then provide a theoretical analysis of this empirical observation, based on which a corpus refining strategy is proposed. Specifically, OCR/ASR-based data samples are fed into a well-calibrated CSC model trained on random replacement-based corpora and then filtered based on prediction confidence. By learning a simple BERT-based model on the refined OCR/ASR-based corpus, we set up impressive state-of-the-art performance on three widely-used benchmarks, while significantly alleviating over-correction (e.g., lowering false positive predictions).
A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives
Authors: Zihao Li, Shaoxiong Ji, Timothee Mickus, Vincent Segonne, Jörg Tiedemann
Date: 2024-07-22 00:00:00
Tasks: Language Modeling, Language Modelling

Pretrained language models (PLMs) display impressive performances and have captured the attention of the NLP community. Establishing best practices in pretraining has, therefore, become a major focus of NLP research, especially since insights gained from monolingual English models may not necessarily apply to more complex multilingual models. One significant caveat of the current state of the art is that different works are rarely comparable: they often discuss different parameter counts, training data, and evaluation methodology. This paper proposes a comparison of multilingual pretraining objectives in a controlled methodological environment. We ensure that training data and model architectures are comparable, and discuss the downstream performances across 6 languages that we observe in probing and fine-tuning scenarios. We make two key observations: (1) the architecture dictates which pretraining objective is optimal; (2) multilingual translation is a very effective pretraining objective under the right conditions. We make our code, data, and model weights available at \texttt{\url{https://github.com/Helsinki-NLP/lm-vs-mt}}.
Developing a Reliable, Fast, General-Purpose Hallucination Detection and Mitigation Service
Authors: Song Wang, Xun Wang, Jie Mei, Yujia Xie, Sean Muarray, Zhang Li, Lingfeng Wu, Si-Qing Chen, Wayne Xiong
Date: 2024-07-22 00:00:00
Tasks: Hallucination, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Natural Language Inference, NER

Hallucination, a phenomenon where large language models (LLMs) produce output that is factually incorrect or unrelated to the input, is a major challenge for LLM applications that require accuracy and dependability. In this paper, we introduce a reliable and high-speed production system aimed at detecting and rectifying the hallucination issue within LLMs. Our system encompasses named entity recognition (NER), natural language inference (NLI), span-based detection (SBD), and an intricate decision tree-based process to reliably detect a wide range of hallucinations in LLM responses. Furthermore, we have crafted a rewriting mechanism that maintains an optimal mix of precision, response time, and cost-effectiveness. We detail the core elements of our framework and underscore the paramount challenges tied to response time, availability, and performance metrics, which are crucial for real-world deployment of these technologies. Our extensive evaluation, utilizing offline data and live production traffic, confirms the efficacy of our proposed framework and service.
Construction and Hedging of Equity Index Options Portfolios
Authors: Maciej Wysocki, Robert Ślepaczuk
Date: 2024-07-18 00:00:00

This research presents a comprehensive evaluation of systematic index option-writing strategies, focusing on S&P500 index options. We compare the performance of hedging strategies using the Black-Scholes-Merton (BSM) model and the Variance-Gamma (VG) model, emphasizing varying moneyness levels and different sizing methods based on delta and the VIX Index. The study employs 1-minute data of S&P500 index options and index quotes spanning from 2018 to 2023. The analysis benchmarks hedged strategies against buy-and-hold and naked option-writing strategies, with a focus on risk-adjusted performance metrics including transaction costs. Portfolio delta approximations are derived using implied volatility for the BSM model and market-calibrated parameters for the VG model. Key findings reveal that systematic option-writing strategies can potentially yield superior returns compared to buy-and-hold benchmarks. The BSM model generally provided better hedging outcomes than the VG model, although the VG model showed profitability in certain naked strategies as a tool for position sizing. In terms of rehedging frequency, we found that intraday hedging in 130-minute intervals provided both reliable protection against adverse market movements and a satisfactory returns profile.
LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models
Authors: Xi Chen, Songyang Zhang, Qibing Bai, Kai Chen, Satoshi Nakamura
Date: 2024-07-22 00:00:00
Tasks: Data Augmentation, Language Modeling, Language Modelling, Large Language Model, Speech-to-Text, Speech-to-Text Translation, Translation

We introduces LLaST, a framework for building high-performance Large Language model based Speech-to-text Translation systems. We address the limitations of end-to-end speech translation(E2E ST) models by exploring model architecture design and optimization techniques tailored for LLMs. Our approach includes LLM-based speech translation architecture design, ASR-augmented training, multilingual data augmentation, and dual-LoRA optimization. Our approach demonstrates superior performance on the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. We believe this effective method will serve as a strong baseline for speech translation and provide insights for future improvements of the LLM-based speech translation framework. We release the data, code and models in https://github.com/openaudiolab/LLaST.
The Development of a Comprehensive Spanish Dictionary for Phonetic and Lexical Tagging in Socio-phonetic Research (ESPADA)
Authors: Simon Gonzalez
Date: 2024-07-22 00:00:00
Proceeding: LREC (LAW) 2022 6

Pronunciation dictionaries are an important component in the process of speech forced alignment. The accuracy of these dictionaries has a strong effect on the aligned speech data since they help the mapping between orthographic transcriptions and acoustic signals. In this paper, I present the creation of a comprehensive pronunciation dictionary in Spanish (ESPADA) that can be used in most of the dialect variants of Spanish data. Current dictionaries focus on specific regional variants, but with the flexible nature of our tool, it can be readily applied to capture the most common phonetic differences across major dialectal variants. We propose improvements to current pronunciation dictionaries as well as mapping other relevant annotations such as morphological and lexical information. In terms of size, it is currently the most complete dictionary with more than 628,000 entries, representing words from 16 countries. All entries come with their corresponding pronunciations, morphological and lexical tagging, and other relevant information for phonetic analysis: stress patterns, phonotactics, IPA transcriptions, and more. This aims to equip socio-phonetic researchers with a complete open-source tool that enhances dialectal research within socio-phonetic frameworks in the Spanish language.
Dissecting Multiplication in Transformers: Insights into LLMs
Authors: Luyu Qiu, Jianing Li, Chi Su, Chen Jason Zhang, Lei Chen
Date: 2024-07-22 00:00:00

Transformer-based large language models have achieved remarkable performance across various natural language processing tasks. However, they often struggle with seemingly easy tasks like arithmetic despite their vast capabilities. This stark disparity raise human's concerns about their safe and ethical use, hinder their widespread adoption.In this paper, we focus on a typical arithmetic task, integer multiplication, to explore and explain the imperfection of transformers in this domain. We provide comprehensive analysis of a vanilla transformer trained to perform n-digit integer multiplication. Our observations indicate that the model decomposes multiplication task into multiple parallel subtasks, sequentially optimizing each subtask for each digit to complete the final multiplication. Based on observation and analysis, we infer the reasons of transformers deficiencies in multiplication tasks lies in their difficulty in calculating successive carryovers and caching intermediate results, and confirmed this inference through experiments. Guided by these findings, we propose improvements to enhance transformers performance on multiplication tasks. These enhancements are validated through rigorous testing and mathematical modeling, not only enhance transformer's interpretability, but also improve its performance, e.g., we achieve over 99.9% accuracy on 5-digit integer multiplication with a tiny transformer, outperform LLMs GPT-4. Our method contributes to the broader fields of model understanding and interpretability, paving the way for analyzing more complex tasks and Transformer models. This work underscores the importance of explainable AI, helping to build trust in large language models and promoting their adoption in critical applications.
UF-HOBI at "Discharge Me!": A Hybrid Solution for Discharge Summary Generation Through Prompt-based Tuning of GatorTronGPT Models
Authors: Mengxian Lyu, Cheng Peng, Daniel Paredes, Ziyi Chen, Aokun Chen, Jiang Bian, Yonghui Wu
Date: 2024-07-22 00:00:00
Tasks: NER

Automatic generation of discharge summaries presents significant challenges due to the length of clinical documentation, the dispersed nature of patient information, and the diverse terminology used in healthcare. This paper presents a hybrid solution for generating discharge summary sections as part of our participation in the "Discharge Me!" Challenge at the BioNLP 2024 Shared Task. We developed a two-stage generation method using both extractive and abstractive techniques, in which we first apply name entity recognition (NER) to extract key clinical concepts, which are then used as input for a prompt-tuning-based GatorTronGPT model to generate coherent text for two important sections including "Brief Hospital Course" and "Discharge Instructions". Our system was ranked 5th in this challenge, achieving an overall score of 0.284. The results demonstrate the effectiveness of our hybrid solution in improving the quality of automated discharge section generation.
MAVEN-Fact: A Large-scale Event Factuality Detection Dataset
Authors: Chunyang Li, Hao Peng, Xiaozhi Wang, Yunjia Qi, Lei Hou, Bin Xu, Juanzi Li
Date: 2024-07-22 00:00:00
Tasks: Hallucination

Event Factuality Detection (EFD) task determines the factuality of textual events, i.e., classifying whether an event is a fact, possibility, or impossibility, which is essential for faithfully understanding and utilizing event knowledge. However, due to the lack of high-quality large-scale data, event factuality detection is under-explored in event understanding research, which limits the development of EFD community. To address these issues and provide faithful event understanding, we introduce MAVEN-Fact, a large-scale and high-quality EFD dataset based on the MAVEN dataset. MAVEN-Fact includes factuality annotations of 112,276 events, making it the largest EFD dataset. Extensive experiments demonstrate that MAVEN-Fact is challenging for both conventional fine-tuned models and large language models (LLMs). Thanks to the comprehensive annotations of event arguments and relations in MAVEN, MAVEN-Fact also supports some further analyses and we find that adopting event arguments and relations helps in event factuality detection for fine-tuned models but does not benefit LLMs. Furthermore, we preliminarily study an application case of event factuality detection and find it helps in mitigating event-related hallucination in LLMs. Our dataset and codes can be obtained from \url{https://github.com/lcy2723/MAVEN-FACT}
Improving Minimum Bayes Risk Decoding with Multi-Prompt
Authors: David Heineman, Yao Dou, Wei Xu
Date: 2024-07-22 00:00:00

While instruction fine-tuned LLMs are effective text generators, sensitivity to prompt construction makes performance unstable and sub-optimal in practice. Relying on a single "best" prompt cannot capture all differing approaches to a generation problem. Using this observation, we propose multi-prompt decoding, where many candidate generations are decoded from a prompt bank at inference-time. To ensemble candidates, we use Minimum Bayes Risk (MBR) decoding, which selects a final output using a trained value metric. We show multi-prompt improves MBR across a comprehensive set of conditional generation tasks, and show this is a result of estimating a more diverse and higher quality candidate space than that of a single prompt. Further experiments confirm multi-prompt improves generation across tasks, models and metrics.