pwc-archive/papers-with-abstracts huggingface dataset:
Beyond Text: Unveiling Multimodal Proficiency of Large Language Models with MultiAPI Benchmark
Authors: Xiao Liu, Jianfeng Lin, Jiawei Zhang
Date: 2023-11-21 00:00:00
Tasks: Decision Making

The proliferation of Large Language Models like ChatGPT has significantly advanced language understanding and generation, impacting a broad spectrum of applications. However, these models predominantly excel in text-based tasks, overlooking the complexity of real-world multimodal information. This study introduces MultiAPI, a pioneering comprehensive large-scale API benchmark dataset aimed at expanding LLMs' proficiency in multimodal contexts. Developed collaboratively through ChatGPT, MultiAPI consists of 235 diverse API calls and 2,038 contextual prompts, offering a unique platform evaluation of tool-augmented LLMs handling multimodal tasks. Through comprehensive experiments, our findings reveal that while LLMs demonstrate proficiency in API call decision-making, they face challenges in domain identification, function selection, and argument generation. What's more, we surprisingly notice that auxiliary context can actually impair the performance. An in-depth error analysis paves the way for a new paradigm to address these challenges, suggesting a potential direction for future LLM research.
Attribution and Alignment: Effects of Local Context Repetition on Utterance Production and Comprehension in Dialogue
Authors: Aron Molnar, Jaap Jumelet, Mario Giulianelli, Arabella Sinclair
Date: 2023-11-21 00:00:00
Tasks: Dialogue Generation, Language Modeling, Language Modelling

Language models are often used as the backbone of modern dialogue systems. These models are pre-trained on large amounts of written fluent language. Repetition is typically penalised when evaluating language model generations. However, it is a key component of dialogue. Humans use local and partner specific repetitions; these are preferred by human users and lead to more successful communication in dialogue. In this study, we evaluate (a) whether language models produce human-like levels of repetition in dialogue, and (b) what are the processing mechanisms related to lexical re-use they use during comprehension. We believe that such joint analysis of model production and comprehension behaviour can inform the development of cognitively inspired dialogue generation systems.
Perceptual Structure in the Absence of Grounding for LLMs: The Impact of Abstractedness and Subjectivity in Color Language
Authors: Pablo Loyola, Edison Marrese-Taylor, Andres Hoyos-Idobro
Date: 2023-11-22 00:00:00
Tasks: Language Modeling, Language Modelling

The need for grounding in language understanding is an active research topic. Previous work has suggested that color perception and color language appear as a suitable test bed to empirically study the problem, given its cognitive significance and showing that there is considerable alignment between a defined color space and the feature space defined by a language model. To further study this issue, we collect a large scale source of colors and their descriptions, containing almost a 1 million examples , and perform an empirical analysis to compare two kinds of alignments: (i) inter-space, by learning a mapping between embedding space and color space, and (ii) intra-space, by means of prompting comparatives between color descriptions. Our results show that while color space alignment holds for monolexemic, highly pragmatic color descriptions, this alignment drops considerably in the presence of examples that exhibit elements of real linguistic usage such as subjectivity and abstractedness, suggesting that grounding may be required in such cases.
Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
Authors: Chengyu Wang, Junbing Yan, Wei zhang, Jun Huang
Date: 2023-11-22 00:00:00
Tasks: Model Compression, parameter-efficient fine-tuning, Position

This paper delves into the pressing need in Parameter-Efficient Fine-Tuning (PEFT) for Large Language Models (LLMs). While LLMs possess remarkable capabilities, their extensive parameter requirements and associated computational demands hinder their practicality and scalability for real-world applications. Our position paper highlights current states and the necessity of further studying into the topic, and recognizes significant challenges and open issues that must be addressed to fully harness the powerful abilities of LLMs. These challenges encompass novel efficient PEFT architectures, PEFT for different learning settings, PEFT combined with model compression techniques, and the exploration of PEFT for multi-modal LLMs. By presenting this position paper, we aim to stimulate further research and foster discussions surrounding more efficient and accessible PEFT for LLMs.
On the Calibration of Large Language Models and Alignment
Authors: Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, Zhendong Mao
Date: 2023-11-22 00:00:00

As large language models attract increasing attention and find widespread application, concurrent challenges of reliability also arise at the same time. Confidence calibration, an effective analysis method for gauging the reliability of deep models, serves as a crucial tool for assessing and improving their reliability. However, such investigation has been comparatively underexplored. In this work, we conduct a systematic examination of the calibration of aligned language models throughout the entire construction process, including pretraining and alignment training. At each stage, we investigate how different training settings, such as parameter scales and training data, affect model calibration. To thoroughly assess model calibration, we evaluate models on three most concerned aspects: generation, factuality and understanding. Our work sheds light on whether popular LLMs are well-calibrated and how the training process influences model calibration.
CoachLM: Automatic Instruction Revisions Improve the Data Quality in LLM Instruction Tuning
Authors: Yilun Liu, Shimin Tao, Xiaofeng Zhao, Ming Zhu, Wenbing Ma, Junhao Zhu, Chang Su, Yutai Hou, Miao Zhang, Min Zhang, Hongxia Ma, Li Zhang, Hao Yang, Yanfei Jiang
Date: 2023-11-22 00:00:00
Tasks: Instruction Following

Instruction tuning is crucial for enabling Language Learning Models (LLMs) in responding to human instructions. The quality of instruction pairs used for tuning greatly affects the performance of LLMs. However, the manual creation of high-quality instruction datasets is costly, leading to the adoption of automatic generation of instruction pairs by LLMs as a popular alternative. To ensure the high quality of LLM-generated instruction datasets, several approaches have been proposed. Nevertheless, existing methods either compromise dataset integrity by filtering a large proportion of samples, or are unsuitable for industrial applications. In this paper, instead of discarding low-quality samples, we propose CoachLM, a novel approach to enhance the quality of instruction datasets through automatic revisions on samples in the dataset. CoachLM is trained from the samples revised by human experts and significantly increases the proportion of high-quality samples in the dataset from 17.7% to 78.9%. The effectiveness of CoachLM is further assessed on various real-world instruction test sets. The results show that CoachLM improves the instruction-following capabilities of the instruction-tuned LLM by an average of 29.9%, which even surpasses larger LLMs with nearly twice the number of parameters. Furthermore, CoachLM is successfully deployed in a data management system for LLMs at Huawei, resulting in an efficiency improvement of up to 20% in the cleaning of 40k real-world instruction pairs. We release various assets of CoachLM, including the training data, code and test set (https://github.com/lunyiliu/CoachLM).
Comparative Experimentation of Accuracy Metrics in Automated Medical Reporting: The Case of Otitis Consultations
Authors: Wouter Faber, Renske Eline Bootsma, Tom Huibers, Sandra van Dulmen, Sjaak Brinkkemper
Date: 2023-11-22 00:00:00

Generative Artificial Intelligence (AI) can be used to automatically generate medical reports based on transcripts of medical consultations. The aim is to reduce the administrative burden that healthcare professionals face. The accuracy of the generated reports needs to be established to ensure their correctness and usefulness. There are several metrics for measuring the accuracy of AI generated reports, but little work has been done towards the application of these metrics in medical reporting. A comparative experimentation of 10 accuracy metrics has been performed on AI generated medical reports against their corresponding General Practitioner's (GP) medical reports concerning Otitis consultations. The number of missing, incorrect, and additional statements of the generated reports have been correlated with the metric scores. In addition, we introduce and define a Composite Accuracy Score which produces a single score for comparing the metrics within the field of automated medical reporting. Findings show that based on the correlation study and the Composite Accuracy Score, the ROUGE-L and Word Mover's Distance metrics are the preferred metrics, which is not in line with previous work. These findings help determine the accuracy of an AI generated medical report, which aids the development of systems that generate medical reports for GPs to reduce the administrative burden.
Enhancing Summarization Performance through Transformer-Based Prompt Engineering in Automated Medical Reporting
Authors: Daphne van Zandvoort, Laura Wiersema, Tom Huibers, Sandra van Dulmen, Sjaak Brinkkemper
Date: 2023-11-22 00:00:00
Tasks: Prompt Engineering

Customized medical prompts enable Large Language Models (LLM) to effectively address medical dialogue summarization. The process of medical reporting is often time-consuming for healthcare professionals. Implementing medical dialogue summarization techniques presents a viable solution to alleviate this time constraint by generating automated medical reports. The effectiveness of LLMs in this process is significantly influenced by the formulation of the prompt, which plays a crucial role in determining the quality and relevance of the generated reports. In this research, we used a combination of two distinct prompting strategies, known as shot prompting and pattern prompting to enhance the performance of automated medical reporting. The evaluation of the automated medical reports is carried out using the ROUGE score and a human evaluation with the help of an expert panel. The two-shot prompting approach in combination with scope and domain context outperforms other methods and achieves the highest score when compared to the human reference set by a general practitioner. However, the automated reports are approximately twice as long as the human references, due to the addition of both redundant and relevant statements that are added to the report.
Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting
Authors: Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, Le Sun
Date: 2023-11-22 00:00:00
Tasks: Hallucination, Language Modeling, Language Modelling, Large Language Model

Incorporating factual knowledge in knowledge graph is regarded as a promising approach for mitigating the hallucination of large language models (LLMs). Existing methods usually only use the user's input to query the knowledge graph, thus failing to address the factual hallucination generated by LLMs during its reasoning process. To address this problem, this paper proposes Knowledge Graph-based Retrofitting (KGR), a new framework that incorporates LLMs with KGs to mitigate factual hallucination during the reasoning process by retrofitting the initial draft responses of LLMs based on the factual knowledge stored in KGs. Specifically, KGR leverages LLMs to extract, select, validate, and retrofit factual statements within the model-generated responses, which enables an autonomous knowledge verifying and refining procedure without any additional manual efforts. Experiments show that KGR can significantly improve the performance of LLMs on factual QA benchmarks especially when involving complex reasoning processes, which demonstrates the necessity and effectiveness of KGR in mitigating hallucination and enhancing the reliability of LLMs.
PaSS: Parallel Speculative Sampling
Authors: Giovanni Monea, Armand Joulin, Edouard Grave
Date: 2023-11-22 00:00:00

Scaling the size of language models to tens of billions of parameters has led to impressive performance on a wide range of tasks. At generation, these models are used auto-regressively, requiring a forward pass for each generated token, and thus reading the full set of parameters from memory. This memory access forms the primary bottleneck for generation and it worsens as the model size increases. Moreover, executing a forward pass for multiple tokens in parallel often takes nearly the same time as it does for just one token. These two observations lead to the development of speculative sampling, where a second smaller model is used to draft a few tokens, that are then validated or rejected using a single forward pass of the large model. Unfortunately, this method requires two models that share the same tokenizer and thus limits its adoption. As an alternative, we propose to use parallel decoding as a way to draft multiple tokens from a single model with no computational cost, nor the need for a second model. Our approach only requires an additional input token that marks the words that will be generated simultaneously. We show promising performance (up to $30\%$ speed-up) while requiring only as few as $O(d_{emb})$ additional parameters.
Intention and Context Elicitation with Large Language Models in the Legal Aid Intake Process
Authors: Nick Goodson, Rongfei Lu
Date: 2023-11-22 00:00:00

Large Language Models (LLMs) and chatbots show significant promise in streamlining the legal intake process. This advancement can greatly reduce the workload and costs for legal aid organizations, improving availability while making legal assistance more accessible to a broader audience. However, a key challenge with current LLMs is their tendency to overconfidently deliver an immediate 'best guess' to a client's question based on the output distribution learned over the training data. This approach often overlooks the client's actual intentions or the specifics of their legal situation. As a result, clients may not realize the importance of providing essential additional context or expressing their underlying intentions, which are crucial for their legal cases. Traditionally, logic based decision trees have been used to automate intake for specific access to justice issues, such as immigration and eviction. But those solutions lack scalability. We demonstrate a proof-of-concept using LLMs to elicit and infer clients' underlying intentions and specific legal circumstances through free-form, language-based interactions. We also propose future research directions to use supervised fine-tuning or offline reinforcement learning to automatically incorporate intention and context elicitation in chatbots without explicit prompting.
Cross-project Defect Prediction with An Enhanced Transfer Boosting Algorithm
Authors: Nazgol Nikravesh; Mohammad Reza Keyvanpour
Date: 2022-11-29 00:00:00
Proceeding: International Conference on Computer and Knowledge Engineering (ICCKE) 2022 11
Tasks: Prediction, Transfer Learning

A growing number of software projects makes it increasingly crucial to predict software defects. If adequate historical data are accessible, within-project defect prediction models can be reliable. However, during the early phases of software development, sufficient data are not available to train an effective predictor. Cross-project defect prediction (CPDP) utilizes information from previous mature projects (source data) for predicting whether new software modules (target data) will be defective. CPDP models must take into account the fact that data distributions between source and target projects are different. Cross-project defect prediction often reduces distribution differences by either selecting training data or using transfer learning methods. Using transfer learning effectively reduces distribution differences in recent CPDP models, yet none of them have taken into account the possibility that negative transfer may occur as a result of the imbalanced nature of defect data. In this paper, a four-step model is proposed, of which three steps are dedicated to the preparation of training data and their initial weights for use in the fourth step, which involves an enhanced version of the transfer boosting algorithm. In this algorithm, the imbalance nature of data is considered and the weighting of the source data is updated to enhance the prediction performance. Therefore, aside from reducing distribution discrepancy between source and target data, this model also addresses the issues related to defect data class imbalance. As compared to four state-of-the-art CPDP models, this model provided consistent and accurate predictions for fifteen projects from PROMISE, AEEEM, and SOFTLAB. Our proposed model provided the best average results for both AUC and F-measure and in some datasets, the improvements were more than 5%.
Tackling the Qubit Mapping Problem for NISQ-Era Quantum Devices
Authors: Gushu Li, Yufei Ding, Yuan Xie
Date: 2019-04-01 00:00:00
Proceeding: International Conference on Architectural Support for Programming Languages and Operating Systems 2019 4
Tasks: Heuristic Search

Due to little consideration in the hardware constraints, e.g., limited connections between physical qubits to enable two-qubit gates, most quantum algorithms cannot be directly executed on the Noisy Intermediate-Scale Quantum (NISQ) devices. Dynamically remapping logical qubits to physical qubits in the compiler is needed to enable the two-qubit
gates in the algorithm, which introduces additional operations and inevitably reduces the fidelity of the algorithm.
Previous solutions in finding such remapping suffer from high complexity, poor initial mapping quality, and limited flexibility and controllability.
To address these drawbacks mentioned above, this paper proposes a SWAP-based BidiREctional heuristic search algorithm (SABRE), which is applicable to NISQ devices with arbitrary connections between qubits. By optimizing every
search attempt, globally optimizing the initial mapping using a novel reverse traversal technique, introducing the decay
effect to enable the trade-off between the depth and the number of gates of the entire algorithm, SABRE outperforms
=the best known algorithm with exponential speedup and comparable or better results on various benchmarks
Physics constrained nonlinear regression models for time series
Authors: Andrew J. Majda1 and John Harlim
Date: 2012-11-20 00:00:00
Proceeding: Nonlinearity 2012 11
Tasks: regression, Time Series

A central issue in contemporary science is the development of data driven statistical nonlinear dynamical models for time series of partial observations of nature or a complex physical model. It has been established recently that ad-hoc quadratic multi-level regression models can have finite-time blow up of statistical solutions and/or pathological behavior of their invariant measure. Here a new class of physics constrained multi-level quadratic regression models are introduced, analyzed,
and applied to build reduced stochastic models from data of nonlinear systems. These models have the advantages of incorporating memory effects in time as well as the nonlinear noise from energy conserving nonlinear interactions. The mathematical guidelines for the performance and behavior of these physics constrained multi-level regression models as well as filtering algorithms for their implementation are developed here. Data driven applications of these new multi-level nonlinear regression models are developed for test models involving a nonlinear oscillator with memory effects and the difficult test case of the truncated Burgers-Hopf (TBH) model. These new physics constrained quadratic multi-level regression models are proposed here as process models for Bayesian estimation through Markov Chain Monte Carlo algorithms of low frequency behavior in complex physical data.
Vast TVB parameter space exploration: A Modular Framework for Accelerating the Multi-Scale Simulation of Human Brain Dynamics
Authors: Michiel van der Vlag, Lionel Kusch, Alain Destexhe, Viktor Jirsa, Sandra Diaz-Pier, Jennifer S. Goldman
Date: 2023-11-22 00:00:00
Tasks: GPU

Global neural dynamics emerge from multi-scale brain structures, with neurons communicating through synapses to form transiently communicating networks. Network activity arises from intercellular communication that depends on the structure of connectome tracts and local connection, intracellular signalling cascades, and the extracellular molecular milieu that regulate cellular properties. Multi-scale models of brain function have begun to directly link the emergence of global brain dynamics in conscious and unconscious brain states to microscopic changes at the level of cells. In particular, AdEx mean-field models representing statistical properties of local populations of neurons have been connected following human tractography data to represent multi-scale neural phenomena in simulations using The Virtual Brain (TVB). While mean-field models can be run on personal computers for short simulations, or in parallel on high-performance computing (HPC) architectures for longer simulations and parameter scans, the computational burden remains high and vast areas of the parameter space remain unexplored. In this work, we report that our TVB-HPC framework, a modular set of methods used here to implement the TVB-AdEx model for GPU and analyze emergent dynamics, notably accelerates simulations and substantially reduces computational resource requirements. The framework preserves the stability and robustness of the TVB-AdEx model, thus facilitating finer resolution exploration of vast parameter spaces as well as longer simulations previously near impossible to perform. Given that simulation and analysis toolkits are made public as open-source packages, our framework serves as a template onto which other models can be easily scripted and personalized datasets can be used for studies of inter-individual variability of parameters related to functional brain dynamics.