pwc-archive/papers-with-abstracts huggingface dataset:
Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs
Authors: Anirudh Phukan, Divyansh, Harshit Kumar Morj, Vaishnavi, Apoorv Saxena, Koustava Goswami
Date: 2024-11-28 00:00:00
Tasks: Attribute, Hallucination, Optical Character Recognition (OCR), Question Answering, Semantic Segmentation, Visual Question Answering

The rapid development of Large Multimodal Models (LMMs) has significantly advanced multimodal understanding by harnessing the language abilities of Large Language Models (LLMs) and integrating modality-specific encoders. However, LMMs are plagued by hallucinations that limit their reliability and adoption. While traditional methods to detect and mitigate these hallucinations often involve costly training or rely heavily on external models, recent approaches utilizing internal model features present a promising alternative. In this paper, we critically assess the limitations of the state-of-the-art training-free technique, the logit lens, in handling generalized visual hallucinations. We introduce a refined method that leverages contextual token embeddings from middle layers of LMMs. This approach significantly improves hallucination detection and grounding across diverse categories, including actions and OCR, while also excelling in tasks requiring contextual understanding, such as spatial relations and attribute comparison. Our novel grounding technique yields highly precise bounding boxes, facilitating a transition from Zero-Shot Object Segmentation to Grounded Visual Question Answering. Our contributions pave the way for more reliable and interpretable multimodal models.
Integration of Contextual Descriptors in Ontology Alignment for Enrichment of Semantic Correspondence
Authors: Eduard Manziuk, Oleksander Barmak, Pavlo Radiuk, Vladislav Kuznetsov, Iurii Krak, Sergiy Yakovlev
Date: 2024-11-28 00:00:00
Tasks: Semantic correspondence

This paper proposes a novel approach to semantic ontology alignment using contextual descriptors. A formalization was developed that enables the integration of essential and contextual descriptors to create a comprehensive knowledge model. The hierarchical structure of the semantic approach and the mathematical apparatus for analyzing potential conflicts between concepts, particularly in the example of "Transparency" and "Privacy" in the context of artificial intelligence, are demonstrated. Experimental studies showed a significant improvement in ontology alignment metrics after the implementation of contextual descriptors, especially in the areas of privacy, responsibility, and freedom & autonomy. The application of contextual descriptors achieved an average overall improvement of approximately 4.36%. The results indicate the effectiveness of the proposed approach for more accurately reflecting the complexity of knowledge and its contextual dependence.
Pralekha: An Indic Document Alignment Evaluation Benchmark
Authors: Sanjay Suryanarayanan, Haiyue Song, Mohammed Safi Ur Rahman Khan, Anoop Kunchukuttan, Mitesh M. Khapra, Raj Dabre
Date: 2024-11-28 00:00:00
Tasks: Sentence, Sentence Embedding, Sentence-Embedding

Mining parallel document pairs poses a significant challenge because existing sentence embedding models often have limited context windows, preventing them from effectively capturing document-level information. Another overlooked issue is the lack of concrete evaluation benchmarks comprising high-quality parallel document pairs for assessing document-level mining approaches, particularly for Indic languages. In this study, we introduce Pralekha, a large-scale benchmark for document-level alignment evaluation. Pralekha includes over 2 million documents, with a 1:2 ratio of unaligned to aligned pairs, covering 11 Indic languages and English. Using Pralekha, we evaluate various document-level mining approaches across three dimensions: the embedding models, the granularity levels, and the alignment algorithm. To address the challenge of aligning documents using sentence and chunk-level alignments, we propose a novel scoring method, Document Alignment Coefficient (DAC). DAC demonstrates substantial improvements over baseline pooling approaches, particularly in noisy scenarios, achieving average gains of 20-30% in precision and 15-20% in F1 score. These results highlight DAC's effectiveness in parallel document mining for Indic languages.
Talking to oneself in CMC: a study of self replies in Wikipedia talk pages
Authors: Ludovic Tanguy, Céline Poudat, Lydia-Mai Ho-Dac
Date: 2024-11-28 00:00:00

This study proposes a qualitative analysis of self replies in Wikipedia talk pages, more precisely when the first two messages of a discussion are written by the same user. This specific pattern occurs in more than 10% of threads with two messages or more and can be explained by a number of reasons. After a first examination of the lexical specificities of second messages, we propose a seven categories typology and use it to annotate two reference samples (English and French) of 100 threads each. Finally, we analyse and compare the performance of human annotators (who reach a reasonable global efficiency) and instruction-tuned LLMs (which encounter important difficulties with several categories).
Zero-shot Slot Filling in the Age of LLMs for Dialogue Systems
Authors: Mansi Rana, Kadri Hacioglu, Sindhuja Gopalan, Maragathamani Boothalingam
Date: 2024-11-28 00:00:00
Tasks: Knowledge Distillation, Natural Language Understanding, slot-filling, Slot Filling, Zero-shot Slot Filling

Zero-shot slot filling is a well-established subtask of Natural Language Understanding (NLU). However, most existing methods primarily focus on single-turn text data, overlooking the unique complexities of conversational dialogue. Conversational data is highly dynamic, often involving abrupt topic shifts, interruptions, and implicit references that make it difficult to directly apply zero-shot slot filling techniques, even with the remarkable capabilities of large language models (LLMs). This paper addresses these challenges by proposing strategies for automatic data annotation with slot induction and black-box knowledge distillation (KD) from a teacher LLM to a smaller model, outperforming vanilla LLMs on internal datasets by 26% absolute increase in F1 score. Additionally, we introduce an efficient system architecture for call center product settings that surpasses off-the-shelf extractive models by 34% relative F1 score, enabling near real-time inference on dialogue streams with higher accuracy, while preserving low latency.
Rephrasing Electronic Health Records for Pretraining Clinical Language Models
Authors: Jinghui Liu, Anthony Nguyen
Date: 2024-11-28 00:00:00
Tasks: Decoder, Language Modeling, Language Modelling

Clinical language models are important for many applications in healthcare, but their development depends on access to extensive clinical text for pretraining. However, obtaining clinical notes from electronic health records (EHRs) at scale is challenging due to patient privacy concerns. In this study, we rephrase existing clinical notes using LLMs to generate synthetic pretraining corpora, drawing inspiration from previous work on rephrasing web data. We examine four popular small-sized LLMs (<10B) to create synthetic clinical text to pretrain both decoder-based and encoder-based language models. The method yields better results in language modeling and downstream tasks than previous synthesis approaches without referencing real clinical text. We find that augmenting original clinical notes with synthetic corpora from different LLMs improves performances even at a small token budget, showing the potential of this method to support pretraining at the institutional level or be scaled to synthesize large-scale clinical corpora.
The Impact of Example Selection in Few-Shot Prompting on Automated Essay Scoring Using GPT Models
Authors: Lui Yoshida
Date: 2024-11-28 00:00:00
Tasks: Automated Essay Scoring

This study investigates the impact of example selection on the performance of au-tomated essay scoring (AES) using few-shot prompting with GPT models. We evaluate the effects of the choice and order of examples in few-shot prompting on several versions of GPT-3.5 and GPT-4 models. Our experiments involve 119 prompts with different examples, and we calculate the quadratic weighted kappa (QWK) to measure the agreement between GPT and human rater scores. Regres-sion analysis is used to quantitatively assess biases introduced by example selec-tion. The results show that the impact of example selection on QWK varies across models, with GPT-3.5 being more influenced by examples than GPT-4. We also find evidence of majority label bias, which is a tendency to favor the majority la-bel among the examples, and recency bias, which is a tendency to favor the label of the most recent example, in GPT-generated essay scores and QWK, with these biases being more pronounced in GPT-3.5. Notably, careful example selection enables GPT-3.5 models to outperform some GPT-4 models. However, among the GPT models, the June 2023 version of GPT-4, which is not the latest model, exhibits the highest stability and performance. Our findings provide insights into the importance of example selection in few-shot prompting for AES, especially in GPT-3.5 models, and highlight the need for individual performance evaluations of each model, even for minor versions.
Sneaking Syntax into Transformer Language Models with Tree Regularization
Authors: Ananjan Nandi, Christopher D. Manning, Shikhar Murty
Date: 2024-11-28 00:00:00
Tasks: Inductive Bias

While compositional accounts of human language understanding are based on a hierarchical tree-like process, neural models like transformers lack a direct inductive bias for such tree structures. Introducing syntactic inductive biases could unlock more robust and data-efficient learning in transformer language models (LMs), but existing methods for incorporating such structure greatly restrict models, either limiting their expressivity or increasing inference complexity. This work instead aims to softly inject syntactic inductive biases into given transformer circuits, through a structured regularizer. We introduce TreeReg, an auxiliary loss function that converts bracketing decisions from silver parses into a set of differentiable orthogonality constraints on vector hidden states. TreeReg integrates seamlessly with the standard LM objective, requiring no architectural changes. LMs pre-trained with TreeReg on natural language corpora such as WikiText-103 achieve up to 10% lower perplexities on out-of-distribution data and up to 9.5 point improvements in syntactic generalization, requiring less than half the training data to outperform standard LMs. TreeReg still provides gains for pre-trained LLMs: Continued pre-training of Sheared Llama with TreeReg results in improved syntactic generalization, and fine-tuning on MultiNLI with TreeReg mitigates degradation of performance on adversarial NLI benchmarks by 41.2 points. We release all code to guide future research.
An indicator for effectiveness of text-to-image guardrails utilizing the Single-Turn Crescendo Attack (STCA)
Authors: Ted Kwartler, Nataliia Bagan, Ivan Banny, Alan Aqrawi, Arian Abbasi
Date: 2024-11-27 00:00:00

The Single-Turn Crescendo Attack (STCA), first introduced in Aqrawi and Abbasi [2024], is an innovative method designed to bypass the ethical safeguards of text-to-text AI models, compelling them to generate harmful content. This technique leverages a strategic escalation of context within a single prompt, combined with trust-building mechanisms, to subtly deceive the model into producing unintended outputs. Extending the application of STCA to text-to-image models, we demonstrate its efficacy by compromising the guardrails of a widely-used model, DALL-E 3, achieving outputs comparable to outputs from the uncensored model Flux Schnell, which served as a baseline control. This study provides a framework for researchers to rigorously evaluate the robustness of guardrails in text-to-image models and benchmark their resilience against adversarial attacks.
A Cross-Corpus Speech Emotion Recognition Method Based on Supervised Contrastive Learning
Authors: Xiang minjie
Date: 2024-11-25 00:00:00
Tasks: Contrastive Learning, Cross-corpus, Emotion Recognition, Speech Emotion Recognition

Research on Speech Emotion Recognition (SER) often faces challenges such as the lack of large-scale public datasets and limited generalization capability when dealing with data from different distributions. To solve this problem, this paper proposes a cross-corpus speech emotion recognition method based on supervised contrast learning. The method employs a two-stage fine-tuning process: first, the self-supervised speech representation model is fine-tuned using supervised contrastive learning on multiple speech emotion datasets; then, the classifier is fine-tuned on the target dataset. The experimental results show that the WavLM-based model achieved unweighted accuracy (UA) of 77.41% on the IEMOCAP dataset and 96.49% on the CASIA dataset, outperforming the state-of-the-art results on the two datasets.
Parallel Stacked Aggregated Network for Voice Authentication in IoT-Enabled Smart Devices
Authors: Awais Khan, Ijaz Ul Haq, Khalid Mahmood Malik
Date: 2024-11-29 00:00:00
Tasks: Voice Anti-spoofing, Voice Cloning

Voice authentication on IoT-enabled smart devices has gained prominence in recent years due to increasing concerns over user privacy and security. The current authentication systems are vulnerable to different voice-spoofing attacks (e.g., replay, voice cloning, and audio deepfakes) that mimic legitimate voices to deceive authentication systems and enable fraudulent activities (e.g., impersonation, unauthorized access, financial fraud, etc.). Existing solutions are often designed to tackle a single type of attack, leading to compromised performance against unseen attacks. On the other hand, existing unified voice anti-spoofing solutions, not designed specifically for IoT, possess complex architectures and thus cannot be deployed on IoT-enabled smart devices. Additionally, most of these unified solutions exhibit significant performance issues, including higher equal error rates or lower accuracy for specific attacks. To overcome these issues, we present the parallel stacked aggregation network (PSA-Net), a lightweight framework designed as an anti-spoofing defense system for voice-controlled smart IoT devices. The PSA-Net processes raw audios directly and eliminates the need for dataset-dependent handcrafted features or pre-computed spectrograms. Furthermore, PSA-Net employs a split-transform-aggregate approach, which involves the segmentation of utterances, the extraction of intrinsic differentiable embeddings through convolutions, and the aggregation of them to distinguish legitimate from spoofed audios. In contrast to existing deep Resnet-oriented solutions, we incorporate cardinality as an additional dimension in our network, which enhances the PSA-Net ability to generalize across diverse attacks. The results show that the PSA-Net achieves more consistent performance for different attacks that exist in current anti-spoofing solutions.
Integrated Artificial Neurons from Metal Halide Perovskites
Authors: Jeroen J. de Boer, Bruno Ehrler
Date: 2024-11-29 00:00:00

Hardware neural networks could perform certain computational tasks orders of magnitude more energy-efficiently than conventional computers. Artificial neurons are a key component of these networks and are currently implemented with electronic circuits based on capacitors and transistors. However, artificial neurons based on memristive devices are a promising alternative, owing to their potentially smaller size and inherent stochasticity. But despite their promise, demonstrations of memristive artificial neurons have so far been limited. Here we demonstrate a fully on-chip artificial neuron based on microscale electrodes and halide perovskite semiconductors as the active layer. By connecting a halide perovskite memristive device in series with a capacitor, the device demonstrates stochastic leaky integrate-and-fire behavior, with an energy consumption of 20 to 60 pJ per spike, lower than that of a biological neuron. We simulate populations of our neuron and show that the stochastic firing allows the detection of sub-threshold inputs. The neuron can easily be integrated with previously-demonstrated halide perovskite artificial synapses in energy-efficient neural networks.
Core Placement Optimization of Many-core Brain-Inspired Near-Storage Systems for Spiking Neural Network Training
Authors: Xueke Zhu, Wenjie Lin, Yanyu Lin, Yunhao Ma, Wenxiang Cheng, Zhengyu Ma, Yonghong Tian, Huihui Zhou
Date: 2024-11-29 00:00:00
Tasks: Deep Reinforcement Learning

With the increasing application scope of spiking neural networks (SNN), the complexity of SNN models has surged, leading to an exponential growth in demand for AI computility. As the new generation computing architecture of the neural networks, the efficiency and power consumption of distributed storage and parallel computing in the many-core near-memory computing system have attracted much attention. Among them, the mapping problem from logical cores to physical cores is one of the research hotspots. In order to improve the computing parallelism and system throughput of the many-core near-memory computing system, and to reduce power consumption, we propose a SNN training many-core deployment optimization method based on Off-policy Deterministic Actor-Critic. We utilize deep reinforcement learning as a nonlinear optimizer, treating the many-core topology as network graph features and using graph convolution to input the many-core structure into the policy network. We update the parameters of the policy network through near-end policy optimization to achieve deployment optimization of SNN models in the many-core near-memory computing architecture to reduce chip power consumption. To handle large-dimensional action spaces, we use continuous values matching the number of cores as the output of the policy network and then discretize them again to obtain new deployment schemes. Furthermore, to further balance inter-core computation latency and improve system throughput, we propose a model partitioning method with a balanced storage and computation strategy. Our method overcomes the problems such as uneven computation and storage loads between cores, and the formation of local communication hotspots, significantly reducing model training time, communication costs, and average flow load between cores in the many-core near-memory computing architecture.
Wafer2Spike: Spiking Neural Network for Wafer Map Pattern Classification
Authors: Abhishek Mishra, Suman Kumar, Anush Lingamoorthy, Anup Das, Nagarajan Kandasamy
Date: 2024-11-29 00:00:00
Tasks: Computational Efficiency

In integrated circuit design, the analysis of wafer map patterns is critical to improve yield and detect manufacturing issues. We develop Wafer2Spike, an architecture for wafer map pattern classification using a spiking neural network (SNN), and demonstrate that a well-trained SNN achieves superior performance compared to deep neural network-based solutions. Wafer2Spike achieves an average classification accuracy of 98\% on the WM-811k wafer benchmark dataset. It is also superior to existing approaches for classifying defect patterns that are underrepresented in the original dataset. Wafer2Spike achieves this improved precision with great computational efficiency.
A spiking photonic neural network of 40.000 neurons, trained with rank-order coding for leveraging sparsity
Authors: Ria Talukder, Anas Skalli, Xavier Porte, Simon Thorpe, Daniel Brunner
Date: 2024-11-28 00:00:00

piking neural networks are neuromorphic systems that emulate certain aspects of biological neurons, offering potential advantages in energy efficiency and speed by for example leveraging sparsity. While CMOS-based electronic SNN hardware has shown promise, scalability and parallelism challenges remain. Photonics provides a promising platform for SNNs due to the speed of excitable photonic devices standing in as neurons and the parallelism and low-latency of optical signal conduction. Here, we present a photonic SNN comprising 40,000 neurons using off-the-shelf components, including a spatial light modulator and a CMOS camera, enabling scalable and cost-effective implementations for photonic SNN proof of concept studies. The system is governed by a modified Ikeda map, were adding additional inhibitory feedback forcing introduces excitability akin to biological dynamics. Using latency encoding and sparsity, the network achieves 83.5% accuracy on MNIST using 22% of neurons, and 77.5% with 8.5% neuron utilization. Training is performed via liquid state machine concepts combined with the hardware-compatible SPSA algorithm, marking its first use in photonic neural networks. This demonstration integrates photonic nonlinearity, excitability, and sparse computation, paving the way for efficient large-scale photonic neuromorphic systems.
Transfer Learning for High-dimensional Quantile Regression with Distribution Shift
Authors: Ruiqi Bai, Yijiao Zhang, Hanbo Yang, Zhongyi Zhu
Date: 2024-11-29 00:00:00
Tasks: quantile regression, regression, Transfer Learning

Information from related source studies can often enhance the findings of a target study. However, the distribution shift between target and source studies can severely impact the efficiency of knowledge transfer. In the high-dimensional regression setting, existing transfer approaches mainly focus on the parameter shift. In this paper, we focus on the high-dimensional quantile regression with knowledge transfer under three types of distribution shift: parameter shift, covariate shift, and residual shift. We propose a novel transferable set and a new transfer framework to address the above three discrepancies. Non-asymptotic estimation error bounds and source detection consistency are established to validate the availability and superiority of our method in the presence of distribution shift. Additionally, an orthogonal debiased approach is proposed for statistical inference with knowledge transfer, leading to sharper asymptotic results. Extensive simulation results as well as real data applications further demonstrate the effectiveness of our proposed procedure.
Geometry of fibers of the multiplication map of deep linear neural networks
Authors: SImon Pepin Lehalleur, Richárd Rimányi
Date: 2024-11-29 00:00:00
Tasks: Learning Theory

We study the geometry of the algebraic set of tuples of composable matrices which multiply to a fixed matrix, using tools from the theory of quiver representations. In particular, we determine its codimension $C$ and the number $\theta$ of its top-dimensional irreducible components. Our solution is presented in three forms: a Poincar\'e series in equivariant cohomology, a quadratic integer program, and an explicit formula. In the course of the proof, we establish a surprising property: $C$ and $\theta$ are invariant under arbitrary permutations of the dimension vector. We also show that the real log-canonical threshold of the function taking a tuple to the square Frobenius norm of its product is $C/2$. These results are motivated by the study of deep linear neural networks in machine learning and Bayesian statistics (singular learning theory) and show that deep linear networks are in a certain sense ``mildly singular".