pwc-archive/papers-with-abstracts huggingface dataset:
Embracing the Enemy
Authors: Álvaro Delgado-Vega, Johannes Schneider
Date: 2024-06-14 00:00:00

We study the repeated interactions between two power-hungry agents, the "friend", and the "enemy," and one power broker, the principal. All three care about the leading agent's policy choice. The principal, who aligns more with the friend, can influence but not fully control leadership allocation. After an initial cordon sanitaire breaks, the principal embraces the enemy, sometimes promising persistent support: she grants the enemy power in exchange for moderation, which benefits the friend who reciprocates. The closer the principal is to the friend, the more she desires to embrace the enemy, but the harder it is to uphold such promises.
Existence and structure of Nash equilibria for supermodular games
Authors: Lu Yu
Date: 2024-06-13 00:00:00

Two theorems announced by Topkis about the topological description of sublattices are proved. They are applied to extend some classical results concerning the existence and the order structure of Nash equilibria of certain supermodular games, with some problems in Zhou's proof corrected.
Equilibria and Group Welfare in Vote Trading Systems
Authors: Matthew I. Jones
Date: 2024-06-13 00:00:00

We introduce a new framework to study the group dynamics and game-theoretic considerations when voters in a committee are allowed to trade votes. This model represents a significant step forward by considering vote-for-vote trades in a low-information environment where voters do not know the preferences of their trading partners. All voters draw their preference intensities on two issues from a common probability distribution and then consider offering to trade with an anonymous partner. The result is a strategic game between two voters that can be studied analytically. We compute the Nash equilibria for this game and derive several interesting results involving symmetry, group heterogeneity, and more. This framework allows us to determine that trades are typically detrimental to the welfare of the group as a whole, but there are exceptions. We also expand our model to allow all voters to trade votes and derive approximate results for this more general scenario. Finally, we emulate vote trading in real groups by forming simulated committees using real voter preference intensity data and computing the resulting equilibria and associated welfare gains or losses.
Speak in the Scene: Diffusion-based Acoustic Scene Transfer toward Immersive Speech Generation
Authors: Miseul Kim, Soo-Whan Chung, Youna Ji, Hong-Goo Kang, Min-Seok Choi
Date: 2024-06-18 00:00:00

This paper introduces a novel task in generative speech processing, Acoustic Scene Transfer (AST), which aims to transfer acoustic scenes of speech signals to diverse environments. AST promises an immersive experience in speech perception by adapting the acoustic scene behind speech signals to desired environments. We propose AST-LDM for the AST task, which generates speech signals accompanied by the target acoustic scene of the reference prompt. Specifically, AST-LDM is a latent diffusion model conditioned by CLAP embeddings that describe target acoustic scenes in either audio or text modalities. The contributions of this paper include introducing the AST task and implementing its baseline model. For AST-LDM, we emphasize its core framework, which is to preserve the input speech and generate audio consistently with both the given speech and the target acoustic environment. Experiments, including objective and subjective tests, validate the feasibility and efficacy of our approach.
Transcribe, Align and Segment: Creating speech datasets for low-resource languages
Authors: Taras Sereda
Date: 2024-06-18 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

In this work, we showcase a cost-effective method for generating training data for speech processing tasks. First, we transcribe unlabeled speech using a state-of-the-art Automatic Speech Recognition (ASR) model. Next, we align generated transcripts with the audio and apply segmentation on short utterances. Our focus is on ASR for low-resource languages, such as Ukrainian, using podcasts as a source of unlabeled speech. We release a new dataset UK-PODS that features modern conversational Ukrainian language. It contains over 50 hours of text audio-pairs as well as uk-pods-conformer, a 121 M parameters ASR model that is trained on MCV-10 and UK-PODS and achieves 3x reduction of Word Error Rate (WER) on podcasts comparing to publically available uk-nvidia-citrinet while maintaining comparable WER on MCV-10 test split. Both dataset UK-PODS https://huggingface.co/datasets/taras-sereda/uk-pods and ASR uk-pods-conformer https://huggingface.co/taras-sereda/uk-pods-conformer are available on the hugging-face hub.
Challenging margin-based speaker embedding extractors by using the variational information bottleneck
Authors: Themos Stafylakis, Anna Silnova, Johan Rohdin, Oldrich Plchot, Lukas Burget
Date: 2024-06-18 00:00:00
Tasks: Speaker Recognition

Speaker embedding extractors are typically trained using a classification loss over the training speakers. During the last few years, the standard softmax/cross-entropy loss has been replaced by the margin-based losses, yielding significant improvements in speaker recognition accuracy. Motivated by the fact that the margin merely reduces the logit of the target speaker during training, we consider a probabilistic framework that has a similar effect. The variational information bottleneck provides a principled mechanism for making deterministic nodes stochastic, resulting in an implicit reduction of the posterior of the target speaker. We experiment with a wide range of speaker recognition benchmarks and scoring methods and report competitive results to those obtained with the state-of-the-art Additive Angular Margin loss.
Unsupervised Online Continual Learning for Automatic Speech Recognition
Authors: Steven Vander Eeckt, Hugo Van hamme
Date: 2024-06-18 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Continual Learning, Domain Adaptation, speech-recognition, Speech Recognition

Adapting Automatic Speech Recognition (ASR) models to new domains leads to Catastrophic Forgetting (CF) of previously learned information. This paper addresses CF in the challenging context of Online Continual Learning (OCL), with tasks presented as a continuous data stream with unknown boundaries. We extend OCL for ASR into the unsupervised realm, by leveraging self-training (ST) to facilitate unsupervised adaptation, enabling models to adapt continually without label dependency and without forgetting previous knowledge. Through comparative analysis of various OCL and ST methods across two domain adaptation experiments, we show that UOCL suffers from significantly less forgetting compared to supervised OCL, allowing UOCL methods to approach the performance levels of supervised OCL. Our proposed UOCL extensions further boosts UOCL's efficacy. Our findings represent a significant step towards continually adaptable ASR systems, capable of leveraging unlabeled data across diverse domains.
Text-aware Speech Separation for Multi-talker Keyword Spotting
Authors: Haoyu Li, Baochen Yang, Yu Xi, Linfeng Yu, Tian Tan, Hao Li, Kai Yu
Date: 2024-06-18 00:00:00
Tasks: Keyword Spotting, Speech Separation

For noisy environments, ensuring the robustness of keyword spotting (KWS) systems is essential. While much research has focused on noisy KWS, less attention has been paid to multi-talker mixed speech scenarios. Unlike the usual cocktail party problem where multi-talker speech is separated using speaker clues, the key challenge here is to extract the target speech for KWS based on text clues. To address it, this paper proposes a novel Text-aware Permutation Determinization Training method for multi-talker KWS with a clue-based Speech Separation front-end (TPDT-SS). Our research highlights the critical role of SS front-ends and shows that incorporating keyword-specific clues into these models can greatly enhance the effectiveness. TPDT-SS shows remarkable success in addressing permutation problems in mixed keyword speech, thereby greatly boosting the performance of the backend. Additionally, fine-tuning our system on unseen mixed speech results in further performance improvement.
Multi-Channel Multi-Speaker ASR Using Target Speaker's Solo Segment
Authors: Yiwen Shao, Shi-Xiong Zhang, Yong Xu, Meng Yu, Dong Yu, Daniel Povey, Sanjeev Khudanpur
Date: 2024-06-13 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

In the field of multi-channel, multi-speaker Automatic Speech Recognition (ASR), the task of discerning and accurately transcribing a target speaker's speech within background noise remains a formidable challenge. Traditional approaches often rely on microphone array configurations and the information of the target speaker's location or voiceprint. This study introduces the Solo Spatial Feature (Solo-SF), an innovative method that utilizes a target speaker's isolated speech segment to enhance ASR performance, thereby circumventing the need for conventional inputs like microphone array layouts. We explore effective strategies for selecting optimal solo segments, a crucial aspect for Solo-SF's success. Through evaluations conducted on the AliMeeting dataset and AISHELL-1 simulations, Solo-SF demonstrates superior performance over existing techniques, significantly lowering Character Error Rates (CER) in various test conditions. Our findings highlight Solo-SF's potential as an effective solution for addressing the complexities of multi-channel, multi-speaker ASR tasks.
An Exploration of Length Generalization in Transformer-Based Speech Enhancement
Authors: Qiquan Zhang, Hongxu Zhu, Xinyuan Qian, Eliathamby Ambikairajah, Haizhou Li
Date: 2024-06-17 00:00:00
Tasks: Position, Speech Enhancement

The use of Transformer architectures has facilitated remarkable progress in speech enhancement. Training Transformers using substantially long speech utterances is often infeasible as self-attention suffers from quadratic complexity. It is a critical and unexplored challenge for a Transformer-based speech enhancement model to learn from short speech utterances and generalize to longer ones. In this paper, we conduct comprehensive experiments to explore the length generalization problem in speech enhancement with Transformer. Our findings first establish that position embedding provides an effective instrument to alleviate the impact of utterance length on Transformer-based speech enhancement. Specifically, we explore four different position embedding schemes to enable length generalization. The results confirm the superiority of relative position embeddings (RPEs) over absolute PE (APEs) in length generalization.
Spatially constrained vs. unconstrained filtering in neural spatiospectral filters for multichannel speech enhancement
Authors: Annika Briegleb, Walter Kellermann
Date: 2024-06-17 00:00:00
Tasks: Speech Enhancement

When using artificial neural networks for multichannel speech enhancement, filtering is often achieved by estimating a complex-valued mask that is applied to all or one reference channel of the input signal. The estimation of this mask is based on the noisy multichannel signal and, hence, can exploit spatial and spectral cues simultaneously. While it has been shown that exploiting spatial and spectral cues jointly is beneficial for the speech enhancement result, the mechanics of the interplay of the two inside the neural network are still largely unknown. In this contribution, we investigate how two conceptually different neural spatiospectral filters (NSSFs) exploit spatial cues depending on the training target signal and show that, while one NSSF always performs spatial filtering, the other one is selective in leveraging spatial information depending on the task at hand. These insights provide better understanding of the information the NSSFs use to make their prediction and, thus, allow to make informed decisions regarding their design and deployment.
AlignNet: Learning dataset score alignment functions to enable better training of speech quality estimators
Authors: Jaden Pieper, Stephen D. Voran
Date: 2024-06-14 00:00:00

We develop two complementary advances for training no-reference (NR) speech quality estimators with independent datasets. Multi-dataset finetuning (MDF) pretrains an NR estimator on a single dataset and then finetunes it on multiple datasets at once, including the dataset used for pretraining. AlignNet uses an AudioNet to generate intermediate score estimates before using the Aligner to map intermediate estimates to the appropriate score range. AlignNet is agnostic to the choice of AudioNet so any successful NR speech quality estimator can benefit from its Aligner. The methods can be used in tandem, and we use two studies to show that they improve on current solutions: one study uses nine smaller datasets and the other uses four larger datasets. AlignNet with MDF improves on other solutions because it efficiently and effectively removes misalignments that impair the learning process, and thus enables successful training with larger amounts of more diverse data.
ROAR: Reinforcing Original to Augmented Data Ratio Dynamics for Wav2Vec2.0 Based ASR
Authors: Vishwanath Pratap Singh, Federico Malato, Ville Hautamaki, Md. Sahidullah, Tomi Kinnunen
Date: 2024-06-14 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Data Augmentation, Reinforcement Learning (RL), speech-recognition, Speech Recognition

While automatic speech recognition (ASR) greatly benefits from data augmentation, the augmentation recipes themselves tend to be heuristic. In this paper, we address one of the heuristic approach associated with balancing the right amount of augmented data in ASR training by introducing a reinforcement learning (RL) based dynamic adjustment of original-to-augmented data ratio (OAR). Unlike the fixed OAR approach in conventional data augmentation, our proposed method employs a deep Q-network (DQN) as the RL mechanism to learn the optimal dynamics of OAR throughout the wav2vec2.0 based ASR training. We conduct experiments using the LibriSpeech dataset with varying amounts of training data, specifically, the 10Min, 1H, 10H, and 100H splits to evaluate the efficacy of the proposed method under different data conditions. Our proposed method, on average, achieves a relative improvement of 4.96% over the open-source wav2vec2.0 base model on standard LibriSpeech test sets.
Low algorithmic delay implementation of convolutional beamformer for online joint source separation and dereverberation
Authors: Kaien Mo, Xianrui Wang, Yichen Yang, Shoji Makino, Jingdong Chen
Date: 2024-06-14 00:00:00
Tasks: Audio Source Separation

Blind-audio-source-separation (BASS) techniques, particularly those with low latency, play an important role in a wide range of real-time systems, e.g., hearing aids, in-car hand-free voice communication, real-time human-machine interaction, etc. Most existing BASS algorithms are deduced to run on batch mode, and therefore large latency is unavoidable. Recently, some online algorithms were developed, which achieve separation on a frame-by-frame basis in the short-time-Fourier-transform (STFT) domain and the latency is significantly reduced as compared to those batch methods. However, the latency with these algorithms may still be too long for many real-time systems to bear. To further reduce latency while achieving good separation performance, we propose in this work to integrate a weighted prediction error (WPE) module into a non-causal sample-truncating-based independent vector analysis (NST-IVA). The resulting algorithm can maintain the algorithmic delay as NST-IVA if the delay with WPE is appropriately controlled while achieving significantly better performance, which is validated by simulations.
Enhanced Deep Speech Separation in Clustered Ad Hoc Distributed Microphone Environments
Authors: Jihyun Kim, Stijn Kindt, Nilesh Madhu, Hong-Goo Kang
Date: 2024-06-14 00:00:00
Tasks: Deep Learning, Speech Separation

Ad-hoc distributed microphone environments, where microphone locations and numbers are unpredictable, present a challenge to traditional deep learning models, which typically require fixed architectures. To tailor deep learning models to accommodate arbitrary array configurations, the Transform-Average-Concatenate (TAC) layer was previously introduced. In this work, we integrate TAC layers with dual-path transformers for speech separation from two simultaneous talkers in realistic settings. However, the distributed nature makes it hard to fuse information across microphones efficiently. Therefore, we explore the efficacy of blindly clustering microphones around sources of interest prior to enhancement. Experimental results show that this deep cluster-informed approach significantly improves the system's capacity to cope with the inherent variability observed in ad-hoc distributed microphone environments.
A Multimodal Framework for the Assessment of the Schizophrenia Spectrum
Authors: Gowtham Premananth, Yashish M. Siriwardena, Philip Resnik, Sonia Bansal, Deanna L. Kelly, Carol Espy-Wilson
Date: 2024-06-14 00:00:00

This paper presents a novel multimodal framework to distinguish between different symptom classes of subjects in the schizophrenia spectrum and healthy controls using audio, video, and text modalities. We implemented Convolution Neural Network and Long Short Term Memory based unimodal models and experimented on various multimodal fusion approaches to come up with the proposed framework. We utilized a minimal Gated multimodal unit (mGMU) to obtain a bi-modal intermediate fusion of the features extracted from the input modalities before finally fusing the outputs of the bimodal fusions to perform subject-wise classifications. The use of mGMU units in the multimodal framework improved the performance in both weighted f1-score and weighted AUC-ROC scores.
Efficient Personalization of Amplification in Hearing Aids via Multi-band Bayesian Machine Learning
Authors: Aoxin Ni, Edward Lobarinas, Nasser Kehtarnavaz
Date: 2024-06-13 00:00:00

Personalization of the amplification function of hearing aids has been shown to be of benefit to hearing aid users in previous studies. Several machine learning-based personalization approaches have been introduced in the literature. This paper presents a machine learning personalization approach with the advantage of being efficient in its training based on paired comparisons which makes it practical and field deployable. The training efficiency of this approach is the result of treating frequency bands independent of one another and by simultaneously carrying out Bayesian machine learning in each band across all of the frequency bands. Simulation results indicate that this approach leads to an estimated hearing preference function close to the true hearing preference function in fewer number of paired comparisons relative to the previous machine learning approaches. In addition, a clinical experiment conducted on eight subjects with hearing impairment indicate that this training efficient personalization approach provides personalized gain settings which are on average six times more preferred over the standard prescriptive gain settings.
Difference Autocorrelation: A Novel Approach to Estimate Shear Wave Speed in the Presence of Compression Waves
Authors: Hamidreza Asemani, Jannick P. Rolland, Kevin J. Parker
Date: 2024-06-18 00:00:00

In share wave elastography (SWE), the aim is to measure the velocity of shear waves, however unwanted compression waves and bulk tissue motion pose challenges in evaluating tissue stiffness. Conventional approaches often struggle to discriminate between shear and compression waves, leading to inaccurate shear wave speed (SWS) estimation. In this study, we propose a novel approach known as the difference autocorrelation estimator to accurately estimate reverberant SWS in the presence of compression waves and noise. Methods: The difference autocorrelation estimator, unlike conventional techniques, computes the subtraction of velocity between neighboring particles, effectively minimizing the impact of long wavelength compression waves and other wide-area movements such as those caused by respiration. We evaluated the effectiveness of the integrated difference autocorrelation (IDA) by: (1) using k-Wave simulations of a branching cylinder in a soft background, (2) using ultrasound elastography on a breast phantom, (3) using ultrasound elastography in the human liver-kidney region, and (4) using magnetic resonance elastography (MRE) on a brain phantom. Results: By applying IDA on the unfiltered contaminated wave fields of simulation and elastography experiments, the estimated SWSs are in good agreement with the ground truth values (i.e., less than 2% error for the simulation, 9% error for ultrasound elastography of breast phantom and 19% error for MRE). Conclusion: Our results demonstrate that IDA accurately estimates SWS, revealing the existence of a lesion, even in the presence of strong compression waves. Significance: IDA exhibits consistency in SWS estimation across different modalities and excitation scenarios, highlighting its robustness and potential clinical utility.
Weakly Supervised Learning of Cortical Surface Reconstruction from Segmentations
Authors: Qiang Ma, Liu Li, Emma C. Robinson, Bernhard Kainz, Daniel Rueckert
Date: 2024-06-18 00:00:00
Tasks: Surface Reconstruction, Weakly-supervised Learning

Existing learning-based cortical surface reconstruction approaches heavily rely on the supervision of pseudo ground truth (pGT) cortical surfaces for training. Such pGT surfaces are generated by traditional neuroimage processing pipelines, which are time consuming and difficult to generalize well to low-resolution brain MRI, e.g., from fetuses and neonates. In this work, we present CoSeg, a learning-based cortical surface reconstruction framework weakly supervised by brain segmentations without the need for pGT surfaces. CoSeg introduces temporal attention networks to learn time-varying velocity fields from brain MRI for diffeomorphic surface deformations, which fit an initial surface to target cortical surfaces within only 0.11 seconds for each brain hemisphere. A weakly supervised loss is designed to reconstruct pial surfaces by inflating the white surface along the normal direction towards the boundary of the cortical gray matter segmentation. This alleviates partial volume effects and encourages the pial surface to deform into deep and challenging cortical sulci. We evaluate CoSeg on 1,113 adult brain MRI at 1mm and 2mm resolution. CoSeg achieves superior geometric and morphological accuracy compared to existing learning-based approaches. We also verify that CoSeg can extract high-quality cortical surfaces from fetal brain MRI on which traditional pipelines fail to produce acceptable results.
Energy Reduction Opportunities in HDR Video Encoding
Authors: Christian Herglotz, Steven Le Moan, Alexandre Mercat
Date: 2024-06-17 00:00:00

This paper investigates the energy consumption of video encoding for high dynamic range videos. Specifically, we compare the energy consumption of the compression process using 10-bit input sequences, a tone-mapped 8-bit input sequence at 10-bit internal bit depth, and encoding an 8-bit input sequence using an encoder with an internal bit depth of 8 bit. We find that linear scaling of the luminance and chrominance values leads to degradations of the visual quality, but that significant encoding complexity and thus encoding energy can be saved. An important reason for this is the availability of vector instructions, which are not available for the 10-bit encoder. Furthermore, we find that at sufficiently low target bitrates, the compression efficiency at an internal bit depth of 8 bit exceeds the compression efficiency of regular 10-bit encoding.