pwc-archive/papers-with-abstracts huggingface dataset:
EDSep: An Effective Diffusion-Based Method for Speech Source Separation
Authors: Jinwei Dong, Xinsheng Wang, Qirong Mao
Date: 2025-01-27 00:00:00
Tasks: Speech Separation

Generative models have attracted considerable attention for speech separation tasks, and among these, diffusion-based methods are being explored. Despite the notable success of diffusion techniques in generation tasks, their adaptation to speech separation has encountered challenges, notably slow convergence and suboptimal separation outcomes. To address these issues and enhance the efficacy of diffusion-based speech separation, we introduce EDSep, a novel single-channel method grounded in score matching via stochastic differential equation (SDE). This method enhances generative modeling for speech source separation by optimizing training and sampling efficiency. Specifically, a novel denoiser function is proposed to approximate data distributions, which obtains ideal denoiser outputs. Additionally, a stochastic sampler is carefully designed to resolve the reverse SDE during the sampling process, gradually separating speech from mixtures. Extensive experiments on databases such as WSJ0-2mix, LRS2-2mix, and VoxCeleb2-2mix demonstrate our proposed method's superior performance over existing diffusion and discriminative models, validating its efficacy.
Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning
Authors: Jisi Zhang, Pablo Peso Parada, Md Asif Jalal, Karthikeyan Saravanan
Date: 2025-01-24 00:00:00
Tasks: FAD, Language Modeling, Language Modelling, Large Language Model, Music Generation, Text-to-Music Generation

Diffusion based Text-To-Music (TTM) models generate music corresponding to text descriptions. Typically UNet based diffusion models condition on text embeddings generated from a pre-trained large language model or from a cross-modality audio-language representation model. This work proposes a diffusion based TTM, in which the UNet is conditioned on both (i) a uni-modal language model (e.g., T5) via cross-attention and (ii) a cross-modal audio-language representation model (e.g., CLAP) via Feature-wise Linear Modulation (FiLM). The diffusion model is trained to exploit both a local text representation from the T5 and a global representation from the CLAP. Furthermore, we propose modifications that extract both global and local representations from the T5 through pooling mechanisms that we call mean pooling and self-attention pooling. This approach mitigates the need for an additional encoder (e.g., CLAP) to extract a global representation, thereby reducing the number of model parameters. Our results show that incorporating the CLAP global embeddings to the T5 local embeddings enhances text adherence (KL=1.47) compared to a baseline model solely relying on the T5 local embeddings (KL=1.54). Alternatively, extracting global text embeddings directly from the T5 local embeddings through the proposed mean pooling approach yields superior generation quality (FAD=1.89) while exhibiting marginally inferior text adherence (KL=1.51) against the model conditioned on both CLAP and T5 text embeddings (FAD=1.94 and KL=1.47). Our proposed solution is not only efficient but also compact in terms of the number of parameters required.
Introducing RIFT: A Hierarchical Entropic Filtering Scheme for Ideal Time-Frequency Reconstruction
Authors: James M. Cozens, Simon J. Godsill
Date: 2025-01-27 00:00:00

In this paper, we introduce the Reconstructive Ideal Fractional Transform (RIFT), an entropy-based probabilistic filtering algorithm formulated to reconstruct the Ideal Time-Frequency Representation (ITFR). RIFT surpasses the limitations imposed by the Gabor uncertainty principle for linear transforms, achieving the bilinear transform accuracy present in the Wigner-Ville Distribution (WVD) while effectively suppressing cross-terms. The algorithm utilises a hierarchical fractional wavelet-based scheme to account for localised time-frequency trajectory curvature. This scheme is optimised through an entropic-based filtering method that probabilistically extracts auto-terms while retaining the resolution of the WVD. This is achieved by employing a spatially varying, positively constrained deconvolution algorithm (Lucy-Richardson) with regularisation for adequate noise suppression. Additionally, the optimisation yields an Instantaneous Phase Direction field, which allows the localised curvature in speech or music extracts to be visualised and utilised within a Kalman tracking scheme, enabling the extraction of signal component trajectories. Evaluation demonstrates the algorithm's ability to eradicate cross-terms effectively and achieve superior time-frequency precision. This advance holds significant potential for a wide range of applications requiring high-resolution cross-term-free time-frequency analysis.
Noise disturbance and lack of privacy: Modeling acoustic dissatisfaction in open-plan offices
Authors: Manuj Yadav, Jungsoo Kim, Valtteri Hongisto, Densil Cabrera, Richard de Dear
Date: 2025-01-27 00:00:00

Open-plan offices are well-known to be adversely affected by acoustic issues. This study aims to model acoustic dissatisfaction using measurements of room acoustics, sound environment during occupancy, and occupant surveys (n = 349) in 28 offices representing a diverse range of workplace parameters. As latent factors, the contribution of $\textit{lack of privacy}$ (LackPriv) was 25% higher than $\textit{noise disturbance}$ (NseDstrb) in predicting $\textit{acoustic dissatisfaction}$ (AcDsat). Room acoustic metrics based on sound pressure level (SPL) decay of speech ($L_{\text{p,A,s,4m}}$ and $r_{\text{C}}$) were better in predicting these factors than distraction distance ($r_{\text{D}}$) based on speech transmission index. This contradicts previous findings, and the trends for SPL-based metrics in predicting AcDsat and LackPriv go against expectations based on ISO 3382-3. For sound during occupation, $L_{\text{A,90}}$ and psychoacoustic loudness ($N_{\text{90}}$) predicted AcDsat, and a SPL fluctuation metric ($M_{\text{A,eq}}$) predicted LackPriv. However, these metrics were weaker predictors than ISO 3382-3 metrics. Medium-sized offices exhibited higher dissatisfaction than larger ($\geq$50 occupants) offices. Dissatisfaction varied substantially across parameters including ceiling heights, number of workstations, and years of work, but not between offices with fixed seating compared to more flexible and activity-based working configurations. Overall, these findings highlight the complexities in characterizing occupants' perceptions using instrumental acoustic measurements.
Variational Bayesian Adaptive Learning of Deep Latent Variables for Acoustic Knowledge Transfer
Authors: Hu Hu, Sabato Marco Siniscalchi, Chao-Han Huck Yang, Chin-Hui Lee
Date: 2025-01-26 00:00:00
Tasks: Acoustic Scene Classification, Scene Classification, Spoken Command Recognition, Transfer Learning, Variational Inference

In this work, we propose a novel variational Bayesian adaptive learning approach for cross-domain knowledge transfer to address acoustic mismatches between training and testing conditions, such as recording devices and environmental noise. Different from the traditional Bayesian approaches that impose uncertainties on model parameters risking the curse of dimensionality due to the huge number of parameters, we focus on estimating a manageable number of latent variables in deep neural models. Knowledge learned from a source domain is thus encoded in prior distributions of deep latent variables and optimally combined, in a Bayesian sense, with a small set of adaptation data from a target domain to approximate the corresponding posterior distributions. Two different strategies are proposed and investigated to estimate the posterior distributions: Gaussian mean-field variational inference, and empirical Bayes. These strategies address the presence or absence of parallel data in the source and target domains. Furthermore, structural relationship modeling is investigated to enhance the approximation. We evaluated our proposed approaches on two acoustic adaptation tasks: 1) device adaptation for acoustic scene classification, and 2) noise adaptation for spoken command recognition. Experimental results show that the proposed variational Bayesian adaptive learning approach can obtain good improvements on target domain data, and consistently outperforms state-of-the-art knowledge transfer methods.
End-to-End Target Speaker Speech Recognition Using Context-Aware Attention Mechanisms for Challenging Enrollment Scenario
Authors: Mohsen Ghane, Mohammad Sadegh Safari
Date: 2025-01-26 00:00:00
Tasks: Decoder, speech-recognition, Speech Recognition

This paper presents a novel streaming end-to-end target-speaker speech recognition that addresses two critical limitations in systems: the handling of noisy enrollment utterances and specific enrollment phrase requirements. This paper proposes a robust Target-Speaker Recurrent Neural Network Transducer (TS-RNNT) with dual attention mechanisms for contextual biasing and overlapping enrollment processing. The model incorporates a text decoder and attention mechanism specifically designed to extract relevant speaker characteristics from noisy, overlapping enrollment audio. Experimental results on a synthesized dataset demonstrate the model's resilience, maintaining a Word Error Rate (WER) of 16.44% even with overlapping enrollment at 5dB Signal-to-Interference Ratio (SIR), compared to conventional approaches that degrade to WERs above 75% under similar conditions. This significant performance improvement, coupled with the model's semi-text-dependent enrollment capabilities, represents a substantial advancement toward more practical and versatile voice-controlled devices.
Enhancing Intelligibility for Generative Target Speech Extraction via Joint Optimization with Target Speaker ASR
Authors: Hao Ma, Rujin Chen, Xiao-Lei Zhang, Ju Liu, Xuelong Li
Date: 2025-01-24 00:00:00
Tasks: Speech Extraction

Target speech extraction (TSE) isolates the speech of a specific speaker from a multi-talker overlapped speech mixture. Most existing TSE models rely on discriminative methods, typically predicting a time-frequency spectrogram mask for the target speech. However, imperfections in these masks often result in over-/under-suppression of target/non-target speech, degrading perceptual quality. Generative methods, by contrast, re-synthesize target speech based on the mixture and target speaker cues, achieving superior perceptual quality. Nevertheless, these methods often overlook speech intelligibility, leading to alterations or loss of semantic content in the re-synthesized speech. Inspired by the Whisper model's success in target speaker ASR, we propose a generative TSE framework based on the pre-trained Whisper model to address the above issues. This framework integrates semantic modeling with flow-based acoustic modeling to achieve both high intelligibility and perceptual quality. Results from multiple benchmarks demonstrate that the proposed method outperforms existing generative and discriminative baselines. We present speech samples on https://aisaka0v0.github.io/GenerativeTSE_demo/.
FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integration
Authors: Kai-Tuo Xu, Feng-Long Xie, Xu Tang, Yao Hu
Date: 2025-01-24 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Computational Efficiency, Decoder, Large Language Model, speech-recognition, Speech Recognition

We present FireRedASR, a family of large-scale automatic speech recognition (ASR) models for Mandarin, designed to meet diverse requirements in superior performance and optimal efficiency across various applications. FireRedASR comprises two variants: FireRedASR-LLM: Designed to achieve state-of-the-art (SOTA) performance and to enable seamless end-to-end speech interaction. It adopts an Encoder-Adapter-LLM framework leveraging large language model (LLM) capabilities. On public Mandarin benchmarks, FireRedASR-LLM (8.3B parameters) achieves an average Character Error Rate (CER) of 3.05%, surpassing the latest SOTA of 3.33% with an 8.4% relative CER reduction (CERR). It demonstrates superior generalization capability over industrial-grade baselines, achieving 24%-40% CERR in multi-source Mandarin ASR scenarios such as video, live, and intelligent assistant. FireRedASR-AED: Designed to balance high performance and computational efficiency and to serve as an effective speech representation module in LLM-based speech models. It utilizes an Attention-based Encoder-Decoder (AED) architecture. On public Mandarin benchmarks, FireRedASR-AED (1.1B parameters) achieves an average CER of 3.18%, slightly worse than FireRedASR-LLM but still outperforming the latest SOTA model with over 12B parameters. It offers a more compact size, making it suitable for resource-constrained applications. Moreover, both models exhibit competitive results on Chinese dialects and English speech benchmarks and excel in singing lyrics recognition. To advance research in speech processing, we release our models and inference code at https://github.com/FireRedTeam/FireRedASR.
Characteristic-Specific Partial Fine-Tuning for Efficient Emotion and Speaker Adaptation in Codec Language Text-to-Speech Models
Authors: Tianrui Wang, Meng Ge, Cheng Gong, Chunyu Qiang, Haoyu Wang, Zikang Huang, Yu Jiang, Xiaobao Wang, Xie Chen, Longbiao Wang, Jianwu Dang
Date: 2025-01-24 00:00:00
Tasks: Emotion Classification, Speaker Identification, text-to-speech, Text to Speech

Recently, emotional speech generation and speaker cloning have garnered significant interest in text-to-speech (TTS). With the open-sourcing of codec language TTS models trained on massive datasets with large-scale parameters, adapting these general pre-trained TTS models to generate speech with specific emotional expressions and target speaker characteristics has become a topic of great attention. Common approaches, such as full and adapter-based fine-tuning, often overlook the specific contributions of model parameters to emotion and speaker control. Treating all parameters uniformly during fine-tuning, especially when the target data has limited content diversity compared to the pre-training corpus, results in slow training speed and an increased risk of catastrophic forgetting. To address these challenges, we propose a characteristic-specific partial fine-tuning strategy, short as CSP-FT. First, we use a weighted-sum approach to analyze the contributions of different Transformer layers in a pre-trained codec language TTS model for emotion and speaker control in the generated speech. We then selectively fine-tune the layers with the highest and lowest characteristic-specific contributions to generate speech with target emotional expression and speaker identity. Experimental results demonstrate that our method achieves performance comparable to, or even surpassing, full fine-tuning in generating speech with specific emotional expressions and speaker identities. Additionally, CSP-FT delivers approximately 2x faster training speeds, fine-tunes only around 8% of parameters, and significantly reduces catastrophic forgetting. Furthermore, we show that codec language TTS models perform competitively with self-supervised models in speaker identification and emotion classification tasks, offering valuable insights for developing universal speech processing models.
Generalizable Audio Deepfake Detection via Latent Space Refinement and Augmentation
Authors: Wen Huang, Yanmei Gu, Zhiming Wang, Huijia Zhu, Yanmin Qian
Date: 2025-01-24 00:00:00
Tasks: Audio Deepfake Detection, DeepFake Detection, Face Swapping, Speech Synthesis, text-to-speech, Text to Speech, Voice Conversion

Advances in speech synthesis technologies, like text-to-speech (TTS) and voice conversion (VC), have made detecting deepfake speech increasingly challenging. Spoofing countermeasures often struggle to generalize effectively, particularly when faced with unseen attacks. To address this, we propose a novel strategy that integrates Latent Space Refinement (LSR) and Latent Space Augmentation (LSA) to improve the generalization of deepfake detection systems. LSR introduces multiple learnable prototypes for the spoof class, refining the latent space to better capture the intricate variations within spoofed data. LSA further diversifies spoofed data representations by applying augmentation techniques directly in the latent space, enabling the model to learn a broader range of spoofing patterns. We evaluated our approach on four representative datasets, i.e. ASVspoof 2019 LA, ASVspoof 2021 LA and DF, and In-The-Wild. The results show that LSR and LSA perform well individually, and their integration achieves competitive results, matching or surpassing current state-of-the-art methods.
Faster Configuration Performance Bug Testing with Neural Dual-level Prioritization
Authors: Youpeng Ma, Tao Chen, Ke Li
Date: 2025-01-26 00:00:00

As software systems become more complex and configurable, more performance problems tend to arise from the configuration designs. This has caused some configuration options to unexpectedly degrade performance which deviates from their original expectations designed by the developers. Such discrepancies, namely configuration performance bugs (CPBugs), are devastating and can be deeply hidden in the source code. Yet, efficiently testing CPBugs is difficult, not only due to the test oracle is hard to set, but also because the configuration measurement is expensive and there are simply too many possible configurations to test. As such, existing testing tools suffer from lengthy runtime or have been ineffective in detecting CPBugs when the budget is limited, compounded by inaccurate test oracle. In this paper, we seek to achieve significantly faster CPBug testing by neurally prioritizing the testing at both the configuration option and value range levels with automated oracle estimation. Our proposed tool, dubbed NDP, is a general framework that works with different heuristic generators. The idea is to leverage two neural language models: one to estimate the CPBug types that serve as the oracle while, more vitally, the other to infer the probabilities of an option being CPBug-related, based on which the options and the value ranges to be searched can be prioritized. Experiments on several widely-used systems of different versions reveal that NDP can, in general, better predict CPBug type in 87% cases and find more CPBugs with up to 88.88x testing efficiency speedup over the state-of-the-art tools.
ED-Filter: Dynamic Feature Filtering for Eating Disorder Classification
Authors: Mehdi Naseriparsa, Suku Sukunesan, Zhen Cai, Osama Alfarraj, Amr Tolba, Saba Fathi Rabooki, Feng Xia
Date: 2025-01-04 00:00:00
Tasks: feature selection

Eating disorders (ED) are critical psychiatric problems that have alarmed the mental health community. Mental health professionals are increasingly recognizing the utility of data derived from social media platforms such as Twitter. However, high dimensionality and extensive feature sets of Twitter data present remarkable challenges for ED classification. To overcome these hurdles, we introduce a novel method, an informed branch and bound search technique known as ED-Filter. This strategy significantly improves the drawbacks of conventional feature selection algorithms such as filters and wrappers. ED-Filter iteratively identifies an optimal set of promising features that maximize the eating disorder classification accuracy. In order to adapt to the dynamic nature of Twitter ED data, we enhance the ED-Filter with a hybrid greedy-based deep learning algorithm. This algorithm swiftly identifies sub-optimal features to accommodate the ever-evolving data landscape. Experimental results on Twitter eating disorder data affirm the effectiveness and efficiency of ED-Filter. The method demonstrates significant improvements in classification accuracy and proves its value in eating disorder detection on social media platforms.
Leveraging Social Media Data and Artificial Intelligence for Improving Earthquake Response Efforts
Authors: Kalin Kopanov, Velizar Varbanov, Tatiana Atanasova
Date: 2024-12-28 00:00:00
Tasks: Disaster Response, Management

The integration of social media and artificial intelligence (AI) into disaster management, particularly for earthquake response, represents a profound evolution in emergency management practices. In the digital age, real-time information sharing has reached unprecedented levels, with social media platforms emerging as crucial communication channels during crises. This shift has transformed traditional, centralized emergency services into more decentralized, participatory models of disaster situational awareness. Our study includes an experimental analysis of 8,900 social media interactions, including 2,920 posts and 5,980 replies on X (formerly Twitter), following a magnitude 5.1 earthquake in Oklahoma on February 2, 2024. The analysis covers data from the immediate aftermath and extends over the following seven days, illustrating the critical role of digital platforms in modern disaster response. The results demonstrate that social media platforms can be effectively used as real-time situational awareness tools, delivering critical information to society and authorities during emergencies.
A Dynamic Similarity Index for Assessing Voltage Source Behaviour in Power Systems
Authors: Onur Alican, Dionysios Moutevelis, Josep Arevalo-Soler, Carlos Collados-Rodriguez, Jaume Amorós, Oriol Gomis-Bellmunt, Marc Cheah-Mane, Eduardo Prieto-Araujo
Date: 2025-01-27 00:00:00
Tasks: Electrical Engineering

Due to the fundamental transition to a power electronic dominated power system, the increasing diversity of dynamic elements underscores the need to assess their similarity to mature electrical engineering models. This article addresses the concept of the Dynamic Similarity Index (DSI) for its use in, power electronics-dominated networks. The DSI is a multipurpose tool developed to be used by different stakeholders (e.g., converter manufacturers and system operators). Such an index is calculated per frequency, which serves to anticipate potential differences in particular frequency ranges of interest between the model under study and the reference model. Within the scope of this study, the dynamic similarity of inverter-based generators to an ideal voltage source behind an impedance is assessed, due to the relevance of this fundamental circuit in the representation of generation units in power system studies. The article presents two potential applications based on this mathematical framework. First, for manufacturers to evaluate control performance compared to a reference model and second, it enables operators to diagnose buses with voltage vulnerability based on a user-defined reference Short-Circuit Ratio (SCR) value. The DSI results for these two case studies are validated using Matlab Simulink simulations.