pwc-archive/papers-with-abstracts huggingface dataset:
A Unified Pilot Design for Integrated Sensing and Communications
Authors: Pu Yuan
Date: 2024-06-12 00:00:00
Tasks: ISAC

This paper investigates a unified pilot signal design in an orthogonal frequency division modulation (OFDM)-based integrated sensing and communications (ISAC) system. The novel designed two-dimensional (2D) pilot signal is generated on the delay-Doppler (DD) plane for sensing, while its time-frequency (TF) plane transformation acts as the demodulation reference signal (DMRS) for the OFDM data. The well-designed pilot signal preserves orthogonality with the data in terms of resource occupancy in the TF plane and quasi-orthogonality in terms of codeword in the DD plane. Leveraging these nice properties, we are allowed to implement sensing detection in the DD plane using a simple 2D correlation, taking advantage of the favorable auto-correlation properties of the 2D pilot. In the communication part, the transformed pilot in the TF plane serves as a known DMRS for channel estimation and equalization. The 2D pilot design demonstrates good scalability and can adapt to different delay and Doppler resolution requirements without violating the OFDM data detection and can overcome the fractional Doppler with limited sensing resources. Experimental results show the effective sensing performance of the proposed pilot, with only a small fraction of power shared from the OFDM data,while maintaining satisfactory symbol detection performance in communication.
Adaptive Actor-Critic Based Optimal Regulation for Drift-Free Uncertain Nonlinear Systems
Authors: Ashwin P. Dani, Shubhendu Bhasin
Date: 2024-06-13 00:00:00
Tasks: Reinforcement Learning (RL)

In this paper, a continuous-time adaptive actor-critic reinforcement learning (RL) controller is developed for drift-free nonlinear systems. Practical examples of such systems are image-based visual servoing (IBVS) and wheeled mobile robots (WMR), where the system dynamics includes a parametric uncertainty in the control effectiveness matrix with no drift term. The uncertainty in the input term poses a challenge for developing a continuous-time RL controller using existing methods. In this paper, an actor-critic or synchronous policy iteration (PI)-based RL controller is presented with a concurrent learning (CL)-based parameter update law for estimating the unknown parameters of the control effectiveness matrix. An infinite-horizon value function minimization objective is achieved by regulating the current states to the desired with near-optimal control efforts. The proposed controller guarantees closed-loop stability and simulation results validate the proposed theory using IBVS and WMR examples.
Broadband MEMS Microphone Arrays with Reduced Aperture Through 3D-Printed Waveguides
Authors: Dennis Laurijssen, Walter Daems, Jan Steckel
Date: 2024-06-11 00:00:00
Tasks: Sound Source Localization

In this paper we present a passive and cost-effective method for increasing the frequency range of ultrasound MEMS microphone arrays when using beamforming techniques. By applying a 3D-printed construction that reduces the acoustic aperture of the MEMS microphones we can create a regularly spaced microphone array layout with much smaller inter-element spacing than could be accomplished on a printed circuit board due to the physical size of the MEMS elements. This method allows the use of ultrasound sensors incorporating microphone arrays in combination with beamforming techniques without aliases due to grating lobes in applications such as sound source localization or the emulation of bat HRTFs.
FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching
Authors: Chaeyoung Jung, Suyeon Lee, Ji-Hoon Kim, Joon Son Chung
Date: 2024-06-13 00:00:00
Tasks: Speech Enhancement

This work proposes an efficient method to enhance the quality of corrupted speech signals by leveraging both acoustic and visual cues. While existing diffusion-based approaches have demonstrated remarkable quality, their applicability is limited by slow inference speeds and computational complexity. To address this issue, we present FlowAVSE which enhances the inference speed and reduces the number of learnable parameters without degrading the output quality. In particular, we employ a conditional flow matching algorithm that enables the generation of high-quality speech in a single sampling step. Moreover, we increase efficiency by optimizing the underlying U-net architecture of diffusion-based systems. Our experiments demonstrate that FlowAVSE achieves 22 times faster inference speed and reduces the model size by half while maintaining the output quality. The demo page is available at: https://cyongong.github.io/FlowAVSE.github.io/
ToneUnit: A Speech Discretization Approach for Tonal Language Speech Synthesis
Authors: Dehua Tao, Daxin Tan, Yu Ting Yeung, Xiao Chen, Tan Lee
Date: 2024-06-13 00:00:00
Tasks: Quantization, Speech Synthesis

Representing speech as discretized units has numerous benefits in supporting downstream spoken language processing tasks. However, the approach has been less explored in speech synthesis of tonal languages like Mandarin Chinese. Our preliminary experiments on Chinese speech synthesis reveal the issue of "tone shift", where a synthesized speech utterance contains correct base syllables but incorrect tones. To address the issue, we propose the ToneUnit framework, which leverages annotated data with tone labels as CTC supervision to learn tone-aware discrete speech units for Mandarin Chinese speech. Our findings indicate that the discrete units acquired through the TonUnit resolve the "tone shift" issue in synthesized Chinese speech and yield favorable results in English synthesis. Moreover, the experimental results suggest that finite scalar quantization enhances the effectiveness of ToneUnit. Notably, ToneUnit can work effectively even with minimal annotated data.
Cascaded noise reduction and acoustic echo cancellation based on an extended noise reduction
Authors: Arnout Roebben, Toon van Waterschoot, Marc Moonen
Date: 2024-06-13 00:00:00
Tasks: Acoustic echo cancellation

In many speech recording applications, the recorded desired speech is corrupted by both noise and acoustic echo, such that combined noise reduction (NR) and acoustic echo cancellation (AEC) is called for. A common cascaded design corresponds to NR filters preceding AEC filters. These NR filters aim at reducing the near-end room noise (and possibly partially the echo) and operate on the microphones only, consequently requiring the AEC filters to model both the echo paths and the NR filters. In this paper, however, we propose a design with extended NR (NRext) filters preceding AEC filters under the assumption of the echo paths being additive maps, thus preserving the addition operation. Here, the NRext filters aim at reducing both the near-end room noise and the far-end room noise component in the echo, and operate on both the microphones and loudspeakers. We show that the succeeding AEC filters remarkably become independent of the NRext filters, such that the AEC filters are only required to model the echo paths, improving the AEC performance. Further, the degrees of freedom in the NRext filters scale with the number of loudspeakers, which is not the case for the NR filters, resulting in an improved NR performance.
On Improving Error Resilience of Neural End-to-End Speech Coders
Authors: Kishan Gupta, Nicola Pia, Srikanth Korse, Andreas Brendel, Guillaume Fuchs, Markus Multrus
Date: 2024-06-13 00:00:00
Tasks: Packet Loss Concealment

Error resilient tools like Packet Loss Concealment (PLC) and Forward Error Correction (FEC) are essential to maintain a reliable speech communication for applications like Voice over Internet Protocol (VoIP), where packets are frequently delayed and lost. In recent times, end-to-end neural speech codecs have seen a significant rise, due to their ability to transmit speech signal at low bitrates but few considerations were made about their error resilience in a real system. Recently introduced Neural End-to-End Speech Codec (NESC) can reproduce high quality natural speech at low bitrates. We extend its robustness to packet losses by adding a low complexity network to predict the codebook indices in latent space. Furthermore, we propose a method to add an in-band FEC at an additional bitrate of 0.8 kbps. Both subjective and objective assessment indicate the effectiveness of proposed methods, and demonstrate that coupling PLC and FEC provide significant robustness against packet losses.
Refining Self-Supervised Learnt Speech Representation using Brain Activations
Authors: Hengyu Li, Kangdi Mei, Zhaoci Liu, Yang Ai, Liping Chen, Jie Zhang, ZhenHua Ling
Date: 2024-06-12 00:00:00
Tasks: Automatic Speech Recognition, Speaker Verification, speech-recognition, Speech Recognition

It was shown in literature that speech representations extracted by self-supervised pre-trained models exhibit similarities with brain activations of human for speech perception and fine-tuning speech representation models on downstream tasks can further improve the similarity. However, it still remains unclear if this similarity can be used to optimize the pre-trained speech models. In this work, we therefore propose to use the brain activations recorded by fMRI to refine the often-used wav2vec2.0 model by aligning model representations toward human neural responses. Experimental results on SUPERB reveal that this operation is beneficial for several downstream tasks, e.g., speaker verification, automatic speech recognition, intent classification.One can then consider the proposed method as a new alternative to improve self-supervised speech models.
DubWise: Video-Guided Speech Duration Control in Multimodal LLM-based Text-to-Speech for Dubbing
Authors: Neha Sahipjohn, Ashishkumar Gudmalwar, Nirmesh Shah, Pankaj Wasnik, Rajiv Ratn Shah
Date: 2024-06-13 00:00:00
Tasks: Language Modeling, Language Modelling, Large Language Model, text-to-speech, Text to Speech, Voice Cloning

Audio-visual alignment after dubbing is a challenging research problem. To this end, we propose a novel method, DubWise Multi-modal Large Language Model (LLM)-based Text-to-Speech (TTS), which can control the speech duration of synthesized speech in such a way that it aligns well with the speakers lip movements given in the reference video even when the spoken text is different or in a different language. To accomplish this, we propose to utilize cross-modal attention techniques in a pre-trained GPT-based TTS. We combine linguistic tokens from text, speaker identity tokens via a voice cloning network, and video tokens via a proposed duration controller network. We demonstrate the effectiveness of our system on the Lip2Wav-Chemistry and LRS2 datasets. Also, the proposed method achieves improved lip sync and naturalness compared to the SOTAs for the same language but different text (i.e., non-parallel) and the different language, different text (i.e., cross-lingual) scenarios.
LAFMA: A Latent Flow Matching Model for Text-to-Audio Generation
Authors: Wenhao Guan, Kaidi Wang, Wangjin Zhou, Yang Wang, Feng Deng, Hui Wang, Lin Li, Qingyang Hong, Yong Qin
Date: 2024-06-12 00:00:00
Tasks: Audio Generation

Recently, the application of diffusion models has facilitated the significant development of speech and audio generation. Nevertheless, the quality of samples generated by diffusion models still needs improvement. And the effectiveness of the method is accompanied by the extensive number of sampling steps, leading to an extended synthesis time necessary for generating high-quality audio. Previous Text-to-Audio (TTA) methods mostly used diffusion models in the latent space for audio generation. In this paper, we explore the integration of the Flow Matching (FM) model into the audio latent space for audio generation. The FM is an alternative simulation-free method that trains continuous normalization flows (CNF) based on regressing vector fields. We demonstrate that our model significantly enhances the quality of generated audio samples, achieving better performance than prior models. Moreover, it reduces the number of inference steps to ten steps almost without sacrificing performance.
Low-Complexity Acoustic Scene Classification Using Parallel Attention-Convolution Network
Authors: Yanxiong Li, Jiaxin Tan, Guoqing Chen, Jialong Li, Yongjie Si, Qianhua He
Date: 2024-06-12 00:00:00
Tasks: Acoustic Scene Classification, Data Augmentation, Knowledge Distillation, Scene Classification

This work is an improved system that we submitted to task 1 of DCASE2023 challenge. We propose a method of low-complexity acoustic scene classification by a parallel attention-convolution network which consists of four modules, including pre-processing, fusion, global and local contextual information extraction. The proposed network is computationally efficient to capture global and local contextual information from each audio clip. In addition, we integrate other techniques into our method, such as knowledge distillation, data augmentation, and adaptive residual normalization. When evaluated on the official dataset of DCASE2023 challenge, our method obtains the highest accuracy of 56.10% with parameter number of 5.21 kilo and multiply-accumulate operations of 1.44 million. It exceeds the top two systems of DCASE2023 challenge in accuracy and complexity, and obtains state-of-the-art result. Code is at: https://github.com/Jessytan/Low-complexity-ASC.
VECL-TTS: Voice identity and Emotional style controllable Cross-Lingual Text-to-Speech
Authors: Ashishkumar Gudmalwar, Nirmesh Shah, Sai Akarsh, Pankaj Wasnik, Rajiv Ratn Shah
Date: 2024-06-12 00:00:00
Tasks: text-to-speech, Text to Speech

Despite the significant advancements in Text-to-Speech (TTS) systems, their full utilization in automatic dubbing remains limited. This task necessitates the extraction of voice identity and emotional style from a reference speech in a source language and subsequently transferring them to a target language using cross-lingual TTS techniques. While previous approaches have mainly concentrated on controlling voice identity within the cross-lingual TTS framework, there has been limited work on incorporating emotion and voice identity together. To this end, we introduce an end-to-end Voice Identity and Emotional Style Controllable Cross-Lingual (VECL) TTS system using multilingual speakers and an emotion embedding network. Moreover, we introduce content and style consistency losses to enhance the quality of synthesized speech further. The proposed system achieved an average relative improvement of 8.83\% compared to the state-of-the-art (SOTA) methods on a database comprising English and three Indian languages (Hindi, Telugu, and Marathi).
DCASE 2024 Task 4: Sound Event Detection with Heterogeneous Data and Missing Labels
Authors: Samuele Cornell, Janek Ebbers, Constance Douwes, Irene Martín-Morató, Manu Harju, Annamaria Mesaros, Romain Serizel
Date: 2024-06-12 00:00:00
Tasks: Event Detection, Missing Labels, Sound Event Detection

The Detection and Classification of Acoustic Scenes and Events Challenge Task 4 aims to advance sound event detection (SED) systems in domestic environments by leveraging training data with different supervision uncertainty. Participants are challenged in exploring how to best use training data from different domains and with varying annotation granularity (strong/weak temporal resolution, soft/hard labels), to obtain a robust SED system that can generalize across different scenarios. Crucially, annotation across available training datasets can be inconsistent and hence sound labels of one dataset may be present but not annotated in the other one and vice-versa. As such, systems will have to cope with potentially missing target labels during training. Moreover, as an additional novelty, systems will also be evaluated on labels with different granularity in order to assess their robustness for different applications. To lower the entry barrier for participants, we developed an updated baseline system with several caveats to address these aforementioned problems. Results with our baseline system indicate that this research direction is promising and is possible to obtain a stronger SED system by using diverse domain training data with missing labels compared to training a SED system for each domain separately.
Target Speaker Extraction with Curriculum Learning
Authors: Yun Liu, Xuechen Liu, Xiaoxiao Miao, Junichi Yamagishi
Date: 2024-06-12 00:00:00
Tasks: Target Speaker Extraction

This paper presents a novel approach to target speaker extraction (TSE) using Curriculum Learning (CL) techniques, addressing the challenge of distinguishing a target speaker's voice from a mixture containing interfering speakers. For efficient training, we propose designing a curriculum that selects subsets of increasing complexity, such as increasing similarity between target and interfering speakers, and that selects training data strategically. Our CL strategies include both variants using predefined difficulty measures (e.g. gender, speaker similarity, and signal-to-distortion ratio) and ones using the TSE's standard objective function, each designed to expose the model gradually to more challenging scenarios. Comprehensive testing on the Libri2talker dataset demonstrated that our CL strategies for TSE improved the performance, and the results markedly exceeded baseline models without CL about 1 dB.
DB3V: A Dialect Dominated Dataset of Bird Vocalisation for Cross-corpus Bird Species Recognition
Authors: Xin Jing, Luyang Zhang, Jiangjian Xie, Alexander Gebhard, Alice Baird, Bjoern Schuller
Date: 2024-06-11 00:00:00
Tasks: Benchmarking, Cross-corpus

In ornithology, bird species are known to have variedit's widely acknowledged that bird species display diverse dialects in their calls across different regions. Consequently, computational methods to identify bird species onsolely through their calls face critsignificalnt challenges. There is growing interest in understanding the impact of species-specific dialects on the effectiveness of bird species recognition methods. Despite potential mitigation through the expansion of dialect datasets, the absence of publicly available testing data currently impedes robust benchmarking efforts. This paper presents the Dialect Dominated Dataset of Bird Vocalisation, the first cross-corpus dataset that focuses on dialects in bird vocalisations. The DB3V comprises more than 25 hours of audio recordings from 10 bird species distributed across three distinct regions in the contiguous United States (CONUS). In addition to presenting the dataset, we conduct analyses and establish baseline models for cross-corpus bird recognition. The data and code are publicly available online: https://zenodo.org/records/11544734
Biharmonic Distance of Graphs and its Higher-Order Variants: Theoretical Properties with Applications to Centrality and Clustering
Authors: Mitchell Black, Lucy Lin, Amir Nayyeri, Weng-Keen Wong
Date: 2024-06-04 00:00:00
Tasks: Clustering, Graph Clustering

Effective resistance is a distance between vertices of a graph that is both theoretically interesting and useful in applications. We study a variant of effective resistance called the biharmonic distance. While the effective resistance measures how well-connected two vertices are, we prove several theoretical results supporting the idea that the biharmonic distance measures how important an edge is to the global topology of the graph. Our theoretical results connect the biharmonic distance to well-known measures of connectivity of a graph like its total resistance and sparsity. Based on these results, we introduce two clustering algorithms using the biharmonic distance. Finally, we introduce a further generalization of the biharmonic distance that we call the $k$-harmonic distance. We empirically study the utility of biharmonic and $k$-harmonic distance for edge centrality and graph clustering.
A Hybrid Modelling of a Water and Air Injector in a Subsonic Icing Wind Tunnel
Authors: César Hernández-Hernández, Thomas Chevet, Rihab el Houda Thabet, Nicolas Langlois
Date: 2024-06-13 00:00:00

The study of droplet generation in wind tunnels in conducting icing experiments is of great importance in determining ice formation on structures or surfaces, where parameters such as Liquid Water Content (LWC) and Median Volumetric Diameter (MVD) play a relevant role. The measurement of these parameters requires specialised instrumentation. In this paper, several experiments have been carried out in a subsonic wind tunnel facility to study the parameters that are part of the icing process in structures. Furthermore, a mathematical modelling of the constituent subsystems of the plant study that allow us to have a comprehensive understanding of the behaviour of the system is developed using techniques based on first principles and machine learning techniques such as regression trees and neural networks. The simulation results show that the implementation of the model manages to obtain prominent expected values of LWC and MVD within the range of values obtained in the real experimental data.
Joint Observer Gain and Input Design for Asymptotic Active Fault Diagnosis
Authors: Feng Xu, Yiming Wan, Ye Wang, Vicenc Puig
Date: 2024-06-13 00:00:00
Tasks: Fault Diagnosis

This paper proposes a joint gain and input design method for observer-based asymptotic active fault diagnosis, which is based on a newly-defined notion named the excluding degree of the origin from a zonotope. Using the excluding degree, a quantitative specification is obtained to characterize the performance of set-based robust fault diagnosis. Furthermore, a single gain design method and a joint gain and input design method are proposed, respectively. This is the first work to achieve a joint observer gain and input design for set-based active fault diagnosis. Compared with the existing methods that design gains and input separately, the proposed joint gain and input design method has advantages to exploit the fault diagnosis potential of observer-based schemes. Finally, several examples are used to illustrate the effectiveness of the proposed methods.
Adaptive Cooperative Streaming of Holographic Video Over Wireless Networks: A Proximal Policy Optimization Solution
Authors: Wanli Wen, Jiping Yan, Yulu Zhang, Zhen Huang, Liang Liang, Yunjian Jia
Date: 2024-06-13 00:00:00

Adapting holographic video streaming to fluctuating wireless channels is essential to maintain consistent and satisfactory Quality of Experience (QoE) for users, which, however, is a challenging task due to the dynamic and uncertain characteristics of wireless networks. To address this issue, we propose a holographic video cooperative streaming framework designed for a generic wireless network in which multiple access points can cooperatively transmit video with different bitrates to multiple users. Additionally, we model a novel QoE metric tailored specifically for holographic video streaming, which can effectively encapsulate the nuances of holographic video quality, quality fluctuations, and rebuffering occurrences simultaneously. Furthermore, we formulate a formidable QoE maximization problem, which is a non-convex mixed integer nonlinear programming problem. Using proximal policy optimization (PPO), a new class of reinforcement learning algorithms, we devise a joint beamforming and bitrate control scheme, which can be wisely adapted to fluctuations in the wireless channel. The numerical results demonstrate the superiority of the proposed scheme over representative baselines.