pwc-archive/papers-with-abstracts huggingface dataset:
NTT Multi-Speaker ASR System for the DASR Task of CHiME-8 Challenge
Authors: Naoyuki Kamo, Naohiro Tawara, Atsushi Ando, Takatomo Kano, Hiroshi Sato, Rintaro Ikeshita, Takafumi Moriya, Shota Horiguchi, Kohei Matsuura, Atsunori Ogawa, Alexis Plaquet, Takanori Ashihara, Tsubasa Ochiai, Masato Mimura, Marc Delcroix, Tomohiro Nakatani, Taichi Asami, Shoko Araki
Date: 2024-09-09 00:00:00
Tasks: Action Detection, Activity Detection, Automatic Speech Recognition, speech-recognition, Speech Recognition

We present a distant automatic speech recognition (DASR) system developed for the CHiME-8 DASR track. It consists of a diarization first pipeline. For diarization, we use end-to-end diarization with vector clustering (EEND-VC) followed by target speaker voice activity detection (TS-VAD) refinement. To deal with various numbers of speakers, we developed a new multi-channel speaker counting approach. We then apply guided source separation (GSS) with several improvements to the baseline system. Finally, we perform ASR using a combination of systems built from strong pre-trained models. Our proposed system achieves a macro tcpWER of 21.3 % on the dev set, which is a 57 % relative improvement over the baseline.
Transferable Selective Virtual Sensing Active Noise Control Technique Based on Metric Learning
Authors: Boxiang Wang, Dongyuan Shi, Zhengding Luo, Xiaoyi Shen, Junwei Ji, Woon-Seng Gan
Date: 2024-09-09 00:00:00
Tasks: Metric Learning

Virtual sensing (VS) technology enables active noise control (ANC) systems to attenuate noise at virtual locations distant from the physical error microphones. Appropriate auxiliary filters (AF) can significantly enhance the effectiveness of VS approaches. The selection of appropriate AF for various types of noise can be automatically achieved using convolutional neural networks (CNNs). However, training the CNN model for different ANC systems is often labour-intensive and time-consuming. To tackle this problem, we propose a novel method, Transferable Selective VS, by integrating metric-learning technology into CNN-based VS approaches. The Transferable Selective VS method allows a pre-trained CNN to be applied directly to new ANC systems without requiring retraining, and it can handle unseen noise types. Numerical simulations demonstrate the effectiveness of the proposed method in attenuating sudden-varying broadband noises and real-world noises.
Low-Complexity Own Voice Reconstruction for Hearables with an In-Ear Microphone
Authors: Mattes Ohlenbusch, Christian Rollwage, Simon Doclo
Date: 2024-09-06 00:00:00
Tasks: Data Augmentation

Hearable devices, equipped with one or more microphones, are commonly used for speech communication. Here, we consider the scenario where a hearable is used to capture the user's own voice in a noisy environment. In this scenario, own voice reconstruction (OVR) is essential for enhancing the quality and intelligibility of the recorded noisy own voice signals. In previous work, we developed a deep learning-based OVR system, aiming to reduce the amount of device-specific recordings for training by using data augmentation with phoneme-dependent models of own voice transfer characteristics. Given the limited computational resources available on hearables, in this paper we propose low-complexity variants of an OVR system based on the FT-JNF architecture and investigate the required amount of device-specific recordings for effective data augmentation and fine-tuning. Simulation results show that the proposed OVR system considerably improves speech quality, even under constraints of low complexity and a limited amount of device-specific recordings.
NPU-NTU System for Voice Privacy 2024 Challenge
Authors: Jixun Yao, Nikita Kuzmin, Qing Wang, Pengcheng Guo, Ziqian Ning, Dake Guo, Kong Aik Lee, Eng-Siong Chng, Lei Xie
Date: 2024-09-06 00:00:00
Tasks: Disentanglement, Speaker anonymization

Speaker anonymization is an effective privacy protection solution that conceals the speaker's identity while preserving the linguistic content and paralinguistic information of the original speech. To establish a fair benchmark and facilitate comparison of speaker anonymization systems, the VoicePrivacy Challenge (VPC) was held in 2020 and 2022, with a new edition planned for 2024. In this paper, we describe our proposed speaker anonymization system for VPC 2024. Our system employs a disentangled neural codec architecture and a serial disentanglement strategy to gradually disentangle the global speaker identity and time-variant linguistic content and paralinguistic information. We introduce multiple distillation methods to disentangle linguistic content, speaker identity, and emotion. These methods include semantic distillation, supervised speaker distillation, and frame-level emotion distillation. Based on these distillations, we anonymize the original speaker identity using a weighted sum of a set of candidate speaker identities and a randomly generated speaker identity. Our system achieves the best trade-off of privacy protection and emotion preservation in VPC 2024.
ZSDEVC: Zero-Shot Diffusion-based Emotional Voice Conversion with Disentangled Mechanism
Authors: Hsing-Hang Chou, Yun-Shao Lin, Ching-Chin Sung, Yu Tsao, Chi-Chun Lee
Date: 2024-09-05 00:00:00
Tasks: Emotion Classification, Voice Conversion

The human voice conveys not just words but also emotional states and individuality. Emotional voice conversion (EVC) modifies emotional expressions while preserving linguistic content and speaker identity, improving applications like human-machine interaction. While deep learning has advanced EVC models for specific target speakers on well-crafted emotional datasets, existing methods often face issues with emotion accuracy and speech distortion. In addition, the zero-shot scenario, in which emotion conversion is applied to unseen speakers, remains underexplored. This work introduces a novel diffusion framework with disentangled mechanisms and expressive guidance, trained on a large emotional speech dataset and evaluated on unseen speakers across in-domain and out-of-domain datasets. Experimental results show that our method produces expressive speech with high emotional accuracy, naturalness, and quality, showcasing its potential for broader EVC applications.
Speaker and Style Disentanglement of Speech Based on Contrastive Predictive Coding Supported Factorized Variational Autoencoder
Authors: Yuying Xie, Michael Kuhlmann, Frederik Rautenberg, Zheng-Hua Tan, Reinhold Haeb-Umbach
Date: 2024-09-05 00:00:00
Tasks: Disentanglement, Voice Conversion

Speech signals encompass various information across multiple levels including content, speaker, and style. Disentanglement of these information, although challenging, is important for applications such as voice conversion. The contrastive predictive coding supported factorized variational autoencoder achieves unsupervised disentanglement of a speech signal into speaker and content embeddings by assuming speaker info to be temporally more stable than content-induced variations. However, this assumption may introduce other temporal stable information into the speaker embeddings, like environment or emotion, which we call style. In this work, we propose a method to further disentangle non-content features into distinct speaker and style features, notably by leveraging readily accessible and well-defined speaker labels without the necessity for style labels. Experimental results validate the proposed method's effectiveness on extracting disentangled features, thereby facilitating speaker, style, or combined speaker-style conversion.
A spherical harmonic-domain spatial audio signal enhancement method based on minimum variance distortionless response
Authors: Huawei Zhang, Jihui, Zhang, Huiyuan, Sun, Prasanga Samarasinghe
Date: 2024-09-05 00:00:00

Spatial audio signal enhancement aims to reduce interfering source contributions while preserving the desired sound field with its spatial cues intact. Existing methods generally rely on impractical assumptions (e.g. no reverberation or accurate estimations of impractical information) or have limited applicability. This paper presents a spherical harmonic (SH)-domain minimum variance distortionless response (MVDR)-based spatial signal enhancer using Relative Harmonic Coefficients (ReHCs) to extract clean SH coefficients from noisy ones in reverberant environments. A simulation study shows the proposed method achieves lower estimation error, higher speech-distortion-ratio (SDR), and comparable noise reduction (NR) within the sweet area in a reverberant environment, compared to a beamforming-and-projection method as the baseline.
BELT-2: Bootstrapping EEG-to-Language representation alignment for multi-task brain decoding
Authors: Jinzhao Zhou, Yiqun Duan, Fred Chang, Thomas Do, Yu-Kai Wang, Chin-Teng Lin
Date: 2024-08-28 00:00:00
Tasks: Brain Decoding, EEG

The remarkable success of large language models (LLMs) across various multi-modality applications is well established. However, integrating large language models with humans, or brain dynamics, remains relatively unexplored. In this paper, we introduce BELT-2, a pioneering multi-task model designed to enhance both encoding and decoding performance from EEG signals. To bolster the quality of the EEG encoder, BELT-2 is the first work to innovatively 1) adopt byte-pair encoding (BPE)-level EEG-language alignment and 2) integrate multi-task training and decoding in the EEG domain. Inspired by the idea of \textbf{\textit{Bridging the Brain with GPT}}, we further connect the multi-task EEG encoder with LLMs by utilizing prefix-tuning on intermediary output from the EEG encoder. These innovative efforts make BELT-2 a pioneering breakthrough, making it the first work in the field capable of decoding coherent and readable sentences from non-invasive brain signals. Our experiments highlight significant advancements over prior techniques in both quantitative and qualitative measures, achieving a decoding performance with a BLEU-1 score of 52.2\% on the ZuCo dataset. Furthermore, BELT-2 shows a remarkable improvement ranging from 31\% to 162\% on other translation benchmarks. Codes can be accessed via the provided anonymous link~\footnote{https://anonymous.4open.science/r/BELT-2-0048}.
Dual channel CW nnU-Net for 3D PET-CT Lesion Segmentation in 2024 autoPET III Challenge
Authors: Ching-Wei Wang, Ting-Sheng Su, Keng-Wei Liu
Date: 2024-09-11 00:00:00
Tasks: Lesion Segmentation, Segmentation

PET/CT is extensively used in imaging malignant tumors because it highlights areas of increased glucose metabolism, indicative of cancerous activity. Accurate 3D lesion segmentation in PET/CT imaging is essential for effective oncological diagnostics and treatment planning. In this study, we developed an advanced 3D residual U-Net model for the Automated Lesion Segmentation in Whole-Body PET/CT - Multitracer Multicenter Generalization (autoPET III) Challenge, which will be held jointly with 2024 Medical Image Computing and Computer Assisted Intervention (MICCAI) conference at Marrakesh, Morocco. Proposed model incorporates a novel sample attention boosting technique to enhance segmentation performance by adjusting the contribution of challenging cases during training, improving generalization across FDG and PSMA tracers. The proposed model outperformed the challenge baseline model in the preliminary test set on the Grand Challenge platform, and our team is currently ranking in the 2nd place among 497 participants worldwide from 53 countries (accessed date: 2024/9/4), with Dice score of 0.8700, False Negative Volume of 19.3969 and False Positive Volume of 1.0857.
RATNUS: Rapid, Automatic Thalamic Nuclei Segmentation using Multimodal MRI inputs
Authors: Anqi Feng, Zhangxing Bian, Blake E. Dewey, Alexa Gail Colinco, Jiachen Zhuo, Jerry L. Prince
Date: 2024-09-10 00:00:00
Tasks: Segmentation

Accurate segmentation of thalamic nuclei is important for better understanding brain function and improving disease treatment. Traditional segmentation methods often rely on a single T1-weighted image, which has limited contrast in the thalamus. In this work, we introduce RATNUS, which uses synthetic T1-weighted images with many inversion times along with diffusion-derived features to enhance the visibility of nuclei within the thalamus. Using these features, a convolutional neural network is used to segment 13 thalamic nuclei. For comparison with other methods, we introduce a unified nuclei labeling scheme. Our results demonstrate an 87.19% average true positive rate (TPR) against manual labeling. In comparison, FreeSurfer and THOMAS achieve TPRs of 64.25% and 57.64%, respectively, demonstrating the superiority of RATNUS in thalamic nuclei segmentation.
Scanning Electron Microscopy-based Automatic Defect Inspection for Semiconductor Manufacturing: A Systematic Review
Authors: Enrique Dehaerne, Bappaditya Dey, Victor Blanco, Jesse Davis
Date: 2024-09-10 00:00:00
Tasks: Defect Detection

In this review, automatic defect inspection algorithms that analyze Scanning Electron Microscopy (SEM) images for Semiconductor Manufacturing (SM) are identified, categorized, and discussed. This is a topic of critical importance for the SM industry as the continuous shrinking of device patterns has led to increasing defectivity and a greater prevalence of higher-resolution imaging tools such as SEM. Among others, these aspects threaten to increase costs due to increased inspection time-to-solution and decreased yield. Relevant research papers were systematically identified in four popular publication databases in January 2024. A total of 103 papers were selected after screening for novel contributions relating to automatic SEM image analysis algorithms for semiconductor defect inspection. These papers were then categorized based on the inspection tasks they addressed, their evaluation metrics, and the type of algorithms used. A notable finding from this categorization is that reference-based defect detection algorithms were the most popular algorithm type until 2020 when Deep Learning (DL)-based inspection algorithms became more popular, especially for defect classification. Furthermore, four broader research questions were discussed to come to the following conclusions: (i) the key components of inspection algorithms are set up, pre-processing, feature extraction, and final prediction; (ii) the maturity of the manufacturing process affects the data availability and required sensitivity of inspection algorithms; (iii) key challenges for these algorithms relate to the desiderata of minimizing time-to-solution which pushes for high imaging throughput, reducing manual input during algorithm setup, and higher processing throughput; and (iv) three promising directions for future work are suggested based on gaps in the reviewed literature that address key remaining limitations.
Universal End-to-End Neural Network for Lossy Image Compression
Authors: Bouzid Arezki, Fangchen Feng, Anissa Mokraoui
Date: 2024-09-10 00:00:00
Tasks: Image Compression, SSIM

This paper presents variable bitrate lossy image compression using a VAE-based neural network. An adaptable image quality adjustment strategy is proposed. The key innovation involves adeptly adjusting the input scale exclusively during the inference process, resulting in an exceptionally efficient rate-distortion mechanism. Through extensive experimentation, across diverse VAE-based compression architectures (CNN, ViT) and training methodologies (MSE, SSIM), our approach exhibits remarkable universality. This success is attributed to the inherent generalization capacity of neural networks. Unlike methods that adjust model architecture or loss functions, our approach emphasizes simplicity, reducing computational complexity and memory requirements. The experiments not only highlight the effectiveness of our approach but also indicate its potential to drive advancements in variable-rate neural network lossy image compression methodologies.
Stain Normalization of Hematology Slides using Neural Color Transfer
Authors: M. Muneeb Arshad, Hasan Sajid, M. Jawad Khan
Date: 2024-09-10 00:00:00
Tasks: Deep Learning

Deep learning is popularly used for analyzing pathology images, but variations in image properties can limit the effectiveness of the models. The study aims to develop a method that transfers the variability present in the training set to unseen images, improving the model's ability to make accurate inferences. YOLOv5 was trained on peripheral blood and bone marrow sample images and Neural Color Transfer techniques were used to incorporate invariance. The results showed significant improvement in detecting WBCs from untrained samples after normalization, highlighting the potential of deep learning-based normalization techniques for inference robustness.
Outlier Removal in Cryo-EM via Radial Profiles
Authors: Lev Kapnulin, Ayelet Heimowitz, Nir Sharon
Date: 2024-09-09 00:00:00

The process of particle picking, a crucial step in cryo-electron microscopy (cryo-EM) image analysis, often encounters challenges due to outliers, leading to inaccuracies in downstream processing. In response to this challenge, this research introduces an additional automated step to reduce the number of outliers identified by the particle picker. The proposed method enhances both the accuracy and efficiency of particle picking, thereby reducing the overall running time and the necessity for expert intervention in the process. Experimental results demonstrate the effectiveness of the proposed approach in mitigating outlier inclusion and its potential to enhance cryo-EM data analysis pipelines significantly. This work contributes to the ongoing advancement of automated cryo-EM image processing methods, offering novel insights and solutions to challenges in structural biology research.
Physics-augmented Deep Learning with Adversarial Domain Adaptation: Applications to STM Image Denoising
Authors: Jianxin Xie, Wonhee Ko, Rui-Xing Zhang, Bing Yao
Date: 2024-09-08 00:00:00
Tasks: Denoising, Domain Adaptation, Image Denoising, scientific discovery

Image denoising is a critical task in various scientific fields such as medical imaging and material characterization, where the accurate recovery of underlying structures from noisy data is essential. Although supervised denoising techniques have achieved significant advancements, they typically require large datasets of paired clean-noisy images for training. Unsupervised methods, while not reliant on paired data, typically necessitate a set of unpaired clean images for training, which are not always accessible. In this paper, we propose a physics-augmented deep learning with adversarial domain adaption (PDA-Net) framework for unsupervised image denoising, with applications to denoise real-world scanning tunneling microscopy (STM) images. Our PDA-Net leverages the underlying physics to simulate and envision the ground truth for denoised STM images. Additionally, built upon Generative Adversarial Networks (GANs), we incorporate a cycle-consistency module and a domain adversarial module into our PDA-Net to address the challenge of lacking paired training data and achieve information transfer between the simulated and real experimental domains. Finally, we propose to implement feature alignment and weight-sharing techniques to fully exploit the similarity between simulated and real experimental images, thereby enhancing the denoising performance in both the simulation and experimental domains. Experimental results demonstrate that the proposed PDA-Net successfully enhances the quality of STM images, offering promising applications to enhance scientific discovery and accelerate experimental quantum material research.
Cluster-based Random Radial Basis Kernel Function for Hyperspectral Data Classification
Authors: Saeid Niazmardi
Date: 2024-09-08 00:00:00
Tasks: Classification

Kernel-based classification methods, particularly the support vector machine (SVM), are among the most common algorithms for hyperspectral data classification. The Radial Basis function (RBF) kernel has earned great popularity in hyperspectral data classification due to its superior performance among other available kernel functions. Nonetheless, the cross-validation technique usually used for tunning the RBF parameter can be time-consuming and may result in sub-optimal values for the parameter. This paper proposed the cluster-based random radial basis function (CRRBF) kernel function as an alternative to the RBF kernel to achieve similar performance with a more manageable parameter, which is the number of clusters. The CRRBF kernel initially clusters the hyperspectral bands and then constructs an RBF kernel with a randomly assigned value as the kernel parameter from each cluster of bands. The final CRRBF kernel is constructed by adding up these basis RBF kernels. We have designed several experiments to evaluate the SVM performance trained with the CRRBF kernel considering a different number of clusters and training samples, using three hyperspectral data sets. The obtained results showed that the CRRBF kernel could provide comparable or better results than the RBF. The results also showed that the classification performance is pretty robust to the number of clusters, as the only open parameter of the CRRBF kernel.
Unrolling Plug-and-Play Network for Hyperspectral Unmixing
Authors: Min Zhao, Linruize Tang, Jie Chen
Date: 2024-09-07 00:00:00
Tasks: Hyperspectral Unmixing

Deep learning based unmixing methods have received great attention in recent years and achieve remarkable performance. These methods employ a data-driven approach to extract structure features from hyperspectral image, however, they tend to be less physical interpretable. Conventional unmixing methods are with much more interpretability, whereas they require manually designing regularization and choosing penalty parameters. To overcome these limitations, we propose a novel unmixing method by unrolling the plug-and-play unmixing algorithm to conduct the deep architecture. Our method integrates both inner and outer priors. The carefully designed unfolding deep architecture is used to learn the spectral and spatial information from the hyperspectral image, which we refer to as inner priors. Additionally, our approach incorporates deep denoisers that have been pretrained on a large volume of image data to leverage the outer priors. Secondly, we design a dynamic convolution to model the multiscale information. Different scales are fused using an attention module. Experimental results of both synthetic and real datasets demonstrate that our method outperforms compared methods.