pwc-archive/papers-with-abstracts huggingface dataset:
Target Active Speaker Detection with Audio-visual Cues
Authors: Yidi Jiang, Ruijie Tao, Zexu Pan, Haizhou Li
Date: 2023-05-22 00:00:00
Tasks: Active Speaker Detection, Audio-Visual Synchronization

In active speaker detection (ASD), we would like to detect whether an on-screen person is speaking based on audio-visual cues. Previous studies have primarily focused on modeling audio-visual synchronization cue, which depends on the video quality of the lip region of a speaker. In real-world applications, it is possible that we can also have the reference speech of the on-screen speaker. To benefit from both facial cue and reference speech, we propose the Target Speaker TalkNet (TS-TalkNet), which leverages a pre-enrolled speaker embedding to complement the audio-visual synchronization cue in detecting whether the target speaker is speaking. Our framework outperforms the popular model, TalkNet on two datasets, achieving absolute improvements of 1.6% in mAP on the AVA-ActiveSpeaker validation set, and 0.8%, 0.4%, and 0.8% in terms of AP, AUC and EER on the ASW test set, respectively. Code is available at https://github.com/Jiang-Yidi/TS-TalkNet/.
ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer
Authors: Huadai Liu, Rongjie Huang, Xuan Lin, Wenqiang Xu, Maozong Zheng, Hong Chen, Jinzheng He, Zhou Zhao
Date: 2023-05-22 00:00:00
Tasks: Decoder, Denoising, Rhythm, Self-Supervised Learning, text-to-speech, Text to Speech

Text-to-speech(TTS) has undergone remarkable improvements in performance, particularly with the advent of Denoising Diffusion Probabilistic Models (DDPMs). However, the perceived quality of audio depends not solely on its content, pitch, rhythm, and energy, but also on the physical environment. In this work, we propose ViT-TTS, the first visual TTS model with scalable diffusion transformers. ViT-TTS complement the phoneme sequence with the visual information to generate high-perceived audio, opening up new avenues for practical applications of AR and VR to allow a more immersive and realistic audio experience. To mitigate the data scarcity in learning visual acoustic information, we 1) introduce a self-supervised learning framework to enhance both the visual-text encoder and denoiser decoder; 2) leverage the diffusion transformer scalable in terms of parameters and capacity to learn visual scene information. Experimental results demonstrate that ViT-TTS achieves new state-of-the-art results, outperforming cascaded systems and other baselines regardless of the visibility of the scene. With low-resource data (1h, 2h, 5h), ViT-TTS achieves comparative results with rich-resource baselines.~\footnote{Audio samples are available at \url{https://ViT-TTS.github.io/.}}
Towards Robust Family-Infant Audio Analysis Based on Unsupervised Pretraining of Wav2vec 2.0 on Large-Scale Unlabeled Family Audio
Authors: Jialu Li, Mark Hasegawa-Johnson, Nancy L. McElwain
Date: 2023-05-21 00:00:00
Tasks: speaker-diarization, Speaker Diarization

To perform automatic family audio analysis, past studies have collected recordings using phone, video, or audio-only recording devices like LENA, investigated supervised learning methods, and used or fine-tuned general-purpose embeddings learned from large pretrained models. In this study, we advance the audio component of a new infant wearable multi-modal device called LittleBeats (LB) by learning family audio representation via wav2vec 2.0 (W2V2) pertaining. We show given a limited number of labeled LB home recordings, W2V2 pretrained using 1k-hour of unlabeled home recordings outperforms oracle W2V2 pretrained on 960-hour unlabeled LibriSpeech in terms of parent/infant speaker diarization (SD) and vocalization classifications (VC) at home. Extra relevant external unlabeled and labeled data further benefit W2V2 pretraining and fine-tuning. With SpecAug and environmental speech corruptions, we obtain 12% relative gain on SD and moderate boost on VC. Code and model weights are available.
CASA-ASR: Context-Aware Speaker-Attributed ASR
Authors: Mohan Shi, Zhihao Du, Qian Chen, Fan Yu, Yangze Li, Shiliang Zhang, Jie Zhang, Li-Rong Dai
Date: 2023-05-21 00:00:00
Tasks: Automatic Speech Recognition, speech-recognition, Speech Recognition

Recently, speaker-attributed automatic speech recognition (SA-ASR) has attracted a wide attention, which aims at answering the question ``who spoke what''. Different from modular systems, end-to-end (E2E) SA-ASR minimizes the speaker-dependent recognition errors directly and shows a promising applicability. In this paper, we propose a context-aware SA-ASR (CASA-ASR) model by enhancing the contextual modeling ability of E2E SA-ASR. Specifically, in CASA-ASR, a contextual text encoder is involved to aggregate the semantic information of the whole utterance, and a context-dependent scorer is employed to model the speaker discriminability by contrasting with speakers in the context. In addition, a two-pass decoding strategy is further proposed to fully leverage the contextual modeling ability resulting in a better recognition performance. Experimental results on AliMeeting corpus show that the proposed CASA-ASR model outperforms the original E2E SA-ASR system with a relative improvement of 11.76% in terms of speaker-dependent character error rate.
Semantic VAD: Low-Latency Voice Activity Detection for Speech Interaction
Authors: Mohan Shi, Yuchun Shu, Lingyun Zuo, Qian Chen, Shiliang Zhang, Jie Zhang, Li-Rong Dai
Date: 2023-05-21 00:00:00
Tasks: Action Detection, Activity Detection, Automatic Speech Recognition, Automatic Speech Recognition (ASR), Segmentation, speech-recognition, Speech Recognition

For speech interaction, voice activity detection (VAD) is often used as a front-end. However, traditional VAD algorithms usually need to wait for a continuous tail silence to reach a preset maximum duration before segmentation, resulting in a large latency that affects user experience. In this paper, we propose a novel semantic VAD for low-latency segmentation. Different from existing methods, a frame-level punctuation prediction task is added to the semantic VAD, and the artificial endpoint is included in the classification category in addition to the often-used speech presence and absence. To enhance the semantic information of the model, we also incorporate an automatic speech recognition (ASR) related semantic loss. Evaluations on an internal dataset show that the proposed method can reduce the average latency by 53.3% without significant deterioration of character error rate in the back-end ASR compared to the traditional VAD approach.
DualVC: Dual-mode Voice Conversion using Intra-model Knowledge Distillation and Hybrid Predictive Coding
Authors: Ziqian Ning, Yuepeng Jiang, Pengcheng Zhu, Jixun Yao, Shuai Wang, Lei Xie, Mengxiao Bi
Date: 2023-05-21 00:00:00
Tasks: Data Augmentation, Decoder, Knowledge Distillation, Voice Conversion

Voice conversion is an increasingly popular technology, and the growing number of real-time applications requires models with streaming conversion capabilities. Unlike typical (non-streaming) voice conversion, which can leverage the entire utterance as full context, streaming voice conversion faces significant challenges due to the missing future information, resulting in degraded intelligibility, speaker similarity, and sound quality. To address this challenge, we propose DualVC, a dual-mode neural voice conversion approach that supports both streaming and non-streaming modes using jointly trained separate network parameters. Furthermore, we propose intra-model knowledge distillation and hybrid predictive coding (HPC) to enhance the performance of streaming conversion. Additionally, we incorporate data augmentation to train a noise-robust autoregressive decoder, improving the model's performance on long-form speech conversion. Experimental results demonstrate that the proposed model outperforms the baseline models in the context of streaming voice conversion, while maintaining comparable performance to the non-streaming topline system that leverages the complete context, albeit with a latency of only 252.8 ms.
DCCRN-KWS: an audio bias based model for noise robust small-footprint keyword spotting
Authors: Shubo Lv, Xiong Wang, Sining Sun, Long Ma, Lei Xie
Date: 2023-05-21 00:00:00
Tasks: Denoising, Keyword Spotting, Multi-Task Learning, Small-Footprint Keyword Spotting, Speech Enhancement, speech-recognition, Speech Recognition

Real-world complex acoustic environments especially the ones with a low signal-to-noise ratio (SNR) will bring tremendous challenges to a keyword spotting (KWS) system. Inspired by the recent advances of neural speech enhancement and context bias in speech recognition, we propose a robust audio context bias based DCCRN-KWS model to address this challenge. We form the whole architecture as a multi-task learning framework for both denosing and keyword spotting, where the DCCRN encoder is connected with the KWS model. Helped with the denoising task, we further introduce an audio context bias module to leverage the real keyword samples and bias the network to better iscriminate keywords in noisy conditions. Feature merge and complex context linear modules are also introduced to strength such discrimination and to effectively leverage contextual information respectively. Experiments on the internal challenging dataset and the HIMIYA public dataset show that our DCCRN-KWS system is superior in performance, while ablation study demonstrates the good design of the whole model.
Joint Generative-Contrastive Representation Learning for Anomalous Sound Detection
Authors: Xiao-Min Zeng, Yan Song, Zhu Zhuo, Yu Zhou, Yu-Hong Li, Hui Xue, Li-Rong Dai, Ian McLoughlin
Date: 2023-05-20 00:00:00
Tasks: Contrastive Learning, Representation Learning

In this paper, we propose a joint generative and contrastive representation learning method (GeCo) for anomalous sound detection (ASD). GeCo exploits a Predictive AutoEncoder (PAE) equipped with self-attention as a generative model to perform frame-level prediction. The output of the PAE together with original normal samples, are used for supervised contrastive representative learning in a multi-task framework. Besides cross-entropy loss between classes, contrastive loss is used to separate PAE output and original samples within each class. GeCo aims to better capture context information among frames, thanks to the self-attention mechanism for PAE model. Furthermore, GeCo combines generative and contrastive learning from which we aim to yield more effective and informative representations, compared to existing methods. Extensive experiments have been conducted on the DCASE2020 Task2 development dataset, showing that GeCo outperforms state-of-the-art generative and discriminative methods.
Robust dynamic operating envelopes for flexibility operation using only local voltage measurement
Authors: Md Umar Hashmi, Dirk Van Hertem
Date: 2023-05-22 00:00:00

With growing intermittency and uncertainty in distribution networks around the world, ensuring operational integrity is becoming challenging. Recent use cases of dynamic operating envelopes (DOEs) indicate that they can be utilized for network awareness for autonomous operation of flexibility, maximizing distributed generation integration, coordinating flexibility in different power networks and in resource planning. To this end, a novel framework is presented for generating decentralized DOEs in real-time using only the nodal voltage measurement and partially decentralized, risk-averse, robust DOEs in a time-ahead setting using voltage forecast scenarios. Chance constraint level is analytically implemented for avoiding extremely restrictive time-ahead DOEs with insufficient feasible regions for local energy optimization. Since the proposed DOE calculation framework uses none or limited centralized feedback, it is resilient to cyberattacks, communication failures, missing data and errors in network layout information. Numerical results showcase the DOE calculation framework in real-time using voltage magnitude measurements and in day-ahead timeframe using forecasted voltage scenarios. Furthermore, the DOEs are extended to form P-Q charts while considering power factor and converter capacity limits.
Integrated Guidance and Gimbal Control for Coverage Planning With Visibility Constraints
Authors: Savvas Papaioannou, Panayiotis Kolios, Theocharis Theocharides, Christos G. Panayiotou, Marios M. Polycarpou
Date: 2023-05-22 00:00:00

Coverage path planning with unmanned aerial vehicles (UAVs) is a core task for many services and applications including search and rescue, precision agriculture, infrastructure inspection and surveillance. This work proposes an integrated guidance and gimbal control coverage path planning (CPP) approach, in which the mobility and gimbal inputs of an autonomous UAV agent are jointly controlled and optimized to achieve full coverage of a given object of interest, according to a specified set of optimality criteria. The proposed approach uses a set of visibility constraints to integrate the physical behavior of sensor signals (i.e., camera-rays) into the coverage planning process, thus generating optimized coverage trajectories that take into account which parts of the scene are visible through the agent's camera at any point in time. The integrated guidance and gimbal control CPP problem is posed in this work as a constrained optimal control problem which is then solved using mixed integer programming (MIP) optimization. Extensive numerical experiments demonstrate the effectiveness of the proposed approach.
Lagrangian-based online safe reinforcement learning for state-constrained systems
Authors: Soutrik Bandyopadhyay, Shubhendu Bhasin
Date: 2023-05-22 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Safe Reinforcement Learning

This paper proposes a safe reinforcement learning (RL) algorithm that approximately solves the state-constrained optimal control problem for continuous-time uncertain nonlinear systems. We formulate the safe RL problem as the minimization of a Lagrangian that includes the cost functional and a user-defined barrier Lyapunov function (BLF) encoding the state constraints. We show that the analytical solution obtained by the application of Karush-Kuhn-Tucker (KKT) conditions contains a state-dependent expression for the Lagrange multiplier, which is a function of uncertain terms in the system dynamics. We argue that a naive estimation of the Lagrange multiplier may lead to safety constraint violations. To obviate this challenge, we propose an Actor-Critic-Identifier-Lagrangian (ACIL) algorithm that learns optimal control policies from online data without compromising safety. We provide safety and boundedness guarantees with the proposed algorithm and compare its performance with existing offline/online RL methods via a simulation study.
Position Control of Single Link Flexible Manipulator: A Functional Observer Based Sliding Mode Approach
Authors: Atul Sharma, S. Janardhanan
Date: 2023-05-22 00:00:00
Tasks: Position

This paper proposes a functional observer-based sliding mode control technique for position control of a single-link flexible manipulator. The proposed method considers the unmodelled system dynamics as uncertainty and aims to achieve accurate position control. The functional observer is used to directly estimate the sliding mode control design components and a sliding mode controller to generate the control signal, which guarantees the system's robustness and stability. The proposed control scheme is validated using numerical simulations.
Adaptive Parameter Estimation under Finite Excitation
Authors: Siyu Chen, Jing Na, Yingbo Huang
Date: 2023-05-22 00:00:00
Tasks: parameter estimation

Although persistent excitation is often acknowledged as a sufficient condition to exponentially converge in the field of adaptive parameter estimation, it must be noted that in practical applications this may be unguaranteed. Recently, more attention has turned to another relaxed condition, i.e., finite excitation. In this paper, for a class of nominal nonlinear systems with unknown constant parameters, a novel method that combines the Newton algorithm and the time-varying factor is proposed, which can achieve exponential convergence under finite excitation. First, by introducing pre-filtering, the nominal system is transformed to a linear parameterized form. Then the detailed mathematical derivation is outlined from an estimation error accumulated cost function. And it is given that the theoretical analysis of the proposed method in stability and robustness. Finally, comparative numerical simulations are given to illustrate the superiority of the proposed method.
GreenEVT: Greensboro Electric Vehicle Testbed
Authors: Gustav Nilsson, Alejandro D. Owen Aquino, Samuel Coogan, Daniel K. Molzahn
Date: 2023-05-22 00:00:00

The ongoing electrification of the transportation fleet will increase the load on the electric power grid. Since both the transportation network and the power grid already experience periods of significant stress, joint analyses of both infrastructures will most likely be necessary to ensure acceptable operation in the future. To enable such analyses, this paper presents an open-source testbed that jointly simulates high-fidelity models of both the electric distribution system and the transportation network. The testbed utilizes two open-source simulators, OpenDSS to simulate the electric distribution system and the microscopic traffic simulator SUMO to simulate the traffic dynamics. Electric vehicle charging links the electric distribution system and the transportation network models at vehicle locations determined using publicly available parcel data. Leveraging high-fidelity synthetic electric distribution system data from the SMART-DS project and transportation system data from OpenStreetMap, this testbed models the city of Greensboro, NC down to the household level. Moreover, the methodology and the supporting scripts released with the testbed allow adaption to other areas where high-fidelity geolocated OpenDSS datasets are available. After describing the components and usage of the testbed, we exemplify applications enabled by the testbed via two scenarios modeling the extreme stresses encountered during evacuations.
Safety Challenges and Analysis of Autonomous Electric Vehicle Development: Insights from On-Road Testing and Accident Reports
Authors: Qasim Ajao, Olukotun Oludamilare
Date: 2023-05-22 00:00:00

Autonomous electric vehicles (AEVs) hold great promise for the future of automotive engineering, but safety remains a significant challenge in their development and commercialization. Therefore, conducting a comprehensive analysis of AEV development and reported accidents is crucial. This paper reviews the levels of automation in AEVs, their disengagement frequencies, and on-road accident reports. According to the report, numerous manufacturers thoroughly tested AEVs across a distance of more than 3.9 million miles between 2014 and 2022. Disengagement frequencies vary among manufacturers, and approximately 65% of accidents during this period occurred while AEVs were operating in autonomous mode. Notably, the majority of accidents (90%) were caused by other road users, with only a small fraction (approximately 8%) directly attributed to AEVs. Enhancing AEVs' ability to detect and mitigate safety risks from external sources has the potential to significantly improve their safety. This paper provides valuable insights into AEV safety by emphasizing the importance of comprehensively understanding AEV development and reported accidents. Through the analysis of disengagement and accident reports, the study highlights the prevalence of passive accidents caused by other road users. Future research should concentrate on enabling AEVs to effectively detect and respond to safety risks originating from external sources to enhance AEV safety. Overall, this analysis contributes to the ongoing efforts in AEV development and provides guidance for strategies aimed at improving their safety features.
Wavefront Engineering: Realizing Efficient Terahertz Band Communications in 6G and Beyond
Authors: Arjun Singh, Vitaly Petrov, Hichem Guerboukha, Innem V. A. K. Reddy, Edward W. Knightly, Daniel M. Mittleman, Josep M. Jornet
Date: 2023-05-22 00:00:00

Terahertz (THz) band communications is envisioned as a key technology for future wireless standards. Substantial progress has been made in this field, with advances in hardware design, channel models, and signal processing. High-rate backhaul links operating at sub-THz frequencies have been experimentally demonstrated. However, there are inherent challenges in making the next great leap for adopting the THz band in widespread communication systems, such as cellular access and wireless local area networks. Primarily, such systems have to be both: (i) wideband, to maintain desired data rate and sensing resolution; and, more importantly, (ii) operate in the massive near field of the high-gain devices required to overcome the propagation losses. In this article, it is first explained why the state-of-the-art techniques from lower frequencies, including millimeter-wave, cannot be simply repurposed to realize THz band communication systems. Then, a vision of wavefront engineering is presented to address these shortfalls. Further, it is illustrated how novel implementations of specific wavefronts, such as Bessel beams and Airy beams, offer attractive advantages in creating THz links over state-of-the-art far-field beamforming and near-field beamfocusing techniques. The paper ends by discussing novel problems and challenges in this new and exciting research area. Index Terms - Terahertz Communications; 6G; Wavefront Engineering; Bessel beams; Near field; Orbital Angular Momentum
Tunable Experimental Testbed for Evaluating Load Coordination Methods
Authors: Drew A. Geller, Johanna L. Mathieu
Date: 2023-05-21 00:00:00

Driven by the need to offset the variability of wind and solar generation on the electrical grid, development of load controls is a highly active field in the engineering literature. However, practical use of residential loads for grid balancing and ancillary services remains rare, in part due to the relative cost of communicating with hordes of small loads and also due to the limited experimentation done so far to demonstrate reliable operation. To establish a basis for the safe and reliable use of fleets of small compressor loads as distributed energy resources (DERs), we have constructed an experimental testbed in a laboratory, so that load coordination schemes can be tested at extreme conditions within a laboratory environment. This experiment can be used to tune a simulation testbed to which it can then be linked, thereby augmenting the effective size of the ensemble of loads. Control algorithms can simply be plugged in for testing. Modeling of the system was done both to demonstrate the experimental testbed's behavior and also to understand how to tune the behavior of each participating model house in the system. Implementing this testbed has been useful for the rapid turnaround of experiments on various control types, and it enables testing year-round without the constraints and limitations arising in seasonal field tests with real human participants.
Atomic Anatomy of Low-Inertia Power Systems
Authors: Subham Sahoo, Arpan Malkhandi, Kristian Skafte Jensen
Date: 2023-05-21 00:00:00
Tasks: Anatomy, Quantization

In this article, we determine a fundamental anatomical modeling parallelism between low-inertia power systems and Bohr's atomic model. The proposed atomic architecture will serve as a microscopic building block, where we validate the structural analogy of low-inertia power systems using semi-classical quantum approximations in IEEE 9-bus & 39-bus systems. As a future scope of work, detailed modeling & system stability will be investigated by using pre-quantization and geometric quantization methods.