pwc-archive/papers-with-abstracts huggingface dataset:
Long-Term Conversation Analysis: Privacy-Utility Trade-off under Noise and Reverberation
Authors: Jule Pohlhausen, Francesco Nespoli, Joerg Bitzer
Date: 2024-08-01 00:00:00
Tasks: Action Detection, Activity Detection, Edge-computing, Privacy Preserving, Speaker anonymization, speaker-diarization, Speaker Diarization, Speaker Recognition

Recordings in everyday life require privacy preservation of the speech content and speaker identity. This contribution explores the influence of noise and reverberation on the trade-off between privacy and utility for low-cost privacy-preserving methods feasible for edge computing. These methods compromise spectral and temporal smoothing, speaker anonymization using the McAdams coefficient, sampling with a very low sampling rate, and combinations. Privacy is assessed by automatic speech and speaker recognition, while our utility considers voice activity detection and speaker diarization. Overall, our evaluation shows that additional noise degrades the performance of all models more than reverberation. This degradation corresponds to enhanced speech privacy, while utility is less deteriorated for some methods.
Towards EMG-to-Speech with a Necklace Form Factor
Authors: Peter Wu, Ryan Kaveh, Raghav Nautiyal, Christine Zhang, Albert Guo, Anvitha Kachinthaya, Tavish Mishra, Bohan Yu, Alan W Black, Rikky Muller, Gopala Krishna Anumanchipalli
Date: 2024-07-31 00:00:00
Tasks: Electromyography (EMG), Form

Electrodes for decoding speech from electromyography (EMG) are typically placed on the face, requiring adhesives that are inconvenient and skin-irritating if used regularly. We explore a different device form factor, where dry electrodes are placed around the neck instead. 11-word, multi-speaker voiced EMG classifiers trained on data recorded with this device achieve 92.7% accuracy. Ablation studies reveal the importance of having more than two electrodes on the neck, and phonological analyses reveal similar classification confusions between neck-only and neck-and-face form factors. Finally, speech-EMG correlation experiments demonstrate a linear relationship between many EMG spectrogram frequency bins and self-supervised speech representation dimensions.
$T\bar{a}laGen:$ A System for Automatic $T\bar{a}la$ Identification and Generation
Authors: Rahul Bapusaheb Kodag, Himanshu Jindal, Vipul Arora
Date: 2024-07-30 00:00:00
Tasks: Meta-Learning

In Hindustani classical music, the tabla plays an important role as a rhythmic backbone and accompaniment. In applications like computer-based music analysis, learning singing, and learning musical instruments, tabla stroke transcription, $t\bar{a}la$ identification, and generation are crucial. This paper proposes a comprehensive system aimed at addressing these challenges. For tabla stroke transcription, we propose a novel approach based on model-agnostic meta-learning (MAML) that facilitates the accurate identification of tabla strokes using minimal data. Leveraging these transcriptions, the system introduces two novel $t\bar{a}la$ identification methods based on the sequence analysis of tabla strokes. \par Furthermore, the paper proposes a framework for $t\bar{a}la$ generation to bridge traditional and modern learning methods. This framework utilizes finite state transducers (FST) and linear time-invariant (LTI) filters to generate $t\bar{a}las$ with real-time tempo control through user interaction, enhancing practice sessions and musical education. Experimental evaluations on tabla solo and concert datasets demonstrate the system's exceptional performance on real-world data and its ability to outperform existing methods. Additionally, the proposed $t\bar{a}la$ identification methods surpass state-of-the-art techniques. The contributions of this paper include a combined approach to tabla stroke transcription, innovative $t\bar{a}la$ identification techniques, and a robust framework for $t\bar{a}la$ generation that handles the rhythmic complexities of Hindustani music.
Blind Acoustic Parameter Estimation Through Task-Agnostic Embeddings Using Latent Approximations
Authors: Philipp Götz, Cagdas Tuna, Andreas Brendel, Andreas Walther, Emanuël A. P. Habets
Date: 2024-07-29 00:00:00
Tasks: parameter estimation, regression

We present a method for blind acoustic parameter estimation from single-channel reverberant speech. The method is structured into three stages. In the first stage, a variational auto-encoder is trained to extract latent representations of acoustic impulse responses represented as mel-spectrograms. In the second stage, a separate speech encoder is trained to estimate low-dimensional representations from short segments of reverberant speech. Finally, the pre-trained speech encoder is combined with a small regression model and evaluated on two parameter regression tasks. Experimentally, the proposed method is shown to outperform a fully end-to-end trained baseline model.
Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognition
Authors: Jingjing Xu, Wei Zhou, Zijian Yang, Eugen Beck, Ralf Schlueter
Date: 2024-07-10 00:00:00
Tasks: speech-recognition, Speech Recognition

Varying-size models are often required to deploy ASR systems under different hardware and/or application constraints such as memory and latency. To avoid redundant training and optimization efforts for individual models of different sizes, we present the dynamic encoder size approach, which jointly trains multiple performant models within one supernet from scratch. These subnets of various sizes are layer-wise pruned from the supernet, and thus, enjoy full parameter sharing. By combining score-based pruning with supernet training, we propose two novel methods, Simple-Top-k and Iterative-Zero-Out, to automatically select the best-performing subnets in a data-driven manner, avoiding resource-intensive search efforts. Our experiments using CTC on both Librispeech and TED-LIUM-v2 corpora show that our methods can achieve on-par performance as individually trained models of each size category. Also, our approach consistently brings small performance improvements for the full-size supernet.
Explaining Spectrograms in Machine Learning: A Study on Neural Networks for Speech Classification
Authors: Jesin James, Balamurali B. T., Binu Abeysinghe, Junchen Liu
Date: 2024-07-10 00:00:00
Tasks: Classification, speech-recognition, Speech Recognition, Vowel Classification

This study investigates discriminative patterns learned by neural networks for accurate speech classification, with a specific focus on vowel classification tasks. By examining the activations and features of neural networks for vowel classification, we gain insights into what the networks "see" in spectrograms. Through the use of class activation mapping, we identify the frequencies that contribute to vowel classification and compare these findings with linguistic knowledge. Experiments on a American English dataset of vowels showcases the explainability of neural networks and provides valuable insights into the causes of misclassifications and their characteristics when differentiating them from unvoiced speech. This study not only enhances our understanding of the underlying acoustic cues in vowel classification but also offers opportunities for improving speech recognition by bridging the gap between abstract representations in neural networks and established linguistic knowledge
BVI-AOM: A New Training Dataset for Deep Video Compression Optimization
Authors: Jakub Nawała, YuXuan Jiang, Fan Zhang, Xiaoqing Zhu, Joel Sole, David Bull
Date: 2024-08-06 00:00:00
Tasks: Video Compression

Deep learning is now playing an important role in enhancing the performance of conventional hybrid video codecs. These learning-based methods typically require diverse and representative training material for optimization in order to achieve model generalization and optimal coding performance. However, existing datasets either offer limited content variability or come with restricted licensing terms constraining their use to research purposes only. To address these issues, we propose a new training dataset, named BVI-AOM, which contains 956 uncompressed sequences at various resolutions from 270p to 2160p, covering a wide range of content and texture types. The dataset comes with more flexible licensing terms and offers competitive performance when used as a training set for optimizing deep video coding tools. The experimental results demonstrate that when used as a training set to optimize two popular network architectures for two different coding tools, the proposed dataset leads to additional bitrate savings of up to 0.29 and 2.98 percentage points in terms of PSNR-Y and VMAF, respectively, compared to an existing training dataset, BVI-DVC, which has been widely used for deep video coding. The BVI-AOM dataset is available at https://github.com/fan-aaron-zhang/bvi-aom
Image Quality Transfer of Diffusion MRI Guided By High-Resolution Structural MRI
Authors: Alp G. Cicimen, Henry F. J. Tregidgo, Matteo Figini, Eirini Messaritaki, Carolyn B. McNabb, Marco Palombo, C. John Evans, Mara Cercignani, Derek K. Jones, Daniel C. Alexander
Date: 2024-08-06 00:00:00
Tasks: Diffusion  MRI

Prior work on the Image Quality Transfer on Diffusion MRI (dMRI) has shown significant improvement over traditional interpolation methods. However, the difficulty in obtaining ultra-high resolution Diffusion MRI scans poses a problem in training neural networks to obtain high-resolution dMRI scans. Here we hypothesise that the inclusion of structural MRI images, which can be acquired at much higher resolutions, can be used as a guide to obtaining a more accurate high-resolution dMRI output. To test our hypothesis, we have constructed a novel framework that incorporates structural MRI scans together with dMRI to obtain high-resolution dMRI scans. We set up tests which evaluate the validity of our claim through various configurations and compare the performance of our approach against a unimodal approach. Our results show that the inclusion of structural MRI scans do lead to an improvement in high-resolution image prediction when T1w data is incorporated into the model input.
Constructing Per-Shot Bitrate Ladders using Visual Information Fidelity
Authors: Krishna Srikar Durbha, Alan C. Bovik
Date: 2024-08-04 00:00:00

Adaptive video streaming allows for the construction of bitrate ladders that deliver perceptually optimized visual quality to viewers under bandwidth constraints. Two common approaches to adaptation are per-title encoding and per-shot encoding. The former involves encoding each program, movie, or other content in a manner that is perceptually- and bandwidth-optimized for that content but is otherwise fixed. The latter is a more granular approach that optimizes the encoding parameters for each scene or shot (however defined) of a video content. Per-shot video encoding, as pioneered by Netflix, encodes on a per-shot basis using the Dynamic Optimizer (DO). Under the control of the VMAF perceptual video quality prediction engine, the DO delivers high-quality videos to millions of viewers at considerably reduced bitrates than per-title or fixed bitrate ladder encoding. A variety of per-title and per-shot encoding techniques have been recently proposed that seek to reduce computational overhead and to construct optimal bitrate ladders more efficiently using low-level features extracted from source videos. Here we develop a perceptually optimized method of constructing optimal per-shot bitrate and quality ladders, using an ensemble of low-level features and Visual Information Fidelity (VIF) features extracted from different scales and subbands. We compare the performance of our model, which we call VIF-ladder, against other content-adaptive bitrate ladder prediction methods, counterparts of them that we designed to construct quality ladders, a fixed bitrate ladder, and bitrate ladders constructed via exhaustive encoding using Bjontegaard delta metrics.
Deep Learning for Super-resolution Ultrasound Imaging with Spatiotemporal Data
Authors: Arthur David Redfern, Katherine G. Brown
Date: 2024-07-29 00:00:00
Tasks: Deep Learning, Super-Resolution

Super-resolution ultrasound imaging (SRUS) is an active area of research as it brings up to a ten-fold improvement in the resolution of microvascular structures. The limitations to the clinical adoption of SRUS include long acquisition times and long image processing times. Both these limitations can be alleviated with deep learning approaches to the processing of SRUS images. In this study we propose an optimized architecture based on modern improvements to convolutional neural networks from the ConvNeXt architecture and further customize the choice of features to improve performance on the specific tasks of both MB detection and localization within a single network. We employ a spatiotemporal input of up to five successive image frames to increase the number of MBs detected. The output structure produces three classifications: a MB detection Boolean for each pixel in the central image frame, as well as x and z offsets at 4-fold subpixel resolution for each MB detected. Ultrasound simulations generated images based on the L22-14v transducer (Verasonics) for training and testing of the proposed SRUS-ConvNeXt network. In vivo image data of a mouse brain was used as further validation of the architecture. The proposed network had the highest performance as measured by F1 score when configured for a 3-frame spatiotemporal input. The smallest localization error of {\lambda}/22 was achieved when the network was configured for a single input frame. The flexibility of the proposed architecture allows extension to 10-fold upscaling for SRUS images with a much lower impact to number of parameters and subsequent increase in inference time than typical U-Net style approaches. This network is promising in the quest to develop a SRUS deep network architecture for real time image formation.
Pre-processing and quality control of large clinical CT head datasets for intracranial arterial calcification segmentation
Authors: Benjamin Jin, Maria del C. Valdés Hernández, Alessandro Fontanella, Wenwen Li, Eleanor Platt, Paul Armitage, Amos Storkey, Joanna M. Wardlaw, Grant Mair
Date: 2024-08-02 00:00:00

As a potential non-invasive biomarker for ischaemic stroke, intracranial arterial calcification (IAC) could be used for stroke risk assessment on CT head scans routinely acquired for other reasons (e.g. trauma, confusion). Artificial intelligence methods can support IAC scoring, but they have not yet been developed for clinical imaging. Large heterogeneous clinical CT datasets are necessary for the training of such methods, but they exhibit expected and unexpected data anomalies. Using CTs from a large clinical trial, the third International Stroke Trial (IST-3), we propose a pipeline that uses as input non-enhanced CT scans to output regions of interest capturing selected large intracranial arteries for IAC scoring. Our method uses co-registration with templates. We focus on quality control, using information presence along the z-axis of the imaging to group and apply similarity measures (structural similarity index measure) to triage assessment of individual image series. Additionally, we propose superimposing thresholded binary masks of the series to inspect large quantities of data in parallel. We identify and exclude unrecoverable samples and registration failures. In total, our pipeline processes 10,659 CT series, rejecting 4,322 (41%) in the entire process, 1,450 (14% of the total) during quality control, and outputting 6,337 series. Our pipeline enables effective and efficient region of interest localisation for targeted IAC segmentation.
Sentinel-1 SAR Based Weakly Supervised Learning For Tropical Forest Mapping
Authors: Adugna Mullissa, Sassan Saatchi
Date: 2024-07-31 00:00:00
Tasks: Weakly-supervised Learning

Tropical forests play an important role in regulating the global carbon cycle and are crucial for maintaining the tropical forest biodiversity. Therefore, there is an urgent need to map the extent of tropical forest ecosystems. Recently, deep learning has come out as a powerful tool to map these ecosystems with the caveat of curating high quality reference datasets. Since, manually annotating high quality reference datasets is time consuming and expensive, weakly supervised learning techniques offer the potential to train high quality models without the need for manually annotating large quantities of reference datasets. In this manuscript, we propose two weakly supervised approaches that are based on Sentinel-1 SAR images, sparsely distributed pixel-wise high quality reference labels and densely distributed noisy reference labels. The proposed approaches were tested in a tropical setting in the Brazilian amazon. The results demonstrate that high quality tropical forest maps can be derived from weakly supervised learning without the need for manually annotated labels.
AI-Enabled sensor fusion of time of flight imaging and mmwave for concealed metal detection
Authors: Chaitanya Kaul, Kevin J. Mitchell, Khaled Kassem, Athanasios Tragakis, Valentin Kapitany, Ilya Starshynov, Federica Villa, Roderick Murray-Smith, Daniele Faccio
Date: 2024-08-01 00:00:00
Tasks: Sensor Fusion

In the field of detection and ranging, multiple complementary sensing modalities may be used to enrich the information obtained from a dynamic scene. One application of this sensor fusion is in public security and surveillance, whose efficacy and privacy protection measures must be continually evaluated. We present a novel deployment of sensor fusion for the discrete detection of concealed metal objects on persons whilst preserving their privacy. This is achieved by coupling off-the-shelf mmWave radar and depth camera technology with a novel neural network architecture that processes the radar signals using convolutional Long Short-term Memory (LSTM) blocks and the depth signal, using convolutional operations. The combined latent features are then magnified using a deep feature magnification to learn cross-modality dependencies in the data. We further propose a decoder, based on the feature extraction and embedding block, to learn an efficient upsampling of the latent space to learn the location of the concealed object in the spatial domain through radar feature guidance. We demonstrate the detection of presence and inference of 3D location of concealed metal objects with an accuracy of up to 95%, using a technique that is robust to multiple persons. This work provides a demonstration of the potential for cost effective and portable sensor fusion, with strong opportunities for further development.
Goal-Oriented Semantic Communication for Wireless Image Transmission via Stable Diffusion
Authors: Nan Li, Yansha Deng
Date: 2024-08-01 00:00:00
Tasks: Image Reconstruction, Semantic Communication

Efficient image transmission is essential for seamless communication and collaboration within the visually-driven digital landscape. To achieve low latency and high-quality image reconstruction over a bandwidth-constrained noisy wireless channel, we propose a stable diffusion (SD)-based goal-oriented semantic communication (GSC) framework. In this framework, we design a semantic autoencoder that effectively extracts semantic information (SI) from images to reduce the transmission data size while ensuring high-quality reconstruction. Recognizing the impact of wireless channel noise on SI transmission, we propose an SD-based denoiser for GSC (SD-GSC) conditional on an instantaneous channel gain to remove the channel noise from the received noisy SI under known channel. For scenarios with unknown channel, we further propose a parallel SD denoiser for GSC (PSD-GSC) to jointly learn the distribution of channel gains and denoise the received SI. It is shown that, with the known channel, our SD-GSC outperforms state-of-the-art ADJSCC and Latent-Diff DNSC, improving Peak Signal-to-Noise Ratio (PSNR) by 32% and 21%, and reducing Fr\'echet Inception Distance (FID) by 40% and 35%, respectively. With the unknown channel, our PSD-GSC improves PSNR by 8% and reduces FID by 17% compared to MMSE equalizer-enhanced SD-GSC.
LightViz: Autonomous Light-field Surveying and Mapping for Distributed Light Pollution Monitoring
Authors: Sheng-En Huang, Kazi Farha Farzana Suhi, Md Jahidul Islam
Date: 2024-07-31 00:00:00

Existing technologies for distributed light-field mapping and light pollution monitoring (LPM) rely on either remote satellite imagery or manual light surveying with single-point sensors such as SQMs (sky quality meters). These modalities offer low-resolution data that are not informative for dense light-field mapping, pollutant factor identification, or sustainable policy implementation. In this work, we propose LightViz -- an interactive software interface to survey, simulate, and visualize light pollution maps in real-time. As opposed to manual error-prone methods, LightViz (i) automates the light-field data collection and mapping processes; (ii) provides a platform to simulate various light sources and intensity attenuation models; and (iii) facilitates effective policy identification for conservation. To validate the end-to-end computational pipeline, we design a distributed light-field sensor suit, collect data on Florida coasts, and visualize the distributed light-field maps. In particular, we perform a case study at St. Johns County in Florida, which has a two-decade conservation program for lighting ordinances. The experimental results demonstrate that LightViz can offer high-resolution light-field mapping and provide interactive features to simulate and formulate community policies for light pollution mitigation. We also propose a mathematical formulation for light footprint evaluation, which we integrated into LightViz for targeted LPM in vulnerable communities. A test-case of the LightViz software release is available at: https://github.com/uf-robopi/LightViz.
Exploiting Change Blindness for Video Coding: Perspectives from a Less Promising User Study
Authors: Mitra Amiri, Steven Le Moan, Christian Herglotz
Date: 2024-07-31 00:00:00
Tasks: Computational Efficiency, Quantization, Saliency Prediction, Video Compression

What the human visual system can perceive is strongly limited by the capacity of our working memory and attention. Such limitations result in the human observer's inability to perceive large-scale changes in a stimulus, a phenomenon known as change blindness. In this paper, we started with the premise that this phenomenon can be exploited in video coding, especially HDR-video compression where the bitrate is high. We designed an HDR-video encoding approach that relies on spatially and temporally varying quantization parameters within the framework of HEVC video encoding. In the absence of a reliable change blindness prediction model, to extract compression candidate regions (CCR) we used an existing saliency prediction algorithm. We explored different configurations and carried out a subjective study to test our hypothesis. While our methodology did not lead to significantly superior performance in terms of the ratio between perceived quality and bitrate, we were able to determine potential flaws in our methodology, such as the employed saliency model for CCR prediction (chosen for computational efficiency, but eventually not sufficiently accurate), as well as a very strong subjective bias due to observers priming themselves early on in the experiment about the type of artifacts they should look for, thus creating a scenario with little ecological validity.