pwc-archive/papers-with-abstracts huggingface dataset:
Classifying emotions and engagement in online learning based on a single facial expression recognition neural network
Authors: Savchenko A.V., Savchenko L.V., Makarov I
Date: 2022-07-04 00:00:00
Proceeding: IEEE Transactions on Affective Computing 2022 7
Tasks: Efficient Neural Network, Emotion Recognition, Face Detection, Face Identification, Facial Expression Recognition, Facial Expression Recognition (FER)

In this paper, behaviour of students in the e-learning environment is analyzed. The novel pipeline is proposed based on video facial processing. At first, face detection, tracking and clustering techniques are applied to extract the sequences of faces of each student. Next, a single efficient neural network is used to extract emotional features in each frame. This network is pre-trained on face identification and fine-tuned for facial expression recognition on static images from AffectNet using a specially developed robust optimization technique. It is shown that the resulting facial features can be used for fast simultaneous prediction of students’ engagement levels (from disengaged to highly engaged), individual emotions (happy, sad, etc.,) and group-level affect (positive, neutral or negative). This model can be used for real-time video processing even on a mobile device of each student without the need for sending their facial video to the remote server or teacher’s PC. In addition, the possibility to prepare a summary of a lesson is demonstrated by saving short clips of different emotions and engagement of all students. The experimental study on the datasets from EmotiW (Emotion Recognition in the Wild) challenges showed that the proposed network significantly outperforms existing single models.
temporal driver action Localization using action classifications method
Authors: Munirah Alyahya, Shahad Alghannam, Taghreed Alhussan
Date: 2022-06-11 00:00:00
Proceeding: CVPR 2022 6
Tasks: Action Classification, Action Localization, Temporal Action Localization

Driver distraction recognition is an essential computer vision task that can play a key role in increasing traffic safety and reducing traffic accidents. In this paper, we propose a temporal driver action localization (TDAL) framework for classifying driver distraction actions, as well as identifying the start and end time of a given driver action. The TDAL framework consists of three stages: preprocessing, which takes untrimmed video as input and generates multiple clips; action classification, which classifies the clips; and finally, the classifier output is sent to the temporal action localization to generate the start and end times of the distracted actions. The proposed framework achieves an F1 score of 27.06% on Track 3 A2 dataset of NVIDIA AI City 2022 Challenge. The findings show that the TDAL framework contributes to fine-grained driver distraction recognition and paves the way for the development of smart and safe transportation.
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Authors: Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen & Klaus H. Maier-Hein
Date: 2020-12-07 00:00:00
Proceeding: Nature 2020 12
Tasks: Image Segmentation, scientific discovery, Segmentation, Semantic Segmentation

Biomedical imaging is a driver of scientific discovery and a core component of medical care and is being stimulated by the field of deep learning. While semantic segmentation algorithms enable image analysis and quantification in many applications, the design of respective specialized solutions is non-trivial and highly dependent on dataset properties and hardware conditions. We developed nnU-Net, a deep learning-based segmentation method that automatically configures itself, including preprocessing, network architecture, training and post-processing for any new task. The key design choices in this process are modeled as a set of fixed parameters, interdependent rules and empirical decisions. Without manual intervention, nnU-Net surpasses most existing approaches, including highly specialized solutions on 23 public datasets used in international biomedical segmentation competitions. We make nnU-Net publicly available as an out-of-the-box tool, rendering state-of-the-art segmentation accessible to a broad audience by requiring neither expert knowledge nor computing resources beyond standard network training.
Conditioned and Composed Image Retrieval Combining and Partially Fine-Tuning CLIP-Based Features
Authors: Alberto Baldrati, Marco Bertini, Tiberio Uricchio, Alberto del Bimbo
Date: 2022-06-19 00:00:00
Proceeding: CVPRW 2022 6
Tasks: Composed Image Retrieval (CoIR), Content-Based Image Retrieval, Contrastive Learning, Image Retrieval, Retrieval

In this paper, we present an approach for conditioned and composed image retrieval based on CLIP features. In this extension of content-based image retrieval (CBIR), an image is combined with a text that provides information regarding user intentions and is relevant for application domains like e-commerce. The proposed method is based on an initial training stage where a simple combination of visual and textual features is used, to fine-tune the CLIP text encoder. Then in a second training stage, we learn a more complex combiner network that merges visual and textual features. Contrastive learning is used in both stages. The proposed approach obtains state-of-the-art performance for conditioned CBIR on the FashionIQ dataset and for composed CBIR on the more recent CIRR dataset.
Differentially Private Linear Bandits with Partial Distributed Feedback
Authors: Fengjiao Li, Xingyu Zhou, Bo Ji
Date: 2022-07-12 00:00:00

In this paper, we study the problem of global reward maximization with only partial distributed feedback. This problem is motivated by several real-world applications (e.g., cellular network configuration, dynamic pricing, and policy selection) where an action taken by a central entity influences a large population that contributes to the global reward. However, collecting such reward feedback from the entire population not only incurs a prohibitively high cost but often leads to privacy concerns. To tackle this problem, we consider differentially private distributed linear bandits, where only a subset of users from the population are selected (called clients) to participate in the learning process and the central server learns the global model from such partial feedback by iteratively aggregating these clients' local feedback in a differentially private fashion. We then propose a unified algorithmic learning framework, called differentially private distributed phased elimination (DP-DPE), which can be naturally integrated with popular differential privacy (DP) models (including central DP, local DP, and shuffle DP). Furthermore, we prove that DP-DPE achieves both sublinear regret and sublinear communication cost. Interestingly, DP-DPE also achieves privacy protection ``for free'' in the sense that the additional cost due to privacy guarantees is a lower-order additive term. In addition, as a by-product of our techniques, the same results of ``free" privacy can also be achieved for the standard differentially private linear bandits. Finally, we conduct simulations to corroborate our theoretical results and demonstrate the effectiveness of DP-DPE.
RelaxLoss: Defending Membership Inference Attacks without Losing Utility
Authors: Dingfan Chen, Ning Yu, Mario Fritz
Date: 2022-07-12 00:00:00
Proceeding: ICLR 2022 4

As a long-term threat to the privacy of training data, membership inference attacks (MIAs) emerge ubiquitously in machine learning models. Existing works evidence strong connection between the distinguishability of the training and testing loss distributions and the model's vulnerability to MIAs. Motivated by existing results, we propose a novel training framework based on a relaxed loss with a more achievable learning target, which leads to narrowed generalization gap and reduced privacy leakage. RelaxLoss is applicable to any classification model with added benefits of easy implementation and negligible overhead. Through extensive evaluations on five datasets with diverse modalities (images, medical data, transaction records), our approach consistently outperforms state-of-the-art defense mechanisms in terms of resilience against MIAs as well as model utility. Our defense is the first that can withstand a wide range of attacks while preserving (or even improving) the target model's utility. Source code is available at https://github.com/DingfanChen/RelaxLoss
A Conceptual Framework for Using Machine Learning to Support Child Welfare Decisions
Authors: Ka Ho Brian Chor, Kit T. Rodolfa, Rayid Ghani
Date: 2022-07-12 00:00:00
Tasks: BIG-bench Machine Learning

Human services systems make key decisions that impact individuals in the society. The U.S. child welfare system makes such decisions, from screening-in hotline reports of suspected abuse or neglect for child protective investigations, placing children in foster care, to returning children to permanent home settings. These complex and impactful decisions on children's lives rely on the judgment of child welfare decisionmakers. Child welfare agencies have been exploring ways to support these decisions with empirical, data-informed methods that include machine learning (ML). This paper describes a conceptual framework for ML to support child welfare decisions. The ML framework guides how child welfare agencies might conceptualize a target problem that ML can solve; vet available administrative data for building ML; formulate and develop ML specifications that mirror relevant populations and interventions the agencies are undertaking; deploy, evaluate, and monitor ML as child welfare context, policy, and practice change over time. Ethical considerations, stakeholder engagement, and avoidance of common pitfalls underpin the framework's impact and success. From abstract to concrete, we describe one application of this framework to support a child welfare decision. This ML framework, though child welfare-focused, is generalizable to solving other public policy problems.
On How to Not Prove Faulty Controllers Safe in Differential Dynamic Logic
Authors: Yuvaraj Selvaraj, Jonas Krook, Wolfgang Ahrendt, Martin Fabian
Date: 2022-07-12 00:00:00

Cyber-physical systems are often safety-critical and their correctness is crucial, as in the case of automated driving. Using formal mathematical methods is one way to guarantee correctness. Though these methods have shown their usefulness, care must be taken as modeling errors might result in proving a faulty controller safe, which is potentially catastrophic in practice. This paper deals with two such modeling errors in differential dynamic logic. Differential dynamic logic is a formal specification and verification language for hybrid systems, which are mathematical models of cyber-physical systems. The main contribution is to prove conditions that when fulfilled, these two modeling errors cannot cause a faulty controller to be proven safe. The problems are illustrated with a real world example of a safety controller for automated driving, and it is shown that the formulated conditions have the intended effect both for a faulty and a correct controller. It is also shown how the formulated conditions aid in finding a loop invariant candidate to prove properties of hybrid systems with feedback loops. The results are proven using the interactive theorem prover KeYmaera X.
DDPG Learning for Aerial RIS-Assisted MU-MISO Communications
Authors: Aly Sabri Abdalla, Vuk Marojevic
Date: 2022-07-13 00:00:00
Tasks: Q-Learning

This paper defines the problem of optimizing the downlink multi-user multiple input, single output (MU-MISO) sum-rate for ground users served by an aerial reconfigurable intelligent surface (ARIS) that acts as a relay to the terrestrial base station. The deep deterministic policy gradient (DDPG) is proposed to calculate the optimal active beamforming matrix at the base station and the phase shifts of the reflecting elements at the ARIS to maximize the data rate. Simulation results show the superiority of the proposed scheme when compared to deep Q-learning (DQL) and baseline approaches.
Fast Radix-32 Approximate DFTs for 1024-Beam Digital RF Beamforming
Authors: A. Madanayake, R. J. Cintra, N. Akram, V. Ariyarathna, S. Mandal, V. A. Coutinho, F. M. Bayer, D. Coelho, T. S. Rappaport
Date: 2022-07-12 00:00:00

The discrete Fourier transform (DFT) is widely employed for multi-beam digital beamforming. The DFT can be efficiently implemented through the use of fast Fourier transform (FFT) algorithms, thus reducing chip area, power consumption, processing time, and consumption of other hardware resources. This paper proposes three new hybrid DFT 1024-point DFT approximations and their respective fast algorithms. These approximate DFT (ADFT) algorithms have significantly reduced circuit complexity and power consumption compared to traditional FFT approaches while trading off a subtle loss in computational precision which is acceptable for digital beamforming applications in RF antenna implementations. ADFT algorithms have not been introduced for beamforming beyond $N = 32$, but this paper anticipates the need for massively large adaptive arrays for future 5G and 6G systems. Digital CMOS circuit designs for the ADFTs show the resulting improvements in both circuit complexity and power consumption metrics. Simulation results show similar or lower critical path delay with up to 48.5% lower chip area compared to a standard Cooley-Tukey FFT. The time-area and dynamic power metrics are reduced up to 66.0%. The 1024-point ADFT beamformers produce signal-to-noise ratio (SNR) gains between 29.2--30.1 dB, which is a loss of $\le$ 0.9 dB SNR gain compared to exact 1024-point DFT beamformers (worst case) realizable at using an FFT.
Data-driven Control of Agent-based Models: an Equation/Variable-free Machine Learning Approach
Authors: Dimitrios G. Patsatzis, Lucia Russo, Ioannis G. Kevrekidis, Constantinos Siettos
Date: 2022-07-12 00:00:00

We present an Equation/Variable free machine learning (EVFML) framework for the control of the collective dynamics of complex/multiscale systems modelled via microscopic/agent-based simulators. The approach obviates the need for construction of surrogate, reduced-order models.~The proposed implementation consists of three steps: (A) from high-dimensional agent-based simulations, machine learning (in particular, non-linear manifold learning (Diffusion Maps (DMs)) helps identify a set of coarse-grained variables that parametrize the low-dimensional manifold on which the emergent/collective dynamics evolve. The out-of-sample extension and pre-image problems, i.e. the construction of non-linear mappings from the high-dimensional input space to the low-dimensional manifold and back, are solved by coupling DMs with the Nystrom extension and Geometric Harmonics, respectively; (B) having identified the manifold and its coordinates, we exploit the Equation-free approach to perform numerical bifurcation analysis of the emergent dynamics; then (C) based on the previous steps, we design data-driven embedded wash-out controllers that drive the agent-based simulators to their intrinsic, imprecisely known, emergent open-loop unstable steady-states, thus demonstrating that the scheme is robust against numerical approximation errors and modelling uncertainty.~The efficiency of the framework is illustrated by controlling emergent unstable (i) traveling waves of a deterministic agent-based model of traffic dynamics, and (ii) equilibria of a stochastic financial market agent model with mimesis.
Physics-Informed Deep Neural Operator Networks
Authors: Somdatta Goswami, Aniruddha Bora, Yue Yu, George Em Karniadakis
Date: 2022-07-08 00:00:00
Tasks: Uncertainty Quantification

Standard neural networks can approximate general nonlinear operators, represented either explicitly by a combination of mathematical operators, e.g., in an advection-diffusion-reaction partial differential equation, or simply as a black box, e.g., a system-of-systems. The first neural operator was the Deep Operator Network (DeepONet), proposed in 2019 based on rigorous approximation theory. Since then, a few other less general operators have been published, e.g., based on graph neural networks or Fourier transforms. For black box systems, training of neural operators is data-driven only but if the governing equations are known they can be incorporated into the loss function during training to develop physics-informed neural operators. Neural operators can be used as surrogates in design problems, uncertainty quantification, autonomous systems, and almost in any application requiring real-time inference. Moreover, independently pre-trained DeepONets can be used as components of a complex multi-physics system by coupling them together with relatively light training. Here, we present a review of DeepONet, the Fourier neural operator, and the graph neural operator, as well as appropriate extensions with feature expansions, and highlight their usefulness in diverse applications in computational mechanics, including porous media, fluid mechanics, and solid mechanics.
A new hope for network model generalization
Authors: Alexander Dietmüller, Siddhant Ray, Romain Jacob, Laurent Vanbever
Date: 2022-07-12 00:00:00
Tasks: model

Generalizing machine learning (ML) models for network traffic dynamics tends to be considered a lost cause. Hence for every new task, we design new models and train them on model-specific datasets closely mimicking the deployment environments. Yet, an ML architecture called_Transformer_ has enabled previously unimaginable generalization in other domains. Nowadays, one can download a model pre-trained on massive datasets and only fine-tune it for a specific task and context with comparatively little time and data. These fine-tuned models are now state-of-the-art for many benchmarks. We believe this progress could translate to networking and propose a Network Traffic Transformer (NTT), a transformer adapted to learn network dynamics from packet traces. Our initial results are promising: NTT seems able to generalize to new prediction tasks and environments. This study suggests there is still hope for generalization, though it calls for a lot of future research.
Conditional Energy-Based Models for Implicit Policies: The Gap between Theory and Practice
Authors: Duy-Nguyen Ta, Eric Cousineau, Huihua Zhao, Siyuan Feng
Date: 2022-07-12 00:00:00
Tasks: regression

We present our findings in the gap between theory and practice of using conditional energy-based models (EBM) as an implicit representation for behavior-cloned policies. We also clarify several subtle, and potentially confusing, details in previous work in an attempt to help future research in this area. We point out key differences between unconditional and conditional EBMs, and warn that blindly applying training methods for one to the other could lead to undesirable results that do not generalize well. Finally, we emphasize the importance of the Maximum Mutual Information principle as a necessary condition to achieve good generalization in conditional EBMs as implicit models for regression tasks.
MM-ALT: A Multimodal Automatic Lyric Transcription System
Authors: Xiangming Gu, Longshen Ou, Danielle Ong, Ye Wang
Date: 2022-07-13 00:00:00
Tasks: Action Detection, Activity Detection, Automatic Speech Recognition, Automatic Speech Recognition (ASR), Information Retrieval, Music Information Retrieval, Retrieval, speech-recognition, Speech Recognition

Automatic lyric transcription (ALT) is a nascent field of study attracting increasing interest from both the speech and music information retrieval communities, given its significant application potential. However, ALT with audio data alone is a notoriously difficult task due to instrumental accompaniment and musical constraints resulting in degradation of both the phonetic cues and the intelligibility of sung lyrics. To tackle this challenge, we propose the MultiModal Automatic Lyric Transcription system (MM-ALT), together with a new dataset, N20EM, which consists of audio recordings, videos of lip movements, and inertial measurement unit (IMU) data of an earbud worn by the performing singer. We first adapt the wav2vec 2.0 framework from automatic speech recognition (ASR) to the ALT task. We then propose a video-based ALT method and an IMU-based voice activity detection (VAD) method. In addition, we put forward the Residual Cross Attention (RCA) mechanism to fuse data from the three modalities (i.e., audio, video, and IMU). Experiments show the effectiveness of our proposed MM-ALT system, especially in terms of noise robustness. Project page is at https://n20em.github.io.
SATTS: Speaker Attractor Text to Speech, Learning to Speak by Learning to Separate
Authors: Nabarun Goswami, Tatsuya Harada
Date: 2022-07-13 00:00:00
Tasks: Speech Separation, text-to-speech, Text to Speech

The mapping of text to speech (TTS) is non-deterministic, letters may be pronounced differently based on context, or phonemes can vary depending on various physiological and stylistic factors like gender, age, accent, emotions, etc. Neural speaker embeddings, trained to identify or verify speakers are typically used to represent and transfer such characteristics from reference speech to synthesized speech. Speech separation on the other hand is the challenging task of separating individual speakers from an overlapping mixed signal of various speakers. Speaker attractors are high-dimensional embedding vectors that pull the time-frequency bins of each speaker's speech towards themselves while repelling those belonging to other speakers. In this work, we explore the possibility of using these powerful speaker attractors for zero-shot speaker adaptation in multi-speaker TTS synthesis and propose speaker attractor text to speech (SATTS). Through various experiments, we show that SATTS can synthesize natural speech from text from an unseen target speaker's reference signal which might have less than ideal recording conditions, i.e. reverberations or mixed with other speakers.
Cross-Age Speaker Verification: Learning Age-Invariant Speaker Embeddings
Authors: Xiaoyi Qin, Na Li, Chao Weng, Dan Su, Ming Li
Date: 2022-07-13 00:00:00
Tasks: Age Estimation, Speaker Verification

Automatic speaker verification has achieved remarkable progress in recent years. However, there is little research on cross-age speaker verification (CASV) due to insufficient relevant data. In this paper, we mine cross-age test sets based on the VoxCeleb dataset and propose our age-invariant speaker representation(AISR) learning method. Since the VoxCeleb is collected from the YouTube platform, the dataset consists of cross-age data inherently. However, the meta-data does not contain the speaker age label. Therefore, we adopt the face age estimation method to predict the speaker age value from the associated visual data, then label the audio recording with the estimated age. We construct multiple Cross-Age test sets on VoxCeleb (Vox-CA), which deliberately select the positive trials with large age-gap. Also, the effect of nationality and gender is considered in selecting negative pairs to align with Vox-H cases. The baseline system performance drops from 1.939\% EER on the Vox-H test set to 10.419\% on the Vox-CA20 test set, which indicates how difficult the cross-age scenario is. Consequently, we propose an age-decoupling adversarial learning (ADAL) method to alleviate the negative effect of the age gap and reduce intra-class variance. Our method outperforms the baseline system by over 10\% related EER reduction on the Vox-CA20 test set. The source code and trial resources are available on https://github.com/qinxiaoyi/Cross-Age_Speaker_Verification