pwc-archive/papers-with-abstracts huggingface dataset:
End-to-End Lip Synchronisation Based on Pattern Classification
Authors: You Jin Kim, Hee Soo Heo, Soo-Whan Chung, Bong-Jin Lee
Date: 2020-05-18 00:00:00
Tasks: Classification, General Classification

The goal of this work is to synchronise audio and video of a talking face using deep neural network models. Existing works have trained networks on proxy tasks such as cross-modal similarity learning, and then computed similarities between audio and video frames using a sliding window approach. While these methods demonstrate satisfactory performance, the networks are not trained directly on the task. To this end, we propose an end-to-end trained network that can directly predict the offset between an audio stream and the corresponding video stream. The similarity matrix between the two modalities is first computed from the features, then the inference of the offset can be considered to be a pattern recognition problem where the matrix is considered equivalent to an image. The feature extractor and the classifier are trained jointly. We demonstrate that the proposed approach outperforms the previous work by a large margin on LRS2 and LRS3 datasets.
DDD20 End-to-End Event Camera Driving Dataset: Fusing Frames and Events with Deep Learning for Improved Steering Prediction
Authors: Yuhuang Hu, Jonathan Binas, Daniel Neil, Shih-Chii Liu, Tobi Delbruck
Date: 2020-05-18 00:00:00

Neuromorphic event cameras are useful for dynamic vision problems under difficult lighting conditions. To enable studies of using event cameras in automobile driving applications, this paper reports a new end-to-end driving dataset called DDD20. The dataset was captured with a DAVIS camera that concurrently streams both dynamic vision sensor (DVS) brightness change events and active pixel sensor (APS) intensity frames. DDD20 is the longest event camera end-to-end driving dataset to date with 51h of DAVIS event+frame camera and vehicle human control data collected from 4000km of highway and urban driving under a variety of lighting conditions. Using DDD20, we report the first study of fusing brightness change events and intensity frame data using a deep learning approach to predict the instantaneous human steering wheel angle. Over all day and night conditions, the explained variance for human steering prediction from a Resnet-32 is significantly better from the fused DVS+APS frames (0.88) than using either DVS (0.67) or APS (0.77) data alone.
Brain-inspired Distributed Cognitive Architecture
Authors: Leendert A Remmelzwaal, Amit K Mishra, George F R Ellis
Date: 2020-05-18 00:00:00

In this paper we present a brain-inspired cognitive architecture that incorporates sensory processing, classification, contextual prediction, and emotional tagging. The cognitive architecture is implemented as three modular web-servers, meaning that it can be deployed centrally or across a network for servers. The experiments reveal two distinct operations of behaviour, namely high- and low-salience modes of operations, which closely model attention in the brain. In addition to modelling the cortex, we have demonstrated that a bio-inspired architecture introduced processing efficiencies. The software has been published as an open source platform, and can be easily extended by future research teams. This research lays the foundations for bio-realistic attention direction and sensory selection, and we believe that it is a key step towards achieving a bio-realistic artificial intelligent system.
Design Choices for X-vector Based Speaker Anonymization
Authors: Brij Mohan Lal Srivastava, Natalia Tomashenko, Xin Wang, Emmanuel Vincent, Junichi Yamagishi, Mohamed Maouche, Aurélien Bellet, Marc Tommasi
Date: 2020-05-18 00:00:00
Tasks: Speaker anonymization, Speaker Verification

The recently proposed x-vector based anonymization scheme converts any input voice into that of a random pseudo-speaker. In this paper, we present a flexible pseudo-speaker selection technique as a baseline for the first VoicePrivacy Challenge. We explore several design choices for the distance metric between speakers, the region of x-vector space where the pseudo-speaker is picked, and gender selection. To assess the strength of anonymization achieved, we consider attackers using an x-vector based speaker verification system who may use original or anonymized speech for enrollment, depending on their knowledge of the anonymization scheme. The Equal Error Rate (EER) achieved by the attackers and the decoding Word Error Rate (WER) over anonymized data are reported as the measures of privacy and utility. Experiments are performed using datasets derived from LibriSpeech to find the optimal combination of design choices in terms of privacy and utility.
Sequential Recommender via Time-aware Attentive Memory Network
Authors: Wendi Ji, Keqiang Wang, Xiaoling Wang, TingWei Chen, Alexandra Cristea
Date: 2020-05-18 00:00:00
Tasks: Recommendation Systems, Retrieval

Recommendation systems aim to assist users to discover most preferred contents from an ever-growing corpus of items. Although recommenders have been greatly improved by deep learning, they still faces several challenges: (1) Behaviors are much more complex than words in sentences, so traditional attentive and recurrent models may fail in capturing the temporal dynamics of user preferences. (2) The preferences of users are multiple and evolving, so it is difficult to integrate long-term memory and short-term intent. In this paper, we propose a temporal gating methodology to improve attention mechanism and recurrent units, so that temporal information can be considered in both information filtering and state transition. Additionally, we propose a Multi-hop Time-aware Attentive Memory network (MTAM) to integrate long-term and short-term preferences. We use the proposed time-aware GRU network to learn the short-term intent and maintain prior records in user memory. We treat the short-term intent as a query and design a multi-hop memory reading operation via the proposed time-aware attention to generate user representation based on the current intent and long-term memory. Our approach is scalable for candidate retrieval tasks and can be viewed as a non-linear generalization of latent factorization for dot-product based Top-K recommendation. Finally, we conduct extensive experiments on six benchmark datasets and the experimental results demonstrate the effectiveness of our MTAM and temporal gating methodology.
Efficient Wait-k Models for Simultaneous Machine Translation
Authors: Maha Elbayad, Laurent Besacier, Jakob Verbeek
Date: 2020-05-18 00:00:00
Tasks: Machine Translation, Translation

Simultaneous machine translation consists in starting output generation before the entire input sequence is available. Wait-k decoders offer a simple but efficient approach for this problem. They first read k source tokens, after which they alternate between producing a target token and reading another source token. We investigate the behavior of wait-k decoding in low resource settings for spoken corpora using IWSLT datasets. We improve training of these models using unidirectional encoders, and training across multiple values of k. Experiments with Transformer and 2D-convolutional architectures show that our wait-k models generalize well across a wide range of latency levels. We also show that the 2D-convolution architecture is competitive with Transformers for simultaneous translation of spoken language.
Product Insights: Analyzing Product Intents in Web Search
Authors: Nikitha Rao, Chetan Bansal, Subhabrata Mukherjee, Chandra Maddila
Date: 2020-05-18 00:00:00

Web search engines are frequently used to access information about products. This has increased in recent times with the rising popularity of e-commerce. However, there is limited understanding of what users search for and their intents when it comes to product search on the web. In this work, we study search logs from Bing web search engine to characterize user intents and study user behavior for product search. We propose a taxonomy of product intents by analyzing product search queries. This is a challenging task given that only 15%-17% of web search queries are about products. We train machine learning classifiers with query log features to classify queries based on intent with an overall F1-score of 78%. We further analyze various characteristics of product search queries in terms of search metrics like dwell time, success, popularity and session-specific information.
Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
Authors: Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Yen-Hao Chen, Shang-Wen Li, Hung-Yi Lee
Date: 2020-05-18 00:00:00
Tasks: Self-Supervised Learning, Speaker Identification

For self-supervised speech processing, it is crucial to use pretrained models as speech representation extractors. In recent works, increasing the size of the model has been utilized in acoustic model training in order to achieve better performance. In this paper, we propose Audio ALBERT, a lite version of the self-supervised speech representation model. We use the representations with two downstream tasks, speaker identification, and phoneme classification. We show that Audio ALBERT is capable of achieving competitive performance with those huge models in the downstream tasks while utilizing 91\% fewer parameters. Moreover, we use some simple probing models to measure how much the information of the speaker and phoneme is encoded in latent representations. In probing experiments, we find that the latent representations encode richer information of both phoneme and speaker than that of the last layer.
Audio-visual Multi-channel Recognition of Overlapped Speech
Authors: Jianwei Yu, Bo Wu, Rongzhi Gu, Shi-Xiong Zhang, LianWu Chen, Yong Xu. Meng Yu, Dan Su, Dong Yu, Xunying Liu, Helen Meng
Date: 2020-05-18 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Lipreading, Sentence, speech-recognition, Speech Recognition, Speech Separation

Automatic speech recognition (ASR) of overlapped speech remains a highly challenging task to date. To this end, multi-channel microphone array data are widely used in state-of-the-art ASR systems. Motivated by the invariance of visual modality to acoustic signal corruption, this paper presents an audio-visual multi-channel overlapped speech recognition system featuring tightly integrated separation front-end and recognition back-end. A series of audio-visual multi-channel speech separation front-end components based on \textit{TF masking}, \textit{filter\&sum} and \textit{mask-based MVDR} beamforming approaches were developed. To reduce the error cost mismatch between the separation and recognition components, they were jointly fine-tuned using the connectionist temporal classification (CTC) loss function, or a multi-task criterion interpolation with scale-invariant signal to noise ratio (Si-SNR) error cost. Experiments suggest that the proposed multi-channel AVSR system outperforms the baseline audio-only ASR system by up to 6.81\% (26.83\% relative) and 22.22\% (56.87\% relative) absolute word error rate (WER) reduction on overlapped speech constructed using either simulation or replaying of the lipreading sentence 2 (LRS2) dataset respectively.
Quaternion Neural Networks for Multi-channel Distant Speech Recognition
Authors: Xinchi Qiu, Titouan Parcollet, Mirco Ravanelli, Nicholas Lane, Mohamed Morchid
Date: 2020-05-18 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Distant Speech Recognition, speech-recognition, Speech Recognition

Despite the significant progress in automatic speech recognition (ASR), distant ASR remains challenging due to noise and reverberation. A common approach to mitigate this issue consists of equipping the recording devices with multiple microphones that capture the acoustic scene from different perspectives. These multi-channel audio recordings contain specific internal relations between each signal. In this paper, we propose to capture these inter- and intra- structural dependencies with quaternion neural networks, which can jointly process multiple signals as whole quaternion entities. The quaternion algebra replaces the standard dot product with the Hamilton one, thus offering a simple and elegant way to model dependencies between elements. The quaternion layers are then coupled with a recurrent neural network, which can learn long-term dependencies in the time domain. We show that a quaternion long-short term memory neural network (QLSTM), trained on the concatenated multi-channel speech signals, outperforms equivalent real-valued LSTM on two different tasks of multi-channel distant speech recognition.
Learning to Model and Calibrate Optics via a Differentiable Wave Optics Simulator
Authors: Josue Page, Paolo Favaro
Date: 2020-05-18 00:00:00

We present a novel learning-based method to build a differentiable computational model of a real fluorescence microscope. Our model can be used to calibrate a real optical setup directly from data samples and to engineer point spread functions by specifying the desired input-output data. This approach is poised to drastically improve the design of microscopes, because the parameters of current models of optical setups cannot be easily fit to real data. Inspired by the recent progress in deep learning, our solution is to build a differentiable wave optics simulator as a composition of trainable modules, each computing light wave-front (WF) propagation due to a specific optical element. We call our differentiable modules WaveBlocks and show reconstruction results in the case of lenses, wave propagation in air, camera sensors and diffractive elements (e.g., phase-masks).
Omni-supervised Facial Expression Recognition via Distilled Data
Authors: Ping Liu, Yunchao Wei, Zibo Meng, Weihong Deng, Joey Tianyi Zhou, Yi Yang
Date: 2020-05-18 00:00:00
Tasks: Dataset Distillation, Facial Expression Recognition, Facial Expression Recognition (FER)

Facial expression plays an important role in understanding human emotions. Most recently, deep learning based methods have shown promising for facial expression recognition. However, the performance of the current state-of-the-art facial expression recognition (FER) approaches is directly related to the labeled data for training. To solve this issue, prior works employ the pretrain-and-finetune strategy, i.e., utilize a large amount of unlabeled data to pretrain the network and then finetune it by the labeled data. As the labeled data is in a small amount, the final network performance is still restricted. From a different perspective, we propose to perform omni-supervised learning to directly exploit reliable samples in a large amount of unlabeled data for network training. Particularly, a new dataset is firstly constructed using a primitive model trained on a small number of labeled samples to select samples with high confidence scores from a face dataset, i.e., MS-Celeb-1M, based on feature-wise similarity. We experimentally verify that the new dataset created in such an omni-supervised manner can significantly improve the generalization ability of the learned FER model. However, as the number of training samples grows, computational cost and training time increase dramatically. To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images, significantly improving the training efficiency. We have conducted extensive experiments on widely used benchmarks, where consistent performance gains can be achieved under various settings using the proposed framework. More importantly, the distilled dataset has shown its capabilities of boosting the performance of FER with negligible additional computational costs.
Necessary and sufficient conditions for causal feature selection in time series with latent common causes
Authors: Atalanti A. Mastakouri, Bernhard Schölkopf, Dominik Janzing
Date: 2020-05-18 00:00:00
Tasks: feature selection, Time Series, Time Series Analysis

We study the identification of direct and indirect causes on time series and provide conditions in the presence of latent variables, which we prove to be necessary and sufficient under some graph constraints. Our theoretical results and estimation algorithms require two conditional independence tests for each observed candidate time series to determine whether or not it is a cause of an observed target time series. We provide experimental results in simulations, as well as real data. Our results show that our method leads to very low false positives and relatively low false negative rates, outperforming the widely used Granger causality.
Meta-learning with Stochastic Linear Bandits
Authors: Leonardo Cella, Alessandro Lazaric, Massimiliano Pontil
Date: 2020-05-18 00:00:00
Proceeding: ICML 2020 1
Tasks: Meta-Learning

We investigate meta-learning procedures in the setting of stochastic linear bandits tasks. The goal is to select a learning algorithm which works well on average over a class of bandits tasks, that are sampled from a task-distribution. Inspired by recent work on learning-to-learn linear regression, we consider a class of bandit algorithms that implement a regularized version of the well-known OFUL algorithm, where the regularization is a square euclidean distance to a bias vector. We first study the benefit of the biased OFUL algorithm in terms of regret minimization. We then propose two strategies to estimate the bias within the learning-to-learn setting. We show both theoretically and experimentally, that when the number of tasks grows and the variance of the task-distribution is small, our strategies have a significant advantage over learning the tasks in isolation.
Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
Authors: Jen-Yu Liu, Yu-Hua Chen, Yin-Cheng Yeh, Yi-Hsuan Yang
Date: 2020-05-18 00:00:00
Tasks: Audio Generation, Generative Adversarial Network

In a recent paper, we have presented a generative adversarial network (GAN)-based model for unconditional generation of the mel-spectrograms of singing voices. As the generator of the model is designed to take a variable-length sequence of noise vectors as input, it can generate mel-spectrograms of variable length. However, our previous listening test shows that the quality of the generated audio leaves room for improvement. The present paper extends and expands that previous work in the following aspects. First, we employ a hierarchical architecture in the generator to induce some structure in the temporal dimension. Second, we introduce a cycle regularization mechanism to the generator to avoid mode collapse. Third, we evaluate the performance of the new model not only for generating singing voices, but also for generating speech voices. Evaluation result shows that new model outperforms the prior one both objectively and subjectively. We also employ the model to unconditionally generate sequences of piano and violin music and find the result promising. Audio examples, as well as the code for implementing our model, will be publicly available online upon paper publication.
Robust Training of Vector Quantized Bottleneck Models
Authors: Adrian Łańcucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans J. G. A. Dolfing, Sameer Khurana, Tanel Alumäe, Antoine Laurent
Date: 2020-05-18 00:00:00
Tasks: Clustering, Disentanglement, Representation Learning, Voice Conversion

In this paper we demonstrate methods for reliable and efficient training of discrete representation using Vector-Quantized Variational Auto-Encoder models (VQ-VAEs). Discrete latent variable models have been shown to learn nontrivial representations of speech, applicable to unsupervised voice conversion and reaching state-of-the-art performance on unit discovery tasks. For unsupervised representation learning, they became viable alternatives to continuous latent variable models such as the Variational Auto-Encoder (VAE). However, training deep discrete variable models is challenging, due to the inherent non-differentiability of the discretization operation. In this paper we focus on VQ-VAE, a state-of-the-art discrete bottleneck model shown to perform on par with its continuous counterparts. It quantizes encoder outputs with on-line $k$-means clustering. We show that the codebook learning can suffer from poor initialization and non-stationarity of clustered encoder outputs. We demonstrate that these can be successfully overcome by increasing the learning rate for the codebook and periodic date-dependent codeword re-initialization. As a result, we achieve more robust training across different tasks, and significantly increase the usage of latent codewords even for large codebooks. This has practical benefit, for instance, in unsupervised representation learning, where large codebooks may lead to disentanglement of latent representations.
Yseop at SemEval-2020 Task 5: Cascaded BERT Language Model for Counterfactual Statement Analysis
Authors: Hanna Abi Akl, Dominique Mariko, Estelle Labidurie
Date: 2020-05-18 00:00:00
Proceeding: SEMEVAL 2020
Tasks: counterfactual, General Classification, Language Modeling, Language Modelling, Task 2

In this paper, we explore strategies to detect and evaluate counterfactual sentences. We describe our system for SemEval-2020 Task 5: Modeling Causal Reasoning in Language: Detecting Counterfactuals. We use a BERT base model for the classification task and build a hybrid BERT Multi-Layer Perceptron system to handle the sequence identification task. Our experiments show that while introducing syntactic and semantic features does little in improving the system in the classification task, using these types of features as cascaded linear inputs to fine-tune the sequence-delimiting ability of the model ensures it outperforms other similar-purpose complex systems like BiLSTM-CRF in the second task. Our system achieves an F1 score of 85.00% in Task 1 and 83.90% in Task 2.
Automatic Knowledge Acquisition for Object-Oriented Expert Systems
Authors: Joël Colloc, Danielle Boulanger
Date: 2020-05-18 00:00:00
Tasks: Object

We describe an Object Oriented Model for building Expert Systems. This model and the detection of similarities allow to implement reasoning modes as induction, deduction and simulation. We specially focus on similarity and its use in induction. We propose original algorithms which deal with total and partial structural similitude of objects to facilitate knowledge acquisition.