pwc-archive/papers-with-abstracts huggingface dataset:
On Zero-shot Cross-lingual Transfer of Multilingual Neural Machine Translation
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Cross-Lingual Transfer, Machine Translation, Natural Language Inference, NMT, Sentiment Analysis, Sentiment Classification, Translation, Zero-Shot Cross-Lingual Transfer

Transferring representations from large-scale supervised tasks to downstream tasks have shown outstanding results in Machine Learning in both Computer Vision and natural language processing (NLP). One particular example can be sequence-to-sequence models for Machine Translation (Neural Machine Translation - NMT). It is because, once trained in a multilingual setup, NMT systems can translate between multiple languages and are also capable of performing zero-shot translation between unseen source-target pairs at test time. In this paper, we first investigate if we can extend the zero-shot transfer capability of multilingual NMT systems to cross-lingual NLP tasks (tasks other than MT, e.g. sentiment classification and natural language inference). We demonstrate a simple framework by reusing the encoder from a multilingual NMT system, a multilingual Encoder-Classifier, achieves remarkable zero-shot cross-lingual classification performance, almost out-of-the-box on three downstream benchmark tasks - Amazon Reviews, Stanford sentiment treebank (SST) and Stanford natural language inference (SNLI). In order to understand the underlying factors contributing to this finding, we conducted a series of analyses on the effect of the shared vocabulary, the training data type for NMT models, classifier complexity, encoder representation power, and model generalization on zero-shot performance. Our results provide strong evidence that the representations learned from multilingual NMT systems are widely applicable across languages and tasks, and the high, out-of-the-box classification performance is correlated with the generalization capability of such systems.
Training Neural Speech Recognition Systems with Synthetic Speech Augmentation
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Language Modeling, Language Modelling, speech-recognition, Speech Recognition

Building an accurate automatic speech recognition (ASR) system requires a large dataset that contains many hours of labeled speech samples produced by a diverse set of speakers. The lack of such open free datasets is one of the main issues preventing advancements in ASR research. To address this problem, we propose to augment a natural speech dataset with synthetic speech. We train very large end-to-end neural speech recognition models using the LibriSpeech dataset augmented with synthetic speech. These new models achieve state of the art Word Error Rate (WER) for character-level based models without an external language model.
Natural Language Detectors Emerge in Individual Neurons
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Concept Alignment, Translation

Although deep convolutional networks have achieved improved performance in many natural language tasks, they have been treated as black boxes because they are difficult to interpret. Especially, little is known about how they represent language in their intermediate layers. In an attempt to understand the representations of deep convolutional networks trained on language tasks, we show that individual units are selectively responsive to specific morphemes, words, and phrases, rather than responding to arbitrary and uninterpretable patterns. In order to quantitatively analyze such intriguing phenomenon, we propose a concept alignment method based on how units respond to replicated text. We conduct analyses with different architectures on multiple datasets for classification and translation tasks and provide new insights into how deep models understand natural language.
A Gray Box Interpretable Visual Debugging Approach for Deep Sequence Learning Model
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Decision Making, Deep Learning

Deep Learning algorithms are often used as black box type learning and they are too complex to understand. The widespread usability of Deep Learning algorithms to solve various machine learning problems demands deep and transparent understanding of the internal representation as well as decision making. Moreover, the learning models, trained on sequential data, such as audio and video data, have intricate internal reasoning process due to their complex distribution of features. Thus, a visual simulator might be helpful to trace the internal decision making mechanisms in response to adversarial input data, and it would help to debug and design appropriate deep learning models. However, interpreting the internal reasoning of deep learning model is not well studied in the literature. In this work, we have developed a visual interactive web application, namely d-DeVIS, which helps to visualize the internal reasoning of the learning model which is trained on the audio data. The proposed system allows to perceive the behavior as well as to debug the model by interactively generating adversarial audio data point. The web application of d-DeVIS is available at ddevis.herokuapp.com.
Investigation of using disentangled and interpretable representations with language conditioning for cross-lingual voice conversion
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: One-Shot Learning, Voice Conversion

We study the problem of cross-lingual voice conversion in non-parallel speech corpora and one-shot learning setting. Most prior work require either parallel speech corpora or enough amount of training data from a target speaker. However, we convert an arbitrary sentences of an arbitrary source speaker to target speaker's given only one target speaker training utterance. To achieve this, we formulate the problem as learning disentangled speaker-specific and context-specific representations and follow the idea of [1] which uses Factorized Hierarchical Variational Autoencoder (FHVAE). After training FHVAE on multi-speaker training data, given arbitrary source and target speakers' utterance, we estimate those latent representations and then reconstruct the desired utterance of converted voice to that of target speaker. We use multi-language speech corpus to learn a universal model that works for all of the languages. We investigate the use of a one-hot language embedding to condition the model on the language of the utterance being queried and show the effectiveness of the approach. We conduct voice conversion experiments with varying size of training utterances and it was able to achieve reasonable performance with even just one training utterance. We also investigate the effect of using or not using the language conditioning. Furthermore, we visualize the embeddings of the different languages and sexes. Finally, in the subjective tests, for one language and cross-lingual voice conversion, our approach achieved moderately better or comparable results compared to the baseline in speech quality and similarity.
Learning pronunciation from a foreign language in speech synthesis networks
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Speech Synthesis

Although there are more than 65,000 languages in the world, the pronunciations of many phonemes sound similar across the languages. When people learn a foreign language, their pronunciation often reflect their native language's characteristics. That motivates us to investigate how the speech synthesis network learns the pronunciation when multi-lingual dataset is given. In this study, we train the speech synthesis network bilingually in English and Korean, and analyze how the network learns the relations of phoneme pronunciation between the languages. Our experimental result shows that the learned phoneme embedding vectors are located closer if their pronunciations are similar across the languages. Based on the result, we also show that it is possible to train networks that synthesize English speaker's Korean speech and vice versa. In another experiment, we train the network with limited amount of English dataset and large Korean dataset, and analyze the required amount of dataset to train a resource-poor language with the help of resource-rich languages.
Targeted Adversarial Examples for Black Box Audio Systems
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

The application of deep recurrent networks to audio transcription has led to impressive gains in automatic speech recognition (ASR) systems. Many have demonstrated that small adversarial perturbations can fool deep neural networks into incorrectly predicting a specified target with high confidence. Current work on fooling ASR systems have focused on white-box attacks, in which the model architecture and parameters are known. In this paper, we adopt a black-box approach to adversarial generation, combining the approaches of both genetic algorithms and gradient estimation to solve the task. We achieve a 89.25% targeted attack similarity after 3000 generations while maintaining 94.6% audio file similarity.
ROBUST SPEECH COMMAND RECOGNITION USING LABEL-DRIVEN TIME-FREQUENCY MASKING
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Classification, Speech Enhancement, speech-recognition, Speech Recognition

Speech enhancement driven robust Automatic Speech Recognition (ASR) systems typically require parallel corpus with noisy and clean speech utterances for training. Moreover, many studies have reported that such front-ends, even though improve speech quality, do not always improve the recognition performance. On the other hand, the multi-condition training of ASR systems have little visualization or interpretability capabilities of how these systems achieve robustness. In this paper, we propose a novel neural architecture with unified enhancement and sequence classification block, that is trained in an end-to-end manner only using noisy speech without having information of clean speech. The enhancement block is a fully convolutional network that is designed to perform Time Frequency (T-F) masking like operation, followed by an LSTM sequence classification block. The T-F masking formulation enables visualization of learned mask and helps us to visualize the T-F points important for classification of a speech command. Experiments performed on Google Speech Command dataset show that our proposed network achieves better results than the baseline model without an enhancement front-end.
Autoencoding Documents for Topic Modeling with L-2 Sparsity Regularization
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Decoder

We propose a novel yet simple neural network architecture for topic modelling. The method is based on training an autoencoder structure where the bottleneck represents the space of the topics distribution and the decoder outputs represent the space of the words distributions over the topics. We exploit an auxiliary decoder to prevent mode collapsing in our model.  A key feature for an effective topic modelling method is having sparse topics and words distributions, where there is a trade-off between the sparsity level of topics and words. This feature is implemented in our model by L-2 regularization and the model hyperparameters take care of the trade-off.  We show in our experiments that our model achieves competitive results compared to the state-of-the-art deep models for topic modelling, despite its simple architecture and training procedure. The “New York Times” and “20 Newsgroups” datasets are used in the experiments.
Multi-Domain Processing via Hybrid Denoising Networks for Speech Enhancement
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Denoising, Speech Enhancement

We present a hybrid framework that leverages the trade-off between temporal and frequency precision in audio representations to improve the performance of speech enhancement task. We first show that conventional approaches using specific representations such as raw-audio and spectrograms are each effective at targeting different types of noise.
By integrating both approaches, our model can learn multi-scale and multi-domain features, effectively removing noise existing on different regions on the time-frequency space in a complementary way. Experimental results show that the proposed hybrid model yields better performance and robustness than using each model individually.
Adversarial Machine Learning And Speech Emotion Recognition: Utilizing Generative Adversarial Networks For Robustness
Authors: Anonymous
Date: 2018-10-24 00:00:00
Tasks: Adversarial Attack, BIG-bench Machine Learning, Emotion Recognition, Generative Adversarial Network, Speech Emotion Recognition

 Although deep learning has enabled unprecedented improvements in the performance of the state-of-the-art speech emotion recognition (SER) systems, recent research on adversarial examples has cast a shadow of doubt on the robustness of SER systems by showing the susceptibility of deep neural networks to adversarial examples that rely only on small and imperceptible perturbations. In this study, we evaluate how adversarial examples can be used to attack SER systems and propose the first black-box adversarial attack on SER systems. We also explore potential defenses including adversarial training and generative adversarial network (GAN) to enhance robustness. Experimental evaluations suggest various interesting aspects of the effective utilization of adversarial examples that can be useful not only for SER robustness but also other speech-based intelligent systems.
Word Sense Induction using Knowledge Embeddings
Authors: Anonymous
Date: 2018-10-23 00:00:00
Tasks: Word Embeddings, Word Sense Induction

Word Embeddings are able to capture lexico-semantic information but remain flawed in their inability to assign unique representations to different senses of a polysemous words. They also fail to include information from well curated semantic lexicons and dictionaries. Previous approaches that integrate polysemy and knowledge bases fall distinctly under two categories - retrofitting vectors to ontologies or learning from sense tagged corpora. While these embeddings are superior in understanding contextual similarity, they are outperformed by single prototype word vectors on several relatedness tasks. 
In this work, we introduce a new approach that can induce polysemy to any pre-trained embedding space by jointly grounding contextualized sense representations with word embeddings to an ontology and a thesaurus. Along with word sense induction, the resulting representations reduces the effect of vocabulary bias that arises in natural language corpora and in turn embedding spaces. By grounding them to knowledge bases  they are able to learn multi-word representations and are also interpretable. We evaluate our vectors across 12 datasets on several similarity and relatedness tasks along with two extrinsic tasks and find that our approach consistently outperforms current state of the art.
On the Confidence of Neural Network Predictions for some NLP Tasks
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Classification, image-classification, Image Classification, Out-of-Distribution Detection, Sentence, Sentence Classification

Neural networks are known to produce unexpected results on inputs that are far from the training distribution. One approach to tackle this problem is to detect the samples on which the trained network can not answer reliably. ODIN is a recently proposed method for out-of-distribution detection that does not modify the trained network and achieves good performance for various image classification tasks. In this paper we adapt ODIN for sentence classification and word tagging tasks. We show that the scores produced by ODIN can be used as a confidence measure for the predictions on both in-distribution and out-of-distribution datasets.
Adversarial Gain
Authors: Anonymous
Date: 2018-10-22 00:00:00

Adversarial examples can be defined as inputs to a model which induce a mistake -- where the model output is different than that of an oracle, perhaps in surprising or malicious ways. Original models of adversarial attacks are primarily studied in the context of classification and computer vision tasks. While several attacks have been proposed in natural language processing (NLP) settings, they often vary in defining the parameters of an attack and what a successful attack would look like. The goal of this work is to propose a unifying model of adversarial examples suitable for NLP tasks in both generative and classification settings. We define the notion of adversarial gain: based in control theory, it is a measure of the change in the output of a system relative to the perturbation of the input (caused by the so-called adversary) presented to the learner. This definition, as we show, can be used under different feature spaces and distance conditions to determine attack or defense effectiveness across different intuitive manifolds. This notion of adversarial gain not only provides a useful way for evaluating adversaries and defenses, but can act as a building block for future work in robustness under adversaries due to its rooted nature in stability and manifold theory.
A Fully Time-domain Neural Model for Subband-based Speech Synthesizer
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: text-to-speech, Text to Speech

—This  paper  introduces  a  deep  neural  network  model for  subband-based  speech  synthesizer.  The  model  benefits  from the  short  bandwidth  of  the  subband  signals  to  reduce  the complexity  of  the time-domain speech generator. We employed the multi-level  wavelet  analysis/synthesis  to  decompose/reconstruct the  signal  to  subbands  in  time  domain.  Inspired  from  the WaveNet,  a  convolutional  neural  network  (CNN)  model  predicts subband  speech  signals  fully  in  time  domain.  Due  to  the  short bandwidth  of  the  subbands,  a  simple  network  architecture  is enough  to  train  the  simple  patterns  of  the  subbands  accurately.  In the  ground  truth  experiments  with  teacher  forcing,  the  subband synthesizer  outperforms  the  fullband  model  significantly.  In addition,  by  conditioning  the  model  on  the  phoneme  sequence using  a  pronunciation  dictionary,  we  have  achieved  the  first  fully time-domain  neural  text-to-speech  (TTS)  system.  The  generated speech  of  the  subband  TTS  shows  comparable  quality  as  the fullband  one  with  a  slighter  network  architecture  for  each subband.
Improving label efficiency through multi-task learning on auditory data
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Data Augmentation, Multi-Task Learning, Self-Supervised Learning

Collecting high-quality, large scale datasets typically requires significant resources. The aim of the present work is to improve the label efficiency of large neural networks operating on audio data through multitask learning with self-supervised tasks on unlabeled data. To this end, we trained an end-to-end audio feature extractor based on WaveNet that feeds into simple, yet versatile task-specific neural networks. We describe three self-supervised learning tasks that can operate on any large, unlabeled audio corpus. We demonstrate that, in a scenario with limited labeled training data, one can significantly improve the performance of a supervised classification task by simultaneously training it with these additional self-supervised tasks. We show that one can improve performance on a diverse sound events classification task by nearly 6\% when jointly trained with up to three distinct self-supervised tasks. This improvement scales with the number of additional auxiliary tasks as well as the amount of unsupervised data. We also show that incorporating data augmentation into our multitask setting leads to even further gains in performance.
Learning Noise-Invariant Representations for Robust Speech Recognition
Authors: Anonymous
Date: 2018-10-02 00:00:00
Tasks: Data Augmentation, Representation Learning, Robust Speech Recognition, speech-recognition, Speech Recognition

Despite rapid advances in speech recognition, current models remain brittle to superficial perturbations to their inputs. Small amounts of noise can destroy the performance of an otherwise state-of-the-art model. To harden models against background noise, practitioners often perform data augmentation, adding artificially-noised examples to the training set, carrying over the original label. In this paper, we hypothesize that a clean example and its superficially perturbed counterparts shouldn't merely map to the same class--- they should map to the same representation. We propose invariant-representation-learning (IRL): At each training iteration, for each training example, we sample a noisy counterpart. We then apply a penalty term to coerce matched representations at each layer (above some chosen layer). Our key results, demonstrated on the LibriSpeech dataset are the following: (i) IRL significantly reduces character error rates (CER)on both `clean' (3.3% vs 6.5%) and `other' (11.0% vs 18.1%) test sets; (ii) on several out-of-domain noise settings (different from those seen during training), IRL's benefits are even more pronounced. Careful ablations confirm that our results are not simply due to shrinking activations at the chosen layers.
A Neural Compositional Paradigm for Image Captioning
Authors: Anonymous
Date: 2018-09-24 00:00:00
Tasks: Diversity, Image Captioning

Mainstream captioning models often follow a sequential structure to generate cap-
tions, leading to issues such as introduction of irrelevant semantics, lack of diversity
in the generated captions, and inadequate generalization performance. In this paper,
we present an alternative paradigm for image captioning, which factorizes the
captioning procedure into two stages: (1) extracting an explicit semantic represen-
tation from the given image; and (2) constructing the caption based on a recursive
compositional procedure in a bottom-up manner. Compared to conventional ones,
our paradigm better preserves the semantic content through an explicit factorization
of semantics and syntax. By using the compositional generation procedure, caption
construction follows a recursive structure, which naturally fits the properties of
human language. Moreover, the proposed compositional procedure requires less
data to train, generalizes better, and yields more diverse captions.
Improved Speech Enhancement with the Wave-U-Net
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Audio Source Separation, Speech Enhancement, speech-recognition, Speech Recognition

We study the use of the Wave-U-Net architecture for speech enhancement, a model introduced by Stoller et al for the separation of music vocals and accompaniment.  This end-to-end learning method for audio source separation operates directly in the time domain, permitting the integrated modelling of phase information and being able to take large  temporal contexts into account.  Our experiments show that the proposed method improves several metrics, namely PESQ, CSIG, CBAK, COVL and SSNR, over the  state-of-the-art with respect to the speech enhancement task on the Voice Bank corpus (VCTK) dataset. We find that a reduced number of hidden layers is sufficient for speech enhancement in comparison to the original system designed for singing voice separation in music. We see this initial result as an encouraging signal to further explore speech enhancement in the time-domain, both as an end in itself and as a pre-processing step to speech recognition systems.