pwc-archive/papers-with-abstracts huggingface dataset:
Deep Within-Class Covariance Analysis for Robust Deep Audio Representation Learning
Authors: Anonymous
Date: 2018-10-22 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Acoustic Scene Classification, Classification, Representation Learning, Scene Classification

Deep Neural Networks (DNNs) are known for excellent performance in supervised tasks such as classification. Convolutional Neural Networks (CNNs), in particular, can learn effective features and build high-level representations that can be used for
classification, but also for querying and nearest neighbor search. However, CNNs have also been shown to suffer from a performance drop when the distribution of the data changes from training to test data. In this paper we analyze the internal
representations of CNNs and observe that the representations of unseen data in each class, spread more (with higher variance) in the embedding space of the CNN compared to representations of the training data. More importantly, this difference is more extreme if the unseen data comes from a shifted distribution. Based on this observation, we objectively evaluate the degree of representation’s variance in each class by applying eigenvalue decomposition on the within-class covariance of the internal representations of CNNs and observe the same behaviour. This can be problematic as larger variances might lead to mis-classification if the sample crosses the decision boundary of its class. We apply nearest neighbor classification on the representations and empirically show that the embeddings with the high variance actually have significantly worse KNN classification performances, although this could not be foreseen from their end-to-end classification results. To tackle this problem, we propose Deep Within-Class Covariance Analysis (DWCCA), a deep neural network layer that significantly reduces the within-class covariance of a DNN’s representation, improving performance on unseen test data from a shifted distribution. We empirically evaluate DWCCA on two datasets for Acoustic Scene Classification (DCASE2016 and DCASE2017). We demonstrate that not only does DWCCA significantly improve the network’s internal representation, it
also increases the end-to-end classification accuracy, especially when the test set exhibits a slight distribution shift. By adding DWCCA to a VGG neural network, we achieve around 6 percentage points improvement in the case of a distribution
mismatch.
Learning Robust Joint Representations for Multimodal Sentiment Analysis
Authors: Anonymous
Date: 2018-10-22 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Machine Translation, Multimodal Sentiment Analysis, Sentiment Analysis, Translation

Multimodal sentiment analysis is a core research area that studies speaker sentiment expressed from the language, visual, and acoustic modalities. The central challenge in multimodal learning involves inferring joint representations that can process and relate information from these modalities. However, existing work learns joint representations using multiple modalities as input and may be sensitive to noisy or missing modalities at test time. With the recent success of sequence to sequence models in machine translation, there is an opportunity to explore new ways of learning joint representations that may not require all input modalities at test time. In this paper, we propose a method to learn robust joint representations by translating between modalities. Our method is based on the key insight that translation from a source to a target modality provides a method of learning joint representations using only the source modality as input. We augment modality translations with a cycle consistency loss to ensure that our joint representations retain maximal information from all modalities. Once our translation model is trained with paired multimodal data, we only need data from the source modality at test-time for prediction. This ensures that our model remains robust from perturbations or missing target modalities. We train our model with a coupled translation-prediction objective and it achieves new state-of-the-art results on multimodal sentiment analysis datasets: CMU-MOSI, ICT-MMMO, and YouTube. Additional experiments show that our model learns increasingly discriminative joint representations with more input modalities while maintaining robustness to perturbations of all other modalities.
Interpretable Convolutional Filters with SincNet
Authors: Anonymous
Date: 2018-10-21 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Inductive Bias, speech-recognition, Speech Recognition

Deep learning is currently playing a crucial role toward higher levels of artificial intelligence. This paradigm allows neural networks to learn complex and abstract representations, that are progressively obtained by combining simpler ones. Nevertheless, the internal "black-box" representations automatically discovered by current neural architectures often suffer from a lack of interpretability, making of primary interest the study of explainable machine learning techniques.
This paper summarizes our recent efforts to develop a more interpretable neural model for directly processing speech from the raw waveform. In particular, we propose SincNet, a novel Convolutional Neural Network (CNN) that encourages the first layer to discover more meaningful filters by exploiting parametrized sinc functions. In contrast to standard CNNs, which learn all the elements of each filter, only low and high cutoff frequencies of band-pass filters are directly learned from data. This inductive bias offers a very compact way to derive a customized filter-bank front-end, that only depends on some parameters with a clear physical meaning.  Our experiments, conducted on both speaker and speech recognition, show that the proposed architecture converges faster, performs better, and is more interpretable than standard CNNs.
Semi-supervised classification by reaching consensus among modalities
Authors: Anonymous
Date: 2018-10-21 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Classification, Marketing

Deep learning has demonstrated abilities to learn complex structures, but they can be restricted by available data. Recently, Consensus Networks (CNs) were proposed to alleviate data sparsity by utilizing features from multiple modalities, but they too have been limited by the size of labeled data. In this paper, we extend CN to Transductive Consensus Networks (TCNs), suitable for semi-supervised learning. In TCNs, different modalities of input are compressed into latent representations, which we encourage to become indistinguishable during iterative adversarial training. To understand TCNs two mechanisms, consensus and classification, we put forward its three variants in ablation studies on these mechanisms. To further investigate TCN models, we treat the latent representations as probability distributions and measure their similarities as the negative relative Jensen-Shannon divergences. We show that a consensus state beneficial for classification desires a stable but imperfect similarity between the representations. Overall, TCNs outperform or align with the best benchmark algorithms given 20 to 200 labeled samples on the Bank Marketing and the DementiaBank datasets.
DASED: A Multi-Domain Dataset for Sound Event Detection Domain Adaptation
Authors: Anonymous
Date: 2018-10-19 00:00:00
Tasks: Domain Adaptation, Event Detection, Sound Event Detection

In this paper we present the first freely available dataset for the development and evaluation of domain adaptation methods, for the sound event detection task. The dataset contains 40 log mel-band energies extracted from $100$ different synthetic sound event tracks, with additive noise from nine different acoustic scenes (from indoor, outdoor, and vehicle environments), mixed at six different sound-to-noise ratios, SNRs, (from -12 to -27 dB with a step of -3 dB), and totaling to 5400 (9 * 100 * 6) sound files and a total length of 30 564 minutes. We provide the dataset as is, the code to re-create the dataset and remix the sound event tracks and the acoustic scenes with different SNRs, and a baseline method that tests the adaptation performance with the proposed dataset and establishes some first results.
Composing RNNs and FSTs for Small Data: Recovering Missing Characters in Old Hawaiian Text
Authors: Anonymous
Date: 2018-10-15 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Deep Learning, Reading Comprehension, Transliteration

In contrast to the older writing system of the 19th century, modern Hawaiian orthography employs characters for long vowels and glottal stops. These extra characters account for about one-third of the phonemes in Hawaiian, so including them makes a big difference to reading comprehension and pronunciation. However, transliterating between older and newer texts is a laborious task when performed manually. We introduce two related methods to help solve this transliteration problem automatically, given that there were not enough data to train an end-to-end deep learning model. One approach is implemented, end-to-end, using finite state transducers (FSTs). The other is a hybrid deep learning approach which approximately composes an FST with a recurrent neural network (RNN). We find that the hybrid approach outperforms the end-to-end FST by partitioning the original problem into one part that can be modelled by hand, using an FST, and into another part, which is easily solved by an RNN trained on the available data.
Speech Recognition with Quaternion Neural Networks
Authors: Anonymous
Date: 2018-10-15 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

Neural network architectures are at the core of powerful automatic speech recognition systems (ASR). However, while recent researches focus on novel model architectures, the acoustic input features remain almost unchanged. Traditional ASR systems rely on multidimensional acoustic features such as the Mel filter bank energies alongside with the first, and second order derivatives to characterize time-frames that compose the signal sequence. Considering that these components describe three different views of the same element, neural networks have to learn both the internal relations that exist within these features, and external or global dependencies that exist between the time-frames. Quaternion-valued neural networks (QNN), recently received an important interest from researchers to process and learn such relations in multidimensional spaces. Indeed, quaternion numbers and QNNs have shown their efficiency to process multidimensional inputs as entities, to encode internal dependencies, and to solve many tasks with up to four times less learning parameters than real-valued models. We propose to investigate modern quaternion-valued models such as convolutional and recurrent quaternion neural networks in the context of speech recognition with the TIMIT dataset. The experiments show that QNNs always outperform real-valued equivalent models with way less free parameters, leading to a more efficient, compact, and expressive representation of the relevant information.
Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information
Authors: Anonymous
Date: 2018-10-14 00:00:00
Proceeding: NIPS Workshop IRASL 2018
Tasks: Abstractive Text Summarization

Sequence-to-sequence (seq2seq) neural models have been actively investigated for abstractive summarization. Nevertheless, existing neural abstractive systems frequently generate factually incorrect summaries and are vulnerable to adversarial information, suggesting a crucial lack of semantic understanding. In this paper, we propose a novel semantic-aware neural abstractive summarization model that learns to generate high quality summaries through semantic interpretation over salient content. A novel evaluation scheme with adversarial samples is introduced to measure how well a model identifies off-topic information, where our model yields significantly better performance than the popular pointer-generator summarizer. Human evaluation also confirms that our system summaries are uniformly more informative and faithful as well as less redundant than the seq2seq model.
A Better Phone Set for the TIMIT Dataset Discovered in Clustering of Listen, Attend and Spell
Authors: Anonymous
Date: 2018-09-20 00:00:00

Listen, Attend and Spell(LAS)maps a sequence of acoustic spectra directly to a sequence of graphemes, with no explicit internal representation of phones. This paper asks whether LAS can be used as a scientific tool, to discover the phone set of a language whose phone set may be controversial or unknown.  Phonemes have a precise linguistic definition, but phones may be defined in any manner that is convenient for speech technology: we propose that a practical phone set is one that can be inferred from speech following certain procedures, but that is also highly predictive of the word sequence. We demonstrate that such a phone set can be inferred by clustering the hidden node activation vectors of an LAS model during training, thus encouraging the model to learn a hidden representation characterized by acoustically compact clusters that are nevertheless predictive of the word sequence. We further define a metric for the quality of a phone set (sum of conditional entropy of the phone set given graphemes, and given acoustics), and demonstrate that according to this metric, the clustered-LAS phone set is better than the original TIMIT phone set.
How to make someone speak a language that they don't know.
Authors: Anonymous
Date: 2018-10-30 00:00:00
Tasks: Decoder, Speech Synthesis

We present a simple idea that allows to record a speaker in a given language and synthesize their voice in other languages that they may not even know. These techniques open a wide range of potential applications such as cross-language communication, language learning or automatic video dubbing. We call this general problem multi-language speaker-conditioned speech synthesis and we present a simple but strong baseline for it.

Our model architecture is similar to the encoder-decoder Char2Wav model or Tacotron. The main difference is that, instead of conditioning on characters or phonemes that are specific to a given language, we condition on a shared phonetic representation that is universal to all languages. This cross-language phonetic representation of text allows to synthesize speech in any language while preserving the vocal characteristics of the original speaker. Furthermore, we show that fine-tuning the weights of our model allows us to extend our results to speakers outside of the training dataset.
Training Neural Nets to Achieve Audio-to-Score Translation: Opening the Black-Box
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Translation

It is suggested that the task of audio-to-score translation offers an adequate testbed to investigate the division of labor between background knowledge and machine learning in the domain of audio pattern recognition, with a controllable level of difficulty and the ability to synthesize a limitless amount of labelled data.
  
  As a proof of concept, this paper focuses on pitch detection from audio signals. Extensive background knowledge is used to initialize simple convolutional neural nets (NN) and achieve the recognition of single notes with a decent accuracy. The performance achieved by trained NNs, however, is significantly higher. Some tentative interpretations of this fact are obtained by opening the black box and inspecting the modifications of the NN filters due to supervised learning.
Transferable and Configurable Audio Adversarial Attack from Low-Level Features
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Adversarial Attack, Automatic Speech Recognition, Automatic Speech Recognition (ASR), BIG-bench Machine Learning, speech-recognition, Speech Recognition

Recent works revealed that state-of-the-art machine learning based Automatic Speech Recognition systems (ASR) have a considerable vulnerability to the crafted adversarial examples. However, limited by individual ASR system's specific machine learning models, the current audio adversarial attacks still lack certain model transferability as well as configurability for different deployment scenarios. In this work, we propose a novel untargeted adversarial example generation method to ASR systems, which shifts the adversarial example generation from the high-level machine learning models to the low-level feature extraction stage. By taking advantage of the fundamental impact and direct configuration of the low-level features, the proposed method can generate transferable and configurable adversarial examples for ASR system perturbation.
During the evaluation, we use 6 commercial ASR models to test the proposed attack method. The results show that the proposed method can achieve strong transferability and good perturbation effectiveness. Also, it can configure the adversarial examples with desired audio attributes for better scenario adaptation capability.
Skip-Thought GAN: Generating Text through Adversarial Training using Skip-Thought Vectors
Authors: Anonymous
Date: 2018-10-16 00:00:00
Tasks: Image Generation, Sentence, Sentence Embeddings, Style Transfer, Text Generation, Word Embeddings

In the past few years, various advancements have been made in generative models owing to the formulation of Generative Adversarial Networks (GANs). GANs have been shown to perform exceedingly well on a wide variety of tasks pertaining to image generation and style transfer. In the field of Natural Language Processing, word embeddings such as word2vec and GLoVe are state-of-the-art methods for applying neural network models on textual data. Attempts have been made for utilizing GANs with word embeddings for text generation. This work presents an approach to text generation using Skip-Thought sentence embeddings in conjunction with GANs based on gradient penalty functions and f-measures. The results of using sentence embeddings with GANs for generating text conditioned on input information are comparable to the approaches where word embeddings are used.
Variational Autoencoders with implicit priors for short-duration text-independent speaker verification
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Speaker Recognition, Speaker Verification, Text-Independent Speaker Verification, Triplet

In this work, we exploited different strategies to provide prior knowledge to commonly used generative modeling approaches aiming to obtain speaker-dependent low dimensional representations from short-duration segments of speech data, making use of available information of speaker identities. Namely, convolutional variational autoencoders are employed, and statistics of its learned posterior distribution are used as low dimensional representations of fixed length short-duration utterances. In order to enforce speaker dependency in the latent layer, we introduced a variation of the commonly used prior within the variational autoencoders framework, i.e. the model is simultaneously trained for reconstruction of inputs along with a discriminative task performed on top of latent layers outputs. The effectiveness of both triplet loss minimization and speaker recognition are evaluated as implicit priors on the challenging cross-language NIST SRE 2016 setting and compared against fully supervised and unsupervised baselines.
Are You Sure YouWant To Do That? Classification with Interpretable Queries
Authors: Anonymous
Date: 2018-10-16 00:00:00
Tasks: Memorization

Classification systems typically act in isolation, meaning they are required to implicitly memorize the characteristics of all candidate classes in order to classify. The cost of this is increased memory usage and poor sample efficiency. We propose a model which instead verifies using reference images during the classification process, reducing the burden of memorization. The model uses iterative non-differentiable queries in order to classify an image. We demonstrate that such a model is feasible to train and can match baseline accuracy while being more parameter efficient. However, we show that finding the correct balance between image recognition and verification is essential to pushing the model towards desired behavior, suggesting that a pipeline of recognition followed by verification is a more promising approach towards designing more powerful networks with simpler architectures.
Boosting pathology detection in infants by deep transfer learning from adult speech
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Diagnostic, Transfer Learning

Can knowledge extracted from adult speech help improve the performance models developed for infant cry? This work investigates this question in the context of pathology detection in newborns. The analysis of infant crying patterns to detect pathology is of interest as it opens the possibility of more accessible diagnostic tools in resource-constrained settings. Classical machine learning approaches leveraging features extracted as Mel frequency cepstral coefficients, have supported the viability of the infant cry as a diagnostic input, but performance is not yet at a level of clinical utility. The application of deep learning models has been limited due to the unavailability of large infant cry databases which are costly to acquire. This work argues that the transfer of useful knowledge from adult speech is possible because it is driven by the same underlying physiologic process as that of infants. Our experiments demonstrate that on the task of predicting perinatal asphyxia from infant cry, such transfer learning provides an overall improvement of 13.5\% in F1 score over a model trained from random initialization.
Learned in Speech Recognition: Contextual Acoustic Word Embeddings
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Sentence, speech-recognition, Speech Recognition, Spoken Language Understanding, Word Embeddings

End-to-end acoustic-to-word speech recognition models have recently gained popularity because they are easy to train, scale well to large amounts of training data, and do not require a lexicon. In addition, word models may also be easier to integrate with downstream tasks such as spoken language understanding, because inference (search) is much simplified compared to phoneme, character or any other sort of sub-word units. In this paper, we describe methods to construct contextual acoustic word embeddings directly from a supervised sequence-to-sequence acoustic-to-word speech recognition model using the learned attention distribution. On a suite of 16 standard sentence evaluation tasks, our embeddings show competitive performance against a word2vec model trained on the speech transcriptions. In addition, we evaluate these embeddings on a spoken language understanding task and observe that our embeddings match the performance of text-based embeddings in a pipeline of first performing speech recognition and then constructing word embeddings from transcriptions.
Variational learning across domains with triplet information
Authors: Anonymous
Date: 2018-10-22 00:00:00
Tasks: Cross-Lingual Document Classification, Document Classification, Image Generation, Image-to-Image Translation, Metric Learning, Translation, Triplet

The work investigates deep generative models, which allow us to use training data from one domain to build a model for another domain. We propose the Variational Bi-domain Triplet Autoencoder (VBTA) that learns a joint distribution of objects from different domains. We extend the VBTAs objective function by the relative constraints or triplets that sampled from the shared latent space across domains. In other words, we combine the deep generative models with a  metric learning ideas in order to improve the final objective with the triplets information. The performance of the VBTA model is demonstrated on different tasks: image-to-image translation, bi-directional image generation and cross-lingual document classification.