pwc-archive/papers-with-abstracts huggingface dataset:
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
Authors: Chao Zhang, Zichao Yang, Xiaodong He, Li Deng
Date: 2019-11-10 00:00:00
Tasks: Caption Generation, Image Generation, Image to text, Multimodal Deep Learning, Question Answering, Representation Learning, speech-recognition, Speech Recognition, Text to Image Generation, Text-to-Image Generation, Visual Question Answering, Visual Question Answering (VQA)

Deep learning methods have revolutionized speech recognition, image recognition, and natural language processing since 2010. Each of these tasks involves a single modality in their input signals. However, many applications in the artificial intelligence field involve multiple modalities. Therefore, it is of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, we provide a technical review of available models and learning methods for multimodal intelligence. The main focus of this review is the combination of vision and natural language modalities, which has become an important topic in both the computer vision and natural language processing research communities. This review provides a comprehensive analysis of recent works on multimodal deep learning from three perspectives: learning multimodal representations, fusing multimodal signals at various levels, and multimodal applications. Regarding multimodal representation learning, we review the key concepts of embedding, which unify multimodal signals into a single vector space and thereby enable cross-modality signal processing. We also review the properties of many types of embeddings that are constructed and learned for general downstream tasks. Regarding multimodal fusion, this review focuses on special architectures for the integration of representations of unimodal signals for a particular task. Regarding applications, selected areas of a broad interest in the current literature are covered, including image-to-text caption generation, text-to-image generation, and visual question answering. We believe that this review will facilitate future studies in the emerging field of multimodal intelligence for related communities.
On Posterior Collapse and Encoder Feature Dispersion in Sequence VAEs
Authors: Teng Long, Yanshuai Cao, Jackie Chi Kit Cheung
Date: 2019-11-10 00:00:00
Tasks: Language Modeling, Language Modelling

Variational autoencoders (VAEs) hold great potential for modelling text, as they could in theory separate high-level semantic and syntactic properties from local regularities of natural language. Practically, however, VAEs with autoregressive decoders often suffer from posterior collapse, a phenomenon where the model learns to ignore the latent variables, causing the sequence VAE to degenerate into a language model. In this paper, we argue that posterior collapse is in part caused by the lack of dispersion in encoder features. We provide empirical evidence to verify this hypothesis, and propose a straightforward fix using pooling. This simple technique effectively prevents posterior collapse, allowing model to achieve significantly better data log-likelihood than standard sequence VAEs. Comparing to existing work, our proposed method is able to achieve comparable or superior performances while being more computationally efficient.
A Multimodal CNN-based Tool to Censure Inappropriate Video Scenes
Authors: Pedro V. A. de Freitas, Paulo R. C. Mendes, Gabriel N. P. dos Santos, Antonio José G. Busson, Álan Livio Guedes, Sérgio Colcher, Ruy Luiz Milidiú
Date: 2019-11-10 00:00:00

Due to the extensive use of video-sharing platforms and services for their storage, the amount of such media on the internet has become massive. This volume of data makes it difficult to control the kind of content that may be present in such video files. One of the main concerns regarding the video content is if it has an inappropriate subject matter, such as nudity, violence, or other potentially disturbing content. More than telling if a video is either appropriate or inappropriate, it is also important to identify which parts of it contain such content, for preserving parts that would be discarded in a simple broad analysis. In this work, we present a multimodal~(using audio and image features) architecture based on Convolutional Neural Networks (CNNs) for detecting inappropriate scenes in video files. In the task of classifying video files, our model achieved 98.95\% and 98.94\% of F1-score for the appropriate and inappropriate classes, respectively. We also present a censoring tool that automatically censors inappropriate segments of a video file.
IrisNet: Deep Learning for Automatic and Real-time Tongue Contour Tracking in Ultrasound Video Data using Peripheral Vision
Authors: M. Hamed Mozaffari, Md. Aminur Rab Ratul, Won-Sook Lee
Date: 2019-11-10 00:00:00
Tasks: Deep Learning, image-classification, Image Classification

The progress of deep convolutional neural networks has been successfully exploited in various real-time computer vision tasks such as image classification and segmentation. Owing to the development of computational units, availability of digital datasets, and improved performance of deep learning models, fully automatic and accurate tracking of tongue contours in real-time ultrasound data became practical only in recent years. Recent studies have shown that the performance of deep learning techniques is significant in the tracking of ultrasound tongue contours in real-time applications such as pronunciation training using multimodal ultrasound-enhanced approaches. Due to the high correlation between ultrasound tongue datasets, it is feasible to have a general model that accomplishes automatic tongue tracking for almost all datasets. In this paper, we proposed a deep learning model comprises of a convolutional module mimicking the peripheral vision ability of the human eye to handle real-time, accurate, and fully automatic tongue contour tracking tasks, applicable for almost all primary ultrasound tongue datasets. Qualitative and quantitative assessment of IrisNet on different ultrasound tongue datasets and PASCAL VOC2012 revealed its outstanding generalization achievement in compare with similar techniques.
Improved Large-margin Softmax Loss for Speaker Diarisation
Authors: Yassir Fathullah, Chao Zhang, Philip C. Woodland
Date: 2019-11-10 00:00:00

Speaker diarisation systems nowadays use embeddings generated from speech segments in a bottleneck layer, which are needed to be discriminative for unseen speakers. It is well-known that large-margin training can improve the generalisation ability to unseen data, and its use in such open-set problems has been widespread. Therefore, this paper introduces a general approach to the large-margin softmax loss without any approximations to improve the quality of speaker embeddings for diarisation. Furthermore, a novel and simple way to stabilise training, when large-margin softmax is used, is proposed. Finally, to combat the effect of overlapping speech, different training margins are used to reduce the negative effect overlapping speech has on creating discriminative embeddings. Experiments on the AMI meeting corpus show that the use of large-margin softmax significantly improves the speaker error rate (SER). By using all hyper parameters of the loss in a unified way, further improvements were achieved which reached a relative SER reduction of 24.6% over the baseline. However, by training overlapping and single speaker speech samples with different margins, the best result was achieved, giving overall a 29.5% SER reduction relative to the baseline.
SeismoGen: Seismic Waveform Synthesis Using Generative Adversarial Networks
Authors: Tiantong Wang, Daniel Trugman, Youzuo Lin
Date: 2019-11-10 00:00:00
Tasks: Generative Adversarial Network, Time Series, Time Series Analysis

Detecting earthquake events from seismic time series has proved itself a challenging task. Manual detection can be expensive and tedious due to the intensive labor and large scale data set. In recent years, automatic detection methods based on machine learning have been developed to improve accuracy and efficiency. However, the accuracy of those methods relies on a sufficient amount of high-quality training data, which itself can be expensive to obtain due to the requirement of domain knowledge and subject matter expertise. This paper is to resolve this dilemma by answering two questions: (1) provided with a limited number of reliable labels, can we use them to generate more synthetic labels; (2) Can we use those synthetic labels to improve the detectability? Among all the existing generative models, the generative adversarial network (GAN) shows its supreme capability in generating high-quality synthetic samples in multiple domains. We designed our model based on GAN. In particular, we studied several different network structures. By comparing the generated results, our GAN-based generative model yields the highest quality. We further combine the dataset with synthetic samples generated by our generative model and show that the detectability of our earthquake classification model is significantly improved than the one trained without augmenting the training set.
Classical linear logic, cobordisms and categorial grammars
Authors: Sergey Slavnov
Date: 2019-11-10 00:00:00

We propose a categorial grammar based on classical multiplicative linear logic. This can be seen as an extension of abstract categorial grammars (ACG) and is at least as expressive. However, constituents of {\it linear logic grammars (LLG)} are not abstract ${\lambda}$-terms, but simply tuples of words with labeled endpoints and supplied with specific {\it plugging instructions}: the sets of endpoints are subdivided into the {\it incoming} and the {\it outgoing} parts. We call such objects {\it word cobordisms}. A key observation is that word cobordisms can be organized in a category, very similar to the familiar category of topological cobordisms. This category is symmetric monoidal closed and compact closed and thus is a model of linear $\lambda$-calculus and classical, as well as intuitionistic linear logic. This allows us using linear logic as a typing system for word cobordisms. At least, this gives a concrete and intuitive representation of ACG. We think, however, that the category of word cobordisms, which has a rich structure and is independent of any grammar, might be interesting on its own right.
TSK-Streams: Learning TSK Fuzzy Systems on Data Streams
Authors: Ammar Shaker, Eyke Hüllermeier
Date: 2019-11-10 00:00:00
Tasks: regression

The problem of adaptive learning from evolving and possibly non-stationary data streams has attracted a lot of interest in machine learning in the recent past, and also stimulated research in related fields, such as computational intelligence and fuzzy systems. In particular, several rule-based methods for the incremental induction of regression models have been proposed. In this paper, we develop a method that combines the strengths of two existing approaches rooted in different learning paradigms. More concretely, our method adopts basic principles of the state-of-the-art learning algorithm AMRules and enriches them by the representational advantages of fuzzy rules. In a comprehensive experimental study, TSK-Streams is shown to be highly competitive in terms of performance.
Interpretable Multiple-Kernel Prototype Learning for Discriminative Representation and Feature Selection
Authors: Babak Hosseini, Barbara Hammer
Date: 2019-11-10 00:00:00
Tasks: feature selection

Prototype-based methods are of the particular interest for domain specialists and practitioners as they summarize a dataset by a small set of representatives. Therefore, in a classification setting, interpretability of the prototypes is as significant as the prediction accuracy of the algorithm. Nevertheless, the state-of-the-art methods make inefficient trade-offs between these concerns by sacrificing one in favor of the other, especially if the given data has a kernel-based representation. In this paper, we propose a novel interpretable multiple-kernel prototype learning (IMKPL) to construct highly interpretable prototypes in the feature space, which are also efficient for the discriminative representation of the data. Our method focuses on the local discrimination of the classes in the feature space and shaping the prototypes based on condensed class-homogeneous neighborhoods of data. Besides, IMKPL learns a combined embedding in the feature space in which the above objectives are better fulfilled. When the base kernels coincide with the data dimensions, this embedding results in a discriminative features selection. We evaluate IMKPL on several benchmarks from different domains which demonstrate its superiority to the related state-of-the-art methods regarding both interpretability and discriminative representation.
Using LSTMs for climate change assessment studies on droughts and floods
Authors: Frederik Kratzert, Daniel Klotz, Johannes Brandstetter, Pieter-Jan Hoedt, Grey Nearing, Sepp Hochreiter
Date: 2019-11-10 00:00:00
Tasks: Diversity

Climate change affects occurrences of floods and droughts worldwide. However, predicting climate impacts over individual watersheds is difficult, primarily because accurate hydrological forecasts require models that are calibrated to past data. In this work we present a large-scale LSTM-based modeling approach that -- by training on large data sets -- learns a diversity of hydrological behaviors. Previous work shows that this model is more accurate than current state-of-the-art models, even when the LSTM-based approach operates out-of-sample and the latter in-sample. In this work, we show how this model can assess the sensitivity of the underlying systems with regard to extreme (high and low) flows in individual watersheds over the continental US.
SLTR: Simultaneous Localization of Target and Reflector in NLOS Condition Using Beacons
Authors: Muhammad. H Fares, Hadi Moradi, Mahmoud Shahabadi
Date: 2019-11-10 00:00:00

When the direct view between the target and the observer is not available, due to obstacles with non-zero sizes, the observation is received after reflection from a reflector, this is the indirect view or Non-Line-Of Sight condition. Localization of a target in NLOS condition still one of the open problems yet. In this paper, we address this problem by localizing the reflector and the target simultaneously using a single stationary receiver, and a determined number of beacons, in which their placements are also analyzed in an unknown map. The work is done in mirror space, when the receiver is a camera, and the reflector is a planar mirror. Furthermore, the distance from the observer to the target is estimated by size constancy concept, and the angle of coming signal is the same as the orientation of the camera, with respect to a global frame. The results show the validation of the proposed work and the simulation results are matched with the theoretical results.
Language Model-Driven Unsupervised Neural Machine Translation
Authors: Wei Zhang, Youyuan Lin, Ruoran Ren, Xiaodong Wang, Zhenshuang Liang, Zhen Huang
Date: 2019-11-10 00:00:00
Tasks: Language Modeling, Language Modelling, Machine Translation, model, NMT, Translation

Unsupervised neural machine translation(NMT) is associated with noise and errors in synthetic data when executing vanilla back-translations. Here, we explicitly exploits language model(LM) to drive construction of an unsupervised NMT system. This features two steps. First, we initialize NMT models using synthetic data generated via temporary statistical machine translation(SMT). Second, unlike vanilla back-translation, we formulate a weight function, that scores synthetic data at each step of subsequent iterative training; this allows unsupervised training to an improved outcome. We present the detailed mathematical construction of our method. Experimental WMT2014 English-French, and WMT2016 English-German and English-Russian translation tasks revealed that our method outperforms the best prior systems by more than 3 BLEU points.
Can Neural Image Captioning be Controlled via Forced Attention?
Authors: Philipp Sadler, Tatjana Scheffler, David Schlangen
Date: 2019-11-10 00:00:00
Proceeding: WS 2019 10
Tasks: Image Captioning, Text Generation

Learned dynamic weighting of the conditioning signal (attention) has been shown to improve neural language generation in a variety of settings. The weights applied when generating a particular output sequence have also been viewed as providing a potentially explanatory insight into the internal workings of the generator. In this paper, we reverse the direction of this connection and ask whether through the control of the attention of the model we can control its output. Specifically, we take a standard neural image captioning model that uses attention, and fix the attention to pre-determined areas in the image. We evaluate whether the resulting output is more likely to mention the class of the object in that area than the normally generated caption. We introduce three effective methods to control the attention and find that these are producing expected results in up to 28.56% of the cases.
Evaluating Voice Conversion-based Privacy Protection against Informed Attackers
Authors: Brij Mohan Lal Srivastava, Nathalie Vauquier, Md Sahidullah, Aurélien Bellet, Marc Tommasi, Emmanuel Vincent
Date: 2019-11-10 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Speaker Verification, speech-recognition, Speech Recognition, Voice Cloning, Voice Conversion

Speech data conveys sensitive speaker attributes like identity or accent. With a small amount of found data, such attributes can be inferred and exploited for malicious purposes: voice cloning, spoofing, etc. Anonymization aims to make the data unlinkable, i.e., ensure that no utterance can be linked to its original speaker. In this paper, we investigate anonymization methods based on voice conversion. In contrast to prior work, we argue that various linkage attacks can be designed depending on the attackers' knowledge about the anonymization scheme. We compare two frequency warping-based conversion methods and a deep learning based method in three attack scenarios. The utility of converted speech is measured via the word error rate achieved by automatic speech recognition, while privacy protection is assessed by the increase in equal error rate achieved by state-of-the-art i-vector or x-vector based speaker verification. Our results show that voice conversion schemes are unable to effectively protect against an attacker that has extensive knowledge of the type of conversion and how it has been applied, but may provide some protection against less knowledgeable attackers.
Robust Unsupervised Audio-visual Speech Enhancement Using a Mixture of Variational Autoencoders
Authors: Mostafa Sadeghi, Xavier Alameda-Pineda
Date: 2019-11-10 00:00:00
Tasks: Speech Enhancement

Recently, an audio-visual speech generative model based on variational autoencoder (VAE) has been proposed, which is combined with a nonnegative matrix factorization (NMF) model for noise variance to perform unsupervised speech enhancement. When visual data is clean, speech enhancement with audio-visual VAE shows a better performance than with audio-only VAE, which is trained on audio-only data. However, audio-visual VAE is not robust against noisy visual data, e.g., when for some video frames, speaker face is not frontal or lips region is occluded. In this paper, we propose a robust unsupervised audio-visual speech enhancement method based on a per-frame VAE mixture model. This mixture model consists of a trained audio-only VAE and a trained audio-visual VAE. The motivation is to skip noisy visual frames by switching to the audio-only VAE model. We present a variational expectation-maximization method to estimate the parameters of the model. Experiments show the promising performance of the proposed method.
Cooperative Pathfinding based on memory-efficient Multi-agent RRT*
Authors: Jinmingwu Jiang, Kaigui Wu
Date: 2019-11-10 00:00:00

In cooperative pathfinding problems, no-conflicts paths that bring several agents from their start location to their destination need to be planned. This problem can be efficiently solved by Multi-agent RRT*(MA-RRT*) algorithm, which is still state-of-the-art in the field of coupled methods. However, the implementation of this algorithm is hindered in systems with limited memory because the number of nodes in the tree grows indefinitely as the paths get optimized. This paper proposes an improved version of MA-RRT*, called Multi-agent RRT* Fixed Node(MA-RRT*FN), which limits the number of nodes stored in the tree by removing the weak nodes on the path which are not likely to reach the goal. The results show that MA-RRT*FN performs close to MA-RRT* in terms of scalability and solution quality while the memory required is much lower and fixed.
Symmetrical Gaussian Error Linear Units (SGELUs)
Authors: Chao Yu, Zhiguo Su
Date: 2019-11-10 00:00:00

In this paper, a novel neural network activation function, called Symmetrical Gaussian Error Linear Unit (SGELU), is proposed to obtain high performance. It is achieved by effectively integrating the property of the stochastic regularizer in the Gaussian Error Linear Unit (GELU) with the symmetrical characteristics. Combining with these two merits, the proposed unit introduces the capability of the bidirection convergence to successfully optimize the network without the gradient diminishing problem. The evaluations of SGELU against GELU and Linearly Scaled Hyperbolic Tangent (LiSHT) have been carried out on MNIST classification and MNIST auto-encoder, which provide great validations in terms of the performance, the convergence rate among these applications.
A Dynamic Modelling Framework for Human Hand Gesture Task Recognition
Authors: Sara Masoud, Bijoy Chowdhury, Young-Jun Son, Chieri Kubota, Russell Tronstad
Date: 2019-11-10 00:00:00
Tasks: Gesture Recognition

Gesture recognition and hand motion tracking are important tasks in advanced gesture based interaction systems. In this paper, we propose to apply a sliding windows filtering approach to sample the incoming streams of data from data gloves and a decision tree model to recognize the gestures in real time for a manual grafting operation of a vegetable seedling propagation facility. The sequence of these recognized gestures defines the tasks that are taking place, which helps to evaluate individuals' performances and to identify any bottlenecks in real time. In this work, two pairs of data gloves are utilized, which reports the location of the fingers, hands, and wrists wirelessly (i.e., via Bluetooth). To evaluate the performance of the proposed framework, a preliminary experiment was conducted in multiple lab settings of tomato grafting operations, where multiple subjects wear the data gloves while performing different tasks. Our results show an accuracy of 91% on average, in terms of gesture recognition in real time by employing our proposed framework.
Un systeme de lemmatisation pour les applications de TALN
Authors: Sadik Bessou, Mohamed Louail, Allaoua Refoufi, Zehour Kadem, Mohamed Touahria
Date: 2019-11-10 00:00:00

This paper presents a method of stemming for the Arabian texts based on the linguistic techniques of the natural language processing. This method leans on the notion of scheme (one of the strong points of the morphology of the Arabian language). The advantage of this approach is that it doesn't use a dictionary of inflexions but a smart dynamic recognition of the different words of the language.