pwc-archive/papers-with-abstracts huggingface dataset:
An Artificial Neural Network Architecture Based on Context Transformations in Cortical Minicolumns
Authors: Vasily Morzhakov, Alexey Redozubov
Date: 2017-12-16 00:00:00

Cortical minicolumns are considered a model of cortical organization. Their
function is still a source of research and not reflected properly in modern
architecture of nets in algorithms of Artificial Intelligence. We assume its
function and describe it in this article. Furthermore, we show how this
proposal allows to construct a new architecture, that is not based on
convolutional neural networks, test it on MNIST data and receive close to
Convolutional Neural Network accuracy. We also show that the proposed
architecture possesses an ability to train on a small quantity of samples. To
achieve these results, we enable the minicolumns to remember context
transformations.
Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation
Authors: Jean-Benoit Delbrouck, Stéphane Dupont, Omar Seddati
Date: 2017-07-04 00:00:00
Tasks: Dense Captioning, Machine Translation, object-detection, Object Detection, Sentence, Translation, Word Embeddings

In Multimodal Neural Machine Translation (MNMT), a neural model generates a
translated sentence that describes an image, given the image itself and one
source descriptions in English. This is considered as the multimodal image
caption translation task. The images are processed with Convolutional Neural
Network (CNN) to extract visual features exploitable by the translation model.
So far, the CNNs used are pre-trained on object detection and localization
task. We hypothesize that richer architecture, such as dense captioning models,
may be more suitable for MNMT and could lead to improved translations. We
extend this intuition to the word-embeddings, where we compute both linguistic
and visual representation for our corpus vocabulary. We combine and compare
different confi
Tensor Product Generation Networks for Deep NLP Modeling
Authors: Qiuyuan Huang, Paul Smolensky, Xiaodong He, Li Deng, Dapeng Wu
Date: 2017-09-26 00:00:00
Proceeding: NAACL 2018 6
Tasks: Caption Generation

We present a new approach to the design of deep networks for natural language
processing (NLP), based on the general technique of Tensor Product
Representations (TPRs) for encoding and processing symbol structures in
distributed neural networks. A network architecture --- the Tensor Product
Generation Network (TPGN) --- is proposed which is capable in principle of
carrying out TPR computation, but which uses unconstrained deep learning to
design its internal representations. Instantiated in a model for image-caption
generation, TPGN outperforms LSTM baselines when evaluated on the COCO dataset.
The TPR-capable structure enables interpretation of internal representations
and operations, which prove to contain considerable grammatical content. Our
caption-generation model can be interpreted as generating sequences of
grammatical categories and retrieving words by their categories from a plan
encoded as a distributed representation.
SAR: Semantic Analysis for Recommendation
Authors: Han Xiao, Lian Meng
Date: 2017-02-21 00:00:00
Tasks: Matrix Completion

Recommendation system is a common demand in daily life and matrix completion
is a widely adopted technique for this task. However, most matrix completion
methods lack semantic interpretation and usually result in weak-semantic
recommendations. To this end, this paper proposes a $S$emantic $A$nalysis
approach for $R$ecommendation systems $(SAR)$, which applies a two-level
hierarchical generative process that assigns semantic properties and categories
for user and item. $SAR$ learns semantic representations of users/items merely
from user ratings on items, which offers a new path to recommendation by
semantic matching with the learned representations. Extensive experiments
demonstrate $SAR$ outperforms other state-of-the-art baselines substantially.
NDT: Neual Decision Tree Towards Fully Functioned Neural Graph
Authors: Han Xiao
Date: 2017-12-16 00:00:00

Though traditional algorithms could be embedded into neural architectures
with the proposed principle of \cite{xiao2017hungarian}, the variables that
only occur in the condition of branch could not be updated as a special case.
To tackle this issue, we multiply the conditioned branches with Dirac symbol
(i.e. $\mathbf{1}_{x>0}$), then approximate Dirac symbol with the continuous
functions (e.g. $1 - e^{-\alpha|x|}$). In this way, the gradients of
condition-specific variables could be worked out in the back-propagation
process, approximately, making a fully functioned neural graph. Within our
novel principle, we propose the neural decision tree \textbf{(NDT)}, which
takes simplified neural networks as decision function in each branch and
employs complex neural networks to generate the output in each leaf. Extensive
experiments verify our theoretical analysis and demonstrate the effectiveness
of our model.
Differentially Private Variational Dropout
Authors: Beyza Ermis, Ali Taylan Cemgil
Date: 2017-11-30 00:00:00
Tasks: Privacy Preserving

Deep neural networks with their large number of parameters are highly
flexible learning systems. The high flexibility in such networks brings with
some serious problems such as overfitting, and regularization is used to
address this problem. A currently popular and effective regularization
technique for controlling the overfitting is dropout. Often, large data
collections required for neural networks contain sensitive information such as
the medical histories of patients, and the privacy of the training data should
be protected. In this paper, we modify the recently proposed variational
dropout technique which provided an elegant Bayesian interpretation to dropout,
and show that the intrinsic noise in the variational dropout can be exploited
to obtain a degree of differential privacy. The iterative nature of training
neural networks presents a challenge for privacy-preserving estimation since
multiple iterations increase the amount of noise added. We overcome this by
using a relaxed notion of differential privacy, called concentrated
differential privacy, which provides tighter estimates on the overall privacy
loss. We demonstrate the accuracy of our privacy-preserving variational dropout
algorithm on benchmark datasets.
Word Sense Disambiguation with LSTM: Do We Really Need 100 Billion Words?
Authors: Minh Le, Marten Postma, Jacopo Urbani
Date: 2017-12-09 00:00:00
Tasks: Word Sense Disambiguation

Recently, Yuan et al. (2016) have shown the effectiveness of using Long
Short-Term Memory (LSTM) for performing Word Sense Disambiguation (WSD). Their
proposed technique outperformed the previous state-of-the-art with several
benchmarks, but neither the training data nor the source code was released.
This paper presents the results of a reproduction study of this technique using
only openly available datasets (GigaWord, SemCore, OMSTI) and software
(TensorFlow). From them, it emerged that state-of-the-art results can be
obtained with much less data than hinted by Yuan et al. All code and trained
models are made freely available.
A Machine Learning Framework for Resource Allocation Assisted by Cloud Computing
Authors: Jun-Bo Wang, Junyuan Wang, Yongpeng Wu, Jin-Yuan Wang, Huiling Zhu, Min Lin, Jiangzhou Wang
Date: 2017-12-16 00:00:00
Tasks: BIG-bench Machine Learning, Cloud Computing, Philosophy

Conventionally, the resource allocation is formulated as an optimization
problem and solved online with instantaneous scenario information. Since most
resource allocation problems are not convex, the optimal solutions are very
difficult to be obtained in real time. Lagrangian relaxation or greedy methods
are then often employed, which results in performance loss. Therefore, the
conventional methods of resource allocation are facing great challenges to meet
the ever-increasing QoS requirements of users with scarce radio resource.
Assisted by cloud computing, a huge amount of historical data on scenarios can
be collected for extracting similarities among scenarios using machine
learning. Moreover, optimal or near-optimal solutions of historical scenarios
can be searched offline and stored in advance. When the measured data of
current scenario arrives, the current scenario is compared with historical
scenarios to find the most similar one. Then, the optimal or near-optimal
solution in the most similar historical scenario is adopted to allocate the
radio resources for the current scenario. To facilitate the application of new
design philosophy, a machine learning framework is proposed for resource
allocation assisted by cloud computing. An example of beam allocation in
multi-user massive multiple-input-multiple-output (MIMO) systems shows that the
proposed machine-learning based resource allocation outperforms conventional
methods.
CycleGAN, a Master of Steganography
Authors: Casey Chu, Andrey Zhmoginov, Mark Sandler
Date: 2017-12-08 00:00:00

CycleGAN (Zhu et al. 2017) is one recent successful approach to learn a
transformation between two image distributions. In a series of experiments, we
demonstrate an intriguing property of the model: CycleGAN learns to "hide"
information about a source image into the images it generates in a nearly
imperceptible, high-frequency signal. This trick ensures that the generator can
recover the original sample and thus satisfy the cyclic consistency
requirement, while the generated image remains realistic. We connect this
phenomenon with adversarial attacks by viewing CycleGAN's training procedure as
training a generator of adversarial examples and demonstrate that the cyclic
consistency loss causes CycleGAN to be especially vulnerable to adversarial
attacks.
Cyberattack Detection in Mobile Cloud Computing: A Deep Learning Approach
Authors: Khoi Khac Nguyen, Dinh Thai Hoang, Dusit Niyato, Ping Wang, Diep Nguyen, Eryk Dutkiewicz
Date: 2017-12-16 00:00:00
Tasks: Cloud Computing, Deep Learning

With the rapid growth of mobile applications and cloud computing, mobile
cloud computing has attracted great interest from both academia and industry.
However, mobile cloud applications are facing security issues such as data
integrity, users' confidentiality, and service availability. A preventive
approach to such problems is to detect and isolate cyber threats before they
can cause serious impacts to the mobile cloud computing system. In this paper,
we propose a novel framework that leverages a deep learning approach to detect
cyberattacks in mobile cloud environment. Through experimental results, we show
that our proposed framework not only recognizes diverse cyberattacks, but also
achieves a high accuracy (up to 97.11%) in detecting the attacks. Furthermore,
we present the comparisons with current machine learning-based approaches to
demonstrate the effectiveness of our proposed solution.
Successive Embedding and Classification Loss for Aerial Image Classification
Authors: Jiayun Wang, Patrick Virtue, Stella X. Yu
Date: 2017-12-05 00:00:00
Tasks: Classification, Clustering, General Classification, image-classification, Image Classification

Deep neural networks can be effective means to automatically classify aerial images but is easy to overfit to the training data. It is critical for trained neural networks to be robust to variations that exist between training and test environments. To address the overfitting problem in aerial image classification, we consider the neural network as successive transformations of an input image into embedded feature representations and ultimately into a semantic class label, and train neural networks to optimize image representations in the embedded space in addition to optimizing the final classification score. We demonstrate that networks trained with this dual embedding and classification loss outperform networks with classification loss only. %We also study placing the embedding loss on different network layers. We also find that moving the embedding loss from commonly-used feature space to the classifier space, which is the space just before softmax nonlinearization, leads to the best classification performance for aerial images. Visualizations of the network's embedded representations reveal that the embedding loss encourages greater separation between target class clusters for both training and testing partitions of two aerial image classification benchmark datasets, MSTAR and AID. Our code is publicly available on GitHub.
NSML: A Machine Learning Platform That Enables You to Focus on Your Models
Authors: Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngkwan Kim, Gayoung Lee, Dong-Hyun Kwak, Jung-Woo Ha, Sunghun Kim
Date: 2017-12-16 00:00:00
Tasks: BIG-bench Machine Learning, GPU

Machine learning libraries such as TensorFlow and PyTorch simplify model
implementation. However, researchers are still required to perform a
non-trivial amount of manual tasks such as GPU allocation, training status
tracking, and comparison of models with different hyperparameter settings. We
propose a system to handle these tasks and help researchers focus on models. We
present the requirements of the system based on a collection of discussions
from an online study group comprising 25k members. These include automatic GPU
allocation, learning status visualization, handling model parameter snapshots
as well as hyperparameter modification during learning, and comparison of
performance metrics between models via a leaderboard. We describe the system
architecture that fulfills these requirements and present a proof-of-concept
implementation, NAVER Smart Machine Learning (NSML). We test the system and
confirm substantial efficiency improvements for model development.
Automatic Music Highlight Extraction using Convolutional Recurrent Attention Networks
Authors: Jung-Woo Ha, Adrian Kim, Chanju Kim, Jang-Yeon Park, Sunghun Kim
Date: 2017-12-16 00:00:00

Music highlights are valuable contents for music services. Most methods
focused on low-level signal features. We propose a method for extracting
highlights using high-level features from convolutional recurrent attention
networks (CRAN). CRAN utilizes convolution and recurrent layers for sequential
learning with an attention mechanism. The attention allows CRAN to capture
significant snippets for distinguishing between genres, thus being used as a
high-level feature. CRAN was evaluated on over 32,000 popular tracks in Korea
for two months. Experimental results show our method outperforms three baseline
methods through quantitative and qualitative evaluations. Also, we analyze the
effects of attention and sequence information on performance.
RubyStar: A Non-Task-Oriented Mixture Model Dialog System
Authors: Huiting Liu, Tao Lin, Hanfei Sun, Weijian Lin, Chih-Wei Chang, Teng Zhong, Alexander Rudnicky
Date: 2017-11-08 00:00:00
Tasks: Question Answering, Response Generation, Retrieval

RubyStar is a dialog system designed to create "human-like" conversation by
combining different response generation strategies. RubyStar conducts a
non-task-oriented conversation on general topics by using an ensemble of
rule-based, retrieval-based and generative methods. Topic detection, engagement
monitoring, and context tracking are used for managing interaction. Predictable
elements of conversation, such as the bot's backstory and simple question
answering are handled by separate modules. We describe a rating scheme we
developed for evaluating response generation. We find that character-level RNN
is an effective generation model for general responses, with proper parameter
settings; however other kinds of conversation topics might benefit from using
other models.
Sentiment analysis of twitter data
Authors: Hamid Bagheri, Md Johirul Islam
Date: 2017-11-15 00:00:00
Tasks: Sentiment Analysis

Social networks are the main resources to gather information about people's
opinion and sentiments towards different topics as they spend hours daily on
social media and share their opinion. In this technical paper, we show the
application of sentimental analysis and how to connect to Twitter and run
sentimental analysis queries. We run experiments on different queries from
politics to humanity and show the interesting results. We realized that the
neutral sentiments for tweets are significantly high which clearly shows the
limitations of the current works.
Impression Network for Video Object Detection
Authors: Congrui Hetang, Hongwei Qin, Shaohui Liu, Junjie Yan
Date: 2017-12-16 00:00:00
Tasks: Object, object-detection, Object Detection, Video Object Detection

Video object detection is more challenging compared to image object
detection. Previous works proved that applying object detector frame by frame
is not only slow but also inaccurate. Visual clues get weakened by defocus and
motion blur, causing failure on corresponding frames. Multi-frame feature
fusion methods proved effective in improving the accuracy, but they
dramatically sacrifice the speed. Feature propagation based methods proved
effective in improving the speed, but they sacrifice the accuracy. So is it
possible to improve speed and performance simultaneously?
  Inspired by how human utilize impression to recognize objects from blurry
frames, we propose Impression Network that embodies a natural and efficient
feature aggregation mechanism. In our framework, an impression feature is
established by iteratively absorbing sparsely extracted frame features. The
impression feature is propagated all the way down the video, helping enhance
features of low-quality frames. This impression mechanism makes it possible to
perform long-range multi-frame feature fusion among sparse keyframes with
minimal overhead. It significantly improves per-frame detection baseline on
ImageNet VID while being 3 times faster (20 fps). We hope Impression Network
can provide a new perspective on video feature enhancement. Code will be made
available.
Deep Prior
Authors: Alexandre Lacoste, Thomas Boquet, Negar Rostamzadeh, Boris Oreshkin, Wonchang Chung, David Krueger
Date: 2017-12-13 00:00:00

The recent literature on deep learning offers new tools to learn a rich
probability distribution over high dimensional data such as images or sounds.
In this work we investigate the possibility of learning the prior distribution
over neural network parameters using such tools. Our resulting variational
Bayes algorithm generalizes well to new tasks, even when very few training
examples are provided. Furthermore, this learned prior allows the model to
extrapolate correctly far from a given task's training data on a meta-dataset
of periodic signals.
Mitigating Asymmetric Nonlinear Weight Update Effects in Hardware Neural Network based on Analog Resistive Synapse
Authors: Chih-Cheng Chang, Pin-Chun Chen, Teyuh Chou, I-Ting Wang, Boris Hudec, Che-Chia Chang, Chia-Ming Tsai, Tian-Sheuan Chang, Tuo-Hung Hou
Date: 2017-12-16 00:00:00
Tasks: General Classification

Asymmetric nonlinear weight update is considered as one of the major
obstacles for realizing hardware neural networks based on analog resistive
synapses because it significantly compromises the online training capability.
This paper provides new solutions to this critical issue through
co-optimization with the hardware-applicable deep-learning algorithms. New
insights on engineering activation functions and a threshold weight update
scheme effectively suppress the undesirable training noise induced by
inaccurate weight update. We successfully trained a two-layer perceptron
network online and improved the classification accuracy of MNIST handwritten
digit dataset to 87.8/94.8% by using 6-bit/8-bit analog synapses, respectively,
with extremely high asymmetric nonlinearity.
Ray: A Distributed Framework for Emerging AI Applications
Authors: Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael. I. Jordan, Ion Stoica
Date: 2017-12-16 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

The next generation of AI applications will continuously interact with the
environment and learn from these interactions. These applications impose new
and demanding systems requirements, both in terms of performance and
flexibility. In this paper, we consider these requirements and present Ray---a
distributed system to address them. Ray implements a unified interface that can
express both task-parallel and actor-based computations, supported by a single
dynamic execution engine. To meet the performance requirements, Ray employs a
distributed scheduler and a distributed and fault-tolerant store to manage the
system's control state. In our experiments, we demonstrate scaling beyond 1.8
million tasks per second and better performance than existing specialized
systems for several challenging reinforcement learning applications.
On reproduction of On the regularization of Wasserstein GANs
Authors: Junghoon Seo, Taegyun Jeon
Date: 2017-12-16 00:00:00

This report has several purposes. First, our report is written to investigate
the reproducibility of the submitted paper On the regularization of Wasserstein
GANs (2018). Second, among the experiments performed in the submitted paper,
five aspects were emphasized and reproduced: learning speed, stability,
robustness against hyperparameter, estimating the Wasserstein distance, and
various sampling method. Finally, we identify which parts of the contribution
can be reproduced, and at what cost in terms of resources. All source code for
reproduction is open to the public.
Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems
Authors: Kexin Pei, Linjie Zhu, Yinzhi Cao, Junfeng Yang, Carl Vondrick, Suman Jana
Date: 2017-12-05 00:00:00
Tasks: BIG-bench Machine Learning, Medical Diagnosis

Due to the increasing usage of machine learning (ML) techniques in security- and safety-critical domains, such as autonomous systems and medical diagnosis, ensuring correct behavior of ML systems, especially for different corner cases, is of growing importance. In this paper, we propose a generic framework for evaluating security and robustness of ML systems using different real-world safety properties. We further design, implement and evaluate VeriVis, a scalable methodology that can verify a diverse set of safety properties for state-of-the-art computer vision systems with only blackbox access. VeriVis leverage different input space reduction techniques for efficient verification of different safety properties. VeriVis is able to find thousands of safety violations in fifteen state-of-the-art computer vision systems including ten Deep Neural Networks (DNNs) such as Inception-v3 and Nvidia's Dave self-driving system with thousands of neurons as well as five commercial third-party vision APIs including Google vision and Clarifai for twelve different safety properties. Furthermore, VeriVis can successfully verify local safety properties, on average, for around 31.7% of the test images. VeriVis finds up to 64.8x more violations than existing gradient-based methods that, unlike VeriVis, cannot ensure non-existence of any violations. Finally, we show that retraining using the safety violations detected by VeriVis can reduce the average number of violations up to 60.2%.
An MPI-Based Python Framework for Distributed Training with Keras
Authors: Dustin Anderson, Jean-Roch Vlimant, Maria Spiropulu
Date: 2017-12-16 00:00:00
Tasks: BIG-bench Machine Learning

We present a lightweight Python framework for distributed training of neural
networks on multiple GPUs or CPUs. The framework is built on the popular Keras
machine learning library. The Message Passing Interface (MPI) protocol is used
to coordinate the training process, and the system is well suited for job
submission at supercomputing sites. We detail the software's features, describe
its use, and demonstrate its performance on systems of varying sizes on a
benchmark problem drawn from high-energy physics research.