pwc-archive/papers-with-abstracts huggingface dataset:
Data-parallel distributed training of very large models beyond GPU capacity
Authors: Samuel Matzek, Max Grossman, Minsik Cho, Anar Yusifov, Bryant Nelson, Amit Juneja
Date: 2018-11-29 00:00:00
Tasks: CPU, GPU

GPUs have limited memory and it is difficult to train wide and/or deep models
that cause the training process to go out of memory. It is shown in this paper
how an open source tool called Large Model Support (LMS) can utilize a high
bandwidth NVLink connection between CPUs and GPUs to accomplish training of
deep convolutional networks. LMS performs tensor swapping between CPU memory
and GPU memory such that only a minimal number of tensors required in a
training step are kept in the GPU memory. It is also shown how LMS can be
combined with an MPI based distributed deep learning module to train models in
a data-parallel fashion across multiple GPUs, such that each GPU is utilizing
the CPU memory for tensor swapping. The hardware architecture that enables the
high bandwidth GPU link with the CPU is discussed as well as the associated set
of software tools that are available as the PowerAI package.
Efficient Coarse-to-Fine Non-Local Module for the Detection of Small Objects
Authors: Hila Levi, Shimon Ullman
Date: 2018-11-29 00:00:00
Tasks: object-detection, Object Detection, Relational Reasoning

An image is not just a collection of objects, but rather a graph where each object is related to other objects through spatial and semantic relations. Using relational reasoning modules, such as the non-local module \cite{wang2017non}, can therefore improve object detection. Current schemes apply such dedicated modules either to a specific layer of the bottom-up stream, or between already-detected objects. We show that the relational process can be better modeled in a coarse-to-fine manner and present a novel framework, applying a non-local module sequentially to increasing resolution feature maps along the top-down stream. In this way, information can naturally passed from larger objects to smaller related ones. Applying the module to fine feature maps further allows the information to pass between the small objects themselves, exploiting repetitions of instances of the same class. In practice, due to the expensive memory utilization of the non-local module, it is infeasible to apply the module as currently used to high-resolution feature maps. We redesigned the non local module, improved it in terms of memory and number of operations, allowing it to be placed anywhere along the network. We further incorporated relative spatial information into the module, in a manner that can be incorporated into our efficient implementation. We show the effectiveness of our scheme by improving the results of detecting small objects on COCO by 1-2 AP points over Faster and Mask RCNN and by 1 AP over using non-local module on the bottom-up stream.
Parameter-Free Spatial Attention Network for Person Re-Identification
Authors: Haoran Wang, Yue Fan, Zexin Wang, Licheng Jiao, Bernt Schiele
Date: 2018-11-29 00:00:00
Tasks: Person Re-Identification

Global average pooling (GAP) allows to localize discriminative information
for recognition [40]. While GAP helps the convolution neural network to attend
to the most discriminative features of an object, it may suffer if that
information is missing e.g. due to camera viewpoint changes. To circumvent this
issue, we argue that it is advantageous to attend to the global configuration
of the object by modeling spatial relations among high-level features. We
propose a novel architecture for Person Re-Identification, based on a novel
parameter-free spatial attention layer introducing spatial relations among the
feature map activations back to the model. Our spatial attention layer
consistently improves the performance over the model without it. Results on
four benchmarks demonstrate a superiority of our model over the
state-of-the-art achieving rank-1 accuracy of 94.7% on Market-1501, 89.0% on
DukeMTMC-ReID, 74.9% on CUHK03-labeled and 69.7% on CUHK03-detected.
Improving Robustness of Neural Dialog Systems in a Data-Efficient Way with Turn Dropout
Authors: Igor Shalyminov, Sungjin Lee
Date: 2018-11-29 00:00:00

Neural network-based dialog models often lack robustness to anomalous,
out-of-domain (OOD) user input which leads to unexpected dialog behavior and
thus considerably limits such models' usage in mission-critical production
environments. The problem is especially relevant in the setting of dialog
system bootstrapping with limited training data and no access to OOD examples.
In this paper, we explore the problem of robustness of such systems to
anomalous input and the associated to it trade-off in accuracies on seen and
unseen data. We present a new dataset for studying the robustness of dialog
systems to OOD input, which is bAbI Dialog Task 6 augmented with OOD content in
a controlled way. We then present turn dropout, a simple yet efficient negative
sampling-based technique for improving robustness of neural dialog models. We
demonstrate its effectiveness applied to Hybrid Code Network-family models
(HCNs) which reach state-of-the-art results on our OOD-augmented dataset as
well as the original one. Specifically, an HCN trained with turn dropout
achieves state-of-the-art performance of more than 75% per-utterance accuracy
on the augmented dataset's OOD turns and 74% F1-score as an OOD detector.
Furthermore, we introduce a Variational HCN enhanced with turn dropout which
achieves more than 56.5% accuracy on the original bAbI Task 6 dataset, thus
outperforming the initially reported HCN's result.
Game Tree Search in a Robust Multistage Optimization Framework: Exploiting Pruning Mechanisms
Authors: Michael Hartisch, Ulf Lorenz
Date: 2018-11-29 00:00:00

We investigate pruning in search trees of so-called quantified integer linear
programs (QIPs). QIPs consist of a set of linear inequalities and a minimax
objective function, where some variables are existentially and others are
universally quantified. They can be interpreted as two-person zero-sum games
between an existential and a universal player on the one hand, or multistage
optimization problems under uncertainty on the other hand. Solutions are
so-called winning strategies for the existential player that specify how to
react on moves of the universal player - i.e. certain assignments of
universally quantified variables - to certainly win the game.
  QIPs can be solved with the help of game tree search that is enhanced with
non-chronological back-jumping. We develop and theoretically substantiate
pruning techniques based upon (algebraic) properties similar to pruning
mechanisms known from linear programming and quantified boolean formulas. The
presented Strategic Copy-Pruning mechanism allows to \textit{implicitly} deduce
the existence of a strategy in linear time (by static examination of the
QIP-matrix) without explicitly traversing the strategy itself. We show that the
implementation of our findings can massively speed up the search process.
Learning to Reason with Third-Order Tensor Products
Authors: Imanol Schlag, Jürgen Schmidhuber
Date: 2018-11-29 00:00:00

We combine Recurrent Neural Networks with Tensor Product Representations to
learn combinatorial representations of sequential data. This improves symbolic
interpretation and systematic generalisation. Our architecture is trained
end-to-end through gradient descent on a variety of simple natural language
reasoning tasks, significantly outperforming the latest state-of-the-art models
in single-task and all-tasks settings. We also augment a subset of the data
such that training and test data exhibit large systematic differences and show
that our approach generalises better than the previous state-of-the-art.
Global optimization of expensive black-box models based on asynchronous hybrid-criterion with interval reduction
Authors: Chunlin Gong, Xu Li, Hua Su, Jinlei Guo, Liangxian Gu
Date: 2018-11-29 00:00:00
Tasks: global-optimization

In this paper, a new sequential surrogate-based optimization (SSBO) algorithm
is developed, which aims to improve the global search ability and local search
efficiency for the global optimization of expensive black-box models. The
proposed method involves three basic sub-criteria to infill new samples
asynchronously to balance the global exploration and local exploitation. First,
to capture the promising possible global optimal region, searching for the
global optimum with genetic algorithm (GA) based on the current surrogate
models of the objective and constraint functions. Second, to infill samples in
the region with sparse samples to improve the global accuracy of the surrogate
models, a grid searching with Latin hypercube sampling (LHS) with the current
surrogate model is adopted to explore the sample space. Third, to accelerate
the local searching efficiency, searching for a local optimum with sequential
quadratic programming (SQP) based on the local surrogate models in the reduced
interval, which involves some samples near the current optimum. When the new
sample is too close to the existing ones, the new sample should be abandoned,
due to the poor additional information. According to the three sub-criteria,
the new samples are placed in the regions which have not been fully explored
and includes the possible global optimum point. When a possible global optimum
point is found, the local searching sub-criterion captures the local optimum
around it rapidly. Numerical and engineering examples are used to verify the
efficiency of the proposed method. The statistical results show that the
proposed method has good global searching ability and efficiency.
Two-level Attention with Two-stage Multi-task Learning for Facial Emotion Recognition
Authors: Xiaohua Wang, Muzi Peng, Lijuan Pan, Min Hu, Chunhua Jin, Fuji Ren
Date: 2018-11-29 00:00:00
Tasks: Emotion Recognition, Facial Emotion Recognition, Multi-Task Learning, Vocal Bursts Valence Prediction

Compared with facial emotion recognition on categorical model, the
dimensional emotion recognition can describe numerous emotions of the real
world more accurately. Most prior works of dimensional emotion estimation only
considered laboratory data and used video, speech or other multi-modal
features. The effect of these methods applied on static images in the real
world is unknown. In this paper, a two-level attention with two-stage
multi-task learning (2Att-2Mt) framework is proposed for facial emotion
estimation on only static images. Firstly, the features of corresponding
region(position-level features) are extracted and enhanced automatically by
first-level attention mechanism. In the following, we utilize Bi-directional
Recurrent Neural Network(Bi-RNN) with self-attention(second-level attention) to
make full use of the relationship features of different layers(layer-level
features) adaptively. Owing to the inherent complexity of dimensional emotion
recognition, we propose a two-stage multi-task learning structure to exploited
categorical representations to ameliorate the dimensional representations and
estimate valence and arousal simultaneously in view of the correlation of the
two targets. The quantitative results conducted on AffectNet dataset show
significant advancement on Concordance Correlation Coefficient(CCC) and Root
Mean Square Error(RMSE), illustrating the superiority of the proposed
framework. Besides, extensive comparative experiments have also fully
demonstrated the effectiveness of different components.
Scaling up Probabilistic Inference in Linear and Non-Linear Hybrid Domains by Leveraging Knowledge Compilation
Authors: Anton Fuxjaeger, Vaishak Belle
Date: 2018-11-29 00:00:00
Tasks: Attribute

Weighted model integration (WMI) extends weighted model counting (WMC) in providing a computational abstraction for probabilistic inference in mixed discrete-continuous domains. WMC has emerged as an assembly language for state-of-the-art reasoning in Bayesian networks, factor graphs, probabilistic programs and probabilistic databases. In this regard, WMI shows immense promise to be much more widely applicable, especially as many real-world applications involve attribute and feature spaces that are continuous and mixed. Nonetheless, state-of-the-art tools for WMI are limited and less mature than their propositional counterparts. In this work, we propose a new implementation regime that leverages propositional knowledge compilation for scaling up inference. In particular, we use sentential decision diagrams, a tractable representation of Boolean functions, as the underlying model counting and model enumeration scheme. Our regime performs competitively to state-of-the-art WMI systems but is also shown to handle a specific class of non-linear constraints over non-linear potentials.
A Machine-Learning Phase Classification Scheme for Anomaly Detection in Signals with Periodic Characteristics
Authors: Lia Ahrens, Julian Ahrens, Hans D. Schotten
Date: 2018-11-29 00:00:00
Tasks: Anomaly Detection, BIG-bench Machine Learning, General Classification, Intrusion Detection, Time Series, Time Series Analysis

In this paper we propose a novel machine-learning method for anomaly
detection applicable to data with periodic characteristics where randomly
varying period lengths are explicitly allowed. A multi-dimensional time series
analysis is conducted by training a data-adapted classifier consisting of deep
convolutional neural networks performing phase classification. The entire
algorithm including data pre-processing, period detection, segmentation, and
even dynamic adjustment of the neural networks is implemented for fully
automatic execution. The proposed method is evaluated on three example datasets
from the areas of cardiology, intrusion detection, and signal processing,
presenting reasonable performance.
Non-entailed subsequences as a challenge for natural language inference
Authors: R. Thomas McCoy, Tal Linzen
Date: 2018-11-29 00:00:00
Tasks: Natural Language Inference, Sentence

Neural network models have shown great success at natural language inference
(NLI), the task of determining whether a premise entails a hypothesis. However,
recent studies suggest that these models may rely on fallible heuristics rather
than deep language understanding. We introduce a challenge set to test whether
NLI systems adopt one such heuristic: assuming that a sentence entails all of
its subsequences, such as assuming that "Alice believes Mary is lying" entails
"Alice believes Mary." We evaluate several competitive NLI models on this
challenge set and find strong evidence that they do rely on the subsequence
heuristic.
Bootstrapping Deep Neural Networks from Approximate Image Processing Pipelines
Authors: Kilho Son, Jesse Hostetler, Sek Chai
Date: 2018-11-29 00:00:00

Complex image processing and computer vision systems often consist of a
processing pipeline of functional modules. We intend to replace parts or all of
a target pipeline with deep neural networks to achieve benefits such as
increased accuracy or reduced computational requirement. To acquire a large
amount of labeled data necessary to train the deep neural network, we propose a
workflow that leverages the target pipeline to create a significantly larger
labeled training set automatically, without prior domain knowledge of the
target pipeline. We show experimentally that despite the noise introduced by
automated labeling and only using a very small initially labeled data set, the
trained deep neural networks can achieve similar or even better performance
than the components they replace, while in some cases also reducing
computational requirements.
Generating Easy-to-Understand Referring Expressions for Target Identifications
Authors: Mikihiro Tanaka, Takayuki Itamochi, Kenichi Narioka, Ikuro Sato, Yoshitaka Ushiku, Tatsuya Harada
Date: 2018-11-29 00:00:00
Proceeding: ICCV 2019 10
Tasks: Referring Expression

This paper addresses the generation of referring expressions that not only refer to objects correctly but also let humans find them quickly. As a target becomes relatively less salient, identifying referred objects itself becomes more difficult. However, the existing studies regarded all sentences that refer to objects correctly as equally good, ignoring whether they are easily understood by humans. If the target is not salient, humans utilize relationships with the salient contexts around it to help listeners to comprehend it better. To derive this information from human annotations, our model is designed to extract information from the target and from the environment. Moreover, we regard that sentences that are easily understood are those that are comprehended correctly and quickly by humans. We optimized this by using the time required to locate the referred objects by humans and their accuracies. To evaluate our system, we created a new referring expression dataset whose images were acquired from Grand Theft Auto V (GTA V), limiting targets to persons. Experimental results show the effectiveness of our approach. Our code and dataset are available at https://github.com/mikittt/easy-to-understand-REG.
Networks for Nonlinear Diffusion Problems in Imaging
Authors: Simon Arridge, Andreas Hauptmann
Date: 2018-11-29 00:00:00

A multitude of imaging and vision tasks have seen recently a major
transformation by deep learning methods and in particular by the application of
convolutional neural networks. These methods achieve impressive results, even
for applications where it is not apparent that convolutions are suited to
capture the underlying physics.
  In this work we develop a network architecture based on nonlinear diffusion
processes, named DiffNet. By design, we obtain a nonlinear network architecture
that is well suited for diffusion related problems in imaging. Furthermore, the
performed updates are explicit, by which we obtain better interpretability and
generalisability compared to classical convolutional neural network
architectures. The performance of DiffNet tested on the inverse problem of
nonlinear diffusion with the Perona-Malik filter on the STL-10 image dataset.
We obtain competitive results to the established U-Net architecture, with a
fraction of parameters and necessary training data.
A Polynomial-time Fragment of Epistemic Probabilistic Argumentation (Technical Report)
Authors: Nico Potyka
Date: 2018-11-29 00:00:00

Probabilistic argumentation allows reasoning about argumentation problems in
a way that is well-founded by probability theory. However, in practice, this
approach can be severely limited by the fact that probabilities are defined by
adding an exponential number of terms. We show that this exponential blowup can
be avoided in an interesting fragment of epistemic probabilistic argumentation
and that some computational problems that have been considered intractable can
be solved in polynomial time. We give efficient convex programming formulations
for these problems and explore how far our fragment can be extended without
loosing tractability.
Deep Haar Scattering Networks in Pattern Recognition: A promising approach
Authors: Fernando Fernandes Neto, Alemayehu Admasu Solomon, Rodrigo de Losso, Claudio Garcia, Pedro Delano Cavalcanti
Date: 2018-11-29 00:00:00
Tasks: Classification, General Classification, regression, Time Series, Time Series Analysis

The aim of this paper is to discuss the use of Haar scattering networks,
which is a very simple architecture that naturally supports a large number of
stacked layers, yet with very few parameters, in a relatively broad set of
pattern recognition problems, including regression and classification tasks.
This architecture, basically, consists of stacking convolutional filters, that
can be thought as a generalization of Haar wavelets, followed by non-linear
operators which aim to extract symmetries and invariances that are later fed in
a classification/regression algorithm. We show that good results can be
obtained with the proposed method for both kind of tasks. We have outperformed
the best available algorithms in 4 out of 18 important data classification
problems, and have obtained a more robust performance than ARIMA and ETS time
series methods in regression problems for data with strong periodicities.
Multi-Scale Distributed Representation for Deep Learning and its Application to b-Jet Tagging
Authors: Jason Lee, Inkyu Park, Sangnam Park
Date: 2018-11-29 00:00:00
Tasks: Binarization, Classification with Binary Neural Network, General Classification, Jet Tagging

Recently machine learning algorithms based on deep layered artificial neural
networks (DNNs) have been applied to a wide variety of high energy physics
problems such as jet tagging or event classification. We explore a simple but
effective preprocessing step which transforms each real-valued observational
quantity or input feature into a binary number with a fixed number of digits.
Each binary digit represents the quantity or magnitude in different scales. We
have shown that this approach improves the performance of DNNs significantly
for some specific tasks without any further complication in feature
engineering. We apply this multi-scale distributed binary representation to
deep learning on b-jet tagging using daughter particles' momenta and vertex
information.
TEA-DNN: the Quest for Time-Energy-Accuracy Co-optimized Deep Neural Networks
Authors: Lile Cai, Anne-Maelle Barneche, Arthur Herbout, Chuan Sheng Foo, Jie Lin, Vijay Ramaseshan Chandrasekhar, Mohamed M. Sabry
Date: 2018-11-29 00:00:00
Tasks: General Classification, image-classification, Image Classification, Neural Architecture Search

Embedded deep learning platforms have witnessed two simultaneous improvements. First, the accuracy of convolutional neural networks (CNNs) has been significantly improved through the use of automated neural-architecture search (NAS) algorithms to determine CNN structure. Second, there has been increasing interest in developing hardware accelerators for CNNs that provide improved inference performance and energy consumption compared to GPUs. Such embedded deep learning platforms differ in the amount of compute resources and memory-access bandwidth, which would affect performance and energy consumption of CNNs. It is therefore critical to consider the available hardware resources in the network architecture search. To this end, we introduce TEA-DNN, a NAS algorithm targeting multi-objective optimization of execution time, energy consumption, and classification accuracy of CNN workloads on embedded architectures. TEA-DNN leverages energy and execution time measurements on embedded hardware when exploring the Pareto-optimal curves across accuracy, execution time, and energy consumption and does not require additional effort to model the underlying hardware. We apply TEA-DNN for image classification on actual embedded platforms (NVIDIA Jetson TX2 and Intel Movidius Neural Compute Stick). We highlight the Pareto-optimal operating points that emphasize the necessity to explicitly consider hardware characteristics in the search process. To the best of our knowledge, this is the most comprehensive study of Pareto-optimal models across a range of hardware platforms using actual measurements on hardware to obtain objective values.