pwc-archive/papers-with-abstracts huggingface dataset:
An Estimation and Analysis Framework for the Rasch Model
Authors: Andrew S. Lan, Mung Chiang, Christoph Studer
Date: 2018-06-09 00:00:00
Proceeding: ICML 2018 7
Tasks: Collaborative Filtering, parameter estimation, Recommendation Systems

The Rasch model is widely used for item response analysis in applications
ranging from recommender systems to psychology, education, and finance. While a
number of estimators have been proposed for the Rasch model over the last
decades, the available analytical performance guarantees are mostly asymptotic.
This paper provides a framework that relies on a novel linear minimum
mean-squared error (L-MMSE) estimator which enables an exact, nonasymptotic,
and closed-form analysis of the parameter estimation error under the Rasch
model. The proposed framework provides guidelines on the number of items and
responses required to attain low estimation errors in tests or surveys. We
furthermore demonstrate its efficacy on a number of real-world collaborative
filtering datasets, which reveals that the proposed L-MMSE estimator performs
on par with state-of-the-art nonlinear estimators in terms of predictive
performance.
Linear Spectral Estimators and an Application to Phase Retrieval
Authors: Ramina Ghods, Andrew S. Lan, Tom Goldstein, Christoph Studer
Date: 2018-06-09 00:00:00
Proceeding: ICML 2018 7
Tasks: Retrieval

Phase retrieval refers to the problem of recovering real- or complex-valued
vectors from magnitude measurements. The best-known algorithms for this problem
are iterative in nature and rely on so-called spectral initializers that
provide accurate initialization vectors. We propose a novel class of estimators
suitable for general nonlinear measurement systems, called linear spectral
estimators (LSPEs), which can be used to compute accurate initialization
vectors for phase retrieval problems. The proposed LSPEs not only provide
accurate initialization vectors for noisy phase retrieval systems with
structured or random measurement matrices, but also enable the derivation of
sharp and nonasymptotic mean-squared error bounds. We demonstrate the efficacy
of LSPEs on synthetic and real-world phase retrieval problems, and show that
our estimators significantly outperform existing methods for structured
measurement systems that arise in practice.
Hierarchical Bi-level Multi-Objective Evolution of Single- and Multi-layer Echo State Network Autoencoders for Data Representations
Authors: Naima Chouikhi, Boudour Ammar, Adel M. ALIMI
Date: 2018-06-04 00:00:00

Echo State Network (ESN) presents a distinguished kind of recurrent neural
networks. It is built upon a sparse, random and large hidden infrastructure
called reservoir. ESNs have succeeded in dealing with several non-linear
problems such as prediction, classification, etc. Thanks to its rich dynamics,
ESN is used as an Autoencoder (AE) to extract features from original data
representations. ESN is not only used with its basic single layer form but also
with the recently proposed Multi-Layer (ML) architecture. The well setting of
ESN (basic and ML) architectures and training parameters is a crucial and hard
labor task. Generally, a number of parameters (hidden neurons, sparsity rates,
input scaling) is manually altered to achieve minimum learning error. However,
this randomly hand crafted task, on one hand, may not guarantee best training
results and on the other hand, it can raise the network's complexity. In this
paper, a hierarchical bi-level evolutionary optimization is proposed to deal
with these issues. The first level includes a multi-objective architecture
optimization providing maximum learning accuracy while sustaining the
complexity at a reduced standard. Multi-objective Particle Swarm Optimization
(MOPSO) is used to optimize ESN structure in a way to provide a trade-off
between the network complexity decreasing and the accuracy increasing. A
pareto-front of optimal solutions is generated by the end of the MOPSO process.
These solutions present the set of candidates that succeeded in providing a
compromise between different objectives (learning error and network
complexity). At the second level, each of the solutions already found undergo a
mono-objective weights optimization to enhance the obtained pareto-front.
Empirical results ensure the effectiveness of the evolved ESN recurrent AEs
(basic and ML) for noisy and noise free data.
Not All Samples Are Created Equal: Deep Learning with Importance Sampling
Authors: Angelos Katharopoulos, François Fleuret
Date: 2018-03-02 00:00:00
Proceeding: ICML 2018 7
Tasks: All, image-classification, Image Classification

Deep neural network training spends most of the computation on examples that are properly handled, and could be ignored. We propose to mitigate this phenomenon with a principled importance sampling scheme that focuses computation on "informative" examples, and reduces the variance of the stochastic gradients during training. Our contribution is twofold: first, we derive a tractable upper bound to the per-sample gradient norm, and second we derive an estimator of the variance reduction achieved with importance sampling, which enables us to switch it on when it will result in an actual speedup. The resulting scheme can be used by changing a few lines of code in a standard SGD procedure, and we demonstrate experimentally, on image classification, CNN fine-tuning, and RNN training, that for a fixed wall-clock time budget, it provides a reduction of the train losses of up to an order of magnitude and a relative improvement of test errors between 5% and 17%.
Representation Learning on Graphs with Jumping Knowledge Networks
Authors: Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, Stefanie Jegelka
Date: 2018-06-09 00:00:00
Proceeding: ICML 2018 7
Tasks: Graph Attention, Node Classification, Node Property Prediction, Representation Learning

Recent deep learning approaches for representation learning on graphs follow
a neighborhood aggregation procedure. We analyze some important properties of
these models, and propose a strategy to overcome those. In particular, the
range of "neighboring" nodes that a node's representation draws from strongly
depends on the graph structure, analogous to the spread of a random walk. To
adapt to local neighborhood properties and tasks, we explore an architecture --
jumping knowledge (JK) networks -- that flexibly leverages, for each node,
different neighborhood ranges to enable better structure-aware representation.
In a number of experiments on social, bioinformatics and citation networks, we
demonstrate that our model achieves state-of-the-art performance. Furthermore,
combining the JK framework with models like Graph Convolutional Networks,
GraphSAGE and Graph Attention Networks consistently improves those models'
performance.
Cell Detection with Star-convex Polygons
Authors: Uwe Schmidt, Martin Weigert, Coleman Broaddus, Gene Myers
Date: 2018-06-09 00:00:00
Tasks: Cell Detection, Cell Segmentation, Medical Image Segmentation, Segmentation, valid

Automatic detection and segmentation of cells and nuclei in microscopy images
is important for many biological applications. Recent successful learning-based
approaches include per-pixel cell segmentation with subsequent pixel grouping,
or localization of bounding boxes with subsequent shape refinement. In
situations of crowded cells, these can be prone to segmentation errors, such as
falsely merging bordering cells or suppressing valid cell instances due to the
poor approximation with bounding boxes. To overcome these issues, we propose to
localize cell nuclei via star-convex polygons, which are a much better shape
representation as compared to bounding boxes and thus do not need shape
refinement. To that end, we train a convolutional neural network that predicts
for every pixel a polygon for the cell instance at that position. We
demonstrate the merits of our approach on two synthetic datasets and one
challenging dataset of diverse fluorescence microscopy images.
Learning to Search in Long Documents Using Document Structure
Authors: Mor Geva, Jonathan Berant
Date: 2018-06-09 00:00:00
Proceeding: COLING 2018 8
Tasks: Information Retrieval, Question Answering, Reading Comprehension, Retrieval

Reading comprehension models are based on recurrent neural networks that
sequentially process the document tokens. As interest turns to answering more
complex questions over longer documents, sequential reading of large portions
of text becomes a substantial bottleneck. Inspired by how humans use document
structure, we propose a novel framework for reading comprehension. We represent
documents as trees, and model an agent that learns to interleave quick
navigation through the document tree with more expensive answer extraction. To
encourage exploration of the document tree, we propose a new algorithm, based
on Deep Q-Network (DQN), which strategically samples tree nodes at training
time. Empirically we find our algorithm improves question answering performance
compared to DQN and a strong information-retrieval (IR) baseline, and that
ensembling our model with the IR baseline results in further gains in
performance.
Second Language Acquisition Modeling: An Ensemble Approach
Authors: Anton Osika, Susanna Nilsson, Andrii Sydorchuk, Faruk Sahin, Anders Huss
Date: 2018-06-09 00:00:00
Proceeding: WS 2018 6
Tasks: Language Acquisition

Accurate prediction of students knowledge is a fundamental building block of
personalized learning systems. Here, we propose a novel ensemble model to
predict student knowledge gaps. Applying our approach to student trace data
from the online educational platform Duolingo we achieved highest score on both
evaluation metrics for all three datasets in the 2018 Shared Task on Second
Language Acquisition Modeling. We describe our model and discuss relevance of
the task compared to how it would be setup in a production environment for
personalized education.
A Taxonomy of Network Threats and the Effect of Current Datasets on Intrusion Detection Systems
Authors: Hanan Hindy, David Brosset, Ethan Bayne, Amar Seeam, Christos Tachtatzis, Robert Atkinson, Xavier Bellekens
Date: 2018-06-09 00:00:00
Tasks: Anomaly Detection, Intrusion Detection, Network Intrusion Detection

As the world moves towards being increasingly dependent on computers and automation, building secure applications, systems and networks are some of the main challenges faced in the current decade. The number of threats that individuals and businesses face is rising exponentially due to the increasing complexity of networks and services of modern networks. To alleviate the impact of these threats, researchers have proposed numerous solutions for anomaly detection; however, current tools often fail to adapt to ever-changing architectures, associated threats and zero-day attacks. This manuscript aims to pinpoint research gaps and shortcomings of current datasets, their impact on building Network Intrusion Detection Systems (NIDS) and the growing number of sophisticated threats. To this end, this manuscript provides researchers with two key pieces of information; a survey of prominent datasets, analyzing their use and impact on the development of the past decade's Intrusion Detection Systems (IDS) and a taxonomy of network threats and associated tools to carry out these attacks. The manuscript highlights that current IDS research covers only 33.3% of our threat taxonomy. Current datasets demonstrate a clear lack of real-network threats, attack representation and include a large number of deprecated threats, which together limit the detection accuracy of current machine learning IDS approaches. The unique combination of the taxonomy and the analysis of the datasets provided in this manuscript aims to improve the creation of datasets and the collection of real-world data. As a result, this will improve the efficiency of the next generation IDS and reflect network threats more accurately within new datasets.
Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising
Authors: Junwei Pan, Jian Xu, Alfonso Lobos Ruiz, Wenliang Zhao, Shengjun Pan, Yu Sun, Quan Lu
Date: 2018-06-09 00:00:00
Tasks: Click-Through Rate Prediction, Prediction

Click-through rate (CTR) prediction is a critical task in online display advertising. The data involved in CTR prediction are typically multi-field categorical data, i.e., every feature is categorical and belongs to one and only one field. One of the interesting characteristics of such data is that features from one field often interact differently with features from different other fields. Recently, Field-aware Factorization Machines (FFMs) have been among the best performing models for CTR prediction by explicitly modeling such difference. However, the number of parameters in FFMs is in the order of feature number times field number, which is unacceptable in the real-world production systems. In this paper, we propose Field-weighted Factorization Machines (FwFMs) to model the different feature interactions between different fields in a much more memory-efficient way. Our experimental evaluations show that FwFMs can achieve competitive prediction performance with only as few as 4% parameters of FFMs. When using the same number of parameters, FwFMs can bring 0.92% and 0.47% AUC lift over FFMs on two real CTR prediction data sets.
Feature Pyramid Network for Multi-Class Land Segmentation
Authors: Selim S. Seferbekov, Vladimir I. Iglovikov, Alexander V. Buslaev, Alexey A. Shvets
Date: 2018-06-09 00:00:00
Tasks: Decoder, General Classification, Land Cover Classification, Segmentation, Semantic Segmentation

Semantic segmentation is in-demand in satellite imagery processing. Because
of the complex environment, automatic categorization and segmentation of land
cover is a challenging problem. Solving it can help to overcome many obstacles
in urban planning, environmental engineering or natural landscape monitoring.
In this paper, we propose an approach for automatic multi-class land
segmentation based on a fully convolutional neural network of feature pyramid
network (FPN) family. This network is consisted of pre-trained on ImageNet
Resnet50 encoder and neatly developed decoder. Based on validation results,
leaderboard score and our own experience this network shows reliable results
for the DEEPGLOBE - CVPR 2018 land cover classification sub-challenge.
Moreover, this network moderately uses memory that allows using GTX 1080 or
1080 TI video cards to perform whole training and makes pretty fast
predictions.
On the Universal Approximation Property and Equivalence of Stochastic Computing-based Neural Networks and Binary Neural Networks
Authors: Yanzhi Wang, Zheng Zhan, Jiayu Li, Jian Tang, Bo Yuan, Liang Zhao, Wujie Wen, Siyue Wang, Xue Lin
Date: 2018-03-14 00:00:00

Large-scale deep neural networks are both memory intensive and
computation-intensive, thereby posing stringent requirements on the computing
platforms. Hardware accelerations of deep neural networks have been extensively
investigated in both industry and academia. Specific forms of binary neural
networks (BNNs) and stochastic computing based neural networks (SCNNs) are
particularly appealing to hardware implementations since they can be
implemented almost entirely with binary operations. Despite the obvious
advantages in hardware implementation, these approximate computing techniques
are questioned by researchers in terms of accuracy and universal applicability.
Also it is important to understand the relative pros and cons of SCNNs and BNNs
in theory and in actual hardware implementations. In order to address these
concerns, in this paper we prove that the "ideal" SCNNs and BNNs satisfy the
universal approximation property with probability 1 (due to the stochastic
behavior). The proof is conducted by first proving the property for SCNNs from
the strong law of large numbers, and then using SCNNs as a "bridge" to prove
for BNNs. Based on the universal approximation property, we further prove that
SCNNs and BNNs exhibit the same energy complexity. In other words, they have
the same asymptotic energy consumption with the growing of network size. We
also provide a detailed analysis of the pros and cons of SCNNs and BNNs for
hardware implementations and conclude that SCNNs are more suitable for
hardware.
A Fast and Scalable Joint Estimator for Integrating Additional Knowledge in Learning Multiple Related Sparse Gaussian Graphical Models
Authors: Beilun Wang, Arshdeep Sekhon, Yanjun Qi
Date: 2018-06-01 00:00:00
Proceeding: ICML 2018 7
Tasks: 2k, Computational Efficiency, Structured Prediction

We consider the problem of including additional knowledge in estimating
sparse Gaussian graphical models (sGGMs) from aggregated samples, arising often
in bioinformatics and neuroimaging applications. Previous joint sGGM estimators
either fail to use existing knowledge or cannot scale-up to many tasks (large
$K$) under a high-dimensional (large $p$) situation. In this paper, we propose
a novel \underline{J}oint \underline{E}lementary \underline{E}stimator
incorporating additional \underline{K}nowledge (JEEK) to infer multiple related
sparse Gaussian Graphical models from large-scale heterogeneous data. Using
domain knowledge as weights, we design a novel hybrid norm as the minimization
objective to enforce the superposition of two weighted sparsity constraints,
one on the shared interactions and the other on the task-specific structural
patterns. This enables JEEK to elegantly consider various forms of existing
knowledge based on the domain at hand and avoid the need to design
knowledge-specific optimization. JEEK is solved through a fast and entry-wise
parallelizable solution that largely improves the computational efficiency of
the state-of-the-art $O(p^5K^4)$ to $O(p^2K^4)$. We conduct a rigorous
statistical analysis showing that JEEK achieves the same convergence rate
$O(\log(Kp)/n_{tot})$ as the state-of-the-art estimators that are much harder
to compute. Empirically, on multiple synthetic datasets and two real-world
data, JEEK outperforms the speed of the state-of-arts significantly while
achieving the same level of prediction accuracy. Available as R tool @
http://jointnets.org/
Emulating dynamic non-linear simulators using Gaussian processes
Authors: Hossein Mohammadi, Peter Challenor, Marc Goodfellow
Date: 2018-02-21 00:00:00
Tasks: Gaussian Processes, Time Series, Time Series Analysis

The dynamic emulation of non-linear deterministic computer codes where the
output is a time series, possibly multivariate, is examined. Such computer
models simulate the evolution of some real-world phenomenon over time, for
example models of the climate or the functioning of the human brain. The models
we are interested in are highly non-linear and exhibit tipping points,
bifurcations and chaotic behaviour. However, each simulation run could be too
time-consuming to perform analyses that require many runs, including
quantifying the variation in model output with respect to changes in the
inputs. Therefore, Gaussian process emulators are used to approximate the
output of the code. To do this, the flow map of the system under study is
emulated over a short time period. Then, it is used in an iterative way to
predict the whole time series. A number of ways are proposed to take into
account the uncertainty of inputs to the emulators, after fixed initial
conditions, and the correlation between them through the time series. The
methodology is illustrated with two examples: the highly non-linear dynamical
systems described by the Lorenz and Van der Pol equations. In both cases, the
predictive performance is relatively high and the measure of uncertainty
provided by the method reflects the extent of predictability in each system.
Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks
Authors: Zhihao Jia, Sina Lin, Charles R. Qi, Alex Aiken
Date: 2018-02-14 00:00:00

The past few years have witnessed growth in the computational requirements
for training deep convolutional neural networks. Current approaches parallelize
training onto multiple devices by applying a single parallelization strategy
(e.g., data or model parallelism) to all layers in a network. Although easy to
reason about, these approaches result in suboptimal runtime performance in
large-scale distributed training, since different layers in a network may
prefer different parallelization strategies. In this paper, we propose
layer-wise parallelism that allows each layer in a network to use an individual
parallelization strategy. We jointly optimize how each layer is parallelized by
solving a graph search problem. Our evaluation shows that layer-wise
parallelism outperforms state-of-the-art approaches by increasing training
throughput, reducing communication costs, achieving better scalability to
multiple GPUs, while maintaining original network accuracy.
RUSSE'2018: A Shared Task on Word Sense Induction for the Russian Language
Authors: Alexander Panchenko, Anastasiya Lopukhina, Dmitry Ustalov, Konstantin Lopukhin, Nikolay Arefyev, Alexey Leontyev, Natalia Loukachevitch
Date: 2018-03-15 00:00:00
Tasks: Word Sense Induction

The paper describes the results of the first shared task on word sense
induction (WSI) for the Russian language. While similar shared tasks were
conducted in the past for some Romance and Germanic languages, we explore the
performance of sense induction and disambiguation methods for a Slavic language
that shares many features with other Slavic languages, such as rich morphology
and virtually free word order. The participants were asked to group contexts of
a given word in accordance with its senses that were not provided beforehand.
For instance, given a word "bank" and a set of contexts for this word, e.g.
"bank is a financial institution that accepts deposits" and "river bank is a
slope beside a body of water", a participant was asked to cluster such contexts
in the unknown in advance number of clusters corresponding to, in this case,
the "company" and the "area" senses of the word "bank". For the purpose of this
evaluation campaign, we developed three new evaluation datasets based on sense
inventories that have different sense granularity. The contexts in these
datasets were sampled from texts of Wikipedia, the academic corpus of Russian,
and an explanatory dictionary of Russian. Overall, 18 teams participated in the
competition submitting 383 models. Multiple teams managed to substantially
outperform competitive state-of-the-art baselines from the previous years based
on sense embeddings.
Generalized Earley Parser: Bridging Symbolic Grammars and Sequence Data for Future Prediction
Authors: Siyuan Qi, Baoxiong Jia, Song-Chun Zhu
Date: 2018-06-09 00:00:00
Proceeding: ICML 2018 7
Tasks: Activity Prediction, Future prediction

Future predictions on sequence data (e.g., videos or audios) require the
algorithms to capture non-Markovian and compositional properties of high-level
semantics. Context-free grammars are natural choices to capture such
properties, but traditional grammar parsers (e.g., Earley parser) only take
symbolic sentences as inputs. In this paper, we generalize the Earley parser to
parse sequence data which is neither segmented nor labeled. This generalized
Earley parser integrates a grammar parser with a classifier to find the optimal
segmentation and labels, and makes top-down future predictions. Experiments
show that our method significantly outperforms other approaches for future
human activity prediction.