pwc-archive/papers-with-abstracts huggingface dataset:
Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets
Authors: Nelson F. Liu, Roy Schwartz, Noah A. Smith
Date: 2019-04-04 00:00:00
Proceeding: NAACL 2019 6

Several datasets have recently been constructed to expose brittleness in
models trained on existing benchmarks. While model performance on these
challenge datasets is significantly lower compared to the original benchmark,
it is unclear what particular weaknesses they reveal. For example, a challenge
dataset may be difficult because it targets phenomena that current models
cannot capture, or because it simply exploits blind spots in a model's specific
training set. We introduce inoculation by fine-tuning, a new analysis method
for studying challenge datasets by exposing models (the metaphorical patient)
to a small amount of data from the challenge dataset (a metaphorical pathogen)
and assessing how well they can adapt. We apply our method to analyze the NLI
"stress tests" (Naik et al., 2018) and the Adversarial SQuAD dataset (Jia and
Liang, 2017). We show that after slight exposure, some of these datasets are no
longer challenging, while others remain difficult. Our results indicate that
failures on challenge datasets may lead to very different conclusions about
models, training datasets, and the challenge datasets themselves.
Subject Cross Validation in Human Activity Recognition
Authors: Akbar Dehghani, Tristan Glatard, Emad Shihab
Date: 2019-04-04 00:00:00
Tasks: Activity Recognition, Human Activity Recognition

K-fold Cross Validation is commonly used to evaluate classifiers and tune
their hyperparameters. However, it assumes that data points are Independent and
Identically Distributed (i.i.d.) so that samples used in the training and test
sets can be selected randomly and uniformly. In Human Activity Recognition
datasets, we note that the samples produced by the same subjects are likely to
be correlated due to diverse factors. Hence, k-fold cross validation may
overestimate the performance of activity recognizers, in particular when
overlapping sliding windows are used. In this paper, we investigate the effect
of Subject Cross Validation on the performance of Human Activity Recognition,
both with non-overlapping and with overlapping sliding windows. Results show
that k-fold cross validation artificially increases the performance of
recognizers by about 10%, and even by 16% when overlapping windows are used. In
addition, we do not observe any performance gain from the use of overlapping
windows. We conclude that Human Activity Recognition systems should be
evaluated by Subject Cross Validation, and that overlapping windows are not
worth their extra computational cost.
Frustratingly Poor Performance of Reading Comprehension Models on Non-adversarial Examples
Authors: Soham Parikh, Ananya B. Sai, Preksha Nema, Mitesh M. Khapra
Date: 2019-04-04 00:00:00
Tasks: Reading Comprehension

When humans learn to perform a difficult task (say, reading comprehension
(RC) over longer passages), it is typically the case that their performance
improves significantly on an easier version of this task (say, RC over shorter
passages). Ideally, we would want an intelligent agent to also exhibit such a
behavior. However, on experimenting with state of the art RC models using the
standard RACE dataset, we observe that this is not true. Specifically, we see
counter-intuitive results wherein even when we show frustratingly easy examples
to the model at test time, there is hardly any improvement in its performance.
We refer to this as non-adversarial evaluation as opposed to adversarial
evaluation. Such non-adversarial examples allow us to assess the utility of
specialized neural components. For example, we show that even for easy examples
where the answer is clearly embedded in the passage, the neural components
designed for paying attention to relevant portions of the passage fail to serve
their intended purpose. We believe that the non-adversarial dataset created as
a part of this work would complement the research on adversarial evaluation and
give a more realistic assessment of the ability of RC models. All the datasets
and codes developed as a part of this work will be made publicly available.
Empirical Bayes Regret Minimization
Authors: Chih-Wei Hsu, Branislav Kveton, Ofer Meshi, Martin Mladenov, Csaba Szepesvari
Date: 2019-04-04 00:00:00

Most bandit algorithm designs are purely theoretical. Therefore, they have strong regret guarantees, but also are often too conservative in practice. In this work, we pioneer the idea of algorithm design by minimizing the empirical Bayes regret, the average regret over problem instances sampled from a known distribution. We focus on a tractable instance of this problem, the confidence interval and posterior width tuning, and propose an efficient algorithm for solving it. The tuning algorithm is analyzed and evaluated in multi-armed, linear, and generalized linear bandits. We report several-fold reductions in Bayes regret for state-of-the-art bandit algorithms, simply by optimizing over a small sample from a distribution.
Algebraic Characterization of Essential Matrices and Their Averaging in Multiview Settings
Authors: Yoni Kasten, Amnon Geifman, Meirav Galun, Ronen Basri
Date: 2019-04-04 00:00:00
Proceeding: ICCV 2019 10

Essential matrix averaging, i.e., the task of recovering camera locations and orientations in calibrated, multiview settings, is a first step in global approaches to Euclidean structure from motion. A common approach to essential matrix averaging is to separately solve for camera orientations and subsequently for camera positions. This paper presents a novel approach that solves simultaneously for both camera orientations and positions. We offer a complete characterization of the algebraic conditions that enable a unique Euclidean reconstruction of $n$ cameras from a collection of $(^n_2)$ essential matrices. We next use these conditions to formulate essential matrix averaging as a constrained optimization problem, allowing us to recover a consistent set of essential matrices given a (possibly partial) set of measured essential matrices computed independently for pairs of images. We finally use the recovered essential matrices to determine the global positions and orientations of the $n$ cameras. We test our method on common SfM datasets, demonstrating high accuracy while maintaining efficiency and robustness, compared to existing methods.
Recommendations for Datasets for Source Code Summarization
Authors: Alexander LeClair, Collin McMillan
Date: 2019-04-04 00:00:00
Proceeding: NAACL 2019 6
Tasks: Code Summarization, Sentence, Source Code Summarization

Source Code Summarization is the task of writing short, natural language
descriptions of source code. The main use for these descriptions is in software
documentation e.g. the one-sentence Java method descriptions in JavaDocs. Code
summarization is rapidly becoming a popular research problem, but progress is
restrained due to a lack of suitable datasets. In addition, a lack of community
standards for creating datasets leads to confusing and unreproducible research
results -- we observe swings in performance of more than 33% due only to
changes in dataset design. In this paper, we make recommendations for these
standards from experimental results. We release a dataset based on prior work
of over 2.1m pairs of Java methods and one sentence method descriptions from
over 28k Java projects. We describe the dataset and point out key differences
from natural language data, to guide and support future researchers.
On Direct Distribution Matching for Adapting Segmentation Networks
Authors: Georg Pichler, Jose Dolz, Ismail Ben Ayed, Pablo Piantanida
Date: 2019-04-04 00:00:00
Proceeding: MIDL 2019 7
Tasks: Brain Segmentation, Domain Adaptation, image-classification, Image Classification, Segmentation

Minimization of distribution matching losses is a principled approach to domain adaptation in the context of image classification. However, it is largely overlooked in adapting segmentation networks, which is currently dominated by adversarial models. We propose a class of loss functions, which encourage direct kernel density matching in the network-output space, up to some geometric transformations computed from unlabeled inputs. Rather than using an intermediate domain discriminator, our direct approach unifies distribution matching and segmentation in a single loss. Therefore, it simplifies segmentation adaptation by avoiding extra adversarial steps, while improving both the quality, stability and efficiency of training. We juxtapose our approach to state-of-the-art segmentation adaptation via adversarial training in the network-output space. In the challenging task of adapting brain segmentation across different magnetic resonance images (MRI) modalities, our approach achieves significantly better results both in terms of accuracy and stability.
ElimiNet: A Model for Eliminating Options for Reading Comprehension with Multiple Choice Questions
Authors: Soham Parikh, Ananya B. Sai, Preksha Nema, Mitesh M. Khapra
Date: 2019-04-04 00:00:00
Proceeding: ICLR 2018 1
Tasks: Multiple-choice, Reading Comprehension

The task of Reading Comprehension with Multiple Choice Questions, requires a
human (or machine) to read a given passage, question pair and select one of the
n given options. The current state of the art model for this task first
computes a question-aware representation for the passage and then selects the
option which has the maximum similarity with this representation. However, when
humans perform this task they do not just focus on option selection but use a
combination of elimination and selection. Specifically, a human would first try
to eliminate the most irrelevant option and then read the passage again in the
light of this new information (and perhaps ignore portions corresponding to the
eliminated option). This process could be repeated multiple times till the
reader is finally ready to select the correct option. We propose ElimiNet, a
neural network-based model which tries to mimic this process. Specifically, it
has gates which decide whether an option can be eliminated given the passage,
question pair and if so it tries to make the passage representation orthogonal
to this eliminated option (akin to ignoring portions of the passage
corresponding to the eliminated option). The model makes multiple rounds of
partial elimination to refine the passage representation and finally uses a
selection module to pick the best option. We evaluate our model on the recently
released large scale RACE dataset and show that it outperforms the current
state of the art model on 7 out of the $13$ question types in this dataset.
Further, we show that taking an ensemble of our elimination-selection based
method with a selection based method gives us an improvement of 3.1% over the
best-reported performance on this dataset.
Siamese Encoding and Alignment by Multiscale Learning with Self-Supervision
Authors: Eric Mitchell, Stefan Keselj, Sergiy Popovych, Davit Buniatyan, H. Sebastian Seung
Date: 2019-04-04 00:00:00
Tasks: Self-Supervised Learning

We propose a method of aligning a source image to a target image, where the
transform is specified by a dense vector field. The two images are encoded as
feature hierarchies by siamese convolutional nets. Then a hierarchy of aligner
modules computes the transform in a coarse-to-fine recursion. Each module
receives as input the transform that was computed by the module at the level
above, aligns the source and target encodings at the same level of the
hierarchy, and then computes an improved approximation to the transform using a
convolutional net. The entire architecture of encoder and aligner nets is
trained in a self-supervised manner to minimize the squared error between
source and target remaining after alignment. We show that siamese encoding
enables more accurate alignment than the image pyramids of SPyNet, a previous
deep learning approach to coarse-to-fine alignment. Furthermore,
self-supervision applies even without target values for the transform, unlike
the strongly supervised SPyNet. We also show that our approach outperforms
one-shot approaches to alignment, because the fine pathways in the latter
approach may fail to contribute to alignment accuracy when displacements are
large. As shown by previous one-shot approaches, good results from
self-supervised learning require that the loss function additionally penalize
non-smooth transforms. We demonstrate that "masking out" the penalty function
near discontinuities leads to correct recovery of non-smooth transforms. Our
claims are supported by empirical comparisons using images from serial section
electron microscopy of brain tissue.
Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization
Authors: Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, Christian Daniel
Date: 2019-04-04 00:00:00
Proceeding: ICLR 2020 1
Tasks: Bayesian Optimization, Gaussian Processes, Meta-Learning, Reinforcement Learning, Transfer Learning

Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the field of global black-box optimization. Readily available algorithms are typically designed to be universal optimizers and, therefore, often suboptimal for specific tasks. We propose a novel transfer learning method to obtain customized optimizers within the well-established framework of Bayesian optimization, allowing our algorithm to utilize the proven generalization capabilities of Gaussian processes. Using reinforcement learning to meta-train an acquisition function (AF) on a set of related tasks, the proposed method learns to extract implicit structural information and to exploit it for improved data-efficiency. We present experiments on a simulation-to-real transfer task as well as on several synthetic functions and on two hyperparameter search problems. The results show that our algorithm (1) automatically identifies structural properties of objective functions from available source tasks or simulations, (2) performs favourably in settings with both scarse and abundant source data, and (3) falls back to the performance level of general AFs if no particular structure is present.
Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection
Authors: Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, Anton Van Den Hengel
Date: 2019-04-04 00:00:00
Proceeding: ICCV 2019 10
Tasks: Anomaly Detection, Unsupervised Anomaly Detection

Deep autoencoder has been extensively used for anomaly detection. Training on the normal data, the autoencoder is expected to produce higher reconstruction error for the abnormal inputs than the normal ones, which is adopted as a criterion for identifying anomalies. However, this assumption does not always hold in practice. It has been observed that sometimes the autoencoder "generalizes" so well that it can also reconstruct anomalies well, leading to the miss detection of anomalies. To mitigate this drawback for autoencoder based anomaly detector, we propose to augment the autoencoder with a memory module and develop an improved autoencoder called memory-augmented autoencoder, i.e. MemAE. Given an input, MemAE firstly obtains the encoding from the encoder and then uses it as a query to retrieve the most relevant memory items for reconstruction. At the training stage, the memory contents are updated and are encouraged to represent the prototypical elements of the normal data. At the test stage, the learned memory will be fixed, and the reconstruction is obtained from a few selected memory records of the normal data. The reconstruction will thus tend to be close to a normal sample. Thus the reconstructed errors on anomalies will be strengthened for anomaly detection. MemAE is free of assumptions on the data type and thus general to be applied to different tasks. Experiments on various datasets prove the excellent generalization and high effectiveness of the proposed MemAE.
Clinically Accurate Chest X-Ray Report Generation
Authors: Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, Marzyeh Ghassemi
Date: 2019-04-04 00:00:00
Tasks: Reinforcement Learning, Text Generation

The automatic generation of radiology reports given medical radiographs has significant potential to operationally and improve clinical patient care. A number of prior works have focused on this problem, employing advanced methods from computer vision and natural language generation to produce readable reports. However, these works often fail to account for the particular nuances of the radiology domain, and, in particular, the critical importance of clinical accuracy in the resulting generated reports. In this work, we present a domain-aware automatic chest X-ray radiology report generation system which first predicts what topics will be discussed in the report, then conditionally generates sentences corresponding to these topics. The resulting system is fine-tuned using reinforcement learning, considering both readability and clinical accuracy, as assessed by the proposed Clinically Coherent Reward. We verify this system on two datasets, Open-I and MIMIC-CXR, and demonstrate that our model offers marked improvements on both language generation metrics and CheXpert assessed accuracy over a variety of competitive baselines.
A Learned Representation for Scalable Vector Graphics
Authors: Raphael Gontijo Lopes, David Ha, Douglas Eck, Jonathon Shlens
Date: 2019-04-04 00:00:00
Proceeding: ICCV 2019 10
Tasks: Vector Graphics

Dramatic advances in generative models have resulted in near photographic
quality for artificially rendered faces, animals and other objects in the
natural world. In spite of such advances, a higher level understanding of
vision and imagery does not arise from exhaustively modeling an object, but
instead identifying higher-level attributes that best summarize the aspects of
an object. In this work we attempt to model the drawing process of fonts by
building sequential generative models of vector graphics. This model has the
benefit of providing a scale-invariant representation for imagery whose latent
representation may be systematically manipulated and exploited to perform style
propagation. We demonstrate these results on a large dataset of fonts and
highlight how such a model captures the statistical dependencies and richness
of this dataset. We envision that our model can find use as a tool for graphic
designers to facilitate font design.
End-to-End Video Captioning
Authors: Silvio Olivastri, Gurkirt Singh, Fabio Cuzzolin
Date: 2019-04-04 00:00:00
Tasks: Action Recognition, Caption Generation, Decoder, Machine Translation, Temporal Action Localization, Text Generation, Video Captioning, Video Description

Building correspondences across different modalities, such as video and language, has recently become critical in many visual recognition applications, such as video captioning. Inspired by machine translation, recent models tackle this task using an encoder-decoder strategy. The (video) encoder is traditionally a Convolutional Neural Network (CNN), while the decoding (for language generation) is done using a Recurrent Neural Network (RNN). Current state-of-the-art methods, however, train encoder and decoder separately. CNNs are pretrained on object and/or action recognition tasks and used to encode video-level features. The decoder is then optimised on such static features to generate the video's description. This disjoint setup is arguably sub-optimal for input (video) to output (description) mapping. In this work, we propose to optimise both encoder and decoder simultaneously in an end-to-end fashion. In a two-stage training setting, we first initialise our architecture using pre-trained encoders and decoders -- then, the entire network is trained end-to-end in a fine-tuning stage to learn the most relevant features for video caption generation. In our experiments, we use GoogLeNet and Inception-ResNet-v2 as encoders and an original Soft-Attention (SA-) LSTM as a decoder. Analogously to gains observed in other computer vision problems, we show that end-to-end training significantly improves over the traditional, disjoint training process. We evaluate our End-to-End (EtENet) Networks on the Microsoft Research Video Description (MSVD) and the MSR Video to Text (MSR-VTT) benchmark datasets, showing how EtENet achieves state-of-the-art performance across the board.
Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions
Authors: Awni Hannun, Ann Lee, Qiantong Xu, Ronan Collobert
Date: 2019-04-04 00:00:00
Tasks: Decoder, Language Modeling, Language Modelling, Sequence-To-Sequence Speech Recognition, speech-recognition, Speech Recognition

We propose a fully convolutional sequence-to-sequence encoder architecture
with a simple and efficient decoder. Our model improves WER on LibriSpeech
while being an order of magnitude more efficient than a strong RNN baseline.
Key to our approach is a time-depth separable convolution block which
dramatically reduces the number of parameters in the model while keeping the
receptive field large. We also give a stable and efficient beam search
inference procedure which allows us to effectively integrate a language model.
Coupled with a convolutional language model, our time-depth separable
convolution architecture improves by more than 22% relative WER over the best
previously reported sequence-to-sequence results on the noisy LibriSpeech test
set.
Signal-to-Noise Ratio: A Robust Distance Metric for Deep Metric Learning
Authors: Tongtong Yuan, Weihong Deng, Jian Tang, Yinan Tang, Binghui Chen
Date: 2019-04-04 00:00:00
Proceeding: CVPR 2019 6
Tasks: Clustering, Deep Hashing, Image Clustering, Metric Learning, Retrieval, Semantic Similarity, Semantic Textual Similarity

Deep metric learning, which learns discriminative features to process image
clustering and retrieval tasks, has attracted extensive attention in recent
years. A number of deep metric learning methods, which ensure that similar
examples are mapped close to each other and dissimilar examples are mapped
farther apart, have been proposed to construct effective structures for loss
functions and have shown promising results. In this paper, different from the
approaches on learning the loss structures, we propose a robust SNR distance
metric based on Signal-to-Noise Ratio (SNR) for measuring the similarity of
image pairs for deep metric learning. By exploring the properties of our SNR
distance metric from the view of geometry space and statistical theory, we
analyze the properties of our metric and show that it can preserve the semantic
similarity between image pairs, which well justify its suitability for deep
metric learning. Compared with Euclidean distance metric, our SNR distance
metric can further jointly reduce the intra-class distances and enlarge the
inter-class distances for learned features. Leveraging our SNR distance metric,
we propose Deep SNR-based Metric Learning (DSML) to generate discriminative
feature embeddings. By extensive experiments on three widely adopted
benchmarks, including CARS196, CUB200-2011 and CIFAR10, our DSML has shown its
superiority over other state-of-the-art methods. Additionally, we extend our
SNR distance metric to deep hashing learning, and conduct experiments on two
benchmarks, including CIFAR10 and NUS-WIDE, to demonstrate the effectiveness
and generality of our SNR distance metric.