pwc-archive/papers-with-abstracts huggingface dataset:
Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
Authors: Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, Pascal Vincent
Date: 2018-06-11 00:00:00

Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions. For models with many parameters, the covariance matrix they are based on becomes gigantic, making them inapplicable in their original form. This has motivated research into both simple diagonal approximations and more sophisticated factored approximations such as KFAC (Heskes, 2000; Martens & Grosse, 2015; Grosse & Martens, 2016). In the present work we draw inspiration from both to propose a novel approximation that is provably better than KFAC and amendable to cheap partial updates. It consists in tracking a diagonal variance, not in parameter coordinates, but in a Kronecker-factored eigenbasis, in which the diagonal approximation is likely to be more effective. Experiments show improvements over KFAC in optimization speed for several deep network architectures.
Massively Parallel Video Networks
Authors: Joao Carreira, Viorica Patraucean, Laurent Mazare, Andrew Zisserman, Simon Osindero
Date: 2018-06-11 00:00:00
Proceeding: ECCV 2018 9
Tasks: Action Recognition, Temporal Action Localization, Video Understanding

We introduce a class of causal video understanding models that aims to
improve efficiency of video processing by maximising throughput, minimising
latency, and reducing the number of clock cycles. Leveraging operation
pipelining and multi-rate clocks, these models perform a minimal amount of
computation (e.g. as few as four convolutional layers) for each frame per
timestep to produce an output. The models are still very deep, with dozens of
such operations being performed but in a pipelined fashion that enables
depth-parallel computation. We illustrate the proposed principles by applying
them to existing image architectures and analyse their behaviour on two video
tasks: action recognition and human keypoint localisation. The results show
that a significant degree of parallelism, and implicitly speedup, can be
achieved with little loss in performance.
Global Convergence of Block Coordinate Descent in Deep Learning
Authors: Jinshan Zeng, Tim Tsz-Kit Lau, Shao-Bo Lin, Yuan YAO
Date: 2018-03-01 00:00:00
Tasks: Deep Learning

Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in deep neural network (DNN) training. However, theoretical studies on their convergence properties are limited due to the highly nonconvex nature of DNN training. In this paper, we aim at providing a general methodology for provable convergence guarantees for this type of methods. In particular, for most of the commonly used DNN training models involving both two- and three-splitting schemes, we establish the global convergence to a critical point at a rate of ${\cal O}(1/k)$, where $k$ is the number of iterations. The results extend to general loss functions which have Lipschitz continuous gradients and deep residual networks (ResNets). Our key development adds several new elements to the Kurdyka-{\L}ojasiewicz inequality framework that enables us to carry out the global convergence analysis of BCD in the general scenario of deep learning.
A More Human Way to Play Computer Chess
Authors: Kieran Greer
Date: 2015-03-14 00:00:00

This paper suggests a forward-pruning technique for computer chess that uses
'Move Tables', which are like Transposition Tables, but for moves not
positions. They use an efficient memory structure and has put the design into
the context of long and short-term memories. The long-term memory updates a
play path with weight reinforcement, while the short-term memory can be
immediately added or removed. With this, 'long branches' can play a short path,
before returning to a full search at the resulting leaf nodes. Re-using an
earlier search path allows the tree to be forward-pruned, which is known to be
dangerous, because it removes part of the search process. Additional checks are
therefore made and moves can even be re-added when the search result is
unsatisfactory. Automatic feature analysis is now central to the algorithm,
where key squares and related squares can be generated automatically and used
to guide the search process. Using this analysis, if a search result is
inferior, it can re-insert un-played moves that cover these key squares only.
On the tactical side, a type of move that the forward-pruning will fail on is
recognised and a pattern-based solution to that problem is suggested. This has
completed the theory of an earlier paper and resulted in a more human-like
approach to searching for a chess move. Tests demonstrate that the obvious
blunders associated with forward pruning are no longer present and that it can
compete at the top level with regard to playing strength.
DeepFirearm: Learning Discriminative Feature Representation for Fine-grained Firearm Retrieval
Authors: Jiedong Hao, Jing Dong, Wei Wang, Tieniu Tan
Date: 2018-06-08 00:00:00
Tasks: Image Retrieval, Retrieval, Triplet

There are great demands for automatically regulating inappropriate appearance
of shocking firearm images in social media or identifying firearm types in
forensics. Image retrieval techniques have great potential to solve these
problems. To facilitate research in this area, we introduce Firearm 14k, a
large dataset consisting of over 14,000 images in 167 categories. It can be
used for both fine-grained recognition and retrieval of firearm images. Recent
advances in image retrieval are mainly driven by fine-tuning state-of-the-art
convolutional neural networks for retrieval task. The conventional single
margin contrastive loss, known for its simplicity and good performance, has
been widely used. We find that it performs poorly on the Firearm 14k dataset
due to: (1) Loss contributed by positive and negative image pairs is unbalanced
during training process. (2) A huge domain gap exists between this dataset and
ImageNet. We propose to deal with the unbalanced loss by employing a double
margin contrastive loss. We tackle the domain gap issue with a two-stage
training strategy, where we first fine-tune the network for classification, and
then fine-tune it for retrieval. Experimental results show that our approach
outperforms the conventional single margin approach by a large margin (up to
88.5% relative improvement) and even surpasses the strong triplet-loss-based
approach.
Deep Learning for Classification Tasks on Geospatial Vector Polygons
Authors: Rein van 't Veer, Peter Bloem, Erwin Folmer
Date: 2018-06-11 00:00:00
Tasks: BIG-bench Machine Learning, Classification, Deep Learning, General Classification

In this paper, we evaluate the accuracy of deep learning approaches on geospatial vector geometry classification tasks. The purpose of this evaluation is to investigate the ability of deep learning models to learn from geometry coordinates directly. Previous machine learning research applied to geospatial polygon data did not use geometries directly, but derived properties thereof. These are produced by way of extracting geometry properties such as Fourier descriptors. Instead, our introduced deep neural net architectures are able to learn on sequences of coordinates mapped directly from polygons. In three classification tasks we show that the deep learning architectures are competitive with common learning algorithms that require extracted features.
Data augmentation instead of explicit regularization
Authors: Alex Hernández-García, Peter König
Date: 2018-06-11 00:00:00
Proceeding: ICLR 2018 1
Tasks: Data Augmentation, Object Categorization

Contrary to most machine learning models, modern deep artificial neural networks typically include multiple components that contribute to regularization. Despite the fact that some (explicit) regularization techniques, such as weight decay and dropout, require costly fine-tuning of sensitive hyperparameters, the interplay between them and other elements that provide implicit regularization is not well understood yet. Shedding light upon these interactions is key to efficiently using computational resources and may contribute to solving the puzzle of generalization in deep learning. Here, we first provide formal definitions of explicit and implicit regularization that help understand essential differences between techniques. Second, we contrast data augmentation with weight decay and dropout. Our results show that visual object categorization models trained with data augmentation alone achieve the same performance or higher than models trained also with weight decay and dropout, as is common practice. We conclude that the contribution on generalization of weight decay and dropout is not only superfluous when sufficient implicit regularization is provided, but also such techniques can dramatically deteriorate the performance if the hyperparameters are not carefully tuned for the architecture and data set. In contrast, data augmentation systematically provides large generalization gains and does not require hyperparameter re-tuning. In view of our results, we suggest to optimize neural networks without weight decay and dropout to save computational resources, hence carbon emissions, and focus more on data augmentation and other inductive biases to improve performance and robustness.
Synthetic Perfusion Maps: Imaging Perfusion Deficits in DSC-MRI with Deep Learning
Authors: Andreas Hess, Raphael Meier, Johannes Kaesmacher, Simon Jung, Fabien Scalzo, David Liebeskind, Roland Wiest, Richard McKinley
Date: 2018-06-11 00:00:00

In this work, we present a novel convolutional neural net- work based method
for perfusion map generation in dynamic suscepti- bility contrast-enhanced
perfusion imaging. The proposed architecture is trained end-to-end and solely
relies on raw perfusion data for inference. We used a dataset of 151 acute
ischemic stroke cases for evaluation. Our method generates perfusion maps that
are comparable to the target maps used for clinical routine, while being
model-free, fast, and less noisy.
A Multimodal Classifier Generative Adversarial Network for Carry and Place Tasks from Ambiguous Language Instructions
Authors: Aly Magassouba, Komei Sugiura, Hisashi Kawai
Date: 2018-06-11 00:00:00
Tasks: Generative Adversarial Network

This paper focuses on a multimodal language understanding method for
carry-and-place tasks with domestic service robots. We address the case of
ambiguous instructions, that is, when the target area is not specified. For
instance "put away the milk and cereal" is a natural instruction where there is
ambiguity regarding the target area, considering environments in daily life.
Conventionally, this instruction can be disambiguated from a dialogue system,
but at the cost of time and cumbersome interaction. Instead, we propose a
multimodal approach, in which the instructions are disambiguated using the
robot's state and environment context. We develop the Multi-Modal Classifier
Generative Adversarial Network (MMC-GAN) to predict the likelihood of different
target areas considering the robot's physical limitation and the target
clutter. Our approach, MMC-GAN, significantly improves accuracy compared with
baseline methods that use instructions only or simple deep neural networks.
Robustness via Deep Low-Rank Representations
Authors: Amartya Sanyal, Varun Kanade, Philip H. S. Torr, Puneet K. Dokania
Date: 2018-04-19 00:00:00
Proceeding: ICLR 2019 5
Tasks: Clustering, General Classification, Image Classification, Transfer Learning

We investigate the effect of the dimensionality of the representations learned in Deep Neural Networks (DNNs) on their robustness to input perturbations, both adversarial and random. To achieve low dimensionality of learned representations, we propose an easy-to-use, end-to-end trainable, low-rank regularizer (LR) that can be applied to any intermediate layer representation of a DNN. This regularizer forces the feature representations to (mostly) lie in a low-dimensional linear subspace. We perform a wide range of experiments that demonstrate that the LR indeed induces low rank on the representations, while providing modest improvements to accuracy as an added benefit. Furthermore, the learned features make the trained model significantly more robust to input perturbations such as Gaussian and adversarial noise (even without adversarial training). Lastly, the low-dimensionality means that the learned features are highly compressible; thus discriminative features of the data can be stored using very little memory. Our experiments indicate that models trained using the LR learn robust classifiers by discovering subspaces that avoid non-robust features. Algorithmically, the LR is scalable, generic, and straightforward to implement into existing deep learning frameworks.
Multi-Document Summarization using Distributed Bag-of-Words Model
Authors: Kaustubh Mani, Ishan Verma, Hardik Meisheri, Lipika Dey
Date: 2017-10-07 00:00:00
Tasks: Document Summarization, Multi-Document Summarization, Sentence

As the number of documents on the web is growing exponentially,
multi-document summarization is becoming more and more important since it can
provide the main ideas in a document set in short time. In this paper, we
present an unsupervised centroid-based document-level reconstruction framework
using distributed bag of words model. Specifically, our approach selects
summary sentences in order to minimize the reconstruction error between the
summary and the documents. We apply sentence selection and beam search, to
further improve the performance of our model. Experimental results on two
different datasets show significant performance gains compared with the
state-of-the-art baselines.
DMCNN: Dual-Domain Multi-Scale Convolutional Neural Network for Compression Artifacts Removal
Authors: Xiaoshuai Zhang, Wenhan Yang, Yueyu Hu, Jiaying Liu
Date: 2018-06-08 00:00:00
Tasks: Image Compression, JPEG Artifact Correction, JPEG Artifact Removal

JPEG is one of the most commonly used standards among lossy image compression
methods. However, JPEG compression inevitably introduces various kinds of
artifacts, especially at high compression rates, which could greatly affect the
Quality of Experience (QoE). Recently, convolutional neural network (CNN) based
methods have shown excellent performance for removing the JPEG artifacts. Lots
of efforts have been made to deepen the CNNs and extract deeper features, while
relatively few works pay attention to the receptive field of the network. In
this paper, we illustrate that the quality of output images can be
significantly improved by enlarging the receptive fields in many cases. One
step further, we propose a Dual-domain Multi-scale CNN (DMCNN) to take full
advantage of redundancies on both the pixel and DCT domains. Experiments show
that DMCNN sets a new state-of-the-art for the task of JPEG artifact removal.
Bayesian Model-Agnostic Meta-Learning
Authors: Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, Sungjin Ahn
Date: 2018-06-11 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Active Learning, image-classification, Image Classification, Meta-Learning, model, Reinforcement Learning, Variational Inference

Learning to infer Bayesian posterior from a few-shot dataset is an important
step towards robust meta-learning due to the model uncertainty inherent in the
problem. In this paper, we propose a novel Bayesian model-agnostic
meta-learning method. The proposed method combines scalable gradient-based
meta-learning with nonparametric variational inference in a principled
probabilistic framework. During fast adaptation, the method is capable of
learning complex uncertainty structure beyond a point estimate or a simple
Gaussian approximation. In addition, a robust Bayesian meta-update mechanism
with a new meta-loss prevents overfitting during meta-update. Remaining an
efficient gradient-based meta-learner, the method is also model-agnostic and
simple to implement. Experiment results show the accuracy and robustness of the
proposed method in various tasks: sinusoidal regression, image classification,
active learning, and reinforcement learning.
Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction
Authors: Mohit Shridhar, David Hsu
Date: 2018-06-11 00:00:00
Tasks: Question Generation, Question-Generation, Visual Grounding

This paper presents INGRESS, a robot system that follows human natural
language instructions to pick and place everyday objects. The core issue here
is the grounding of referring expressions: infer objects and their
relationships from input images and language expressions. INGRESS allows for
unconstrained object categories and unconstrained language expressions.
Further, it asks questions to disambiguate referring expressions interactively.
To achieve these, we take the approach of grounding by generation and propose a
two-stage neural network model for grounding. The first stage uses a neural
network to generate visual descriptions of objects, compares them with the
input language expression, and identifies a set of candidate objects. The
second stage uses another neural network to examine all pairwise relations
between the candidates and infers the most likely referred object. The same
neural networks are used for both grounding and question generation for
disambiguation. Experiments show that INGRESS outperformed a state-of-the-art
method on the RefCOCO dataset and in robot experiments with humans.
Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositions
Authors: Nadav Cohen, Or Sharir, Yoav Levine, Ronen Tamari, David Yakira, Amnon Shashua
Date: 2017-05-05 00:00:00
Tasks: Inductive Bias

The driving force behind convolutional networks - the most successful deep
learning architecture to date, is their expressive power. Despite its wide
acceptance and vast empirical evidence, formal analyses supporting this belief
are scarce. The primary notions for formally reasoning about expressiveness are
efficiency and inductive bias. Expressive efficiency refers to the ability of a
network architecture to realize functions that require an alternative
architecture to be much larger. Inductive bias refers to the prioritization of
some functions over others given prior knowledge regarding a task at hand. In
this paper we overview a series of works written by the authors, that through
an equivalence to hierarchical tensor decompositions, analyze the expressive
efficiency and inductive bias of various convolutional network architectural
features (depth, width, strides and more). The results presented shed light on
the demonstrated effectiveness of convolutional networks, and in addition,
provide new tools for network design.
Autofocus Layer for Semantic Segmentation
Authors: Yao Qin, Konstantinos Kamnitsas, Siddharth Ancha, Jay Nanavati, Garrison Cottrell, Antonio Criminisi, Aditya Nori
Date: 2018-05-22 00:00:00
Tasks: Brain Tumor Segmentation, Medical Image Segmentation, Organ Segmentation, Segmentation, Semantic Segmentation, Tumor Segmentation

We propose the autofocus convolutional layer for semantic segmentation with
the objective of enhancing the capabilities of neural networks for multi-scale
processing. Autofocus layers adaptively change the size of the effective
receptive field based on the processed context to generate more powerful
features. This is achieved by parallelising multiple convolutional layers with
different dilation rates, combined by an attention mechanism that learns to
focus on the optimal scales driven by context. By sharing the weights of the
parallel convolutions we make the network scale-invariant, with only a modest
increase in the number of parameters. The proposed autofocus layer can be
easily integrated into existing networks to improve a model's representational
power. We evaluate our models on the challenging tasks of multi-organ
segmentation in pelvic CT and brain tumor segmentation in MRI and achieve very
promising performance.
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
Authors: Sanjeev Arora, Nadav Cohen, Elad Hazan
Date: 2018-02-19 00:00:00
Proceeding: ICML 2018 7
Tasks: regression

Conventional wisdom in deep learning states that increasing depth improves
expressiveness but complicates optimization. This paper suggests that,
sometimes, increasing depth can speed up optimization. The effect of depth on
optimization is decoupled from expressiveness by focusing on settings where
additional layers amount to overparameterization - linear neural networks, a
well-studied model. Theoretical analysis, as well as experiments, show that
here depth acts as a preconditioner which may accelerate convergence. Even on
simple convex problems such as linear regression with $\ell_p$ loss, $p>2$,
gradient descent can benefit from transitioning to a non-convex
overparameterized objective, more than it would from some common acceleration
schemes. We also prove that it is mathematically impossible to obtain the
acceleration effect of overparametrization via gradients of any regularizer.
Know What You Don't Know: Unanswerable Questions for SQuAD
Authors: Pranav Rajpurkar, Robin Jia, Percy Liang
Date: 2018-06-11 00:00:00
Proceeding: ACL 2018 7
Tasks: Natural Language Understanding, Question Answering, Reading Comprehension

Extractive reading comprehension systems can often locate the correct answer
to a question in a context document, but they also tend to make unreliable
guesses on questions for which the correct answer is not stated in the context.
Existing datasets either focus exclusively on answerable questions, or use
automatically generated unanswerable questions that are easy to identify. To
address these weaknesses, we present SQuAD 2.0, the latest version of the
Stanford Question Answering Dataset (SQuAD). SQuAD 2.0 combines existing SQuAD
data with over 50,000 unanswerable questions written adversarially by
crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0,
systems must not only answer questions when possible, but also determine when
no answer is supported by the paragraph and abstain from answering. SQuAD 2.0
is a challenging natural language understanding task for existing models: a
strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on
SQuAD 2.0.
Addition of Code Mixed Features to Enhance the Sentiment Prediction of Song Lyrics
Authors: Gangula Rama Rohit Reddy, Radhika Mamidi
Date: 2018-06-11 00:00:00
Tasks: Language Identification, Opinion Mining, Sentiment Analysis

Sentiment analysis, also called opinion mining, is the field of study that
analyzes people's opinions,sentiments, attitudes and emotions. Songs are
important to sentiment analysis since the songs and mood are mutually dependent
on each other. Based on the selected song it becomes easy to find the mood of
the listener, in future it can be used for recommendation. The song lyric is a
rich source of datasets containing words that are helpful in analysis and
classification of sentiments generated from it. Now a days we observe a lot of
inter-sentential and intra-sentential code-mixing in songs which has a varying
impact on audience. To study this impact we created a Telugu songs dataset
which contained both Telugu-English code-mixed and pure Telugu songs. In this
paper, we classify the songs based on its arousal as exciting or non-exciting.
We develop a language identification tool and introduce code-mixing features
obtained from it as additional features. Our system with these additional
features attains 4-5% accuracy greater than traditional approaches on our
dataset.