pwc-archive/papers-with-abstracts huggingface dataset:
A Temporally-Aware Interpolation Network for Video Frame Inpainting
Authors: Ximeng Sun, Ryan Szeto, Jason J. Corso
Date: 2018-03-20 00:00:00
Tasks: Decoder, Prediction, Video Editing, Video Inpainting, Video Prediction

We propose the first deep learning solution to video frame inpainting, a
challenging instance of the general video inpainting problem with applications
in video editing, manipulation, and forensics. Our task is less ambiguous than
frame interpolation and video prediction because we have access to both the
temporal context and a partial glimpse of the future, allowing us to better
evaluate the quality of a model's predictions objectively. We devise a pipeline
composed of two modules: a bidirectional video prediction module, and a
temporally-aware frame interpolation module. The prediction module makes two
intermediate predictions of the missing frames, one conditioned on the
preceding frames and the other conditioned on the following frames, using a
shared convolutional LSTM-based encoder-decoder. The interpolation module
blends the intermediate predictions to form the final result. Specifically, it
utilizes time information and hidden activations from the video prediction
module to resolve disagreements between the predictions. Our experiments
demonstrate that our approach produces more accurate and qualitatively
satisfying results than a state-of-the-art video prediction method and many
strong frame inpainting baselines.
Acoustic feature learning using cross-domain articulatory measurements
Authors: Qingming Tang, Weiran Wang, Karen Livescu
Date: 2018-03-19 00:00:00
Tasks: speech-recognition, Speech Recognition

Previous work has shown that it is possible to improve speech recognition by
learning acoustic features from paired acoustic-articulatory data, for example
by using canonical correlation analysis (CCA) or its deep extensions. One
limitation of this prior work is that the learned feature models are difficult
to port to new datasets or domains, and articulatory data is not available for
most speech corpora. In this work we study the problem of acoustic feature
learning in the setting where we have access to an external, domain-mismatched
dataset of paired speech and articulatory measurements, either with or without
labels. We develop methods for acoustic feature learning in these settings,
based on deep variational CCA and extensions that use both source and target
domain data and labels. Using this approach, we improve phonetic recognition
accuracies on both TIMIT and Wall Street Journal and analyze a number of design
choices.
Momentum-Space Renormalization Group Transformation in Bayesian Image Modeling by Gaussian Graphical Model
Authors: Kazuyuki Tanaka, Masamichi Nakamura, Shun Kataoka, Masayuki Ohzeki, Muneki Yasuda
Date: 2018-03-20 00:00:00

A new Bayesian modeling method is proposed by combining the maximization of
the marginal likelihood with a momentum-space renormalization group
transformation for Gaussian graphical models. Moreover, we present a scheme for
computint the statistical averages of hyperparameters and mean square errors in
our proposed method based on a momentumspace renormalization transformation.
Real-time Burst Photo Selection Using a Light-Head Adversarial Network
Authors: Baoyuan Wang, Noranart Vesdapunt, Utkarsh Sinha, Lei Zhang
Date: 2018-03-20 00:00:00
Tasks: Attribute

We present an automatic moment capture system that runs in real-time on
mobile cameras. The system is designed to run in the viewfinder mode and
capture a burst sequence of frames before and after the shutter is pressed. For
each frame, the system predicts in real-time a "goodness" score, based on which
the best moment in the burst can be selected immediately after the shutter is
released, without any user interference. To solve the problem, we develop a
highly efficient deep neural network ranking model, which implicitly learns a
"latent relative attribute" space to capture subtle visual differences within a
sequence of burst images. Then the overall goodness is computed as a linear
aggregation of the goodnesses of all the latent attributes. The latent relative
attributes and the aggregation function can be seamlessly integrated in one
fully convolutional network and trained in an end-to-end fashion. To obtain a
compact model which can run on mobile devices in real-time, we have explored
and evaluated a wide range of network design choices, taking into account the
constraints of model size, computational cost, and accuracy. Extensive studies
show that the best frame predicted by our model hit users' top-1 (out of 11 on
average) choice for $64.1\%$ cases and top-3 choices for $86.2\%$ cases.
Moreover, the model(only 0.47M Bytes) can run in real time on mobile devices,
e.g. only 13ms on iPhone 7 for one frame prediction.
Resolution and Relevance Trade-offs in Deep Learning
Authors: Juyong Song, Matteo Marsili, Junghyo Jo
Date: 2017-10-31 00:00:00
Tasks: Deep Learning

Deep learning has been successfully applied to various tasks, but its
underlying mechanism remains unclear. Neural networks associate similar inputs
in the visible layer to the same state of hidden variables in deep layers. The
fraction of inputs that are associated to the same state is a natural measure
of similarity and is simply related to the cost in bits required to represent
these inputs. The degeneracy of states with the same information cost provides
instead a natural measure of noise and is simply related the entropy of the
frequency of states, that we call relevance. Representations with minimal
noise, at a given level of similarity (resolution), are those that maximise the
relevance. A signature of such efficient representations is that frequency
distributions follow power laws. We show, in extensive numerical experiments,
that deep neural networks extract a hierarchy of efficient representations from
data, because they i) achieve low levels of noise (i.e. high relevance) and ii)
exhibit power law distributions. We also find that the layer that is most
efficient to reliably generate patterns of training data is the one for which
relevance and resolution are traded at the same price, which implies that
frequency distribution follows Zipf's law.
Time-Contrastive Networks: Self-Supervised Learning from Video
Authors: Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine
Date: 2017-04-23 00:00:00
Tasks: Metric Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Self-Supervised Learning, Video Alignment

We propose a self-supervised approach for learning representations and
robotic behaviors entirely from unlabeled videos recorded from multiple
viewpoints, and study how this representation can be used in two robotic
imitation settings: imitating object interactions from videos of humans, and
imitating human poses. Imitation of human behavior requires a
viewpoint-invariant representation that captures the relationships between
end-effectors (hands or robot grippers) and the environment, object attributes,
and body pose. We train our representations using a metric learning loss, where
multiple simultaneous viewpoints of the same observation are attracted in the
embedding space, while being repelled from temporal neighbors which are often
visually similar but functionally different. In other words, the model
simultaneously learns to recognize what is common between different-looking
images, and what is different between similar-looking images. This signal
causes our model to discover attributes that do not change across viewpoint,
but do change across time, while ignoring nuisance variables such as
occlusions, motion blur, lighting and background. We demonstrate that this
representation can be used by a robot to directly mimic human poses without an
explicit correspondence, and that it can be used as a reward function within a
reinforcement learning algorithm. While representations are learned from an
unlabeled collection of task-related videos, robot behaviors such as pouring
are learned by watching a single 3rd-person demonstration by a human. Reward
functions obtained by following the human demonstrations under the learned
representation enable efficient reinforcement learning that is practical for
real-world robotic systems. Video results, open-source code and dataset are
available at https://sermanet.github.io/imitate
Why not be Versatile? Applications of the SGNMT Decoder for Machine Translation
Authors: Felix Stahlberg, Danielle Saunders, Gonzalo Iglesias, Bill Byrne
Date: 2018-03-20 00:00:00
Proceeding: WS 2018 3
Tasks: Decoder, Machine Translation, Translation

SGNMT is a decoding platform for machine translation which allows paring
various modern neural models of translation with different kinds of constraints
and symbolic models. In this paper, we describe three use cases in which SGNMT
is currently playing an active role: (1) teaching as SGNMT is being used for
course work and student theses in the MPhil in Machine Learning, Speech and
Language Technology at the University of Cambridge, (2) research as most of the
research work of the Cambridge MT group is based on SGNMT, and (3) technology
transfer as we show how SGNMT is helping to transfer research findings from the
laboratory to the industry, eg. into a product of SDL plc.
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
Authors: Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson
Date: 2018-02-27 00:00:00
Proceeding: NeurIPS 2018 12

The loss functions of deep neural networks are complex and their geometric
properties are not well understood. We show that the optima of these complex
loss functions are in fact connected by simple curves over which training and
test accuracy are nearly constant. We introduce a training procedure to
discover these high-accuracy pathways between modes. Inspired by this new
geometric insight, we also propose a new ensembling method entitled Fast
Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in
the time required to train a single model. We achieve improved performance
compared to the recent state-of-the-art Snapshot Ensembles, on CIFAR-10,
CIFAR-100, and ImageNet.
DYAN: A Dynamical Atoms-Based Network for Video Prediction
Authors: Wenqian Liu, Abhishek Sharma, Octavia Camps, Mario Sznaier
Date: 2018-03-20 00:00:00
Proceeding: ECCV 2018 9
Tasks: Decoder, Generative Adversarial Network, Representation Learning, Video Prediction

The ability to anticipate the future is essential when making real time
critical decisions, provides valuable information to understand dynamic natural
scenes, and can help unsupervised video representation learning. State-of-art
video prediction is based on LSTM recursive networks and/or generative
adversarial network learning. These are complex architectures that need to
learn large numbers of parameters, are potentially hard to train, slow to run,
and may produce blurry predictions. In this paper, we introduce DYAN, a novel
network with very few parameters and easy to train, which produces accurate,
high quality frame predictions, significantly faster than previous approaches.
DYAN owes its good qualities to its encoder and decoder, which are designed
following concepts from systems identification theory and exploit the
dynamics-based invariants of the data. Extensive experiments using several
standard video datasets show that DYAN is superior generating frames and that
it generalizes well across domains.
A Multimodal, Full-Surround Vehicular Testbed for Naturalistic Studies and Benchmarking: Design, Calibration and Deployment
Authors: Akshay Rangesh, Kevan Yuen, Ravi Kumar Satzoda, Rakesh Nattoji Rajaram, Pujitha Gunaratne, Mohan M. Trivedi
Date: 2017-09-21 00:00:00
Tasks: Autonomous Driving, Benchmarking

Recent progress in autonomous and semi-autonomous driving has been made
possible in part through an assortment of sensors that provide the intelligent
agent with an enhanced perception of its surroundings. It has been clear for
quite some while now that for intelligent vehicles to function effectively in
all situations and conditions, a fusion of different sensor technologies is
essential. Consequently, the availability of synchronized multi-sensory data
streams are necessary to promote the development of fusion based algorithms for
low, mid and high level semantic tasks. In this paper, we provide a
comprehensive description of LISA-A: our heavily sensorized, full-surround
testbed capable of providing high quality data from a slew of synchronized and
calibrated sensors such as cameras, LIDARs, radars, and the IMU/GPS. The
vehicle has recorded over 100 hours of real world data for a very diverse set
of weather, traffic and daylight conditions. All captured data is accurately
calibrated and synchronized using timestamps, and stored safely in high
performance servers mounted inside the vehicle itself. Details on the testbed
instrumentation, sensor layout, sensor outputs, calibration and synchronization
are described in this paper.
Adaptive Polar Active Contour for Segmentation and Tracking in Ultrasound Videos
Authors: Ebrahim Karami, Mohamed Shehata, Andrew Smith
Date: 2018-03-19 00:00:00
Tasks: Segmentation

Detection of relative changes in circulating blood volume is important to
guide resuscitation and manage a variety of medical conditions including
sepsis, trauma, dialysis and congestive heart failure. Recent studies have
shown that estimates of circulating blood volume can be obtained from the
cross-sectional area (CSA) of the internal jugular vein (IJV) from ultrasound
images. However, accurate segmentation and tracking of the IJV in ultrasound
imaging is a challenging task and is significantly influenced by a number of
parameters such as the image quality, shape, and temporal variation. In this
paper, we propose a novel adaptive polar active contour (Ad-PAC) algorithm for
the segmentation and tracking of the IJV in ultrasound videos. In the proposed
algorithm, the parameters of the Ad-PAC algorithm are adapted based on the
results of segmentation in previous frames. The Ad-PAC algorithm is applied to
65 ultrasound videos captured from 13 healthy subjects, with each video
containing 450 frames. The results show that spatial and temporal adaptation of
the energy function significantly improves segmentation performance when
compared to current state-of-the-art active contour algorithms.
Diagnostic Classification Of Lung Nodules Using 3D Neural Networks
Authors: Raunak Dey, Zhongjie Lu, Yi Hong
Date: 2018-03-19 00:00:00
Tasks: Classification, Computed Tomography (CT), Diagnostic, General Classification, Lung Nodule Classification, Transfer Learning

Lung cancer is the leading cause of cancer-related death worldwide. Early
diagnosis of pulmonary nodules in Computed Tomography (CT) chest scans provides
an opportunity for designing effective treatment and making financial and care
plans. In this paper, we consider the problem of diagnostic classification
between benign and malignant lung nodules in CT images, which aims to learn a
direct mapping from 3D images to class labels. To achieve this goal, four
two-pathway Convolutional Neural Networks (CNN) are proposed, including a basic
3D CNN, a novel multi-output network, a 3D DenseNet, and an augmented 3D
DenseNet with multi-outputs. These four networks are evaluated on the public
LIDC-IDRI dataset and outperform most existing methods. In particular, the 3D
multi-output DenseNet (MoDenseNet) achieves the state-of-the-art classification
accuracy on the task of end-to-end lung nodule diagnosis. In addition, the
networks pretrained on the LIDC-IDRI dataset can be further extended to handle
smaller datasets using transfer learning. This is demonstrated on our dataset
with encouraging prediction accuracy in lung nodule classification.
A Minimalist Approach to Type-Agnostic Detection of Quadrics in Point Clouds
Authors: Tolga Birdal, Benjamin Busam, Nassir Navab, Slobodan Ilic, Peter Sturm
Date: 2018-03-19 00:00:00
Proceeding: CVPR 2018 6
Tasks: Scene Understanding

This paper proposes a segmentation-free, automatic and efficient procedure to
detect general geometric quadric forms in point clouds, where clutter and
occlusions are inevitable. Our everyday world is dominated by man-made objects
which are designed using 3D primitives (such as planes, cones, spheres,
cylinders, etc.). These objects are also omnipresent in industrial
environments. This gives rise to the possibility of abstracting 3D scenes
through primitives, thereby positions these geometric forms as an integral part
of perception and high level 3D scene understanding.
  As opposed to state-of-the-art, where a tailored algorithm treats each
primitive type separately, we propose to encapsulate all types in a single
robust detection procedure. At the center of our approach lies a closed form 3D
quadric fit, operating in both primal & dual spaces and requiring as low as 4
oriented-points. Around this fit, we design a novel, local null-space voting
strategy to reduce the 4-point case to 3. Voting is coupled with the famous
RANSAC and makes our algorithm orders of magnitude faster than its conventional
counterparts. This is the first method capable of performing a generic
cross-type multi-object primitive detection in difficult scenes. Results on
synthetic and real datasets support the validity of our method.
Unveiling the invisible - mathematical methods for restoring and interpreting illuminated manuscripts
Authors: Luca Calatroni, Marie d'Autume, Rob Hocking, Stella Panayotova, Simone Parisotto, Paola Ricciardi, Carola-Bibiane Schönlieb
Date: 2018-03-19 00:00:00
Tasks: Image Restoration

The last fifty years have seen an impressive development of mathematical
methods for the analysis and processing of digital images, mostly in the
context of photography, biomedical imaging and various forms of engineering.
The arts have been mostly overlooked in this process, apart from a few
exceptional works in the last ten years. With the rapid emergence of
digitisation in the arts, however, the arts domain is becoming increasingly
receptive to digital image processing methods and the importance of paying
attention to this therefore increases. In this paper we discuss a range of
mathematical methods for digital image restoration and digital visualisation
for illuminated manuscripts. The latter provide an interesting opportunity for
digital manipulation because they traditionally remain physically untouched. At
the same time they also serve as an example for the possibilities mathematics
and digital restoration offer as a generic and objective toolkit for the arts.
Attention-based Temporal Weighted Convolutional Neural Network for Action Recognition
Authors: Jinliang Zang, Le Wang, Ziyi Liu, Qilin Zhang, Zhenxing Niu, Gang Hua, Nanning Zheng
Date: 2018-03-19 00:00:00
Tasks: Action Recognition, Temporal Action Localization

Research in human action recognition has accelerated significantly since the
introduction of powerful machine learning tools such as Convolutional Neural
Networks (CNNs). However, effective and efficient methods for incorporation of
temporal information into CNNs are still being actively explored in the recent
literature. Motivated by the popular recurrent attention models in the research
area of natural language processing, we propose the Attention-based Temporal
Weighted CNN (ATW), which embeds a visual attention model into a temporal
weighted multi-stream CNN. This attention model is simply implemented as
temporal weighting yet it effectively boosts the recognition performance of
video representations. Besides, each stream in the proposed ATW framework is
capable of end-to-end training, with both network parameters and temporal
weights optimized by stochastic gradient descent (SGD) with backpropagation.
Our experiments show that the proposed attention mechanism contributes
substantially to the performance gains with the more discriminative snippets by
focusing on more relevant video segments.
Sparse Projection Oblique Randomer Forests
Authors: Tyler M. Tomita, James Browne, Cencheng Shen, Jaewon Chung, Jesse L. Patsolic, Benjamin Falk, Jason Yim, Carey E. Priebe, Randal Burns, Mauro Maggioni, Joshua T. Vogelstein
Date: 2015-06-10 00:00:00
Tasks: Computational Efficiency

Decision forests, including Random Forests and Gradient Boosting Trees, have recently demonstrated state-of-the-art performance in a variety of machine learning settings. Decision forests are typically ensembles of axis-aligned decision trees; that is, trees that split only along feature dimensions. In contrast, many recent extensions to decision forests are based on axis-oblique splits. Unfortunately, these extensions forfeit one or more of the favorable properties of decision forests based on axis-aligned splits, such as robustness to many noise dimensions, interpretability, or computational efficiency. We introduce yet another decision forest, called "Sparse Projection Oblique Randomer Forests" (SPORF). SPORF uses very sparse random projections, i.e., linear combinations of a small subset of features. SPORF significantly improves accuracy over existing state-of-the-art algorithms on a standard benchmark suite for classification with >100 problems of varying dimension, sample size, and number of classes. To illustrate how SPORF addresses the limitations of both axis-aligned and existing oblique decision forest methods, we conduct extensive simulated experiments. SPORF typically yields improved performance over existing decision forests, while mitigating computational efficiency and scalability and maintaining interpretability. SPORF can easily be incorporated into other ensemble methods such as boosting to obtain potentially similar gains.
Multi-Scale Video Frame-Synthesis Network with Transitive Consistency Loss
Authors: Zhe Hu, Yinglan Ma, Lizhuang Ma
Date: 2017-12-07 00:00:00
Tasks: global-optimization, Optical Flow Estimation

Traditional approaches to interpolate/extrapolate frames in a video sequence
require accurate pixel correspondences between images, e.g., using optical
flow. Their results stem on the accuracy of optical flow estimation, and could
generate heavy artifacts when flow estimation failed. Recently methods using
auto-encoder has shown impressive progress, however they are usually trained
for specific interpolation/extrapolation settings and lack of flexibility and
In order to reduce these limitations, we propose a unified network to
parameterize the interest frame position and therefore infer
interpolate/extrapolate frames within the same framework. To achieve this, we
introduce a transitive consistency loss to better regularize the network. We
adopt a multi-scale structure for the network so that the parameters can be
shared across multi-layers. Our approach avoids expensive global optimization
of optical flow methods, and is efficient and flexible for video
interpolation/extrapolation applications. Experimental results have shown that
our method performs favorably against state-of-the-art methods.