pwc-archive/papers-with-abstracts huggingface dataset:
Improved Speech Reconstruction from Silent Video
Authors: Ariel Ephrat, Tavi Halperin, Shmuel Peleg
Date: 2017-08-01 00:00:00

Speechreading is the task of inferring phonetic information from visually
observed articulatory facial movements, and is a notoriously difficult task for
humans to perform. In this paper we present an end-to-end model based on a
convolutional neural network (CNN) for generating an intelligible and
natural-sounding acoustic speech signal from silent video frames of a speaking
person. We train our model on speakers from the GRID and TCD-TIMIT datasets,
and evaluate the quality and intelligibility of reconstructed speech using
common objective measurements. We show that speech predictions from the
proposed model attain scores which indicate significantly improved quality over
existing models. In addition, we show promising results towards reconstructing
speech from an unconstrained dictionary.
Adapting Models to Signal Degradation using Distillation
Authors: Jong-Chyi Su, Subhransu Maji
Date: 2016-04-01 00:00:00
Tasks: Domain Adaptation, Knowledge Distillation, Model Compression, Transfer Learning

Model compression and knowledge distillation have been successfully applied
for cross-architecture and cross-domain transfer learning. However, a key
requirement is that training examples are in correspondence across the domains.
We show that in many scenarios of practical importance such aligned data can be
synthetically generated using computer graphics pipelines allowing domain
adaptation through distillation. We apply this technique to learn models for
recognizing low-resolution images using labeled high-resolution images,
non-localized objects using labeled localized objects, line-drawings using
labeled color images, etc. Experiments on various fine-grained recognition
datasets demonstrate that the technique improves recognition performance on the
low-quality data and beats strong baselines for domain adaptation. Finally, we
present insights into workings of the technique through visualizations and
relating it to existing literature.
Visual Cues to Improve Myoelectric Control of Upper Limb Prostheses
Authors: Andrea Gigli, Arjan Gijsberts, Valentina Gregori, Matteo Cognolato, Manfredo Atzori, Barbara Caputo
Date: 2017-08-29 00:00:00
Tasks: General Classification, Object

The instability of myoelectric signals over time complicates their use to
control highly articulated prostheses. To address this problem, studies have
tried to combine surface electromyography with modalities that are less
affected by the amputation and environment, such as accelerometry or gaze
information. In the latter case, the hypothesis is that a subject looks at the
object he or she intends to manipulate and that knowing this object's
affordances allows to constrain the set of possible grasps. In this paper, we
develop an automated way to detect stable fixations and show that gaze
information is indeed helpful in predicting hand movements. In our multimodal
approach, we automatically detect stable gazes and segment an object of
interest around the subject's fixation in the visual frame. The patch extracted
around this object is subsequently fed through an off-the-shelf deep
convolutional neural network to obtain a high level feature representation,
which is then combined with traditional surface electromyography in the
classification stage. Tests have been performed on a dataset acquired from five
intact subjects who performed ten types of grasps on various objects as well as
in a functional setting. They show that the addition of gaze information
increases the classification accuracy considerably. Further analysis
demonstrates that this improvement is consistent for all grasps and
concentrated during the movement onset and offset.
Reasoning about Fine-grained Attribute Phrases using Reference Games
Authors: Jong-Chyi Su, Chenyun Wu, Huaizu Jiang, Subhransu Maji
Date: 2017-08-29 00:00:00
Proceeding: ICCV 2017 10
Tasks: Attribute, Image Retrieval, Retrieval

We present a framework for learning to describe fine-grained visual
differences between instances using attribute phrases. Attribute phrases
capture distinguishing aspects of an object (e.g., "propeller on the nose" or
"door near the wing" for airplanes) in a compositional manner. Instances within
a category can be described by a set of these phrases and collectively they
span the space of semantic attributes for a category. We collect a large
dataset of such phrases by asking annotators to describe several visual
differences between a pair of instances within a category. We then learn to
describe and ground these phrases to images in the context of a *reference
game* between a speaker and a listener. The goal of a speaker is to describe
attributes of an image that allows the listener to correctly identify it within
a pair. Data collected in a pairwise manner improves the ability of the speaker
to generate, and the ability of the listener to interpret visual descriptions.
Moreover, due to the compositionality of attribute phrases, the trained
listeners can interpret descriptions not seen during training for image
retrieval, and the speakers can generate attribute-based explanations for
differences between previously unseen categories. We also show that embedding
an image into the semantic space of attribute phrases derived from listeners
offers 20% improvement in accuracy over existing attribute-based
representations on the FGVC-aircraft dataset.
Can We See Photosynthesis? Magnifying the Tiny Color Changes of Plant Green Leaves Using Eulerian Video Magnification
Authors: Islam A. T. F. Taj-Eddin, Mahmoud Afifi, Mostafa Korashy, Ali H. Ahmed, Ng Yoke Cheng, Evelyng Hernandez, Salma M. Abdel-latif
Date: 2017-06-12 00:00:00

Plant aliveness is proven through laboratory experiments and special
scientific instruments. In this paper, we aim to detect the degree of animation
of plants based on the magnification of the small color changes in the plant's
green leaves using the Eulerian video magnification. Capturing the video under
a controlled environment, e.g., using a tripod and direct current (DC) light
sources, reduces camera movements and minimizes light fluctuations; we aim to
reduce the external factors as much as possible. The acquired video is then
stabilized and a proposed algorithm used to reduce the illumination variations.
Lastly, the Euler magnification is utilized to magnify the color changes on the
light invariant video. The proposed system does not require any special purpose
instruments as it uses a digital camera with a regular frame rate. The results
of magnified color changes on both natural and plastic leaves show that the
live green leaves have color changes in contrast to the plastic leaves. Hence,
we can argue that the color changes of the leaves are due to biological
operations, such as photosynthesis. To date, this is possibly the first work
that focuses on interpreting visually, some biological operations of plants
without any special purpose instruments.
Camera Pose Filtering with Local Regression Geodesics on the Riemannian Manifold of Dual Quaternions
Authors: Benjamin Busam, Tolga Birdal, Nassir Navab
Date: 2017-04-24 00:00:00
Tasks: Pose Tracking, regression, Translation

Time-varying, smooth trajectory estimation is of great interest to the vision
community for accurate and well behaving 3D systems. In this paper, we propose
a novel principal component local regression filter acting directly on the
Riemannian manifold of unit dual quaternions $\mathbb{D} \mathbb{H}_1$. We use
a numerically stable Lie algebra of the dual quaternions together with $\exp$
and $\log$ operators to locally linearize the 6D pose space. Unlike state of
the art path smoothing methods which either operate on $SO\left(3\right)$ of
rotation matrices or the hypersphere $\mathbb{H}_1$ of quaternions, we treat
the orientation and translation jointly on the dual quaternion quadric in the
7-dimensional real projective space $\mathbb{R}\mathbb{P}^7$. We provide an
outlier-robust IRLS algorithm for generic pose filtering exploiting this
manifold structure. Besides our theoretical analysis, our experiments on
synthetic and real data show the practical advantages of the manifold aware
filtering on pose tracking and smoothing.
Sparsity-Based Super Resolution for SEM Images
Authors: Shahar Tsiper, Or Dicker, Idan Kaizerman, Zeev Zohar, Mordechai Segev, Yonina C. Eldar
Date: 2017-08-29 00:00:00
Tasks: Dictionary Learning, Super-Resolution

The scanning electron microscope (SEM) produces an image of a sample by
scanning it with a focused beam of electrons. The electrons interact with the
atoms in the sample, which emit secondary electrons that contain information
about the surface topography and composition. The sample is scanned by the
electron beam point by point, until an image of the surface is formed. Since
its invention in 1942, SEMs have become paramount in the discovery and
understanding of the nanometer world, and today it is extensively used for both
research and in industry. In principle, SEMs can achieve resolution better than
one nanometer. However, for many applications, working at sub-nanometer
resolution implies an exceedingly large number of scanning points. For exactly
this reason, the SEM diagnostics of microelectronic chips is performed either
at high resolution (HR) over a small area or at low resolution (LR) while
capturing a larger portion of the chip. Here, we employ sparse coding and
dictionary learning to algorithmically enhance LR SEM images of microelectronic
chips up to the level of the HR images acquired by slow SEM scans, while
considerably reducing the noise. Our methodology consists of two steps: an
offline stage of learning a joint dictionary from a sequence of LR and HR
images of the same region in the chip, followed by a fast-online
super-resolution step where the resolution of a new LR image is enhanced. We
provide several examples with typical chips used in the microelectronics
industry, as well as a statistical study on arbitrary images with
characteristic structural features. Conceptually, our method works well when
the images have similar characteristics. This work demonstrates that employing
sparsity concepts can greatly improve the performance of SEM, thereby
considerably increasing the scanning throughput without compromising on
analysis quality and resolution.
Semantic Texture for Robust Dense Tracking
Authors: Jan Czarnowski, Stefan Leutenegger, Andrew Davison
Date: 2017-08-29 00:00:00

We argue that robust dense SLAM systems can make valuable use of the layers
of features coming from a standard CNN as a pyramid of `semantic texture' which
is suitable for dense alignment while being much more robust to nuisance
factors such as lighting than raw RGB values. We use a straightforward
Lucas-Kanade formulation of image alignment, with a schedule of iterations over
the coarse-to-fine levels of a pyramid, and simply replace the usual image
pyramid by the hierarchy of convolutional feature maps from a pre-trained CNN.
The resulting dense alignment performance is much more robust to lighting and
other variations, as we show by camera rotation tracking experiments on
time-lapse sequences captured over many hours. Looking towards the future of
scene representation for real-time visual SLAM, we further demonstrate that a
selection using simple criteria of a small number of the total set of features
output by a CNN gives just as accurate but much more efficient tracking
performance.
Sound-Word2Vec: Learning Word Representations Grounded in Sounds
Authors: Ashwin K. Vijayakumar, Ramakrishna Vedantam, Devi Parikh
Date: 2017-03-06 00:00:00
Proceeding: EMNLP 2017 9
Tasks: Retrieval, Word Embeddings

To be able to interact better with humans, it is crucial for machines to
understand sound - a primary modality of human perception. Previous works have
used sound to learn embeddings for improved generic textual similarity
assessment. In this work, we treat sound as a first-class citizen, studying
downstream textual tasks which require aural grounding. To this end, we propose
sound-word2vec - a new embedding scheme that learns specialized word embeddings
grounded in sounds. For example, we learn that two seemingly (semantically)
unrelated concepts, like leaves and paper are similar due to the similar
rustling sounds they make. Our embeddings prove useful in textual tasks
requiring aural reasoning like text-based sound retrieval and discovering foley
sound effects (used in movies). Moreover, our embedding space captures
interesting dependencies between words and onomatopoeia and outperforms prior
work on aurally-relevant word relatedness datasets such as AMEN and ASLex.
4D Multi-atlas Label Fusion using Longitudinal Images
Authors: Yuankai Huo, Susan M. Resnick, Bennett A. Landman
Date: 2017-08-29 00:00:00
Tasks: Image Segmentation, Medical Image Segmentation, Segmentation, Semantic Segmentation

Longitudinal reproducibility is an essential concern in automated medical
image segmentation, yet has proven to be an elusive objective as manual brain
structure tracings have shown more than 10% variability. To improve
reproducibility, lon-gitudinal segmentation (4D) approaches have been
investigated to reconcile tem-poral variations with traditional 3D approaches.
In the past decade, multi-atlas la-bel fusion has become a state-of-the-art
segmentation technique for 3D image and many efforts have been made to adapt it
to a 4D longitudinal fashion. However, the previous methods were either limited
by using application specified energy function (e.g., surface fusion and multi
model fusion) or only considered tem-poral smoothness on two consecutive time
points (t and t+1) under sparsity as-sumption. Therefore, a 4D multi-atlas
label fusion theory for general label fusion purpose and simultaneously
considering temporal consistency on all time points is appealing. Herein, we
propose a novel longitudinal label fusion algorithm, called 4D joint label
fusion (4DJLF), to incorporate the temporal consistency modeling via non-local
patch-intensity covariance models. The advantages of 4DJLF include: (1) 4DJLF
is under the general label fusion framework by simul-taneously incorporating
the spatial and temporal covariance on all longitudinal time points. (2) The
proposed algorithm is a longitudinal generalization of a lead-ing joint label
fusion method (JLF) that has proven adaptable to a wide variety of
applications. (3) The spatial temporal consistency of atlases is modeled in a
prob-abilistic model inspired from both voting based and statistical fusion.
The pro-posed approach improves the consistency of the longitudinal
segmentation while retaining sensitivity compared with original JLF approach
using the same set of atlases. The method is available online in open-source.
Shape Registration with Directional Data
Authors: Mairéad Grogan, Rozenn Dahyot
Date: 2017-08-25 00:00:00
Tasks: Position

We propose several cost functions for registration of shapes encoded with
Euclidean and/or non-Euclidean information (unit vectors). Our framework is
assessed for estimation of both rigid and non-rigid transformations between the
target and model shapes corresponding to 2D contours and 3D surfaces. The
experimental results obtained confirm that using the combination of a point's
position and unit normal vector in a cost function can enhance the registration
results compared to state of the art methods.
Anomaly Detection: Review and preliminary Entropy method tests
Authors: Pelumi Oluwasanya
Date: 2017-08-29 00:00:00
Tasks: Anomaly Detection

Anomalies are strange data points; they usually represent an unusual
occurrence. Anomaly detection is presented from the perspective of Wireless
sensor networks. Different approaches have been taken in the past, as we will
see, not only to identify outliers, but also to establish the statistical
properties of the different methods. The usual goal is to show that the
approach is asymptotically efficient and that the metric used is unbiased or
maybe biased.
  This project is based on a work done by [1]. The approach is based on the
principle that the entropy of the data is increased when an anomalous data
point is measured. The entropy of the data set is thus to be estimated. In this
report however, preliminary efforts at confirming the results of [1] is
presented. To estimate the entropy of the dataset, since no parametric form is
assumed, the probability density function of the data set is first estimated
using data split method. This estimated pdf value is then plugged-in to the
entropy estimation formula to estimate the entropy of the dataset. The data
(test signal) used in this report is Gaussian distributed with zero mean and
variance 4. Results of pdf estimation using the k-nearest neighbour method
using the entire dataset, and a data-split method are presented and compared
based on how well they approximate the probability density function of a
Gaussian with similar mean and variance. The number of nearest neighbours
chosen for the purpose of this report is 8. This is arbitrary, but is
reasonable since the number of anomalies introduced is expected to be less than
this upon data-split. The data-split method is preferred and rightly so.
Autoencoder with recurrent neural networks for video forgery detection
Authors: Dario D'Avino, Davide Cozzolino, Giovanni Poggi, Luisa Verdoliva
Date: 2017-08-29 00:00:00

Video forgery detection is becoming an important issue in recent years,
because modern editing software provide powerful and easy-to-use tools to
manipulate videos. In this paper we propose to perform detection by means of
deep learning, with an architecture based on autoencoders and recurrent neural
networks. A training phase on a few pristine frames allows the autoencoder to
learn an intrinsic model of the source. Then, forged material is singled out as
anomalous, as it does not fit the learned model, and is encoded with a large
reconstruction error. Recursive networks, implemented with the long short-term
memory model, are used to exploit temporal dependencies. Preliminary results on
forged videos show the potential of this approach.
Motion Deblurring in the Wild
Authors: Mehdi Noroozi, Paramanand Chandramouli, Paolo Favaro
Date: 2017-01-05 00:00:00
Tasks: Deblurring, Image Deblurring

The task of image deblurring is a very ill-posed problem as both the image
and the blur are unknown. Moreover, when pictures are taken in the wild, this
task becomes even more challenging due to the blur varying spatially and the
occlusions between the object. Due to the complexity of the general image model
we propose a novel convolutional network architecture which directly generates
the sharp image.This network is built in three stages, and exploits the
benefits of pyramid schemes often used in blind deconvolution. One of the main
difficulties in training such a network is to design a suitable dataset. While
useful data can be obtained by synthetically blurring a collection of images,
more realistic data must be collected in the wild. To obtain such data we use a
high frame rate video camera and keep one frame as the sharp image and frame
average as the corresponding blurred image. We show that this realistic dataset
is key in achieving state-of-the-art performance and dealing with occlusions.
Multi-view Low-rank Sparse Subspace Clustering
Authors: Maria Brbic, Ivica Kopriva
Date: 2017-08-29 00:00:00
Tasks: Clustering, Multi-view Subspace Clustering

Most existing approaches address multi-view subspace clustering problem by
constructing the affinity matrix on each view separately and afterwards propose
how to extend spectral clustering algorithm to handle multi-view data. This
paper presents an approach to multi-view subspace clustering that learns a
joint subspace representation by constructing affinity matrix shared among all
views. Relying on the importance of both low-rank and sparsity constraints in
the construction of the affinity matrix, we introduce the objective that
balances between the agreement across different views, while at the same time
encourages sparsity and low-rankness of the solution. Related low-rank and
sparsity constrained optimization problem is for each view solved using the
alternating direction method of multipliers. Furthermore, we extend our
approach to cluster data drawn from nonlinear subspaces by solving the
corresponding problem in a reproducing kernel Hilbert space. The proposed
algorithm outperforms state-of-the-art multi-view subspace clustering
algorithms on one synthetic and four real-world datasets.
Large-Scale Classification of Structured Objects using a CRF with Deep Class Embedding
Authors: Eran Goldman, Jacob Goldberger
Date: 2017-05-21 00:00:00
Proceeding: ICLR 2019 5
Tasks: General Classification

This paper presents a novel deep learning architecture to classify structured objects in datasets with a large number of visually similar categories. We model sequences of images as linear-chain CRFs, and jointly learn the parameters from both local-visual features and neighboring classes. The visual features are computed by convolutional layers, and the class embeddings are learned by factorizing the CRF pairwise potential matrix. This forms a highly nonlinear objective function which is trained by optimizing a local likelihood approximation with batch-normalization. This model overcomes the difficulties of existing CRF methods to learn the contextual relationships thoroughly when there is a large number of classes and the data is sparse. The performance of the proposed method is illustrated on a huge dataset that contains images of retail-store product displays, taken in varying settings and viewpoints, and shows significantly improved results compared to linear CRF modeling and unnormalized likelihood optimization.
Learning to Identify Ambiguous and Misleading News Headlines
Authors: Wei Wei, Xiaojun Wan
Date: 2017-05-17 00:00:00
Tasks: Diversity

Accuracy is one of the basic principles of journalism. However, it is
increasingly hard to manage due to the diversity of news media. Some editors of
online news tend to use catchy headlines which trick readers into clicking.
These headlines are either ambiguous or misleading, degrading the reading
experience of the audience. Thus, identifying inaccurate news headlines is a
task worth studying. Previous work names these headlines "clickbaits" and
mainly focus on the features extracted from the headlines, which limits the
performance since the consistency between headlines and news bodies is
underappreciated. In this paper, we clearly redefine the problem and identify
ambiguous and misleading headlines separately. We utilize class sequential
rules to exploit structure information when detecting ambiguous headlines. For
the identification of misleading headlines, we extract features based on the
congruence between headlines and bodies. To make use of the large unlabeled
data set, we apply a co-training method and gain an increase in performance.
The experiment results show the effectiveness of our methods. Then we use our
classifiers to detect inaccurate headlines crawled from different sources and
conduct a data analysis.