pwc-archive/papers-with-abstracts huggingface dataset:
Neural Graph Matching Networks for Fewshot 3D Action Recognition
Authors: Michelle Guo, Edward Chou, De-An Huang, Shuran Song, Serena Yeung, Li Fei-Fei
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: 3D Action Recognition, Action Recognition, Few-Shot Learning, Graph Matching, Skeleton Based Action Recognition, Temporal Action Localization

We propose Neural Graph Matching (NGM) Networks, a novel framework that can learn to recognize a previous unseen 3D action class with only a few examples. We achieve this by leveraging the inherent structure of 3D data through a graphical representation. This allows us to modularize our model and lead to strong data-efficiency in few-shot learning. More specifically, NGM Networks jointly learn a graph generator and a graph matching metric function in a end-to-end fashion to directly optimize the few-shot learning objective. We evaluate NGM on two 3D action recognition datasets, CAD-120 and PiGraphs, and show that learning to generate and match graphs both lead to significant improvement of few-shot 3D action recognition over the holistic baselines.
Temporal Modular Networks for Retrieving Complex Compositional Activities in Videos
Authors: Bingbin Liu, Serena Yeung, Edward Chou, De-An Huang, Li Fei-Fei, Juan Carlos Niebles
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Retrieval, Video Retrieval

A major challenge in computer vision is scaling activity understanding to the long tail of complex activities without requiring collecting large quantities of data for new actions. The task of video retrieval using natural language descriptions seeks to address this through rich, unconstrained supervision about complex activities. However, while this formulation offers hope of leveraging underlying compositional structure in activity descriptions, existing approaches typically do not explicitly model compositional reasoning. In this work, we introduce an approach for explicitly and dynamically reasoning about compositional natural language descriptions of activity in videos. We take a modular neural network approach that, given a natural language query, extracts the semantic structure to assemble a compositional neural network layout and corresponding network modules. We show that this approach is able to achieve state-of-the-art results on the DiDeMo video retrieval dataset.
Attention-aware Deep Adversarial Hashing for Cross-Modal Retrieval
Authors: Xi Zhang, Hanjiang Lai , Jiashi Feng 
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Cross-Modal Retrieval, Retrieval

Due to the rapid growth of multi-modal data, hashing methods for cross-modal retrieval have received considerable attention. However, finding content similarities between different modalities of data is still challenging due to an existing heterogeneity gap. To further address this problem, we propose an adversarial hashing network with an attention mechanism to enhance the measurement of content similarities by selectively focusing on the informative parts of multi-modal data. The proposed new deep adversarial network consists of three building blocks: 1) the feature learning module to obtain the feature representations; 2) the attention module to generate an attention mask, which is used to divide the feature representations into the attended and unattended feature representations; and 3) the hashing module to learn hash functions that preserve the similarities between different modalities. In our framework, the attention and hashing modules are trained in an adversarial way: the attention module attempts to make the hashing module unable to preserve the similarities of multi-modal data w.r.t. the unattended feature representations, while the hashing module aims to preserve the similarities of multi-modal data w.r.t. the attended and unattended feature representations. Extensive evaluations on several benchmark datasets demonstrate that the proposed method brings substantial improvements over other state-of-the-art cross-modal hashing methods.
Variable Ring Light Imaging: Capturing Transient Subsurface Scattering with An Ordinary Camera
Authors: Ko Nishino, Art Subpa-Asa, Yuta Asano, Mihoko Shimano, Imari Sato 
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9

Subsurface scattering plays a significant role in determining the appearance of real-world surfaces. A light ray penetrating into the subsurface is repeatedly scattered and absorbed by particles along its path before reemerging from the outer interface, which determines its spectral radiance. We introduce a novel imaging method that enables the decomposition of the appearance of a fronto-parallel real-world surface into images of light with bounded path lengths, i.e., transient subsurface light transport. Our key idea is to observe each surface point under a variable ring light: a circular illumination pattern of increasingly larger radius centered on it. We show that the path length of light captured in each of these observations is naturally lower-bounded by the ring light radius. By taking the difference of ring light images of incrementally larger radii, we compute transient images that encode light with bounded path lengths. Experimental results on synthetic and complex real-world surfaces demonstrate that the recovered transient images reveal the subsurface structure of general translucent inhomogeneous surfaces. We further show that their differences reveal the surface colors at different surface depths. The proposed method is the first to enable the unveiling of dense and continuous subsurface structures from steady-state external appearance using ordinary camera and illumination.
Deep Domain Generalization via Conditional Invariant Adversarial Networks
Authors: Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, DaCheng Tao
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Domain Generalization, Representation Learning

Domain generalization aims to learn a classification model from multiple source domains and generalize it to unseen target domains. A critical problem in domain generalization involves learning domain-invariant representations. Let $X$ and $Y$ denote the features and the labels, respectively. Under the assumption that the conditional distribution $P(Y|X)$ remains unchanged across domains, earlier approaches to domain generalization learned the invariant representation $T(X)$ by minimizing the discrepancy of the marginal distribution $P(T(X))$. However, such an assumption of stable $P(Y|X)$ does not necessarily hold in practice. In addition, the representation learning function $T(X)$ is usually constrained to a simple linear transformation or shallow networks. To address the above two drawbacks, we propose an end-to-end conditional invariant deep domain generalization approach by leveraging deep neural networks for domain-invariant representation learning. The domain-invariance property is guaranteed through a conditional invariant adversarial network that can learn domain-invariant representations w.r.t. the joint distribution $P(T(X),Y)$ if the target domain data are not severely class unbalanced. We perform various experiments to demonstrate the effectiveness of the proposed method.
Deep Cross-Modal Projection Learning for Image-Text Matching
Authors: Ying Zhang, Huchuan Lu 
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Cross-Modal Retrieval, Image-text matching, Text based Person Retrieval, Text Matching

The key point of image-text matching is how to accurately measure the similarity between visual and textual inputs. Despite the great progress of associating the deep cross-modal embeddings with the bi-directional ranking loss, developing the strategies for mining useful triplets and selecting appropriate margins remains a challenge in real applications. In this paper, we propose a cross-modal projection matching (CMPM) loss and a cross-modal projection classification (CMPC) loss for learning discriminative image-text embeddings. The CMPM loss minimizes the KL divergence between the projection compatibility distributions and the normalized matching distributions defined with all the positive and negative samples in a mini-batch. The CMPC loss attempts to categorize the vector projection of representations from one modality onto another with the improved norm-softmax loss, for further enhancing the feature compactness of each class. Extensive analysis and experiments on multiple datasets demonstrate the superiority of the proposed approach.
Large Scale Urban Scene Modeling from MVS Meshes
Authors: Lingjie Zhu, Shuhan Shen, Xiang Gao, Zhanyi Hu
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9

In this paper we present an effcient modeling framework for large scale urban scenes. Taking surface meshes derived from multi- view-stereo systems as input, our algorithm outputs simplied models with semantics at different levels of detail (LODs). Our key observation is that urban building is usually composed of planar roof tops connected with vertical walls. There are two major steps in our framework: segmentation and building modeling. The scene is first segmented into four classes with a Markov random field combining height and image features. In the following modeling step, various 2D line segments sketching the roof boundaries are detected and slice the plane into faces. Through assigning each face with a roof plane, the final model is constructed by extruding the faces to the corresponding planes. By combining geometric and appearance cues together, the proposed method is robust and fast compared to the state-of-the-art algorithms.
Dual-Agent Deep Reinforcement Learning for Deformable Face Tracking
Authors: Minghao Guo, Jiwen Lu, Jie zhou
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Deep Reinforcement Learning, Facial Landmark Detection, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In this paper, we propose a dual-agent deep reinforcement learning (DADRL) method for deformable face tracking, which generates bounding boxes and detects facial landmarks interactively from face videos. Most existing deformable face tracking methods learn models for these two tasks individually, and perform these two procedures subsequently during the testing phase, which ignore the intrinsic connections of these two tasks. Motivated by the fact that the performance of facial landmark detection depends heavily on the accuracy of the generated bounding boxes, we exploit the interactions of these two tasks in probabilistic manner by following a Bayesian model and propose a unified framework for simultaneous bounding box tracking and landmark detection. By formulating it as a Markov decision process, we define two agents to exploit the relationships and pass messages via an adaptive sequence of actions under a deep reinforcement learning framework to iteratively adjust the positions of the bounding boxes and facial landmarks. Our proposed DADRL achieves performance improvements over the state-of-the-art deformable face tracking methods on the most challenging category of the 300-VW dataset.
Deep Reinforcement Learning with Iterative Shift for Visual Tracking
Authors: Liangliang Ren, Xin Yuan, Jiwen Lu, Ming Yang, Jie Zhou 
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Deep Reinforcement Learning, Motion Estimation, Object, Object Tracking, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Visual Tracking

Visual tracking is confronted by the dilemma to locate a target both}accurately and efficiently, and make decisions online whether and how to adapt the appearance model or even restart tracking. In this paper, we propose a deep reinforcement learning with iterative shift (DRL-IS) method for single object tracking, where an actor-critic network is introduced to predict the iterative shifts of object bounding boxes, and evaluate the shifts to take actions on whether to update object models or re-initialize tracking. Since locating an object is achieved by an iterative shift process, rather than online classification on many sampled locations, the proposed method is robust to cope with large deformations and abrupt motion, and computationally efficient since finding a target takes up to 10 shifts. In offline training, the critic network guides to learn how to make decisions jointly on motion estimation and tracking status in an end-to-end manner. Experimental results on the OTB benchmarks with large deformation improve the tracking precision by 1.7%  and runs about 5 times faster than the competing state-of-the-art methods.
Collaborative Deep Reinforcement Learning for Multi-Object Tracking
Authors: Liangliang Ren, Jiwen Lu, Zifeng Wang, Qi Tian, Jie zhou
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Deep Reinforcement Learning, Multi-Object Tracking, Object, Object Tracking, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In this paper, we propose a collaborative deep reinforcement learning (C-DRL) method for multi-object tracking. Most existing multi-object tracking methods employ the tracking-by-detection strategy which first detects objects in each frame and then associates them across different frames. However, the performance of these methods rely heavily on the detection results, which are usually unsatisfied in many real applications, especially in crowded scenes. To address this, we develop a deep prediction-decision network in our C-DRL, which simultaneously detects and predicts objects under a unified network via deep reinforcement learning. Specifically, we consider each object as an agent and track it via the prediction network, and seek the optimal tracked results by exploiting the collaborative interactions of different agents and environments via the decision network, so that the influences of occlusions and noisy detection results can be well alleviated. Experimental results on the challenging MOT15 and MOT16 benchmarks are presented to show the efficiency of our approach.
Deep Variational Metric Learning
Authors: Xudong Lin, Yueqi Duan, Qiyuan Dong, Jiwen Lu, Jie zhou
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Metric Learning

Deep metric learning has been extensively explored recently, which trains a deep neural network to produce discriminative embedding features. Most existing methods usually enforce the model to be indiscriminating to intra-class variance, which makes the model over-fitting to the training set to minimize loss functions on these specific changes and leads to low generalization power on unseen classes. However, these methods ignore a fact that in the central latent space, the distribution of variance within classes is actually independent on classes. In this paper, we propose a deep variational metric learning (DVML) framework to explicitly model the intra-class variance and disentangle the intra-class invariance, namely, the class centers. With the learned distribution of intra-class variance, we can simultaneously generate discriminative samples to improve robustness. Our method is applicable to most of existing metric learning algorithms, and extensive experiments on three benchmark datasets including CUB-200-2011, Cars196 and Stanford Online Products show that our DVML significantly boosts the performance of currently popular deep metric learning methods.
Multi-Scale Context Intertwining for Semantic Segmentation
Authors: Di Lin, Yuanfeng Ji, Dani Lischinski, Daniel Cohen-Or, Hui Huang
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Image Segmentation, Segmentation, Semantic Segmentation

Accurate semantic image segmentation requires the joint consideration of local appearance, semantic information, and global scene context. In todayâs age of pre-trained deep networks and their powerful convolutional features, state-of-the-art semantic segmentation approaches differ mostly in how they choose to combine together these different kinds of information. In this work, we propose a novel scheme for aggregating features from different scales, which we refer to as Multi-Scale Context Intertwining (MSCI). In contrast to previous approaches, which typically propagate information between scales in a one-directional manner, we merge pairs of feature maps in a bidirectional and recurrent fashion, via connections between two LSTM chains. By training the parameters of the LSTM units on the segmentation task, the above approach learns how to extract powerful and effective features for pixel-level semantic segmentation, which are then combined hierarchically. Furthermore, rather than using fixed information propagation routes, we subdivide images into super-pixels, and use the spatial relationship between them in order to perform image-adapted context aggregation. Our extensive evaluation on public benchmarks indicates that all of the aforementioned components of our approach increase the effectiveness of information propagation throughout the network, and significantly improve its eventual segmentation accuracy.
Learning to Fuse Proposals from Multiple Scanline Optimizations in Semi-Global Matching
Authors: Johannes L. Schonberger, Sudipta N. Sinha, Marc Pollefeys 
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: General Classification

Semi-Global Matching (SGM) uses an aggregation scheme to combine costs from multiple 1D scanline optimizations that tends to hurt its accuracy in difficult scenarios. We propose replacing this aggregation scheme with a new learning-based method  that fuses disparity proposals estimated using scanline optimization. Our proposed SGM-Forest algorithm solves this problem using per-pixel classification. SGM-Forest currently ranks 1st on the ETH3D stereo benchmark and is ranked competitively on the Middlebury 2014 and KITTI 2015 benchmarks.  It consistently outperforms SGM in challenging settings and  under difficult training protocols that demonstrate robust generalization, while adding only a small computational overhead to SGM.
Saliency Detection in 360Â° Videos
Authors: Ziheng Zhang, Yanyu Xu, Jingyi Yu, Shenghua Gao 
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Saliency Detection, Video Saliency Detection

This paper presents a novel spherical convolutional neural network based scheme for saliency detection for 360Â° videos. Specifically, in our spherical convolution neural network definition, kernel is defined on a spherical crown, and the convolution involves the rotation of the kernel along the sphere. Considering that the 360Â° videos are usually stored with equirectangular panorama, we propose to implement the spherical convolution on panorama by stretching and rotating the kernel based on the location of patch to be convolved. Compared with existing spherical convolution, our definition has the parameter sharing property, which would greatly reduce the parameters to be learned. We further take the temporal coherence of the viewing process into consideration, and propose a sequential saliency detection by leveraging a spherical U-Net. To validate our approach, we construct a large-scale 360Â° videos saliency detection benchmark that consists of 104 360Â° videos viewed by 20+ human subjects. Comprehensive experiments validate the effectiveness of our spherical U-net for 360Â° video saliency detection.
AugGAN: Cross Domain Adaptation with GAN-based Data Augmentation
Authors: Sheng-Wei Huang, Che-Tsung Lin, Shu-Ping Chen, Yen-Yi Wu, Po-Hao Hsu, Shang-Hong Lai
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Data Augmentation, Domain Adaptation, Generative Adversarial Network, Image-to-Image Translation, Link Prediction, Translation

Deep learning based image-to-image translation methods aim at learning the joint distribution of the two domains and finding transformations between them. Despite recent GAN (Generative Adversarial Network) based methods have shown compelling visual results, they are prone to fail at preserving image-objects and maintaining translation consistency when faced with large and complex domain shifts, which reduces their practicality on tasks such as generating large-scale training data for different domains. To address this problem, we purpose a weakly supervised structure-aware image-to-image translation network, which is composed of encoders, generators, discriminators and parsing nets for the two domains, respectively, in a unified framework. The purposed network generates more visually plausible images of a different domain compared to the competing methods on different image-translation tasks. In addition, we quantitatively evaluate different methods by training Faster-RCNN and YOLO with datasets generated from the image-translation results and demonstrate significant improvement of the detection accuracies by using the proposed image-object preserving network.
Graininess-Aware Deep Feature Learning for Pedestrian Detection
Authors: Chunze Lin, Jiwen Lu, Gang Wang, Jie zhou
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9
Tasks: Pedestrian Detection

In this paper, we propose a graininess-aware deep feature learning method for pedestrian detection. Unlike most existing pedestrian detection methods which only consider low resolution feature maps, we incorporate fine-grained information into convolutional features to make them more discriminative for human body parts. Specifically, we propose a pedestrian attention mechanism which efficiently identifies pedestrian regions. Our method encodes fine-grained attention masks into convolutional feature maps, which significantly suppresses background interference and highlights pedestrians. Hence, our graininess-aware features become more focused on pedestrians, in particular those of small size and with occlusion. We further introduce a zoom-in-zoom-out module, which enhances the features by incorporating local details and context information. We integrate these two modules into a deep neural network, forming an end-to-end trainable pedestrian detector. Comprehensive experimental results on four challenging pedestrian benchmarks demonstrate the effectiveness of the proposed approach.
R2P2: A ReparameteRized Pushforward Policy for Diverse, Precise Generative Path Forecasting
Authors: Nicholas Rhinehart, Kris M. Kitani, Paul Vernaza
Date: 2018-09-01 00:00:00
Proceeding: ECCV 2018 9

We propose a method to forecast a vehicle's ego-motion as a distribution over spatiotemporal paths, conditioned on features (e.g., from LIDAR and images) embedded in an overhead map. The method learns a policy inducing a distribution over simulated trajectories that is both diverse (produces most paths likely under the data) and precise (mostly produces paths likely under the data). This balance is achieved through minimization of a symmetrized cross-entropy between the distribution and demonstration data.  By viewing the simulated-outcome distribution as the pushforward of a simple distribution under a simulation operator, we obtain expressions for the cross-entropy metrics that can be efficiently evaluated and differentiated, enabling stochastic-gradient optimization. We propose concrete policy architectures for this model, discuss our evaluation metrics relative to previously-used metrics, and demonstrate the superiority of our method relative to state-of-the-art methods in both the KITTI dataset and a similar but novel and larger real-world dataset explicitly designed for the vehicle forecasting domain.