pwc-archive/papers-with-abstracts huggingface dataset:
Cross-Modality Personalization for Retrieval
Authors: Nils Murrugarra-Llerena,  Adriana Kovashka
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Cross-Modal Retrieval, Gaze Prediction, Retrieval

Existing captioning and gaze prediction approaches do not consider the multiple facets of personality that affect how a viewer extracts meaning from an image. While there are methods that consider personalized captioning, they do not consider personalized perception across modalities, i.e. how a person's way of looking at an image (gaze) affects the way they describe it (captioning). In this work, we propose a model for modeling cross-modality personalized retrieval. In addition to modeling gaze and captions, we also explicitly model the personality of the users providing these samples. We incorporate constraints that encourage gaze and caption samples on the same image to be close in a learned space; we refer to this as content modeling. We also model style: we encourage samples provided by the same user to be close in a separate embedding space, regardless of the image on which they were provided. To leverage the complementary information that content and style constraints provide, we combine the embeddings from both networks. We show that our combined embeddings achieve better performance than existing approaches for cross-modal retrieval.
Mutual Learning of Complementary Networks via Residual Correction for Improving Semi-Supervised Classification
Authors: Si Wu,  Jichang Li,  Cheng Liu,  Zhiwen Yu,  Hau-San Wong
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: General Classification

Deep mutual learning jointly trains multiple essential networks having similar properties to improve semi-supervised classification. However, the commonly used consistency regularization between the outputs of the networks may not fully leverage the difference between them. In this paper, we explore how to capture the complementary information to enhance mutual learning. For this purpose, we propose a complementary correction network (CCN), built on top of the essential networks, to learn the mapping from the output of one essential network to the ground truth label, conditioned on the features learnt by another. To make the second essential network increasingly complementary to the first one, this network is supervised by the corrected predictions. As a result, minimizing the prediction divergence between the two complementary networks can lead to significant performance gains in semi-supervised learning. Our experimental results demonstrate that the proposed approach clearly improves mutual learning between essential networks, and achieves state-of-the-art results on multiple semi-supervised classification benchmarks. In particular, the test error rates are reduced from previous 21.23% and 14.65% to 12.05% and 10.37% on CIFAR-10 with 1000 and 2000 labels, respectively.
Barrage of Random Transforms for Adversarially Robust Defense
Authors: Edward Raff,  Jared Sylvester,  Steven Forsyth,  Mark McLean
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6

Defenses against adversarial examples, when using the ImageNet dataset, are historically easy to defeat. The common understanding is that a combination of simple image transformations and other various defenses are insufficient to provide the necessary protection when the obfuscated gradient is taken into account. In this paper, we explore the idea of stochastically combining a large number of individually weak defenses into a single barrage of randomized transformations to build a strong defense against adversarial attacks. We show that, even after accounting for obfuscated gradients, the Barrage of Random Transforms (BaRT) is a resilient defense against even the most difficult attacks, such as PGD. BaRT achieves up to a 24x improvement in accuracy compared to previous work, and has even extended effectiveness out to a previously untested maximum adversarial perturbation of e=32.
Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations
Authors: Hao Wu,  Jiayuan Mao,  Yufeng Zhang,  Yuning Jiang,  Lei Li,  Weiwei Sun,  Wei-Ying Ma
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Contrastive Learning, Cross-Modal Retrieval, Retrieval, Sentence

We propose the Unified Visual-Semantic Embeddings (Unified VSE) for learning a joint space of visual representation and textual semantics. The model unifies the embeddings of concepts at different levels: objects, attributes, relations, and full scenes. We view the sentential semantics as a combination of different semantic components such as objects and relations; their embeddings are aligned with different image regions. A contrastive learning approach is proposed for the effective learning of this fine-grained alignment from only image-caption pairs. We also present a simple yet effective approach that enforces the coverage of caption embeddings on the semantic components that appear in the sentence. We demonstrate that the Unified VSE outperforms baselines on cross-modal retrieval tasks; the enforcement of the semantic coverage improves the model's robustness in defending text-domain adversarial attacks. Moreover, our model empowers the use of visual cues to accurately resolve word dependencies in novel sentences.
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering
Authors: Peng Gao,  Zhengkai Jiang,  Haoxuan You,  Pan Lu,  Steven C. H. Hoi,  Xiaogang Wang,  Hongsheng Li
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA)

Learning effective fusion of multi-modality features is at the heart of visual question answering. We propose a novel method of dynamically fuse multi-modal features with intra- and inter-modality information flow, which alternatively pass dynamic information between and across the visual and language modalities. It can robustly capture the high-level interactions  between language and vision domains, thus significantly improves the performance of visual question answering. We also show that, the proposed dynamic intra modality attention flow conditioned on the other modality can dynamically modulate the intra-modality attention of the current modality, which is vital for multimodality feature fusion. Experimental evaluations on the VQA 2.0 dataset show that the proposed method achieves the state-of-the-art VQA performance. Extensive  ablation studies are carried out for the comprehensive analysis of the proposed method.
Acoustic Non-Line-Of-Sight Imaging
Authors: David B. Lindell,  Gordon Wetzstein,  Vladlen Koltun
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: 3D geometry, Seismic Imaging

Non-line-of-sight (NLOS) imaging enables unprecedented capabilities in a wide range of applications, including robotic and machine vision, remote sensing, autonomous vehicle navigation, and medical imaging. Recent approaches to solving this challenging problem employ optical time-of-flight imaging systems with highly sensitive time-resolved photodetectors and ultra-fast pulsed lasers. However, despite recent successes in NLOS imaging using these systems, widespread implementation and adoption of the technology remains a challenge because of the requirement for specialized, expensive hardware. We introduce acoustic NLOS imaging, which is orders of magnitude less expensive than most optical systems and captures hidden 3D geometry at longer ranges with shorter acquisition times compared to state-of-the-art optical methods. Inspired by hardware setups used in radar and algorithmic approaches to model and invert wave-based image formation models developed in the seismic imaging community, we demonstrate a new approach to seeing around corners.
A Theory of Fermat Paths for Non-Line-Of-Sight Shape Reconstruction
Authors: Shumian Xin,  Sotiris Nousias,  Kiriakos N. Kutulakos,  Aswin C. Sankaranarayanan,  Srinivasa G. Narasimhan,  Ioannis Gkioulekas
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Object, PICO

We present a novel theory of Fermat paths of light between a known visible scene and an unknown object not in the line of sight of a transient camera. These light paths either obey specular reflection or are reflected by the object's boundary, and hence encode the shape of the hidden object. We prove that Fermat paths correspond to discontinuities in the transient measurements. We then derive a novel constraint that relates the spatial derivatives of the path lengths at these discontinuities to the surface normal. Based on this theory, we present an algorithm, called Fermat Flow, to estimate the shape of the non-line-of-sight object.  Our method allows, for the first time, accurate shape recovery of complex objects, ranging from diffuse to specular, that are hidden around the corner as well as hidden behind a diffuser. Finally, our approach is agnostic to the particular technology used for transient imaging. As such, we demonstrate mm-scale  shape recovery from pico-second scale transients using a SPAD and ultrafast laser, as well as micron-scale reconstruction  from femto-second scale transients using interferometry. We believe our work is a significant advance over the state-of-the-art in non-line-of-sight imaging.
Underexposed Photo Enhancement Using Deep Illumination Estimation
Authors: Ruixing Wang,  Qing Zhang,  Chi-Wing Fu,  Xiaoyong Shen,  Wei-Shi Zheng,  Jiaya Jia
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6

This paper presents a new neural network for enhancing underexposed photos. Instead of directly learning an image-to-image mapping as previous work, we introduce intermediate illumination in our network to associate the input with expected enhancement result, which augments the network's capability to learn complex photographic adjustment from expert-retouched input/output image pairs. Based on this model, we formulate a loss function that adopts constraints and priors on the illumination, prepare a new dataset of 3,000 underexposed image pairs, and train the network to effectively learn a rich variety of adjustment for diverse lighting conditions. By these means, our network is able to recover clear details, distinct contrast, and natural color in the enhancement results. We perform extensive experiments on the benchmark MIT-Adobe FiveK dataset and our new dataset, and show that our network is effective to deal with previously challenging images.
Enhanced Bayesian Compression via Deep Reinforcement Learning
Authors: Xin Yuan,  Liangliang Ren,  Jiwen Lu,  Jie Zhou
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Deep Reinforcement Learning, Quantization, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In this paper, we propose an Enhanced Bayesian Compression method to flexibly compress the deep networks via reinforcement learning. Unlike the existing Bayesian compression method which cannot explicitly enforce quantization weights during training, our method learns flexible codebooks in each layer for an optimal network quantization. To dynamically adjust the state of codebooks, we employ an Actor-Critic network to collaborate with the original deep network. Different from most existing network quantization methods, our EBC does not require re-training procedures after the quantization. Experimental results show that our method obtains low-bit precision with acceptable accuracy drop on MNIST, CIFAR and ImageNet.
MFAS: Multimodal Fusion Architecture Search
Authors: Juan-Manuel Perez-Rua,  Valentin Vielzeuf,  Stephane Pateux,  Moez Baccouche,  Frederic Jurie
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Action Recognition, General Classification, Neural Architecture Search, Temporal Action Localization

We tackle the problem of finding good architectures for multimodal classification problems. We propose a novel and generic search space that spans a large number of possible fusion architectures. In order to find an optimal architecture for a given dataset in the proposed search space, we leverage an efficient sequential model-based exploration approach that is tailored for the problem. We demonstrate the value of posing multimodal fusion as a neural architecture search problem by extensive experimentation on a toy dataset and two other real multimodal datasets. We discover fusion architectures that exhibit state-of-the-art performance for problems with different domain and dataset size, including the \ntu dataset, the largest multimodal action recognition dataset available.
ShieldNets: Defending Against Adversarial Attacks Using Probabilistic Adversarial Robustness
Authors: Rajkumar Theagarajan,  Ming Chen,  Bir Bhanu,  Jing Zhang
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Adversarial Attack, Adversarial Robustness

Defending adversarial attack is a critical step towards reliable deployment of deep learning empowered solutions for industrial applications. Probabilistic adversarial robustness (PAR), as a theoretical framework, is introduced to neutralize adversarial attacks by concentrating sample probability to adversarial-free zones. Distinct to most of the existing defense mechanisms that require modifying the architecture/training of the target classifier which is not feasible in the real-world scenario, e.g., when a model has already been deployed, PAR is designed in the first place to provide proactive protection to an existing fixed model. ShieldNet is implemented as a demonstration of PAR in this work by using PixelCNN. Experimental results show that this approach is generalizable, robust against adversarial transferability and resistant to a wide variety of attacks on the Fashion-MNIST and CIFAR10 datasets, respectively.
Feature Space Perturbations Yield More Transferable Adversarial Examples
Authors: Nathan Inkawhich,  Wei Wen,  Hai (Helen) Li,  Yiran Chen
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Adversarial Attack

Many recent works have shown that deep learning models are vulnerable to quasi-imperceptible input perturbations, yet practitioners cannot fully explain this behavior. This work describes a transfer-based blackbox targeted adversarial attack of deep feature space representations that also provides insights into cross-model class representations of deep CNNs. The attack is explicitly designed for transferability and drives feature space representation of a source image at layer L towards the representation of a target image at L. The attack yields highly transferable targeted examples, which outperform competition winning methods by over 30% in targeted attack metrics. We also show the choice of L to generate examples from is important, transferability characteristics are blackbox model agnostic, and indicate that well trained deep models have similar highly-abstract representations.
IGE-Net: Inverse Graphics Energy Networks for Human Pose Estimation and Single-View Reconstruction
Authors: Dominic Jack,  Frederic Maire,  Sareh Shirazi,  Anders Eriksson
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Deep Learning, Object Reconstruction, Pose Estimation

Inferring 3D scene information from 2D observations is an open problem in computer vision. We propose using a deep-learning based energy minimization framework to learn a consistency measure between 2D observations and a proposed world model, and demonstrate that this framework can be trained end-to-end to produce consistent and realistic inferences. We evaluate the framework on human pose estimation and voxel-based object reconstruction benchmarks and show competitive results can be achieved with relatively shallow networks with drastically fewer learned parameters and floating point operations than conventional deep-learning approaches.
Knowledge Distillation via Instance Relationship Graph
Authors: Yufan Liu,  Jiajiong Cao,  Bing Li,  Chunfeng Yuan,  Weiming Hu,  Yangxi Li,  Yunqiang Duan
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Knowledge Distillation

The key challenge of knowledge distillation is to extract general, moderate and sufficient knowledge from a teacher network to guide a student network. In this paper, a novel Instance Relationship Graph (IRG) is proposed for knowledge distillation. It models three kinds of knowledge, including instance features, instance relationships and feature space transformation, while the latter two kinds of knowledge are neglected by previous methods. Firstly, the IRG is constructed  to model the distilled knowledge of one network layer, by considering instance features and instance relationships as vertexes and edges respectively. Secondly, an IRG transformation is proposed to models the feature space transformation across layers. It is more moderate than directly mimicking the features at intermediate layers. Finally, hint loss functions are designed to force a student's IRGs to mimic the structures of a teacher's IRGs. The proposed method effectively captures the knowledge along the whole network via IRGs, and thus shows stable convergence and strong robustness to different network architectures. In addition, the proposed method shows superior performance over existing methods on datasets of various scales.
Building Detail-Sensitive Semantic Segmentation Networks With Polynomial Pooling
Authors: Zhen Wei,  Jingyi Zhang,  Li Liu,  Fan Zhu,  Fumin Shen,  Yi Zhou,  Si Liu,  Yao Sun,  Ling Shao
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: General Classification, Segmentation, Semantic Segmentation

Semantic segmentation is an important computer vision task, which aims to allocate a semantic label to each pixel in an image. When training a segmentation model, it is common to fine-tune a classification network pre-trained on a large-scale dataset. However, as an intrinsic property of the classification model, invariance to spatial perturbation resulting from the lose of detail-sensitivity prevents segmentation networks from achieving high performance. The use of standard poolings is one of the key factors for this invariance. The most common standard poolings are max and average pooling. Max pooling can increase both the invariance to spatial perturbations and the non-linearity of the networks. Average pooling, on the other hand, is sensitive to spatial perturbations, but is a linear function. For semantic segmentation, we prefer both the preservation of detailed cues within a local feature region and non-linearity that increases a network's functional complexity. In this work, we propose a polynomial pooling (P-pooling) function that finds an intermediate form between max and average pooling to provide an optimally balanced and self-adjusted pooling strategy for semantic segmentation. The P-pooling is differentiable and can be applied into a variety of pre-trained networks. Extensive studies on the PASCAL VOC, Cityscapes and ADE20k datasets demonstrate the superiority of P-pooling over other poolings. Experiments on various network architectures and state-of-the-art training strategies also show that models with P-pooling layers consistently outperform those directly fine-tuned using pre-trained classification models.
A Main/Subsidiary Network Framework for Simplifying Binary Neural Networks
Authors: Yinghao Xu,  Xin Dong,  Yudian Li,  Hao Su
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Binarization, image-classification, Image Classification

To reduce memory footprint and run-time latency, techniques such as neural net-work pruning and binarization have been explored separately.  However, it is un-clear how to combine the best of the two worlds to get extremely small and efficient models.  In this paper, we, for the first time, define the filter-level pruning problem for binary neural networks, which cannot be solved by simply migrating existing structural pruning methods for full-precision models.  A novel learning-based approach is proposed to prune filters in our main/subsidiary network frame-work, where the main network is responsible for learning representative features to optimize the prediction performance, and the subsidiary component works as a filter selector on the main network. To avoid gradient mismatch when training the subsidiary component, we propose a layer-wise and bottom-up scheme.  We also provide the theoretical and experimental comparison between our learning-based and greedy rule-based methods.  Finally, we empirically demonstrate the effectiveness of our approach applied on several binary models,  including binarizedNIN, VGG-11, and ResNet-18, on various image classification datasets.  For bi-nary ResNet-18 on ImageNet, we use 78.6% filters but can achieve slightly better test error 49.87% (50.02%-0.15%) than the original model
Compact Feature Learning for Multi-Domain Image Classification
Authors: Yajing Liu,  Xinmei Tian,  Ya Li,  Zhiwei Xiong,  Feng Wu
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Classification, domain classification, General Classification, image-classification, Image Classification

   The goal of multi-domain learning is to improve the performance over multiple domains by making full use of all training data from them. However, variations of feature distributions across different domains result in a non-trivial solution of multi-domain learning. The state-of-the-art work regarding multi-domain classification aims to extract domain-invariant features and domain-specific features independently. However, they view the distributions of features from different classes as a general distribution and try to match these distributions across domains, which lead to the mixture of features from different classes across domains and degrade the performance of classification. Additionally, existing works only force the shared features among domains to be orthogonal to the features in the domain-specific network. However, redundant features between the domain-specific networks still remain, which may shrink the discriminative ability of domain-specific features. Therefore, we propose an end-to-end network to obtain the more optimal features, which we call compact features. We propose to extract the domain-invariant features by matching the joint distributions of different domains, which have dis- tinct boundaries between different classes. Moreover, we add an orthogonal constraint between the private features across domains to ensure the discriminative ability of the domain-specific space. The proposed method is validated on three landmark datasets, and the results demonstrate the effectiveness of our method.
Adaptive Transfer Network for Cross-Domain Person Re-Identification
Authors: Jiawei Liu,  Zheng-Jun Zha,  Di Chen,  Richang Hong,  Meng Wang
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Person Re-Identification, Style Transfer

Recent deep learning based person re-identification approaches have steadily improved the performance for benchmarks, however they often fail to generalize well from one domain to another. In this work, we propose a novel adaptive transfer network (ATNet) for effective cross-domain person re-identification. ATNet looks into the essential causes of domain gap and addresses it following the principle of "divide-and-conquer".  It decomposes the complicated cross-domain transfer into a set of factor-wise sub-transfers, each of which concentrates on style transfer with respect to a certain imaging factor, e.g., illumination, resolution and camera view etc. An adaptive ensemble strategy is proposed to fuse factor-wise transfers by perceiving the affect magnitudes of various factors on images. Such "decomposition-and-ensemble" strategy gives ATNet the capability of precise style transfer at factor level and eventually effective transfer across domains. In particular, ATNet consists of a transfer network composed by multiple factor-wise CycleGANs and an ensemble CycleGAN as well as a selection network that infers the affects of different factors on transferring each image. Extensive experimental results on  three widely-used datasets, i.e., Market-1501, DukeMTMC-reID and PRID2011 have demonstrated the effectiveness of the proposed ATNet with significant performance improvements over state-of-the-art methods.