pwc-archive/papers-with-abstracts huggingface dataset:
Gait Lateral Network: Learning Discriminative and Compact Representations for Gait Recognition
Authors: Saihui Hou, Chunshui Cao, Xu Liu, Yongzhen Huang
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Gait Recognition

Gait recognition aims at identifying different people by the walking patterns, which can be conducted at a long distance without the cooperation of subjects. A key challenge for gait recognition is to learn representations from the silhouettes that are invariant to the factors such as clothing, carrying conditions and camera viewpoints. Besides being discriminative for identification, the gait representations should also be compact for storage to keep millions of subjects registered in the gallery. In this work, we propose a novel network named Gait Lateral Network (GLN) which can learn both discriminative and compact representations from the silhouettes for gait recognition. Specifically, GLN leverages the inherent feature pyramid in deep convolutional neural networks to enhance the gait representations. The silhouette-level and set-level features extracted by different stages are merged with the lateral connections in a top-down manner. Besides, GLN is equipped with a Compact Block which can significantly reduce the dimension of the gait representations without hindering the accuracy. Extensive experiments on CASIA-B and OUMVLP show that GLN can achieve state-of-the-art performance using the $256$-dimensional representations. Under the most challenging condition of walking in different clothes on CASIA-B, our method improves the rank-1 accuracy by $6.45\%$.
Inequality-Constrained and Robust 3D Face Model Fitting
Authors: Evangelos Sariyanidi, Casey J. Zampella, Robert T. Schultz, Birkan Tunc
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Face Model, model

Fitting 3D morphable models (3DMMs) on faces is a well-studied problem, motivated by various industrial and research applications. 3DMMs express a 3D facial shape as a linear sum of basis functions. The resulting shape, however, is a plausible face only when the basis coefficients take values within limited intervals. Methods based on unconstrained optimization address this issue with a weighted $ll_2$ penalty on coefficients; however, determining the weight of this penalty is difficult, and the existence of a unique weight that works universally is questionable. We propose a new formulation that does not require the tuning of any weight parameter. Specifically, we formulate 3DMM fitting as an inequality-constrained optimization problem, where the primary constraint is that basis coefficients should not exceed the interval that is learned when the 3DMM is constructed. We employ additional constraints to exploit sparse landmark detectors by enforcing the facial shape to be within the error bounds of a reliable detector. To enable operation ``in-the-wild'', we use a robust objective function, namely gradient correlation. Our approach performs comparably with deep learning (DL) methods on ``in-the-wild'' data, which have inexact ground truth, and better than DL methods on more controlled data with exact ground truth. Since our formulation does not require any learning, it enjoys a versatility that allows it to operate with multiple frames of arbitrary sizes. This study's results encourage further research on 3DMM fitting with inequality-constrained optimization methods, which remained unexplored compared to unconstrained methods.
Conditional Image Repainting via Semantic Bridge and Piecewise Value Function
Authors: Shuchen Weng, Wenbo Li, Dawei Li, Hongxia Jin, Boxin Shi
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8

We study conditional image repainting where a model is trained to generate visual content conditioned on user inputs, and composite the generated content seamlessly onto a user provided image while preserving the semantics of users' inputs. The content generation community have been pursuing to lower the skill barriers. The usage of human language is the rose among horns for this purpose, because the language is friendly to users but poses great difficulties for the model in associating relevant words with the semantically ambiguous regions. To resolve this issue, we propose a delicate mechanism which bridges the semantic chasm between the language input and the generated visual content. The state-of-the-art image compositing techniques pose a latent ceiling of fidelity for the composited content during the adversarial training process. In this work, we improve the compositing by breaking through the latent ceiling using a novel piecewise value function. We demonstrate on two datasets that the proposed techniques can better assist tackling conditional image repainting compared to the existing ones.
HALO: Hardware-Aware Learning to Optimize
Authors: Chaojian Li, Tianlong Chen, Haoran You, Zhangyang Wang, Yingyan Lin
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8

There has been an explosive demand for bringing machine learning (ML) powered intelligence into numerous Internet-of-Things (IoT) devices. However, the effectiveness of such intelligent functionality requires in-situ continuous model adaptation for adapting to new data and environments, while the on-device computing and energy resources are usually extremely constrained. Neither traditional hand-crafted (e.g., SGD, Adagrad, and Adam) nor existing meta optimizers are specifically designed to meet those challenges, as the former requires tedious hyper-parameter tuning while the latter are often costly due to the meta algorithms’ own overhead. To this end, we propose hardware-aware learning to optimize (HALO), a practical meta optimizer dedicated to resource-efficient on-device adaptation. Our HALO optimizer features the following highlights: (1) faster adaptation speed (i.e., taking fewer data or iterations to reach a specified accuracy) by introducing a new regularizer to promote empirical generalization; and (2) lower per-iteration complexity, thanks to a stochastic structural sparsity regularizer being enforced. Furthermore, the optimizer itself is designed as a very light-weight RNN and thus incurs negligible overhead. Ablation studies and experiments on five datasets, six optimizees, and two state-of-the-art (SOTA) edge AI devices validate that, while always achieving a better accuracy (↑0.46% - ↑20.28%), HALO can greatly trim down the energy cost (up to ↓60%) in adaptation, quantified using an IoT device or SOTA simulator. Codes and pre-trained models are at https://github.com/RICE-EIC/HALO .
Interpretable Visual Reasoning via Probabilistic Formulation under Natural Supervision
Authors: Xinzhe Han, Shuhui Wang, Chi Su, Weigang Zhang, Qingming Huang, Qi Tian
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA), Visual Reasoning

Visual reasoning is crucial for visual question answering (VQA). However, without labelled programs, implicit reasoning under natural supervision is still quite challenging and previous models are hard to interpret. In this paper, we rethink implicit reasoning process in VQA, and propose a new formulation which maximizes the log-likelihood of joint distribution for the observed question and predicted answer. Accordingly, we derive a Temporal Reasoning Network (TRN) framework which models the implicit reasoning process as sequential planning in latent space. Our model is interpretable on both model design in probabilist and reasoning process via visualization. We experimentally demonstrate that TRN can support implicit reasoning across various datasets. The experiment results of our model are competitive to existing implicit reasoning models and surpass baseline by large margin on complicated reasoning tasks without extra computation cost in forward stage.
Reconstructing the Noise Variance Manifold for Image Denoising
Authors: Ioannis Marras, Grigorios G. Chrysos, Ioannis Alexiou, Gregory Slabaugh, Stefanos Zafeiriou
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Conditional Image Generation, Denoising, Image Denoising, Image Generation

Deep Convolutional Neural Networks (CNNs) have been successfully used in many low-level vision problems like image denoising. Although the conditional image generation techniques have led to large improvements in this task, there has been little effort in providing conditional generative adversarial networks (cGANs) with an explicit way of understanding the image noise for object-independent denoising reliable for real-world applications. The task of leveraging structures in the target space is unstable due to the complexity of patterns in natural scenes, so the presence of unnatural artifacts or over-smoothed image areas cannot be avoided. To fill the gap, in this work we introduce the idea of a cGAN which explicitly leverages structure in the image noise variance space. By learning directly a low dimensional manifold of the image noise variance, the generator promotes the removal from the noisy image only that information which spans this manifold. This idea brings many advantages while it can be appended at the end of any denoiser to significantly improve its performance. Based on our experiments, our model substantially outperforms existing state-of-the-art architectures, resulting in denoised images with less oversmoothing and better detail.
URIE: Universal Image Enhancement for Visual Recognition in the Wild
Authors: Taeyoung Son Juwon Kang Namyup Kim Sunghyun Cho Suha Kwak
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Image Enhancement

Despite the great advances in visual recognition, it has been witnessed that recognition models trained on clean images of common datasets are not robust against distorted images in the real world. To tackle this issue, we present a Universal and Recognition-friendly Image Enhancement network, dubbed URIE, which is attached in front of existing recognition models and enhances distorted input to improve their performance without retraining them. URIE is universal in that it aims to handle various factors of image degradation and to be incorporated with any arbitrary recognition models. Also, it is recognition-friendly since it is optimized to improve the robustness of following recognition models, instead of perceptual quality of output image. Our experiments demonstrate that URIE can handle various and latent image distortions and improve the performance of existing models for five diverse recognition tasks when input images are degraded.
RhyRNN: Rhythmic RNN for Recognizing Events in Long and Complex Videos
Authors: Tianshu Yu, Yikang Li, Baoxin Li
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Rhythm

Though many successful approaches have been proposed for recognizing events in short and homogeneous videos, doing so with long and complex videos remains a challenge. One particular reason is that events in long and complex videos can consist of multiple heterogeneous sub-activities (in terms of rhythms, activity variants, composition order, etc.) within quite a long period. This fact brings about two main difficulties: excessive/varying length and complex video dynamic/rhythm. To address this, we propose Rhythmic RNN (RhyRNN) which is capable of handling long video sequences (up to 3,000 frames) as well as capturing rhythms at different scales. We also propose two novel modules: diversity-driven pooling (DivPool) and bilinear reweighting (BR), which consistently and hierarchically abstract higher-level information. We study the behavior of RhyRNN and empirically show that our method works well even when mph{only event-level labels are available} in the training stage (compared to algorithms requiring sub-activity labels for recognition), and thus is more practical when the sub-activity labels are missing or difficult to obtain. Extensive experiments on several public datasets demonstrate that, even mph{without fine-tuning the feature backbones}, our method can achieve promising performance for long and complex videos that contain multiple sub-activities.
Layered Neighborhood Expansion for Incremental Multiple Graph Matching
Authors: Zixuan Chen, Zhihui Xie, Junchi Yan Yinqiang Zheng, Xiaokang Yang
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Graph Matching

Graph matching has been a fundamental problem in computer vision and pattern recognition, for its practical flexibility as well as NP hardness challenge. Though the matching between two graphs and among multiple graphs have been intensively studied in literature, the online setting for incremental matching of a stream of graphs has been rarely considered. In this paper, we treat the graphs as graphs on a super-graph, and propose a novel breadth first search based method for expanding the neighborhood on the super-graph for a new coming graph, such that the matching with the new graph can be efficiently performed within the constructed neighborhood. Then depth first search is performed to update the overall pairwise matchings. Moreover, we show our approach can also be readily used in the batch mode setting, by adaptively determining the order of coming graph batch for matching, still under the neighborhood expansion based incremental matching framework. Extensive experiments on both online and offline matching of graph collections show our approach’s state-of-the-art accuracy and efficiency. The source code will be made public available.
A Comprehensive Study of Weight Sharing in Graph Networks for 3D Human Pose Estimation
Authors: Kenkun Liu, Rongqi Ding, Zhiming Zou, Le Wang, Wei Tang
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: 3D Human Pose Estimation, Pose Estimation

Graph convolutional networks (GCNs) have been applied to 3D human pose estimation (HPE) from 2D body joint detections and have shown encouraging performance. One limitation of the vanilla graph convolution is that it models the relationships between neighboring nodes via a shared weight matrix. This is suboptimal for articulated body modeling as the relations between different body joints are different. The objective of this paper is to have a comprehensive and systematic study of weight sharing in GCNs for 3D HPE. We first show there are two different ways to interpret a GCN depending on whether feature transformation occurs before or after feature aggregation. These two interpretations lead to five different weight sharing methods, and three more variants can be derived by decoupling the self-connections with other edges. We conduct extensive ablation study on these weight sharing methods under controlled settings and obtain new conclusions that will benefit the community.
Deep Spiking Neural Network: Energy Efficiency Through Time based Coding
Authors: Bing Han, Kaushik Roy
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8

Spiking Neural Networks (SNNs) are promising for enabling low-power event-driven data analytics. The best performing SNNs for image recognition tasks are obtained by converting a trained deep learning Analog Neural Network (ANN) composed of Rectified Linear Unit (ReLU) activation to SNN consisting of Integrate-and-Fire (IF) neurons with ""proper"" firing thresholds. However, this has come at the cost of accuracy loss and higher inference latency due to lack of a notion of time. In this work, we propose an ANN to SNN conversion methodology that uses a time-based coding scheme, named Temporal-Switch-Coding (TSC), and a corresponding TSC spiking neuron model. Each input image pixel is presented using two spikes and the timing between the two spiking instants is proportional to the pixel intensity. The real-valued ReLU activations in ANN are encoded using the spike-times of the TSC neurons in the converted TSC-SNN. At most two memory accesses and two addition operations are performed for each synapse during the whole inference, which significantly improves the SNN energy efficiency. We demonstrate the proposed TSC-SNN for VGG-16, ResNet-20 and ResNet-34 SNNs on datasets including CIFAR-10 (93.63% top-1), CIFAR-100 (70.97% top-1) and ImageNet (73.46% top-1 accuracy). It surpasses the best inference accuracy of the converted rate-encoded SNN with 7-14.5 times lesser inference latency, and 30-60 times fewer addition operations and memory accesses per inference across datasets.
Utilizing Patch-level Category Activation Patterns for Multiple Class Novelty Detection
Authors: Poojan Oza, Vishal M. Patel
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Novelty Detection

For any recognition system, the ability to identify novel class samples during inference is an important aspect of the system’s robustness. This problem of detecting novel class samples during inference is commonly referred to as Multiple Class Novelty Detection. In this paper, we propose a novel method that makes deep convolutional neural networks robust to novel classes. Specifically, during training one branch performs traditional classification (referred to as global inference), and the other branch provides patch-level information to keep track of the class-specific activation patterns (referred to as local inference). Both global and local branch information are combined to train a novelty detection network, which is used during inference to identify novel classes. We evaluate the proposed method on four datasets (Caltech256, CUB-200, Stanford Dogs and FounderType-200) and show that the proposed method is able to identify novel class samples better compared to the other deep convolutional neural network based methods.
LST-Net: Learning a Convolutional Neural Network with a Learnable Sparse Transform
Authors: Lida Li, Kun Wang, Shuai Li, Xiangchu Feng, Lei Zhang
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8

The 2D convolutional (Conv2d) layer is the fundamental element to a deep convolutional neural network (CNN). Despite the great success of CNN, the conventional Conv2d is still limited in effectively reducing the spatial and channel-wise redundancy of features. In this paper, we propose to mitigate this issue by learning a CNN with a learnable sparse transform (LST), which converts the input features into a more compact and sparser domain so that the spatial and channel-wise redundancy can be more effectively reduced. The proposed LST can be efficiently implemented with existing CNN modules, such as point-wise and depth-wise separable convolutions, and it is portable to existing CNN architectures for seamless training and inference.We further present a hybrid soft thresholding and ReLU (ST-ReLU) activation scheme, making the trained network, namely LST-Net, more robust to image corruptions at the inference stage. Extensive experiments on CIFAR-10/100, ImageNet, ImageNet-C and Places365-Standard datasets validated that the proposed LST-Net can obtain even higher accuracy than its counterpart networks with fewer parameters and less overhead.
CN: Channel Normalization For Point Cloud Recognition
Authors: Zetong Yang, Yanan sun, Shu Liu, Xiaojuan Qi, Jiaya Jia
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8

In 3D recognition, to fuse multi-scale structure information, existing methods apply hierarchical frameworks stacked by multiple fusion layers for integrating current relative locations with structure information from the previous level. In this paper, we deeply analyze these point recognition frameworks and present a factor, called difference ratio, to measure the influence of structure information among different levels on the final representation. We discover that structure information in deeper layers is overwhelmed by information in shallower layers in generating the final features, which prevents the model from understanding the point cloud in a global view. Inspired by this observation, we propose a novel channel normalization scheme to balance structure information among different layers and avoid excessive accumulation of shallow information, which benefits the model in exploiting and integrating multilayer structure information. We evaluate our channel normalization in several core 3D recognition tasks including classification, segmentation and detection. Experimental results show that our channel normalization further boosts the performance of state-of-the-art methods effectively.
Rethinking the Defocus Blur Detection Problem and A Real-Time Deep DBD Model
Authors: Ning Zhang, Junchi Yan
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Data Augmentation, Defocus Blur Detection

Defocus blur detection (DBD) is a classical low level vision task. It has recently attracted attention focusing on designing complex convolutional neural networks (CNN) which make full use of both low level features and high level semantic information. The heavy networks used in these methods lead to low processing speed, resulting difficulty in applying to real-time applications. In this work, we propose novel perspectives on the DBD problem and design convenient approach to build a real-time cost-effective DBD model. First, we observe that the semantic information does not always relate to and sometimes mislead the blur detection. We start from the essential characteristics of the DBD problem and propose a data augmentation method accordingly to destroy the semantic information and enforce the model to learn image blur related features rather than the semantic features. A novel self-supervision training objective is proposed to enhance the model training consistency and stability. Second, by rethinking the relationship between defocus blur detection and salience detection, we identify two previously ignored but common scenarios, based on which we design a hard mining strategy to enhance the DBD model. By using the proposed techniques, our model that uses a slightly modified U-Net as backbone, improves the processing speed by more than 3 times and performs competitively against state of the art methods. Ablation study is also conducted to verify the effectiveness of each part of our proposed methods.
AutoMix: Mixup Networks for Sample Interpolation via Cooperative Barycenter Learning
Authors: Jianchao Zhu, Liangliang Shi, Junchi Yan, Hongyuan Zha
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Data Augmentation

This paper proposes new ways of sample mixing by thinking of the process as generation of barycenter in a metric space for data augmentation. First, we present an optimal-transport-based mixup technique to generate Wasserstein barycenter which works well on images with clean background and is empirically shown complementary to existing mixup methods. Then we generalize mixup to an AutoMix technique by using a learnable network to fit barycenter in a cooperative way between the classifier (a.k.a. discriminator) and generator networks. Experimental results on both multi-class and multi-label prediction tasks show the efficacy of our approach, which is also verified in the presence of unseen categories (open set) and noise.