pwc-archive/papers-with-abstracts huggingface dataset:
TRRNet: Tiered Relation Reasoning for Compositional Visual Question Answering
Authors: Xiaofeng Yang, Guosheng Lin, Fengmao Lv, Fayao Liu
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Object, Question Answering, Relation, Visual Question Answering, Visual Question Answering (VQA), Visual Reasoning

Compositional visual question answering requires reasoning over both semantic and geometry object relations. We propose a novel tiered reasoning method that dynamically selects object level candidates based on language representations and generates robust pairwise relations within the selected candidate objects. The proposed tiered relation reasoning method can be compatible with the majority of the existing visual reasoning frameworks, leading to significant performance improvement with very little extra computational cost. Moreover, we propose a policy network that decides the appropriate reasoning steps based on question complexity and current reasoning status. In experiments, our model achieves state-of-the-art performance on two VQA datasets.
Mining Inter-Video Proposal Relations for Video Object Detection
Authors: Mingfei Han, Yali Wang, Xiaojun Chang, Yu Qiao
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Object, object-detection, Object Detection, Relation, Relation Network, Triplet, Video Object Detection

Recent studies have shown that, context aggregating information from proposals in different frames can clearly enhance the performance of video object detection. However, these approaches mainly exploit the intra-proposal relation within single video, while ignoring the intra-proposal relation among different videos, which can provide important discriminative cues for recognizing confusing objects. To address the limitation, we propose a novel Inter-Video Proposal Relation module. Based on a concise multi-level triplet selection scheme, this module can learn effective object representations via modeling relations of hard proposals among different videos. Moreover, we design a Hierarchical Video Relation Network (HVR-Net), by integrating intra-video and inter-video proposal relations in a hierarchical fashion. This design can progressively exploit both intra and inter contexts to boost video object detection. We examine our method on the large-scale video object detection benchmark, i.e., ImageNet VID, where HVR-Net achieves the SOTA results. Codes and models will be released afterwards.
Large Batch Optimization for Object Detection: Training COCO in 12 Minutes
Authors: Tong Wang, Yousong Zhu, Chaoyang Zhao, Wei Zeng, Yao-Wei Wang, Jinqiao Wang, Ming Tang
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: object-detection, Object Detection

Most of existing object detectors usually adopt a small training batch size ( ~16), which severely hinders the whole community from exploring large-scale datasets due to the extremely long training procedure. In this paper, we propose a versatile large batch optimization framework for object detection, named LargeDet, which successfully scales the batch size to larger than 1K for the first time. Specifically, we present a novel Periodical Moments Decay LAMB (PMD-LAMB) algorithm to effectively reduce the negative effects of the lagging historical gradients. Additionally, the Synchronized Batch Normalization (SyncBN) is utilized to help fast convergence. With LargeDet, we can not only prominently shorten the training period, but also significantly improve the detection accuracy of sparsely annotated large-scale datasets. For instance, we can finish the training of ResNet50 FPN detector on COCO within 12 minutes. Moreover, we achieve 12.2% mAP@0.5 absolute improvement for ResNet50 FPN on Open Images by training with batch size 640.
Towards Practical and Efficient High-Resolution HDR Deghosting with CNN
Authors: K. Ram Prabhakar, Susmit Agrawal, Durgesh Kumar Singh, Balraj Ashwath , R. Venkatesh Babu
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Vocal Bursts Intensity Prediction

Generating High Dynamic Range (HDR) image in the presence of camera and object motion is a tedious task. If uncorrected, these motions will manifest as ghosting artifacts in the fused HDR image. On one end of the spectrum, there exist methods that generate high-quality results that are computationally demanding and too slow. On the other end, there are few faster methods that produce unsatisfactory results. With ever increasing sensor/display resolution, currently we are very much in need of faster methods that produce high-quality images. In this paper, we present a deep neural network based approach to generate high-quality ghost-free HDR for high-resolution images. Our proposed method is fast and fuses a sequence of three high-resolution images (16-megapixel resolution) in about 10 seconds. Through experiments and ablations, on different publicly available datasets, we show that the proposed method achieves state-of-the-art performance in terms of accuracy and speed.
Sparse-to-Dense Depth Completion Revisited: Sampling Strategy and Graph Construction
Authors: Xin Xiong, Haipeng Xiong, Ke Xian, Chen Zhao, Zhiguo Cao, Xin Li
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Depth Completion, graph construction, Missing Values

Depth completion is a widely studied problem of predicting a dense depth map from a sparse set of measurements and a single RGB image. In this work, we approach this problem by addressing two issues that have been under-researched in the open literature: sampling strategy (data term) and graph construction (prior term). First, instead of the popular random sampling strategy, we suggest that Poisson disk sampling is a much more effective solution to create sparse depth map from a dense version. We experimentally compare a class of quasi-random sampling strategies and demonstrate that an optimized sampling strategy can significantly improve the performance of depth completion for the same number of sparse samples. Second, instead of the traditional square kernel, we suggest that dynamic construction of local neighborhood is a better choice for interpolating the missing values. More specifically, we proposed an end-to-end network with a graph convolution module. Since the neighborhood relationship of 3D points is more effectively exploited by our novel graph convolution module, our approach has achieved not only state-of-the-art results for depth completion of indoor scenes but also better generalization ability than other competing methods.
MEAD: A Large-scale Audio-visual Dataset for Emotional Talking-face Generation
Authors: Kaisiyuan Wang Qianyi Wu Linsen Song Zhuoqian Yang Wayne Wu Chen Qian Ran He Yu Qiao Chen Change Loy
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Face Generation, Talking Face Generation, Talking Head Generation, Video Generation

The synthesis of natural emotional reactions is an essentialcriteria in vivid talking-face video generation. This criteria is nevertheless seldom taken into consideration in previous works due to the absence of a large-scale, high-quality emotional audio-visual dataset. To address this issue, we build the Multi-view Emotional Audio-visual Dataset(MEAD) which is a talking-face video corpus featuring 60 actors and actresses talking with 8 different emotions at 3 different intensity levels. High-quality audio-visual clips are captured at 7 different view angles in a strictly-controlled environment. Together with the dataset, we release an emotional talking-face generation baseline which enables the manipulation of both emotion and its intensity. Our dataset will be made public and could benefit a number of different research fields including conditional generation, cross-modal understanding and expression recognition.
Detecting Human-Object Interactions with Action Co-occurrence Priors
Authors: Dong-Jin Kim Xiao Sun Jinsoo Choi Stephen Lin In So Kweon
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Human-Object Interaction Detection

A common problem in human-object interaction (HOI) detection task is that numerous HOI classes have only a small number of labeled examples, resulting in training sets with a long-tailed distribution. The lack of positive labels can lead to low classification accuracy for these classes. Towards addressing this issue, we observe that there exist natural correlations and anti-correlations among human-object interactions. In this paper, we model the correlations as action co-occurrence matrices and present techniques to learn these priors and leverage them for more effective training, especially in rare classes. The utility of our approach is demonstrated experimentally, where the performance of our approach exceeds the state-of-the-art methods on both of the two leading HOI detection benchmark datasets, HICO-Det and V-COCO.
JSTASR: Joint Size and Transparency-Aware Snow Removal Algorithm Based on Modified Partial Convolution and Veiling Effect Removal
Authors: Wei-Ting Chen, Hao-Yu Fang, Jian-Jiun Ding, Cheng-Che Tsai, Sy-Yen Kuo
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: object-detection, Object Detection, Snow Removal

Snow removal usually affects the performance of computer vision. Comparing with other atmospheric phenomenon (e.g., haze and rain), snow is more complicated due to its transparency, various size, and accumulation of veiling effect, which make single image de-snowing more challenging. In this paper, first, we reformulate the snow model. Different from that in the previous works, in the proposed snow model, the veiling effect is included. Second, a novel joint size and transparency-aware snow removal algorithm called JSTASR is proposed. It can classify snow particles according to their sizes and conduct snow removal in different scales. Moreover, to remove the snow with different transparency, the transparency-aware snow removal is developed. It can address both transparent and non-transparent snow particles by applying the modified partial convolution. Experiments show that the proposed method achieves significant improvement on both synthetic and real-world datasets and is very helpful for object detection on snow images.
Object Tracking using Spatio-Temporal Networks for Future Prediction Location
Authors: Yuan Liu, Ruoteng Li, Yu Cheng, Robby T. Tan, Xiubao Sui
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Future prediction, Object, Object Tracking

We introduce an object tracking algorithm that predicts the future locations of the target object and assists the tracker to handle object occlusion. Given a few frames of an object that are extracted from a complete input sequence, we aim to predict the object’s location in the future frames. To facilitate the future prediction ability, we follow three key observations: 1) object motion trajectory is affected significantly by camera motion; 2) the past trajectory of an object can act as a salient cue to estimate the object motion in the spatial domain; 3) previous frames contain the surroundings and appearance of the target object, which is useful for predicting the target object’s future locations. We incorporate these three observations into our method that employs a multi-stream convolutional-LSTM network. By combining the heatmap scores from our tracker (that utilises appearance inference) and the locations of the target object from our trajectory inference, we predict the final target’s location in each frame. Comprehensive evaluations show that our method sets new state-of-the-art performance on a few commonly used tracking benchmarks.
Sparse Adversarial Attack via Perturbation Factorization
Authors: Yanbo Fan, Baoyuan Wu, Tuanhui Li, Yong Zhang, Mingyang Li, Zhifeng Li, Yujiu Yang
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Adversarial Attack

This work studies the sparse adversarial attack, which aims to generate adversarial perturbations onto partial positions of one benign image, such that the perturbed image is incorrectly predicted by one deep neural network (DNN) model. The sparse adversarial attack involves two challenges, i.e., where to perturb, and how to determine the perturbation magnitude. Many existing works determined the perturbed positions manually or heuristically, and then optimized the magnitude using a proper algorithm designed for the dense adversarial attack. In this work, we propose to factorize the perturbation at each pixel to the product of two variables, including the perturbation magnitude and one binary selection factor (i.e., 0 or 1). One pixel is perturbed if its selection factor is 1, otherwise not perturbed. Based on this factorization, we formulate the sparse attack problem as a mixed integer programming (MIP) to jointly optimize the binary selection factors and continuous perturbation magnitudes of all pixels, with a cardinality constraint on selection factors to explicitly control the degree of sparsity. Besides, the perturbation factorization provides the extra flexibility to incorporate other meaningful constraints on selection factors or magnitudes to achieve some desired performance, such as the group-wise sparsity or the enhanced visual imperceptibility. We develop an efficient algorithm by equivalently reformulating the MIP problem as a continuous optimization problem. Experiments on benchmark databases demonstrate the superiority of the proposed method over several state-of-the-art sparse attack methods.
3D Scene Reconstruction from a Single Viewport
Authors: Maximilian Denninger, Rudolph Triebel
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: 3D Scene Reconstruction

We present a novel approach to infer volumetric reconstructions from a single viewport, based only on an RGB image and a reconstructed normal image. To overcome the problem of reconstructing regions in 3D that are occluded in the 2D image, we propose to learn this information from synthetically generated high-resolution data. To do this, we introduce a deep network architecture that is specifically designed for volumetric TSDF data by featuring a specific tree net architecture. Our framework can handle a 3D resolution of $512^3$ by introducing a dedicated compression technique based on a modified autoencoder. Furthermore, we introduce a novel loss shaping technique for 3D data that guides the learning process towards regions where free and occupied space are close to each other. As we show in experiments on synthetic and realistic benchmark data, this leads to very good reconstruction results, both visually and in terms of quantitative measures.
Leveraging Acoustic Images for Effective Self-Supervised Audio Representation Learning
Authors: Valentina Sanguineti, Pietro Morerio, Niccolò Pozzetti, Danilo Greco, Marco Cristani, Vittorio Murino
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Cross-Modal Retrieval, Representation Learning, Retrieval, Scene Understanding, Self-Supervised Learning

In this paper, we propose the use of a new modality characterized by a richer information content, namely acoustic images, for the sake of audio-visual scene understanding. Each pixel in such images is characterized by a spectral signature, associated to a specific direction in space and obtained by processing the audio signals coming from an array of microphones. By coupling such array with a video camera, we obtain spatio-temporal alignment of acoustic images and video frames. This constitutes a powerful source of self-supervision, which can be exploited in the learning pipeline we are proposing, without resorting to expensive data annotations. However, since 2D planar arrays are cumbersome and not as widespread as ordinary microphones, we propose that the richer information content of acoustic images can be distilled, through a self-supervised learning scheme, into more powerful audio and visual feature representations. The learnt feature representations can then be employed for downstream tasks such as classification and cross-modal retrieval, without the need of a microphone array. To prove that, we introduce a novel multimodal dataset consisting in RGB videos, raw audio signals and acoustic images, aligned in space and synchronized in time. Experimental results demonstrate the validity of our hypothesis and the effectiveness of the proposed pipeline, also when tested for tasks and datasets different from those used for training.
Learning Joint Visual Semantic Matching Embeddings for Language-guided Retrieval
Authors: Yanbei Chen, Loris Bazzani
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Image Retrieval, Retrieval, Specificity, Text Retrieval

Interactive image retrieval is an emerging research topic with the objective of integrating inputs from multiple modalities as query for retrieval, e.g., textual feedback from users to guide, modify or refine image retrieval. In this work, we study the problem of composing images and textual modifications for language-guided retrieval in the context of fashion applications. We propose a unified Joint Visual Semantic Matching (JVSM) model that learns image-text compositional embeddings by jointly associating visual and textual modalities in a shared discriminative embedding space via compositional losses. JVSM has been designed with versatility and flexibility in mind, being able to perform multiple image and text tasks in a single model, such as text-image matching and language-guided retrieval. We show the effectiveness of our approach in the fashion domain, where it is difficult to express keyword-based queries given the complex specificity of fashion terms. Our experiments on three datasets (Fashion-200k, UT-Zap50k, and Fashion-iq) show that JVSM achieves state-of-the-art results on language-guided retrieval and additionally we show its capabilities to perform image and text retrieval.
Globally Optimal and Efficient Vanishing Point Estimation in Atlanta World
Authors: Haoang Li, Pyojin Kim, Ji Zhao, Kyungdon Joo, Zhipeng Cai, Zhe Liu , Yun-hui Liu
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8

Atlanta world holds for the scenes composed of a vertical dominant direction and several horizontal dominant directions. Vanishing point (VP) is the intersection of the image lines projected from parallel 3D lines. In Atlanta world, given a set of image lines, we aim to cluster them by the unknown-but-sought VPs whose number is unknown. Existing approaches are prone to missing partial inliers, rely on prior knowledge of the number of VPs, and/or lead to low efficiency. To overcome these limitations, we propose the novel mine-and-stab (MnS) algorithm and embed it in the branch-and-bound (BnB) algorithm. Different from BnB that iteratively branches the full parameter intervals, our MnS directly mines the narrow sub-intervals and then stabs them by probes. We simultaneously search for the vertical VP by BnB and horizontal VPs by MnS. The proposed collaboration between BnB and MnS guarantees global optimality in terms of maximizing the number of inliers. It can also automatically determine the number of VPs. Moreover, its efficiency is suitable for practical applications. Experiments on synthetic and real-world datasets showed that our method outperforms state-of-the-art approaches in terms of accuracy and/or efficiency.
LatticeNet: Towards Lightweight Image Super-resolution with Lattice Block
Authors: Xiaotong Luo, Yuan Xie, Yulun Zhang, Yanyun Qu, Cuihua Li, Yun Fu
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Image Super-Resolution, Super-Resolution

Deep neural networks with a massive number of layers have made a remarkable breakthrough on single image super-resolution (SR), but sacrifice computation complexity and memory storage. To address this problem, we focus on the lightweight models for fast and accurate image SR. Due to the frequent use of residual block (RB) in SR models, we pursue an economical structure to adaptively combine RBs. Drawing lessons from lattice filter bank, we design the lattice block (LB) in which two butterfly structures are applied to combine two RBs. LB has the potential of various linear combinations of two RBs. Each case of LB depends on the combination coefficients which are determined by the attention mechanism. LB favors the lightweight SR model with the reduction of about half amount of the parameters while keeping the similar SR performance. Moreover, we propose a lightweight SR model, LatticeNet, which uses series connection of LBs and the backward feature fusion. Extensive experiments demonstrate that our proposal can achieve superior accuracy on four available benchmark datasets against other state-of-the-art methods, while maintaining relatively low computation and memory requirements.
PG-Net: Pixel to Global Matching Network for Visual Tracking
Authors: Bingyan Liao, Chenye Wang, Yayun Wang, Yaonong Wang, Jun Yin
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Visual Tracking

Siamese neural network has been well investigated by tracking frameworks due to its fast speed and high accuracy. However, very few efforts were spent on background-extraction by those approaches. In this paper, a Pixel to Global Matching Network (PG-Net) is proposed to suppress the influence of background in search image while achieving state-of-the-art tracking performance. To achieve this purpose, each pixel on search feature is utilized to calculate the similarity with global template feature. This calculation method can appropriately reduce the matching area, thus introducing less background interference. In addition, we propose a new tracking framework to perform correlation-shared tracking and multiple losses for training, which not only reduce the computational burden but also improve the performance. We conduct comparison experiments on various public tracking datasets, which obtains state-of-the-art performance while running with fast speed.
Geometric Estimation via Robust Subspace Recovery
Authors: Aoxiang Fan, Xingyu Jiang, Yang Wang, Junjun Jiang, Jiayi Ma
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: Homography Estimation, Pose Estimation

Geometric estimation from image point correspondences is the core procedure of many 3D vision problems, which is prevalently accomplished by random sampling techniques. In this paper, we consider the problem from an optimization perspective, to exploit the intrinsic linear structure of point correspondences to assist estimation. We generalize the conventional method to a robust one and extend the previous analysis for linear structure to develop several new algorithms. The proposed solutions essentially address the estimation problem by solving a subspace recovery problem to identify the inliers. Experiments on real-world image datasets for both fundamental matrix and homography estimation demonstrate the superiority of our method over the state-of-the-art in terms of both robustness and accuracy.
Learning Memory Augmented Cascading Network for Compressed Sensing of Images
Authors: Jiwei Chen, Yubao Sun, Qingshan Liu, Rui Huang
Date: 2020-08-01 00:00:00
Proceeding: ECCV 2020 8
Tasks: compressed sensing

In this paper, we propose a cascading network for compressed sensing of images with progressive reconstruction. Specifically, we decompose the complex reconstruction mapping into the cascade of incremental detail reconstruction (IDR) modules and measurement residual updating (MRU) modules. The IDR module is designed to reconstruct the remaining details from the residual measurement vector, and MRU is employed to update the residual measurement vector and feed it into the next IDR module. The contextual memory module is introduced to augment the capacity of IDR modules, therefore facilitating the information interaction among all the IDR modules. The final reconstruction is calculated by accumulating the outputs of all the IDR modules. Extensive experiments on natural images and magnetic resonance images demonstrate the proposed method achieves better performance against the state-of-the-art methods.