pwc-archive/papers-with-abstracts huggingface dataset:
Adaptive Rank Estimate in Robust Principal Component Analysis
Authors: Zhengqin Xu, Rui He, Shoulie Xie, Shiqian Wu
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1

    Robust principal component analysis (RPCA) and its variants have gained wide applications in computer vision. However, these methods either involve manual adjustment of some parameters, or require the rank of a low-rank matrix to be known a prior. In this paper, an adaptive rank estimate based RPCA (ARE-RPCA) is proposed, which adaptively assigns weights on different singular values via rank estimation. More specifically, we study the characteristics of the low-rank matrix, and develop an improved Gerschgorin disk theorem to estimate the rank of the low-rank matrix accurately. Furthermore in view of the issue occurred in the Gerschgorin disk theorem that adjustment factor need to be manually pre-defined, an adaptive setting method, which greatly facilitates the practical implementation of the rank estimation, is presented. Then, the weights of singular values in the nuclear norm are updated adaptively based on iteratively estimated rank, and the resultant low-rank matrix is close to the target. Experimental results show that the proposed ARE-RPCA outperforms the state-of-the-art methods in various complex scenarios.
Learning Spatially-Variant MAP Models for Non-Blind Image Deblurring
Authors: Jiangxin Dong, Stefan Roth, Bernt Schiele
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Blind Image Deblurring, Deblurring, Image Deblurring

    The classical maximum a-posteriori (MAP) framework for non-blind image deblurring requires defining suitable data and regularization terms, whose interplay yields the desired clear image through optimization. The vast majority of prior work focuses on advancing one of these two crucial ingredients, while keeping the other one standard. Considering the indispensable roles and interplay of both data and regularization terms, we propose a simple and effective approach to jointly learn these two terms, embedding deep neural networks within the constraints of the MAP framework, trained in an end-to-end manner. The neural networks not only yield suitable image-adaptive features for both terms, but actually predict per-pixel spatially-variant features instead of the commonly used spatially-uniform ones. The resulting spatially-variant data and regularization terms particularly improve the restoration of fine-scale structures and detail. Quantitative and qualitative results underline the effectiveness of our approach, substantially outperforming the current state of the art.
Hierarchical Video Prediction Using Relational Layouts for Human-Object Interactions
Authors: Navaneeth Bodla, Gaurav Shrivastava, Rama Chellappa, Abhinav Shrivastava
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Human-Object Interaction Detection, Object, Relational Reasoning, SSIM, Video Generation, Video Prediction

    Learning to model and predict how humans interact with objects while performing an action is challenging, and most of the existing video prediction models are ineffective in modeling complicated human-object interactions. Our work builds on hierarchical video prediction models, which disentangle the video generation process into two stages: predicting a high-level representation, such as pose sequence, and then learning a pose-to-pixels translation model for pixel generation. An action sequence for a human-object interaction task is typically very complicated, involving the evolution of pose, person's appearance, object locations, and object appearances over time. To this end, we propose a Hierarchical Video Prediction model using Relational Layouts. In the first stage, we learn to predict a sequence of layouts. A layout is a high-level representation of the video containing both pose and objects' information for every frame. The layout sequence is learned by modeling the relationships between the pose and objects using relational reasoning and recurrent neural networks. The layout sequence acts as a strong structure prior to the second stage that learns to map the layouts into pixel space. Experimental evaluation of our method on two datasets, UMD-HOI and Bimanual, shows significant improvements in standard video evaluation metrics such as LPIPS, PSNR, and SSIM. We also perform a detailed qualitative analysis of our model to demonstrate various generalizations.
Intrinsic Image Harmonization
Authors: Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, Bing Zheng
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Benchmarking, Image Harmonization

    Compositing an image usually inevitably suffers from inharmony problem that is mainly caused by incompatibility of foreground and background from two different images with distinct surfaces and lights, corresponding to material-dependent and light-dependent characteristics, namely, reflectance and illumination intrinsic images, respectively. Therefore, we seek to solve image harmonization via separable harmonization of reflectance and illumination, i.e., intrinsic image harmonization. Our method is based on an autoencoder that disentangles composite image into reflectance and illumination for further separate harmonization. Specifically, we harmonize reflectance through material-consistency penalty, while harmonize illumination by learning and transferring light from background to foreground, moreover, we model patch relations between foreground and background of composite images in an inharmony-free learning way, to adaptively guide our intrinsic image harmonization. Both extensive experiments and ablation studies demonstrate the power of our method as well as the efficacy of each component. We also contribute a new challenging dataset for benchmarking illumination harmonization. Code and dataset are at https://github.com/zhenglab/IntrinsicHarmony.
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions
Authors: Junbin Xiao, Xindi Shang, Angela Yao, Tat-Seng Chua
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Question Answering, Video Question Answering, Video Understanding, Visual Question Answering (VQA)

    We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions. Based on the dataset, we set up multi-choice and open-ended QA tasks targeting at causal action reasoning, temporal action reasoning and common scene comprehension. Through extensive analysis of baselines and established VideoQA techniques, we find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning. Furthermore, the models that are effective on multi-choice QA, when adapted to open-ended QA, still struggle in generalizing the answers. This raises doubt on the ability of these models to reason and highlights possibilities for improvement. With detailed results for different question types and heuristic observations for future works, we hope NExT-QA will guide the next generation of VQA research to go beyond superficial description towards a deeper understanding of videos.
Adaptive Image Transformer for One-Shot Object Detection
Authors: Ding-Jie Chen, He-Yen Hsieh, Tyng-Luh Liu
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Decoder, Object, object-detection, Object Detection, One-Shot Learning, One-Shot Object Detection

    One-shot object detection tackles a challenging task that aims at identifying within a target image all object instances of the same class, implied by a query image patch. The main difficulty lies in the situation that the class label of the query patch and its respective examples are not available in the training data. Our main idea leverages the concept of language translation to boost metric-learning-based detection methods. Specifically, we emulate the language translation process to adaptively translate the feature of each object proposal to better correlate the given query feature for discriminating the class-similarity among the proposal-query pairs. To this end, we propose the Adaptive Image Transformer (AIT) module that deploys an attention-based encoder-decoder architecture to simultaneously explore intra-coder and inter-coder (i.e., each proposal-query pair) attention. The adaptive nature of our design turns out to be flexible and effective in addressing the one-shot learning scenario. With the informative attention cues, the proposed model excels in predicting the class-similarity between the target image proposals and the query image patch. Though conceptually simple, our model significantly outperforms a state-of-the-art technique, improving the unseen-class object classification from 63.8 mAP and 22.0 AP50 to 72.2 mAP and 24.3 AP50 on the PASCAL-VOC and MS-COCO benchmark datasets, respectively.
Bottom-Up Shift and Reasoning for Referring Image Segmentation
Authors: Sibei Yang, Meng Xia, Guanbin Li, Hong-Yu Zhou, Yizhou Yu
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Image Segmentation, Segmentation, Semantic Segmentation, Visual Reasoning

    Referring image segmentation aims to segment the referent that is the corresponding object or stuff referred by a natural language expression in an image. Its main challenge lies in how to effectively and efficiently differentiate between the referent and other objects of the same category as the referent. In this paper, we tackle the challenge by jointly performing compositional visual reasoning and accurate segmentation in a single stage via the proposed novel Bottom-Up Shift (BUS) and Bidirectional Attentive Refinement (BIAR) modules. Specifically, BUS progressively locates the referent along hierarchical reasoning steps implied by the expression. At each step, it locates the corresponding visual region by disambiguating between similar regions, where the disambiguation bases on the relationships between regions. By the explainable visual reasoning, BUS explicitly aligns linguistic components with visual regions so that it can identify all the mentioned entities in the expression. BIAR fuses multi-level features via a two-way attentive message passing, which captures the visual details relevant to the referent to refine segmentation results. Experimental results demonstrate that the proposed method consisting of BUS and BIAR modules, can not only consistently surpass all existing state-of-the-art algorithms across common benchmark datasets but also visualize interpretable reasoning steps for stepwise segmentation. Code is available at https://github.com/incredibleXM/BUSNet.
Mesh Saliency: An Independent Perceptual Measure or a Derivative of Image Saliency?
Authors: Ran Song, Wei zhang, Yitian Zhao, Yonghuai Liu, Paul L. Rosin
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1

    While mesh saliency aims to predict regional importance of 3D surfaces in agreement with human visual perception and is well researched in computer vision and graphics, latest work with eye-tracking experiments shows that state-of-the-art mesh saliency methods remain poor at predicting human fixations. Cues emerging prominently from these experiments suggest that mesh saliency might associate with the saliency of 2D natural images. This paper proposes a novel deep neural network for learning mesh saliency using image saliency ground truth to 1) investigate whether mesh saliency is an independent perceptual measure or just a derivative of image saliency and 2) provide a weakly supervised method for more accurately predicting mesh saliency. Through extensive experiments, we not only demonstrate that our method outperforms the current state-of-the-art mesh saliency method by 116% and 21% in terms of linear correlation coefficient and AUC respectively, but also reveal that mesh saliency is intrinsically related with both image saliency and object categorical information. Codes are available at https://github.com/rsong/MIMO-GAN.
Lesion-Aware Transformers for Diabetic Retinopathy Grading
Authors: Rui Sun, Yihao Li, Tianzhu Zhang, Zhendong Mao, Feng Wu, Yongdong Zhang
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Decoder, Diabetic Retinopathy Grading, Diversity

    Diabetic retinopathy (DR) is the leading cause of permanent blindness in the working-age population. And automatic DR diagnosis can assist ophthalmologists to design tailored treatments for patients, including DR grading and lesion discovery. However, most of existing methods treat DR grading and lesion discovery as two independent tasks, which require lesion annotations as a learning guidance and limits the actual deployment. To alleviate this problem, we propose a novel lesion-aware transformer (LAT) for DR grading and lesion discovery jointly in a unified deep model via an encoder-decoder structure including a pixel relation based encoder and a lesion filter based decoder. The proposed LAT enjoys several merits. First, to the best of our knowledge, this is the first work to formulate lesion discovery as a weakly supervised lesion localization problem via a transformer decoder. Second, to learn lesion filters well with only image-level labels, we design two effective mechanisms including lesion region importance and lesion region diversity for identifying diverse lesion regions. Extensive experimental results on three challenging benchmarks including Messidor-1, Messidor-2 and EyePACS demonstrate that the proposed LAT performs favorably against state-of-the-art DR grading and lesion discovery methods.
Explicit Knowledge Incorporation for Visual Reasoning
Authors: Yifeng Zhang, Ming Jiang, Qi Zhao
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Visual Reasoning

    Existing explainable and explicit visual reasoning methods only perform reasoning based on visual evidence but do not take into account knowledge beyond what is in the visual scene. To addresses the knowledge gap between visual reasoning methods and the semantic complexity of real-world images, we present the first explicit visual reasoning method that incorporates external knowledge and models high-order relational attention for improved generalizability and explainability. Specifically, we propose a knowledge incorporation network that explicitly creates and includes new graph nodes for entities and predicates from external knowledge bases to enrich the semantics of the scene graph used in explicit reasoning. We then create a novel Graph-Relate module to perform high-order relational attention on the enriched scene graph. By explicitly introducing structured external knowledge and high-order relational attention, our method demonstrates significant generalizability and explainability over the state-of-the-art visual reasoning approaches on the GQA and VQAv2 datasets.
Dual-GAN: Joint BVP and Noise Modeling for Remote Physiological Measurement
Authors: Hao Lu, Hu Han, S. Kevin Zhou
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Disentanglement, Emotion Recognition, Heart Rate Variability

    Remote photoplethysmography (rPPG) based physiological measurement has great application values in health monitoring, emotion analysis, etc. Existing methods mainly focus on how to enhance or extract the very weak blood volume pulse (BVP) signals from face videos, but seldom explicitly model the noises that dominate face video content. Thus, they may suffer from poor generalization ability in unseen scenarios. This paper proposes a novel adversarial learning approach for rPPG based physiological measurement by using Dual Generative Adversarial Networks (Dual-GAN) to model the BVP estimation and noise distribution jointly. The BVP-GAN aims to learn a noise-resistant mapping from input to ground-truth BVP, and the Noise-GAN aims to learn the noise distribution. The dual GANs can promote each other's capability, leading to improved feature disentanglement between BVP and noises. Besides, a plug-and-play block named ROI alignment and fusion (ROI-AF) block is proposed to alleviate the inconsistencies between different ROIs and exploit informative features from a wider receptive field in terms of ROIs. In comparison to state-of-the-art methods, our method achieves better performance in heart rate, heart rate variability, and respiration frequency estimation from face videos.
Bilinear Parameterization for Non-Separable Singular Value Penalties
Authors: Marcus Valtonen Ornhag, Jose Pedro Iglesias, Carl Olsson
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Second-order methods

    Low rank inducing penalties have been proven to successfully uncover fundamental structures considered in computer vision and machine learning; however, such methods generally lead to non-convex optimization problems. Since the resulting objective is non-convex one often resorts to using standard splitting schemes such as Alternating Direction Methods of Multipliers (ADMM), or other subgradient methods, which exhibit slow convergence in the neighbourhood of a local minimum. We propose a method using second order methods, in particular the variable Projection method (VarPro), by replacing the non-convex penalties with a surrogate capable of converting the original objectives to differentiable equivalents. In this way we benefit from faster convergence. The bilinear framework is compatible with a large family of regularizers, and we demonstrate the benefits of our approach on real datasets for rigid and non-rigid structure from motion. The qualitative difference in reconstructions show that many popular non-convex objectives enjoy an advantage in transitioning to the proposed framework.
Composing Photos Like a Photographer
Authors: Chaoyi Hong, Shuaiyuan Du, Ke Xian, Hao Lu, Zhiguo Cao, Weicai Zhong
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Image Cropping

    We show that explicit modeling of composition rules benefits image cropping. Image cropping is considered a promising way to automate aesthetic composition in professional photography. Existing efforts, however, only model such professional knowledge implicitly, e.g., by ranking from comparative candidates. Inspired by the observation that natural composition traits always follow a specific rule, we propose to learn such rules in a discriminative manner, and more importantly, to incorporate learned composition clues explicitly in the model. To this end, we introduce the concept of the key composition map (KCM) to encode the composition rules. The KCM can reveal the common laws hidden behind different composition rules and can inform the cropping model of what is important in composition. With the KCM, we present a novel cropping-by-composition paradigm and instantiate a network to implement composition-aware image cropping. Extensive experiments on two benchmarks justify that our approach enables effective, interpretable, and fast image cropping.
Learning Deep Latent Variable Models by Short-Run MCMC Inference With Optimal Transport Correction
Authors: Dongsheng An, Jianwen Xie, Ping Li
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1

    Learning latent variable models with deep top-down architectures typically requires inferring the latent variables for each training example based on the posterior distribution of these latent variables. The inference step typically relies on either time-consuming long run Markov chain Monte Caro (MCMC) or a separate inference model for variational learning. In this paper, we propose to use short run MCMC, such as Langevin dynamics, as an approximate inference engine, where the bias existing in the output distribution of the short run Langevin dynamics is corrected by optimal transport, which aims at minimizing the Wasserstein distance between the biased distribution produced by the finite step Langevin dynamics and the prior distribution. Our experiments show that the proposed strategy outperforms the variational auto-encoder (VAE) and alternating back-propagation algorithm (ABP) in terms of reconstruction error and synthesis quality.
Double Low-Rank Representation With Projection Distance Penalty for Clustering
Authors: Zhiqiang Fu, Yao Zhao, Dongxia Chang, Xingxing Zhang, Yiming Wang
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1
Tasks: Clustering

    This paper presents a novel, simple yet robust self-representation method, i.e., Double Low-Rank Representation with Projection Distance penalty (DLRRPD) for clustering. With the learned optimal projected representations, DLRRPD is capable of obtaining an effective similarity graph to capture the multi-subspace structure. Besides the global low-rank constraint, the local geometrical structure is additionally exploited via a projection distance penalty in our DLRRPD, thus facilitating a more favorable graph. Moreover, to improve the robustness of DLRRPD to noises, we introduce a Laplacian rank constraint, which can further encourage the learned graph to be more discriminative for clustering tasks. Meanwhile, Frobenius norm (instead of the popularly used nuclear norm) is employed to enforce the graph to be more block-diagonal with lower complexity. Extensive experiments have been conducted on synthetic, real, and noisy data to show that the proposed method outperforms currently available alternatives by a margin of 1.0% 10.1%.
Learning To Segment Actions From Visual and Language Instructions via Differentiable Weak Sequence Alignment
Authors: YuHan Shen, Lu Wang, Ehsan Elhamifar
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1

    We address the problem of unsupervised localization of key-steps and feature learning in instructional videos using both visual and language instructions. Our key observation is that the sequences of visual and linguistic key-steps are weakly aligned: there is an ordered one-to-one correspondence between most visual and language key-steps, while some key-steps in one modality are absent in the other. To recover the two sequences, we develop an ordered prototype learning module, which extracts visual and linguistic prototypes representing key-steps. On the other hand, to find weak alignment and perform feature learning, we develop a differentiable weak sequence alignment (DWSA) method that finds ordered one-to-one matching between sequences while allowing some items in a sequence to stay unmatched. We develop an efficient forward and backward algorithm for computing the alignment and the loss derivative with respect to parameters of visual and language feature learning modules. By experiments on two instructional video datasets, we show that our method significantly improves the state of the art.
Lighting, Reflectance and Geometry Estimation From 360deg Panoramic Stereo
Authors: Junxuan Li, Hongdong Li, Yasuyuki Matsushita
Date: 2021-06-19 00:00:00
Proceeding: CVPR 2021 1

    We propose a method for estimating high-definition spatially-varying lighting, reflectance, and geometry of a scene from 360deg stereo images. Our model takes advantage of the 360deg input to observe the entire scene with geometric detail, then jointly estimates the scene's properties with physical constraints. We first reconstruct a near-field environment light for predicting the lighting at any 3D location within the scene. Then we present a deep learning model that leverages the stereo information to infer the reflectance and surface normal. Lastly, we incorporate the physical constraints between lighting and geometry to refine the reflectance of the scene. Both quantitative and qualitative experiments show that our method, benefiting from the 360deg observation of the scene, outperforms prior state-of-the-art methods and enables more augmented reality applications such as mirror-objects insertion.