pwc-archive/papers-with-abstracts huggingface dataset:
Global-Local Propagation Network for RGB-D Semantic Segmentation
Authors: Sihan Chen, Xinxin Zhu, Wei Liu, Xingjian He, Jing Liu
Date: 2021-01-26 00:00:00
Tasks: Decoder, Scene Segmentation, Segmentation, Semantic Segmentation

Depth information matters in RGB-D semantic segmentation task for providing additional geometric information to color images. Most existing methods exploit a multi-stage fusion strategy to propagate depth feature to the RGB branch. However, at the very deep stage, the propagation in a simple element-wise addition manner can not fully utilize the depth information. We propose Global-Local propagation network (GLPNet) to solve this problem. Specifically, a local context fusion module(L-CFM) is introduced to dynamically align both modalities before element-wise fusion, and a global context fusion module(G-CFM) is introduced to propagate the depth information to the RGB branch by jointly modeling the multi-modal global context features. Extensive experiments demonstrate the effectiveness and complementarity of the proposed fusion modules. Embedding two fusion modules into a two-stream encoder-decoder structure, our GLPNet achieves new state-of-the-art performance on two challenging indoor scene segmentation datasets, i.e., NYU-Depth v2 and SUN-RGBD dataset.
ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning
Authors: Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, Yale Song
Date: 2021-01-26 00:00:00
Proceeding: ICCV 2021 10
Tasks: Representation Learning

The natural association between visual observations and their corresponding sound provides powerful self-supervisory signals for learning video representations, which makes the ever-growing amount of online videos an attractive source of training data. However, large portions of online videos contain irrelevant audio-visual signals because of edited/overdubbed audio, and models trained on such uncurated videos have shown to learn suboptimal representations. Therefore, existing approaches rely almost exclusively on datasets with predetermined taxonomies of semantic concepts, where there is a high chance of audio-visual correspondence. Unfortunately, constructing such datasets require labor intensive manual annotation and/or verification, which severely limits the utility of online videos for large-scale learning. In this work, we present an automatic dataset curation approach based on subset optimization where the objective is to maximize the mutual information between audio and visual channels in videos. We demonstrate that our approach finds videos with high audio-visual correspondence and show that self-supervised models trained on our data achieve competitive performances compared to models trained on existing manually curated datasets. The most significant benefit of our approach is scalability: We release ACAV100M that contains 100 million videos with high audio-visual correspondence, ideal for self-supervised video representation learning.
CPTR: Full Transformer Network for Image Captioning
Authors: Wei Liu, Sihan Chen, Longteng Guo, Xinxin Zhu, Jing Liu
Date: 2021-01-26 00:00:00
Tasks: Decoder, Image Captioning

In this paper, we consider the image captioning task from a new sequence-to-sequence prediction perspective and propose CaPtion TransformeR (CPTR) which takes the sequentialized raw images as the input to Transformer. Compared to the "CNN+Transformer" design paradigm, our model can model global context at every encoder layer from the beginning and is totally convolution-free. Extensive experiments demonstrate the effectiveness of the proposed model and we surpass the conventional "CNN+Transformer" methods on the MSCOCO dataset. Besides, we provide detailed visualizations of the self-attention between patches in the encoder and the "words-to-patches" attention in the decoder thanks to the full Transformer architecture.
Recovery of underdrawings and ghost-paintings via style transfer by deep convolutional neural networks: A digital tool for art scholars
Authors: Anthony Bourached, George Cann, Ryan-Rhys Griffiths, David G. Stork
Date: 2021-01-04 00:00:00
Tasks: Art Analysis, Style Transfer

We describe the application of convolutional neural network style transfer to the problem of improved visualization of underdrawings and ghost-paintings in fine art oil paintings. Such underdrawings and hidden paintings are typically revealed by x-ray or infrared techniques which yield images that are grayscale, and thus devoid of color and full style information. Past methods for inferring color in underdrawings have been based on physical x-ray fluorescence spectral imaging of pigments in ghost-paintings and are thus expensive, time consuming, and require equipment not available in most conservation studios. Our algorithmic methods do not need such expensive physical imaging devices. Our proof-of-concept system, applied to works by Pablo Picasso and Leonardo, reveal colors and designs that respect the natural segmentation in the ghost-painting. We believe the computed images provide insight into the artist and associated oeuvre not available by other means. Our results strongly suggest that future applications based on larger corpora of paintings for training will display color schemes and designs that even more closely resemble works of the artist. For these reasons refinements to our methods should find wide use in art conservation, connoisseurship, and art analysis.
Fast Facial Landmark Detection and Applications: A Survey
Authors: Kostiantyn Khabarlak, Larysa Koriashkina
Date: 2021-01-12 00:00:00
Tasks: Emotion Recognition, Face Reenactment, Facial Landmark Detection, Survey

Dense facial landmark detection is one of the key elements of face processing pipeline. It is used in virtual face reenactment, emotion recognition, driver status tracking, etc. Early approaches were suitable for facial landmark detection in controlled environments only, which is clearly insufficient. Neural networks have shown an astonishing qualitative improvement for in-the-wild face landmark detection problem, and are now being studied by many researchers in the field. Numerous bright ideas are proposed, often complimentary to each other. However, exploration of the whole volume of novel approaches is quite challenging. Therefore, we present this survey, where we summarize state-of-the-art algorithms into categories, provide a comparison of recently introduced in-the-wild datasets (e.g., 300W, AFLW, COFW, WFLW) that contain images with large pose, face occlusion, taken in unconstrained conditions. In addition to quality, applications require fast inference, and preferably on mobile devices. Hence, we include information about algorithm inference speed both on desktop and mobile hardware, which is rarely studied. Importantly, we highlight problems of algorithms, their applications, vulnerabilities, and briefly touch on established methods. We hope that the reader will find many novel ideas, will see how the algorithms are used in applications, which will enable further research.
Exploring Transfer Learning on Face Recognition of Dark Skinned, Low Quality and Low Resource Face Data
Authors: Nuredin Ali
Date: 2021-01-03 00:00:00
Tasks: Face Recognition, Transfer Learning

There is a big difference in the tone of color of skin between dark and light skinned people. Despite this fact, most face recognition tasks almost all classical state-of-the-art models are trained on datasets containing an overwhelming majority of light skinned face images. It is tedious to collect a huge amount of data for dark skinned faces and train a model from scratch. In this paper, we apply transfer learning on VGGFace to check how it works on recognising dark skinned mainly Ethiopian faces. The dataset is of low quality and low resource. Our experimental results show above 95\% accuracy which indicates that transfer learning in such settings works.
Semi-synthesis: A fast way to produce effective datasets for stereo matching
Authors: Ju He, Enyu Zhou, Liusheng Sun, Fei Lei, Chenyang Liu, Wenxiu Sun
Date: 2021-01-26 00:00:00
Tasks: Stereo Matching

Stereo matching is an important problem in computer vision which has drawn tremendous research attention for decades. Recent years, data-driven methods with convolutional neural networks (CNNs) are continuously pushing stereo matching to new heights. However, data-driven methods require large amount of training data, which is not an easy task for real stereo data due to the annotation difficulties of per-pixel ground-truth disparity. Though synthetic dataset is proposed to fill the gaps of large data demand, the fine-tuning on real dataset is still needed due to the domain variances between synthetic data and real data. In this paper, we found that in synthetic datasets, close-to-real-scene texture rendering is a key factor to boost up stereo matching performance, while close-to-real-scene 3D modeling is less important. We then propose semi-synthetic, an effective and fast way to synthesize large amount of data with close-to-real-scene texture to minimize the gap between synthetic data and real data. Extensive experiments demonstrate that models trained with our proposed semi-synthetic datasets achieve significantly better performance than with general synthetic datasets, especially on real data benchmarks with limited training data. With further fine-tuning on the real dataset, we also achieve SOTA performance on Middlebury and competitive results on KITTI and ETH3D datasets.
Revisiting Locally Supervised Learning: an Alternative to End-to-end Training
Authors: Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, Gao Huang
Date: 2021-01-26 00:00:00
Tasks: GPU

Due to the need to store the intermediate activations for back-propagation, end-to-end (E2E) training of deep networks usually suffers from high GPUs memory footprint. This paper aims to address this problem by revisiting the locally supervised learning, where a network is split into gradient-isolated modules and trained with local supervision. We experimentally show that simply training local modules with E2E loss tends to collapse task-relevant information at early layers, and hence hurts the performance of the full model. To avoid this issue, we propose an information propagation (InfoPro) loss, which encourages local modules to preserve as much useful information as possible, while progressively discard task-irrelevant information. As InfoPro loss is difficult to compute in its original form, we derive a feasible upper bound as a surrogate optimization objective, yielding a simple but effective algorithm. In fact, we show that the proposed method boils down to minimizing the combination of a reconstruction loss and a normal cross-entropy/contrastive term. Extensive empirical results on five datasets (i.e., CIFAR, SVHN, STL-10, ImageNet and Cityscapes) validate that InfoPro is capable of achieving competitive performance with less than 40% memory footprint compared to E2E training, while allowing using training data with higher-resolution or larger batch sizes under the same GPU memory constraint. Our method also enables training local modules asynchronously for potential training acceleration. Code is available at: https://github.com/blackfeather-wang/InfoPro-Pytorch.
DataLoc+: A Data Augmentation Technique for Machine Learning in Room-Level Indoor Localization
Authors: Amr E Hilal, Ismail Arai, Samy El-Tawab
Date: 2021-01-21 00:00:00
Tasks: BIG-bench Machine Learning, Data Augmentation, Indoor Localization

Indoor localization has been a hot area of research over the past two decades. Since its advent, it has been steadily utilizing the emerging technologies to improve accuracy, and machine learning has been at the heart of that. Machine learning has been increasingly used in fingerprint-based indoor localization to replace or emulate the radio map that is used to predict locations given a location signature. The prediction quality of a machine learning model primarily depends on how well the model was trained, which relies on the amount and quality of data used to train it. Data augmentation has been used to improve quality of the trained models by synthetically producing more training data, and several approaches were used in the literature that tackles the problem of lack of training data from different angles. In this paper, we propose DataLoc+, a data augmentation technique for room-level indoor localization that combines different approaches in a simple algorithm. We evaluate the technique by comparing it to the typical direct snapshot approach using data collected from a field experiment conducted in a hospital. Our evaluation shows that the model trained using the proposed technique achieves higher accuracy. We also show that the technique adapts to larger problems using a limited dataset while maintaining high accuracy.
The Ikshana Hypothesis of Human Scene Understanding
Authors: Venkata Satya Sai Ajay Daliparthi
Date: 2021-01-21 00:00:00
Tasks: Representation Learning, Scene Understanding, Semantic Segmentation

In recent years, deep neural networks (DNNs) achieved state-of-the-art performance on several computer vision tasks. However, the one typical drawback of these DNNs is the requirement of massive labeled data. Even though few-shot learning methods address this problem, they often use techniques such as meta-learning and metric-learning on top of the existing methods. In this work, we address this problem from a neuroscience perspective by proposing a hypothesis named Ikshana, which is supported by several findings in neuroscience. Our hypothesis approximates the refining process of conceptual gist in the human brain while understanding a natural scene/image. While our hypothesis holds no particular novelty in neuroscience, it provides a novel perspective for designing DNNs for vision tasks. By following the Ikshana hypothesis, we design a novel neural-inspired CNN architecture named IkshanaNet. The empirical results demonstrate the effectiveness of our method by outperforming several baselines on the entire and subsets of the Cityscapes and the CamVid semantic segmentation benchmarks.
Three-Dimensional Investigation of the Metric Properties of Parabolic Double Projection Involving Catadioptric Camera
Authors: Ahmed Hamdy, Ahmed Elsherif, Saiid Shebl
Date: 2021-01-21 00:00:00
Tasks: Camera Calibration

This paper presents an analytical study for the metric properties of the paraboloidal double projection, i.e. central and orthogonal projections used in the catadioptric camera system. Metric properties have not sufficiently studied in previous treatments of such system. These properties incorporate the determination of the true lengths of projected lines and areas bounded by projected lines. The advantageous main gain of determining metric elements of the paraboloidal double projection is studying distortion analysis and camera calibration, which is considered an essential tool in testing camera accuracy. Also, this may be considered as a significant utility in studying comparison analysis between different cameras projection systems.
PConv: Simple yet Effective Convolutional Layer for Generative Adversarial Network
Authors: Seung Park, Yoon-Jae Yeo, Yong-Goo Shin
Date: 2021-01-19 00:00:00
Tasks: Generative Adversarial Network, Memorization

This paper presents a novel convolutional layer, called perturbed convolution (PConv), which focuses on achieving two goals simultaneously: improving the generative adversarial network (GAN) performance and alleviating the memorization problem in which the discriminator memorizes all images from a given dataset as training progresses. In PConv, perturbed features are generated by randomly disturbing an input tensor before performing the convolution operation. This approach is simple but surprisingly effective. First, to produce a similar output even with the perturbed tensor, each layer in the discriminator should learn robust features having a small local Lipschitz value. Second, since the input tensor is randomly perturbed during the training procedure like the dropout in neural networks, the memorization problem could be alleviated. To show the generalization ability of the proposed method, we conducted extensive experiments with various loss functions and datasets including CIFAR-10, CelebA, CelebA-HQ, LSUN, and tiny-ImageNet. The quantitative evaluations demonstrate that PConv effectively boosts the performance of GAN and conditional GAN in terms of Frechet inception distance (FID).
Source-free Domain Adaptation via Distributional Alignment by Matching Batch Normalization Statistics
Authors: Masato Ishii, Masashi Sugiyama
Date: 2021-01-19 00:00:00
Tasks: Domain Adaptation, Source-Free Domain Adaptation

In this paper, we propose a novel domain adaptation method for the source-free setting. In this setting, we cannot access source data during adaptation, while unlabeled target data and a model pretrained with source data are given. Due to lack of source data, we cannot directly match the data distributions between domains unlike typical domain adaptation algorithms. To cope with this problem, we propose utilizing batch normalization statistics stored in the pretrained model to approximate the distribution of unobserved source data. Specifically, we fix the classifier part of the model during adaptation and only fine-tune the remaining feature encoder part so that batch normalization statistics of the features extracted by the encoder match those stored in the fixed classifier. Additionally, we also maximize the mutual information between the features and the classifier's outputs to further boost the classification performance. Experimental results with several benchmark datasets show that our method achieves competitive performance with state-of-the-art domain adaptation methods even though it does not require access to source data.
Deep View Synthesis via Self-Consistent Generative Network
Authors: Zhuoman Liu, Wei Jia, Ming Yang, Peiyao Luo, Yong Guo, Mingkui Tan
Date: 2021-01-19 00:00:00

View synthesis aims to produce unseen views from a set of views captured by two or more cameras at different positions. This task is non-trivial since it is hard to conduct pixel-level matching among different views. To address this issue, most existing methods seek to exploit the geometric information to match pixels. However, when the distinct cameras have a large baseline (i.e., far away from each other), severe geometry distortion issues would occur and the geometric information may fail to provide useful guidance, resulting in very blurry synthesized images. To address the above issues, in this paper, we propose a novel deep generative model, called Self-Consistent Generative Network (SCGN), which synthesizes novel views from the given input views without explicitly exploiting the geometric information. The proposed SCGN model consists of two main components, i.e., a View Synthesis Network (VSN) and a View Decomposition Network (VDN), both employing an Encoder-Decoder structure. Here, the VDN seeks to reconstruct input views from the synthesized novel view to preserve the consistency of view synthesis. Thanks to VDN, SCGN is able to synthesize novel views without using any geometric rectification before encoding, making it easier for both training and applications. Finally, adversarial loss is introduced to improve the photo-realism of novel views. Both qualitative and quantitative comparisons against several state-of-the-art methods on two benchmark tasks demonstrated the superiority of our approach.
Analysis and evaluation of Deep Learning based Super-Resolution algorithms to improve performance in Low-Resolution Face Recognition
Authors: Angelo G. Menezes
Date: 2021-01-19 00:00:00
Tasks: Face Recognition, Face Verification, image-classification, Image Classification, Super-Resolution

Surveillance scenarios are prone to several problems since they usually involve low-resolution footage, and there is no control of how far the subjects may be from the camera in the first place. This situation is suitable for the application of upsampling (super-resolution) algorithms since they may be able to recover the discriminant properties of the subjects involved. While general super-resolution approaches were proposed to enhance image quality for human-level perception, biometrics super-resolution methods seek the best "computer perception" version of the image since their focus is on improving automatic recognition performance. Convolutional neural networks and deep learning algorithms, in general, have been applied to computer vision tasks and are now state-of-the-art for several sub-domains, including image classification, restoration, and super-resolution. However, no work has evaluated the effects that the latest proposed super-resolution methods may have upon the accuracy and face verification performance in low-resolution "in-the-wild" data. This project aimed at evaluating and adapting different deep neural network architectures for the task of face super-resolution driven by face recognition performance in real-world low-resolution images. The experimental results in a real-world surveillance and attendance datasets showed that general super-resolution architectures might enhance face verification performance of deep neural networks trained on high-resolution faces. Also, since neural networks are function approximators and can be trained based on specific objective functions, the use of a customized loss function optimized for feature extraction showed promising results for recovering discriminant features in low-resolution face images.
Indoor Group Activity Recognition using Multi-Layered HMMs
Authors: Vinayak Elangovan
Date: 2021-01-23 00:00:00
Tasks: Activity Recognition, Group Activity Recognition

Discovery and recognition of Group Activities (GA) based on imagery data processing have significant applications in persistent surveillance systems, which play an important role in some Internet services. The process is involved with analysis of sequential imagery data with spatiotemporal associations. Discretion of video imagery requires a proper inference system capable of discriminating and differentiating cohesive observations and interlinking them to known ontologies. We propose an Ontology based GAR with a proper inference model that is capable of identifying and classifying a sequence of events in group activities. A multi-layered Hidden Markov Model (HMM) is proposed to recognize different levels of abstract GA. The multi-layered HMM consists of N layers of HMMs where each layer comprises of M number of HMMs running in parallel. The number of layers depends on the order of information to be extracted. At each layer, by matching and correlating attributes of detected group events, the model attempts to associate sensory observations to known ontology perceptions. This paper demonstrates and compares performance of three different implementation of HMM, namely, concatenated N-HMM, cascaded C-HMM and hybrid H-HMM for building effective multi-layered HMM.