pwc-archive/papers-with-abstracts huggingface dataset:
Domain Adaptation of Networks for Camera Pose Estimation: Learning Camera Pose Estimation Without Pose Labels
Authors: Jack Langerman, Ziming Qiu, Gábor Sörös, Dávid Sebők, Yao Wang, Howard Huang
Date: 2021-11-29 00:00:00
Tasks: Camera Pose Estimation, Domain Adaptation, Image-to-Image Translation, Pose Estimation

One of the key criticisms of deep learning is that large amounts of expensive and difficult-to-acquire training data are required in order to train models with high performance and good generalization capabilities. Focusing on the task of monocular camera pose estimation via scene coordinate regression (SCR), we describe a novel method, Domain Adaptation of Networks for Camera pose Estimation (DANCE), which enables the training of models without access to any labels on the target task. DANCE requires unlabeled images (without known poses, ordering, or scene coordinate labels) and a 3D representation of the space (e.g., a scanned point cloud), both of which can be captured with minimal effort using off-the-shelf commodity hardware. DANCE renders labeled synthetic images from the 3D model, and bridges the inevitable domain gap between synthetic and real images by applying unsupervised image-level domain adaptation techniques (unpaired image-to-image translation). When tested on real images, the SCR model trained with DANCE achieved comparable performance to its fully supervised counterpart (in both cases using PnP-RANSAC for final pose estimation) at a fraction of the cost. Our code and dataset are available at https://github.com/JackLangerman/dance
Searching the Search Space of Vision Transformer
Authors: Minghao Chen, Kan Wu, Bolin Ni, Houwen Peng, Bei Liu, Jianlong Fu, Hongyang Chao, Haibin Ling
Date: 2021-11-29 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: Neural Architecture Search, object-detection, Object Detection, Question Answering, Semantic Segmentation, Visual Question Answering, Visual Question Answering (VQA)

Vision Transformer has shown great visual representation power in substantial vision tasks such as recognition and detection, and thus been attracting fast-growing efforts on manually designing more effective architectures. In this paper, we propose to use neural architecture search to automate this process, by searching not only the architecture but also the search space. The central idea is to gradually evolve different search dimensions guided by their E-T Error computed using a weight-sharing supernet. Moreover, we provide design guidelines of general vision transformers with extensive analysis according to the space searching process, which could promote the understanding of vision transformer. Remarkably, the searched models, named S3 (short for Searching the Search Space), from the searched space achieve superior performance to recently proposed models, such as Swin, DeiT and ViT, when evaluated on ImageNet. The effectiveness of S3 is also illustrated on object detection, semantic segmentation and visual question answering, demonstrating its generality to downstream vision and vision-language tasks. Code and models will be available at https://github.com/microsoft/Cream.
Real-time Attention Span Tracking in Online Education
Authors: Rahul RK, Shanthakumar S, Vykunth P, Sairamnath K
Date: 2021-11-29 00:00:00

Over the last decade, e-learning has revolutionized how students learn by providing them access to quality education whenever and wherever they want. However, students often get distracted because of various reasons, which affect the learning capacity to a great extent. Many researchers have been trying to improve the quality of online education, but we need a holistic approach to address this issue. This paper intends to provide a mechanism that uses the camera feed and microphone input to monitor the real-time attention level of students during online classes. We explore various image processing techniques and machine learning algorithms throughout this study. We propose a system that uses five distinct non-verbal features to calculate the attention score of the student during computer based tasks and generate real-time feedback for both students and the organization. We can use the generated feedback as a heuristic value to analyze the overall performance of students as well as the teaching standards of the lecturers.
DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion
Authors: Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, Ping Luo
Date: 2021-11-29 00:00:00
Proceeding: CVPR 2022 1
Tasks: Multi-Object Tracking, Object, object-detection, Object Detection, Object Localization, Object Tracking

A typical pipeline for multi-object tracking (MOT) is to use a detector for object localization, and following re-identification (re-ID) for object association. This pipeline is partially motivated by recent progress in both object detection and re-ID, and partially motivated by biases in existing tracking datasets, where most objects tend to have distinguishing appearance and re-ID models are sufficient for establishing associations. In response to such bias, we would like to re-emphasize that methods for multi-object tracking should also work when object appearance is not sufficiently discriminative. To this end, we propose a large-scale dataset for multi-human tracking, where humans have similar appearance, diverse motion and extreme articulation. As the dataset contains mostly group dancing videos, we name it "DanceTrack". We expect DanceTrack to provide a better platform to develop more MOT algorithms that rely less on visual discrimination and depend more on motion analysis. We benchmark several state-of-the-art trackers on our dataset and observe a significant performance drop on DanceTrack when compared against existing benchmarks. The dataset, project code and competition server are released at: \url{https://github.com/DanceTrack}.
Graph Embedding via High Dimensional Model Representation for Hyperspectral Images
Authors: Gulsen Taskin, Gustau Camps-Valls
Date: 2021-11-29 00:00:00
Tasks: Graph Embedding, Vocal Bursts Intensity Prediction

Learning the manifold structure of remote sensing images is of paramount relevance for modeling and understanding processes, as well as to encapsulate the high dimensionality in a reduced set of informative features for subsequent classification, regression, or unmixing. Manifold learning methods have shown excellent performance to deal with hyperspectral image (HSI) analysis but, unless specifically designed, they cannot provide an explicit embedding map readily applicable to out-of-sample data. A common assumption to deal with the problem is that the transformation between the high-dimensional input space and the (typically low) latent space is linear. This is a particularly strong assumption, especially when dealing with hyperspectral images due to the well-known nonlinear nature of the data. To address this problem, a manifold learning method based on High Dimensional Model Representation (HDMR) is proposed, which enables to present a nonlinear embedding function to project out-of-sample samples into the latent space. The proposed method is compared to manifold learning methods along with its linear counterparts and achieves promising performance in terms of classification accuracy of a representative set of hyperspectral images.
3D Compositional Zero-shot Learning with DeCompositional Consensus
Authors: Muhammad Ferjad Naeem, Evin Pınar Örnek, Yongqin Xian, Luc van Gool, Federico Tombari
Date: 2021-11-29 00:00:00
Tasks: Benchmarking, Compositional Zero-Shot Learning, Object, Segmentation, Semantic Segmentation, Semantic Similarity, Semantic Textual Similarity, zero-shot-classification, Zero-Shot Learning, Zero Shot Segmentation

Parts represent a basic unit of geometric and semantic similarity across different objects. We argue that part knowledge should be composable beyond the observed object classes. Towards this, we present 3D Compositional Zero-shot Learning as a problem of part generalization from seen to unseen object classes for semantic segmentation. We provide a structured study through benchmarking the task with the proposed Compositional-PartNet dataset. This dataset is created by processing the original PartNet to maximize part overlap across different objects. The existing point cloud part segmentation methods fail to generalize to unseen object classes in this setting. As a solution, we propose DeCompositional Consensus, which combines a part segmentation network with a part scoring network. The key intuition to our approach is that a segmentation mask over some parts should have a consensus with its part scores when each part is taken apart. The two networks reason over different part combinations defined in a per-object part prior to generate the most suitable segmentation mask. We demonstrate that our method allows compositional zero-shot segmentation and generalized zero-shot classification, and establishes the state of the art on both tasks.
Human Performance Capture from Monocular Video in the Wild
Authors: Chen Guo, Xu Chen, Jie Song, Otmar Hilliges
Date: 2021-11-29 00:00:00
Tasks: 3D Human Shape Estimation, Autonomous Driving

Capturing the dynamically deforming 3D shape of clothed human is essential for numerous applications, including VR/AR, autonomous driving, and human-computer interaction. Existing methods either require a highly specialized capturing setup, such as expensive multi-view imaging systems, or they lack robustness to challenging body poses. In this work, we propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses, without any additional input. We first build a 3D template human model of the subject based on a learned regression model. We then track this template model's deformation under challenging body articulations based on 2D image observations. Our method outperforms state-of-the-art methods on an in-the-wild human video dataset 3DPW. Moreover, we demonstrate its efficacy in robustness and generalizability on videos from iPER datasets.
diffConv: Analyzing Irregular Point Clouds with an Irregular View
Authors: Manxi Lin, Aasa Feragen
Date: 2021-11-29 00:00:00
Tasks: 3D Object Classification, 3D Point Cloud Classification, 3D Shape Classification

Standard spatial convolutions assume input data with a regular neighborhood structure. Existing methods typically generalize convolution to the irregular point cloud domain by fixing a regular "view" through e.g. a fixed neighborhood size, where the convolution kernel size remains the same for each point. However, since point clouds are not as structured as images, the fixed neighbor number gives an unfortunate inductive bias. We present a novel graph convolution named Difference Graph Convolution (diffConv), which does not rely on a regular view. diffConv operates on spatially-varying and density-dilated neighborhoods, which are further adapted by a learned masked attention mechanism. Experiments show that our model is very robust to the noise, obtaining state-of-the-art performance in 3D shape classification and scene understanding tasks, along with a faster inference speed.
Buildings Classification using Very High Resolution Satellite Imagery
Authors: Mohammad Dimassi, Abed Ellatif Samhat, Mohammad Zaraket, Jamal Haidar, Mustafa Shukor, Ali J. Ghandour
Date: 2021-11-29 00:00:00
Tasks: Classification, Semantic Segmentation, Transfer Learning, Vocal Bursts Intensity Prediction

Buildings classification using satellite images is becoming more important for several applications such as damage assessment, resource allocation, and population estimation. We focus, in this work, on buildings damage assessment (BDA) and buildings type classification (BTC) of residential and non-residential buildings. We propose to rely solely on RGB satellite images and follow a 2-stage deep learning-based approach, where first, buildings' footprints are extracted using a semantic segmentation model, followed by classification of the cropped images. Due to the lack of an appropriate dataset for the residential/non-residential building classification, we introduce a new dataset of high-resolution satellite images. We conduct extensive experiments to select the best hyper-parameters, model architecture, and training paradigm, and we propose a new transfer learning-based approach that outperforms classical methods. Finally, we validate the proposed approach on two applications showing excellent accuracy and F1-score metrics.
MUNet: Motion Uncertainty-aware Semi-supervised Video Object Segmentation
Authors: Jiadai Sun, Yuxin Mao, Yuchao Dai, Yiran Zhong, Jianyuan Wang
Date: 2021-11-29 00:00:00
Tasks: Object, Semantic Segmentation, Semi-Supervised Video Object Segmentation, Video Object Segmentation, Video Semantic Segmentation

The task of semi-supervised video object segmentation (VOS) has been greatly advanced and state-of-the-art performance has been made by dense matching-based methods. The recent methods leverage space-time memory (STM) networks and learn to retrieve relevant information from all available sources, where the past frames with object masks form an external memory and the current frame as the query is segmented using the mask information in the memory. However, when forming the memory and performing matching, these methods only exploit the appearance information while ignoring the motion information. In this paper, we advocate the return of the \emph{motion information} and propose a motion uncertainty-aware framework (MUNet) for semi-supervised VOS. First, we propose an implicit method to learn the spatial correspondences between neighboring frames, building upon a correlation cost volume. To handle the challenging cases of occlusion and textureless regions during constructing dense correspondences, we incorporate the uncertainty in dense matching and achieve motion uncertainty-aware feature representation. Second, we introduce a motion-aware spatial attention module to effectively fuse the motion feature with the semantic feature. Comprehensive experiments on challenging benchmarks show that \textbf{\textit{using a small amount of data and combining it with powerful motion information can bring a significant performance boost}}. We achieve ${76.5\%}$ $\mathcal{J} \& \mathcal{F}$ only using DAVIS17 for training, which significantly outperforms the \textit{SOTA} methods under the low-data protocol. \textit{The code will be released.}
Urban Radiance Fields
Authors: Konstantinos Rematas, Andrew Liu, Pratul P. Srinivasan, Jonathan T. Barron, Andrea Tagliasacchi, Thomas Funkhouser, Vittorio Ferrari
Date: 2021-11-29 00:00:00
Proceeding: CVPR 2022 1
Tasks: 3D Reconstruction, NeRF, Novel View Synthesis

The goal of this work is to perform 3D reconstruction and novel view synthesis from data captured by scanning platforms commonly deployed for world mapping in urban outdoor environments (e.g., Street View). Given a sequence of posed RGB images and lidar sweeps acquired by cameras and scanners moving through an outdoor scene, we produce a model from which 3D surfaces can be extracted and novel RGB images can be synthesized. Our approach extends Neural Radiance Fields, which has been demonstrated to synthesize realistic novel images for small scenes in controlled settings, with new methods for leveraging asynchronously captured lidar data, for addressing exposure variation between captured images, and for leveraging predicted image segmentations to supervise densities on rays pointing at the sky. Each of these three extensions provides significant performance improvements in experiments on Street View data. Our system produces state-of-the-art 3D surface reconstructions and synthesizes higher quality novel views in comparison to both traditional methods (e.g.~COLMAP) and recent neural representations (e.g.~Mip-NeRF).
ILabel: Interactive Neural Scene Labelling
Authors: Shuaifeng Zhi, Edgar Sucar, Andre Mouton, Iain Haughton, Tristan Laidlow, Andrew J. Davison
Date: 2021-11-29 00:00:00
Tasks: Semantic Segmentation

Joint representation of geometry, colour and semantics using a 3D neural field enables accurate dense labelling from ultra-sparse interactions as a user reconstructs a scene in real-time using a handheld RGB-D sensor. Our iLabel system requires no training data, yet can densely label scenes more accurately than standard methods trained on large, expensively labelled image datasets. Furthermore, it works in an 'open set' manner, with semantic classes defined on the fly by the user. ILabel's underlying model is a multilayer perceptron (MLP) trained from scratch in real-time to learn a joint neural scene representation. The scene model is updated and visualised in real-time, allowing the user to focus interactions to achieve efficient labelling. A room or similar scene can be accurately labelled into 10+ semantic categories with only a few tens of clicks. Quantitative labelling accuracy scales powerfully with the number of clicks, and rapidly surpasses standard pre-trained semantic segmentation methods. We also demonstrate a hierarchical labelling variant.
Weakly-supervised Generative Adversarial Networks for medical image classification
Authors: Jiawei Mao, Xuesong Yin, Yuanqi Chang, Qi Huang
Date: 2021-11-29 00:00:00
Tasks: Classification, Contrastive Learning, image-classification, Image Classification, Medical Image Classification, Weakly-supervised Learning

Weakly-supervised learning has become a popular technology in recent years. In this paper, we propose a novel medical image classification algorithm, called Weakly-Supervised Generative Adversarial Networks (WSGAN), which only uses a small number of real images without labels to generate fake images or mask images to enlarge the sample size of the training set. First, we combine with MixMatch to generate pseudo labels for the fake images and unlabeled images to do the classification. Second, contrastive learning and self-attention mechanism are introduced into the proposed problem to enhance the classification accuracy. Third, the problem of mode collapse is well addressed by cyclic consistency loss. Finally, we design global and local classifiers to complement each other with the key information needed for classification. The experimental results on four medical image datasets show that WSGAN can obtain relatively high learning performance by using few labeled and unlabeled data. For example, the classification accuracy of WSGAN is 11% higher than that of the second-ranked MIXMATCH with 100 labeled images and 1000 unlabeled images on the OCT dataset. In addition, we also conduct ablation experiments to verify the effectiveness of our algorithm.
TransMVSNet: Global Context-aware Multi-view Stereo Network with Transformers
Authors: Yikang Ding, Wentao Yuan, Qingtian Zhu, Haotian Zhang, Xiangyue Liu, Yuanjiang Wang, Xiao Liu
Date: 2021-11-29 00:00:00
Proceeding: CVPR 2022 1
Tasks: 3D Reconstruction, Feature Correlation

In this paper, we present TransMVSNet, based on our exploration of feature matching in multi-view stereo (MVS). We analogize MVS back to its nature of a feature matching task and therefore propose a powerful Feature Matching Transformer (FMT) to leverage intra- (self-) and inter- (cross-) attention to aggregate long-range context information within and across images. To facilitate a better adaptation of the FMT, we leverage an Adaptive Receptive Field (ARF) module to ensure a smooth transit in scopes of features and bridge different stages with a feature pathway to pass transformed features and gradients across different scales. In addition, we apply pair-wise feature correlation to measure similarity between features, and adopt ambiguity-reducing focal loss to strengthen the supervision. To the best of our knowledge, TransMVSNet is the first attempt to leverage Transformer into the task of MVS. As a result, our method achieves state-of-the-art performance on DTU dataset, Tanks and Temples benchmark, and BlendedMVS dataset. The code of our method will be made available at https://github.com/MegviiRobot/TransMVSNet .
Overcoming the Domain Gap in Contrastive Learning of Neural Action Representations
Authors: Semih Günel, Florian Aymanns, Sina Honari, Pavan Ramdya, Pascal Fua
Date: 2021-11-29 00:00:00
Tasks: Contrastive Learning, Domain Adaptation, Markerless Motion Capture, Self-Supervised Learning, Unsupervised Domain Adaptation

A fundamental goal in neuroscience is to understand the relationship between neural activity and behavior. For example, the ability to extract behavioral intentions from neural data, or neural decoding, is critical for developing effective brain machine interfaces. Although simple linear models have been applied to this challenge, they cannot identify important non-linear relationships. Thus, a self-supervised means of identifying non-linear relationships between neural dynamics and behavior, in order to compute neural representations, remains an important open problem. To address this challenge, we generated a new multimodal dataset consisting of the spontaneous behaviors generated by fruit flies, Drosophila melanogaster -- a popular model organism in neuroscience research. The dataset includes 3D markerless motion capture data from six camera views of the animal generating spontaneous actions, as well as synchronously acquired two-photon microscope images capturing the activity of descending neuron populations that are thought to drive actions. Standard contrastive learning and unsupervised domain adaptation techniques struggle to learn neural action representations (embeddings computed from the neural data describing action labels) due to large inter-animal differences in both neural and behavioral modalities. To overcome this deficiency, we developed simple yet effective augmentations that close the inter-animal domain gap, allowing us to extract behaviorally relevant, yet domain agnostic, information from neural data. This multimodal dataset and our new set of augmentations promise to accelerate the application of self-supervised learning methods in neuroscience.
Multi-instance Point Cloud Registration by Efficient Correspondence Clustering
Authors: Weixuan Tang, Danping Zou
Date: 2021-11-29 00:00:00
Proceeding: CVPR 2022 1
Tasks: Clustering, Point Cloud Registration

We address the problem of estimating the poses of multiple instances of the source point cloud within a target point cloud. Existing solutions require sampling a lot of hypotheses to detect possible instances and reject the outliers, whose robustness and efficiency degrade notably when the number of instances and outliers increase. We propose to directly group the set of noisy correspondences into different clusters based on a distance invariance matrix. The instances and outliers are automatically identified through clustering. Our method is robust and fast. We evaluated our method on both synthetic and real-world datasets. The results show that our approach can correctly register up to 20 instances with an F1 score of 90.46% in the presence of 70% outliers, which performs significantly better and at least 10x faster than existing methods