pwc-archive/papers-with-abstracts huggingface dataset:
Stereo Unstructured Magnification: Multiple Homography Image for View Synthesis
Authors: Qi Zhang, Xin Huang, Ying Feng, Xue Wang, Hongdong Li, Qing Wang
Date: 2022-04-01 00:00:00
Tasks: Novel View Synthesis

This paper studies the problem of view synthesis with certain amount of rotations from a pair of images, what we called stereo unstructured magnification. While the multi-plane image representation is well suited for view synthesis with depth invariant, how to generalize it to unstructured views remains a significant challenge. This is primarily due to the depth-dependency caused by camera frontal parallel representation. Here we propose a novel multiple homography image (MHI) representation, comprising of a set of scene planes with fixed normals and distances. A two-stage network is developed for novel view synthesis. Stage-1 is an MHI reconstruction module that predicts the MHIs and composites layered multi-normal images along the normal direction. Stage-2 is a normal-blending module to find blending weights. We also derive an angle-based cost to guide the blending of multi-normal images by exploiting per-normal geometry. Compared with the state-of-the-art methods, our method achieves superior performance for view synthesis qualitatively and quantitatively, especially for cases when the cameras undergo rotations.
An End-to-end Supervised Domain Adaptation Framework for Cross-Domain Change Detection
Authors: Jia Liu, Wenjie Xuan, Yuhang Gan, Juhua Liu, Bo Du
Date: 2022-04-01 00:00:00
Tasks: Change Detection, Change detection for remote sensing images, Domain Adaptation

Existing deep learning-based change detection methods try to elaborately design complicated neural networks with powerful feature representations, but ignore the universal domain shift induced by time-varying land cover changes, including luminance fluctuations and season changes between pre-event and post-event images, thereby producing sub-optimal results. In this paper, we propose an end-to-end Supervised Domain Adaptation framework for cross-domain Change Detection, namely SDACD, to effectively alleviate the domain shift between bi-temporal images for better change predictions. Specifically, our SDACD presents collaborative adaptations from both image and feature perspectives with supervised learning. Image adaptation exploits generative adversarial learning with cycle-consistency constraints to perform cross-domain style transformation, effectively narrowing the domain gap in a two-side generation fashion. As to feature adaptation, we extract domain-invariant features to align different feature distributions in the feature space, which could further reduce the domain gap of cross-domain images. To further improve the performance, we combine three types of bi-temporal images for the final change prediction, including the initial input bi-temporal images and two generated bi-temporal images from the pre-event and post-event domains. Extensive experiments and analyses on two benchmarks demonstrate the effectiveness and universality of our proposed framework. Notably, our framework pushes several representative baseline models up to new State-Of-The-Art records, achieving 97.34% and 92.36% on the CDD and WHU building datasets, respectively. The source code and models are publicly available at https://github.com/Perfect-You/SDACD.
Semi-Weakly Supervised Object Detection by Sampling Pseudo Ground-Truth Boxes
Authors: Akhil Meethal, Marco Pedersoli, Zhongwen Zhu, Francisco Perdigon Romero, Eric Granger
Date: 2022-04-01 00:00:00
Tasks: Data Augmentation, object-detection, Object Detection, Weakly-supervised Learning, Weakly Supervised Object Detection

Semi- and weakly-supervised learning have recently attracted considerable attention in the object detection literature since they can alleviate the cost of annotation needed to successfully train deep learning models. State-of-art approaches for semi-supervised learning rely on student-teacher models trained using a multi-stage process, and considerable data augmentation. Custom networks have been developed for the weakly-supervised setting, making it difficult to adapt to different detectors. In this paper, a weakly semi-supervised training method is introduced that reduces these training challenges, yet achieves state-of-the-art performance by leveraging only a small fraction of fully-labeled images with information in weakly-labeled images. In particular, our generic sampling-based learning strategy produces pseudo-ground-truth (GT) bounding box annotations in an online fashion, eliminating the need for multi-stage training, and student-teacher network configurations. These pseudo GT boxes are sampled from weakly-labeled images based on the categorical score of object proposals accumulated via a score propagation process. Empirical results on the Pascal VOC dataset, indicate that the proposed approach improves performance by 5.0% when using VOC 2007 as fully-labeled, and VOC 2012 as weak-labeled data. Also, with 5-10% fully annotated images, we observed an improvement of more than 10% in mAP, showing that a modest investment in image-level annotation, can substantially improve detection performance.
Model Predictive Control for Fluid Human-to-Robot Handovers
Authors: Wei Yang, Balakumar Sundaralingam, Chris Paxton, Iretiayo Akinola, Yu-Wei Chao, Maya Cakmak, Dieter Fox
Date: 2022-03-31 00:00:00
Tasks: Model Predictive Control, Open-Ended Question Answering

Human-robot handover is a fundamental yet challenging task in human-robot interaction and collaboration. Recently, remarkable progressions have been made in human-to-robot handovers of unknown objects by using learning-based grasp generators. However, how to responsively generate smooth motions to take an object from a human is still an open question. Specifically, planning motions that take human comfort into account is not a part of the human-robot handover process in most prior works. In this paper, we propose to generate smooth motions via an efficient model-predictive control (MPC) framework that integrates perception and complex domain-specific constraints into the optimization problem. We introduce a learning-based grasp reachability model to select candidate grasps which maximize the robot's manipulability, giving it more freedom to satisfy these constraints. Finally, we integrate a neural net force/torque classifier that detects contact events from noisy data. We conducted human-to-robot handover experiments on a diverse set of objects with several users (N=4) and performed a systematic evaluation of each module. The study shows that the users preferred our MPC approach over the baseline system by a large margin. More results and videos are available at https://sites.google.com/nvidia.com/mpc-for-handover.
Real-Time and Robust 3D Object Detection Within Road-Side LiDARs Using Domain Adaptation
Authors: Walter Zimmer, Marcus Grabler, Alois Knoll
Date: 2022-03-31 00:00:00
Tasks: 3D Object Detection, Domain Adaptation, object-detection, Object Detection, Robust 3D Object Detection, Transfer Learning

This work aims to address the challenges in domain adaptation of 3D object detection using infrastructure LiDARs. We design a model DASE-ProPillars that can detect vehicles in infrastructure-based LiDARs in real-time. Our model uses PointPillars as the baseline model with additional modules to improve the 3D detection performance. To prove the effectiveness of our proposed modules in DASE-ProPillars, we train and evaluate the model on two datasets, the open source A9-Dataset and a semi-synthetic infrastructure dataset created within the Regensburg Next project. We do several sets of experiments for each module in the DASE-ProPillars detector that show that our model outperforms the SE-ProPillars baseline on the real A9 test set and a semi-synthetic A9 test set, while maintaining an inference speed of 45 Hz (22 ms). We apply domain adaptation from the semi-synthetic A9-Dataset to the semi-synthetic dataset from the Regensburg Next project by applying transfer learning and achieve a 3D mAP@0.25 of 93.49% on the Car class of the target test set using 40 recall positions.
Perceptual Quality Assessment of UGC Gaming Videos
Authors: Xiangxu Yu, Zhengzhong Tu, Neil Birkbeck, Yilin Wang, Balu Adsumilli, Alan C. Bovik
Date: 2022-03-31 00:00:00
Tasks: Video Quality Assessment, Visual Question Answering (VQA)

In recent years, with the vigorous development of the video game industry, the proportion of gaming videos on major video websites like YouTube has dramatically increased. However, relatively little research has been done on the automatic quality prediction of gaming videos, especially on those that fall in the category of "User-Generated-Content" (UGC). Since current leading general-purpose Video Quality Assessment (VQA) models do not perform well on this type of gaming videos, we have created a new VQA model specifically designed to succeed on UGC gaming videos, which we call the Gaming Video Quality Predictor (GAME-VQP). GAME-VQP successfully predicts the unique statistical characteristics of gaming videos by drawing upon features designed under modified natural scene statistics models, combined with gaming specific features learned by a Convolution Neural Network. We study the performance of GAME-VQP on a very recent large UGC gaming video database called LIVE-YT-Gaming, and find that it both outperforms other mainstream general VQA models as well as VQA models specifically designed for gaming videos. The new model will be made public after paper being accepted.
GALA: Toward Geometry-and-Lighting-Aware Object Search for Compositing
Authors: Sijie Zhu, Zhe Lin, Scott Cohen, Jason Kuen, Zhifei Zhang, Chen Chen
Date: 2022-03-31 00:00:00
Tasks: Object

Compositing-aware object search aims to find the most compatible objects for compositing given a background image and a query bounding box. Previous works focus on learning compatibility between the foreground object and background, but fail to learn other important factors from large-scale data, i.e. geometry and lighting. To move a step further, this paper proposes GALA (Geometry-and-Lighting-Aware), a generic foreground object search method with discriminative modeling on geometry and lighting compatibility for open-world image compositing. Remarkably, it achieves state-of-the-art results on the CAIS dataset and generalizes well on large-scale open-world datasets, i.e. Pixabay and Open Images. In addition, our method can effectively handle non-box scenarios, where users only provide background images without any input bounding box. A web demo (see supplementary materials) is built to showcase applications of the proposed method for compositing-aware search and automatic location/scale prediction for the foreground object.
A Survey of Robust 3D Object Detection Methods in Point Clouds
Authors: Walter Zimmer, Emec Ercelik, Xingcheng Zhou, Xavier Jair Diaz Ortiz, Alois Knoll
Date: 2022-03-31 00:00:00
Tasks: 3D Object Detection, Autonomous Driving, Data Augmentation, Object, object-detection, Object Detection, Robust 3D Object Detection

The purpose of this work is to review the state-of-the-art LiDAR-based 3D object detection methods, datasets, and challenges. We describe novel data augmentation methods, sampling strategies, activation functions, attention mechanisms, and regularization methods. Furthermore, we list recently introduced normalization methods, learning rate schedules and loss functions. Moreover, we also cover advantages and limitations of 10 novel autonomous driving datasets. We evaluate novel 3D object detectors on the KITTI, nuScenes, and Waymo dataset and show their accuracy, speed, and robustness. Finally, we mention the current challenges in 3D object detection in LiDAR point clouds and list some open issues.
TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization
Authors: Sijie Zhu, Mubarak Shah, Chen Chen
Date: 2022-03-31 00:00:00
Proceeding: CVPR 2022 1
Tasks: All, geo-localization, Image-Based Localization

The dominant CNN-based methods for cross-view image geo-localization rely on polar transform and fail to model global correlation. We propose a pure transformer-based approach (TransGeo) to address these limitations from a different perspective. TransGeo takes full advantage of the strengths of transformer related to global information modeling and explicit position information encoding. We further leverage the flexibility of transformer input and propose an attention-guided non-uniform cropping method, so that uninformative image patches are removed with negligible drop on performance to reduce computation cost. The saved computation can be reallocated to increase resolution only for informative patches, resulting in performance improvement with no additional computation cost. This "attend and zoom-in" strategy is highly similar to human behavior when observing images. Remarkably, TransGeo achieves state-of-the-art results on both urban and rural datasets, with significantly less computation cost than CNN-based methods. It does not rely on polar transform and infers faster than CNN-based methods. Code is available at https://github.com/Jeff-Zilence/TransGeo2022.
Tooth Instance Segmentation on Panoramic Dental Radiographs Using U-Nets and Morphological Processing
Authors: Selahattin Serdar Helli, Andac Hamamci
Date: 2022-03-31 00:00:00
Tasks: Binarization, Instance Segmentation, Segmentation, Semantic Segmentation

Automatic teeth segmentation in panoramic x-ray images is an important research subject of the image analysis in dentistry. In this study, we propose a post-processing stage to obtain a segmentation map in which the objects in the image are separated, and apply this technique to tooth instance segmentation with U-Net network. The post-processing consists of grayscale morphological and filtering operations, which are applied to the sigmoid output of the network before binarization. A dice overlap score of 95.4 - 0.3% is obtained in overall teeth segmentation. The proposed post-processing stages reduce the mean error of tooth count to 6.15%, whereas the error without post-processing is 26.81%. The performances of both segmentation and tooth counting are the highest in the literature, to our knowledge. Moreover, this is achieved by using a relatively small training dataset, which consists of 105 images. Although the aim in this study is to segment tooth instances, the presented method is applicable to similar problems in other domains, such as separating the cell instances
Efficient Maximal Coding Rate Reduction by Variational Forms
Authors: Christina Baek, Ziyang Wu, Kwan Ho Ryan Chan, Tianjiao Ding, Yi Ma, Benjamin D. Haeffele
Date: 2022-03-31 00:00:00
Proceeding: CVPR 2022 1
Tasks: image-classification, Image Classification

The principle of Maximal Coding Rate Reduction (MCR$^2$) has recently been proposed as a training objective for learning discriminative low-dimensional structures intrinsic to high-dimensional data to allow for more robust training than standard approaches, such as cross-entropy minimization. However, despite the advantages that have been shown for MCR$^2$ training, MCR$^2$ suffers from a significant computational cost due to the need to evaluate and differentiate a significant number of log-determinant terms that grows linearly with the number of classes. By taking advantage of variational forms of spectral functions of a matrix, we reformulate the MCR$^2$ objective to a form that can scale significantly without compromising training accuracy. Experiments in image classification demonstrate that our proposed formulation results in a significant speed up over optimizing the original MCR$^2$ objective directly and often results in higher quality learned representations. Further, our approach may be of independent interest in other models that require computation of log-determinant forms, such as in system identification or normalizing flow models.
Automatic Classification of Alzheimer's Disease using brain MRI data and deep Convolutional Neural Networks
Authors: Zahraa Sh. Aaraji, Hawraa H. Abbas
Date: 2022-03-31 00:00:00
Tasks: Binary Classification, Deep Learning, Image Segmentation, Medical Image Analysis, MRI classification, Semantic Segmentation

Alzheimer's disease (AD) is one of the most common public health issues the world is facing today. This disease has a high prevalence primarily in the elderly accompanying memory loss and cognitive decline. AD detection is a challenging task which many authors have developed numerous computerized automatic diagnosis systems utilizing neuroimaging and other clinical data. MRI scans provide high-intensity visible features, making these scans the most widely used brain imaging technique. In recent years deep learning has achieved leading success in medical image analysis. But a relatively little investigation has been done to apply deep learning techniques for the brain MRI classification. This paper explores the construction of several deep learning architectures evaluated on brain MRI images and segmented images. The idea behind segmented images investigates the influence of image segmentation step on deep learning classification. The image processing presented a pipeline consisting of pre-processing to enhance the MRI scans and post-processing consisting of a segmentation method for segmenting the brain tissues. The results show that the processed images achieved a better accuracy in the binary classification of AD vs. CN (Cognitively Normal) across four different architectures. ResNet architecture resulted in the highest prediction accuracy amongst the other architectures (90.83% for the original brain images and 93.50% for the processed images).
Digitizing Historical Balance Sheet Data: A Practitioner's Guide
Authors: Sergio Correia, Stephan Luck
Date: 2022-03-31 00:00:00
Tasks: Optical Character Recognition, Optical Character Recognition (OCR)

This paper discusses how to successfully digitize large-scale historical micro-data by augmenting optical character recognition (OCR) engines with pre- and post-processing methods. Although OCR software has improved dramatically in recent years due to improvements in machine learning, off-the-shelf OCR applications still present high error rates which limit their applications for accurate extraction of structured information. Complementing OCR with additional methods can however dramatically increase its success rate, making it a powerful and cost-efficient tool for economic historians. This paper showcases these methods and explains why they are useful. We apply them against two large balance sheet datasets and introduce quipucamayoc, a Python package containing these methods in a unified framework.
Ball 3D Localization From A Single Calibrated Image
Authors: Gabriel Van Zandycke, Christophe De Vleeschouwer
Date: 2022-03-30 00:00:00

Ball 3D localization in team sports has various applications including automatic offside detection in soccer, or shot release localization in basketball. Today, this task is either resolved by using expensive multi-views setups, or by restricting the analysis to ballistic trajectories. In this work, we propose to address the task on a single image from a calibrated monocular camera by estimating ball diameter in pixels and use the knowledge of real ball diameter in meters. This approach is suitable for any game situation where the ball is (even partly) visible. To achieve this, we use a small neural network trained on image patches around candidates generated by a conventional ball detector. Besides predicting ball diameter, our network outputs the confidence of having a ball in the image patch. Validations on 3 basketball datasets reveals that our model gives remarkable predictions on ball 3D localization. In addition, through its confidence output, our model improves the detection rate by filtering the candidates produced by the detector. The contributions of this work are (i) the first model to address 3D ball localization on a single image, (ii) an effective method for ball 3D annotation from single calibrated images, (iii) a high quality 3D ball evaluation dataset annotated from a single viewpoint. In addition, the code to reproduce this research is be made freely available at https://github.com/gabriel-vanzandycke/deepsport.
Learning the conditional law: signatures and conditional GANs in filtering and prediction of diffusion processes
Authors: Fabian Germ, Marc Sabate-Vidales
Date: 2022-04-01 00:00:00

We consider the filtering and prediction problem for a diffusion process. The signal and observation are modeled by stochastic differential equations (SDEs) driven by correlated Wiener processes. In classical estimation theory, measure-valued stochastic partial differential equations (SPDEs) are derived for the filtering and prediction measures. These equations can be hard to solve numerically. We provide an approximation algorithm using conditional generative adversarial networks (GANs) in combination with signatures, an object from rough path theory. The signature of a sufficiently smooth path determines the path completely. As a result, in some cases, GANs based on signatures have been shown to efficiently approximate the law of a stochastic process. For our algorithm we extend this method to sample from the conditional law, given noisy, partial observation. Our generator is constructed using neural differential equations (NDEs), relying on their universal approximator property. We show well-posedness in providing a rigorous mathematical framework. Numerical results show the efficiency of our algorithm.
Monarch: Expressive Structured Matrices for Efficient and Accurate Training
Authors: Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, Christopher Ré
Date: 2022-04-01 00:00:00
Tasks: Language Modeling, Language Modelling, MRI Reconstruction

Large neural networks excel in many domains, but they are expensive to train and fine-tune. A popular approach to reduce their compute or memory requirements is to replace dense weight matrices with structured ones (e.g., sparse, low-rank, Fourier transform). These methods have not seen widespread adoption (1) in end-to-end training due to unfavorable efficiency--quality tradeoffs, and (2) in dense-to-sparse fine-tuning due to lack of tractable algorithms to approximate a given dense weight matrix. To address these issues, we propose a class of matrices (Monarch) that is hardware-efficient (they are parameterized as products of two block-diagonal matrices for better hardware utilization) and expressive (they can represent many commonly used transforms). Surprisingly, the problem of approximating a dense weight matrix with a Monarch matrix, though nonconvex, has an analytical optimal solution. These properties of Monarch matrices unlock new ways to train and fine-tune sparse and dense models. We empirically validate that Monarch can achieve favorable accuracy-efficiency tradeoffs in several end-to-end sparse training applications: speeding up ViT and GPT-2 training on ImageNet classification and Wikitext-103 language modeling by 2x with comparable model quality, and reducing the error on PDE solving and MRI reconstruction tasks by 40%. In sparse-to-dense training, with a simple technique called "reverse sparsification," Monarch matrices serve as a useful intermediate representation to speed up GPT-2 pretraining on OpenWebText by 2x without quality drop. The same technique brings 23% faster BERT pretraining than even the very optimized implementation from Nvidia that set the MLPerf 1.1 record. In dense-to-sparse fine-tuning, as a proof-of-concept, our Monarch approximation algorithm speeds up BERT fine-tuning on GLUE by 1.7x with comparable accuracy.