pwc-archive/papers-with-abstracts huggingface dataset:
Fine-Grained Image Analysis with Deep Learning: A Survey
Authors: Xiu-Shen Wei, Yi-Zhe Song, Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang, Serge Belongie
Date: 2021-11-11 00:00:00
Tasks: Deep Learning, Fine-Grained Image Recognition, Image Retrieval, Retrieval, Survey

Fine-grained image analysis (FGIA) is a longstanding and fundamental problem in computer vision and pattern recognition, and underpins a diverse set of real-world applications. The task of FGIA targets analyzing visual objects from subordinate categories, e.g., species of birds or models of cars. The small inter-class and large intra-class variation inherent to fine-grained image analysis makes it a challenging problem. Capitalizing on advances in deep learning, in recent years we have witnessed remarkable progress in deep learning powered FGIA. In this paper we present a systematic survey of these advances, where we attempt to re-define and broaden the field of FGIA by consolidating two fundamental fine-grained research areas -- fine-grained image recognition and fine-grained image retrieval. In addition, we also review other key issues of FGIA, such as publicly available benchmark datasets and related domain-specific applications. We conclude by highlighting several research directions and open problems which need further exploration from the community.
Open surgery tool classification and hand utilization using a multi-camera system
Authors: Kristina Basiev, Adam Goldbraikh, Carla M Pugh, Shlomi Laufer
Date: 2021-11-11 00:00:00
Tasks: Anatomy, Hand Detection

Purpose: The goal of this work is to use multi-camera video to classify open surgery tools as well as identify which tool is held in each hand. Multi-camera systems help prevent occlusions in open surgery video data. Furthermore, combining multiple views such as a Top-view camera covering the full operative field and a Close-up camera focusing on hand motion and anatomy, may provide a more comprehensive view of the surgical workflow. However, multi-camera data fusion poses a new challenge: a tool may be visible in one camera and not the other. Thus, we defined the global ground truth as the tools being used regardless their visibility. Therefore, tools that are out of the image should be remembered for extensive periods of time while the system responds quickly to changes visible in the video. Methods: Participants (n=48) performed a simulated open bowel repair. A Top-view and a Close-up cameras were used. YOLOv5 was used for tool and hand detection. A high frequency LSTM with a 1 second window at 30 frames per second (fps) and a low frequency LSTM with a 40 second window at 3 fps were used for spatial, temporal, and multi-camera integration. Results: The accuracy and F1 of the six systems were: Top-view (0.88/0.88), Close-up (0.81,0.83), both cameras (0.9/0.9), high fps LSTM (0.92/0.93), low fps LSTM (0.9/0.91), and our final architecture the Multi-camera classifier(0.93/0.94). Conclusion: By combining a system with a high fps and a low fps from the multiple camera array we improved the classification abilities of the global ground truth.
A Survey of Visual Transformers
Authors: Yang Liu, Yao Zhang, Yixin Wang, Feng Hou, Jin Yuan, Jiang Tian, Yang Zhang, Zhongchao shi, Jianping Fan, Zhiqiang He
Date: 2021-11-11 00:00:00
Tasks: Articles, Survey

Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern Convolution Neural Networks (CNNs). In this survey, we have reviewed over one hundred of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where a taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, three promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.
Graph Relation Transformer: Incorporating pairwise object features into the Transformer architecture
Authors: Michael Yang, Aditya Anantharaman, Zachary Kitowski, Derik Clive Robert
Date: 2021-11-11 00:00:00
Tasks: Graph Attention, Question Answering, Relation, Spatial Reasoning, TextVQA, Visual Question Answering, Visual Question Answering (VQA)

Previous studies such as VizWiz find that Visual Question Answering (VQA) systems that can read and reason about text in images are useful in application areas such as assisting visually-impaired people. TextVQA is a VQA dataset geared towards this problem, where the questions require answering systems to read and reason about visual objects and text objects in images. One key challenge in TextVQA is the design of a system that effectively reasons not only about visual and text objects individually, but also about the spatial relationships between these objects. This motivates the use of 'edge features', that is, information about the relationship between each pair of objects. Some current TextVQA models address this problem but either only use categories of relations (rather than edge feature vectors) or do not use edge features within the Transformer architectures. In order to overcome these shortcomings, we propose a Graph Relation Transformer (GRT), which uses edge information in addition to node information for graph attention computation in the Transformer. We find that, without using any other optimizations, the proposed GRT method outperforms the accuracy of the M4C baseline model by 0.65% on the val set and 0.57% on the test set. Qualitatively, we observe that the GRT has superior spatial reasoning ability to M4C.
CodEx: A Modular Framework for Joint Temporal De-blurring and Tomographic Reconstruction
Authors: Soumendu Majee, Selin Aslan, Doga Gursoy, Charles A. Bouman
Date: 2021-11-11 00:00:00
Tasks: Computed Tomography (CT), Deblurring, Object

In many computed tomography (CT) imaging applications, it is important to rapidly collect data from an object that is moving or changing with time. Tomographic acquisition is generally assumed to be step-and-shoot, where the object is rotated to each desired angle, and a view is taken. However, step-and-shoot acquisition is slow and can waste photons, so in practice fly-scanning is done where the object is continuously rotated while collecting data. However, this can result in motion-blurred views and consequently reconstructions with severe motion artifacts. In this paper, we introduce CodEx, a modular framework for joint de-blurring and tomographic reconstruction that can effectively invert the motion blur introduced in sparse view fly-scanning. The method is a synergistic combination of a novel acquisition method with a novel non-convex Bayesian reconstruction algorithm. CodEx works by encoding the acquisition with a known binary code that the reconstruction algorithm then inverts. Using a well chosen binary code to encode the measurements can improve the accuracy of the inversion process. The CodEx reconstruction method uses the alternating direction method of multipliers (ADMM) to split the inverse problem into iterative deblurring and reconstruction sub-problems, making reconstruction practical to implement. We present reconstruction results on both simulated and binned experimental data to demonstrate the effectiveness of our method.
On the Equivalence between Neural Network and Support Vector Machine
Authors: Yilan Chen, Wei Huang, Lam M. Nguyen, Tsui-Wei Weng
Date: 2021-11-11 00:00:00
Proceeding: NeurIPS 2021 12
Tasks: regression

Recent research shows that the dynamics of an infinitely wide neural network (NN) trained by gradient descent can be characterized by Neural Tangent Kernel (NTK) \citep{jacot2018neural}. Under the squared loss, the infinite-width NN trained by gradient descent with an infinitely small learning rate is equivalent to kernel regression with NTK \citep{arora2019exact}. However, the equivalence is only known for ridge regression currently \citep{arora2019harnessing}, while the equivalence between NN and other kernel machines (KMs), e.g. support vector machine (SVM), remains unknown. Therefore, in this work, we propose to establish the equivalence between NN and SVM, and specifically, the infinitely wide NN trained by soft margin loss and the standard soft margin SVM with NTK trained by subgradient descent. Our main theoretical results include establishing the equivalences between NNs and a broad family of $\ell_2$ regularized KMs with finite-width bounds, which cannot be handled by prior work, and showing that every finite-width NN trained by such regularized loss functions is approximately a KM. Furthermore, we demonstrate our theory can enable three practical applications, including (i) \textit{non-vacuous} generalization bound of NN via the corresponding KM; (ii) \textit{non-trivial} robustness certificate for the infinite-width NN (while existing robustness verification methods would provide vacuous bounds); (iii) intrinsically more robust infinite-width NNs than those from previous kernel regression. Our code for the experiments is available at \url{https://github.com/leslie-CH/equiv-nn-svm}.
Indian Licence Plate Dataset in the wild
Authors: Sanchit Tanwar, Ayush Tiwari, Ritesh Chowdhry
Date: 2021-11-11 00:00:00
Tasks: object-detection, Object Detection, Optical Character Recognition (OCR), Semantic Segmentation

Indian Licence Plate Detection is a problem that has not been explored much at an open-source level.There are proprietary solutions available for it, but there is no big open-source dataset that can be used to perform experiments and test different approaches.Most of the large datasets available are for countries like China, Brazil, but the model trained on these datasets does not perform well on Indian plates because the font styles and plate designs used vary significantly from country to country.This paper introduces an Indian license plate dataset with 16192 images and 21683 plate plates annotated with 4 points for each plate and each character in the corresponding plate.We present a benchmark model that uses semantic segmentation to solve number plate detection. We propose a two-stage approach in which the first stage is for localizing the plate, and the second stage is to read the text in cropped plate image.We tested benchmark object detection and semantic segmentation model, for the second stage, we used lprnet based OCR.
Hybrid Saturation Restoration for LDR Images of HDR Scenes
Authors: Chaobing Zheng, Zhengguo Li, Shiqian Wu
Date: 2021-11-11 00:00:00

There are shadow and highlight regions in a low dynamic range (LDR) image which is captured from a high dynamic range (HDR) scene. It is an ill-posed problem to restore the saturated regions of the LDR image. In this paper, the saturated regions of the LDR image are restored by fusing model-based and data-driven approaches. With such a neural augmentation, two synthetic LDR images are first generated from the underlying LDR image via the model-based approach. One is brighter than the input image to restore the shadow regions and the other is darker than the input image to restore the high-light regions. Both synthetic images are then refined via a novel exposedness aware saturation restoration network (EASRN). Finally, the two synthetic images and the input image are combined together via an HDR synthesis algorithm or a multi-scale exposure fusion algorithm. The proposed algorithm can be embedded in any smart phones or digital cameras to produce an information-enriched LDR image.
FINO: Flow-based Joint Image and Noise Model
Authors: Lanqing Guo, Siyu Huang, Haosen Liu, Bihan Wen
Date: 2021-11-11 00:00:00
Tasks: Denoising, Image Restoration, model

One of the fundamental challenges in image restoration is denoising, where the objective is to estimate the clean image from its noisy measurements. To tackle such an ill-posed inverse problem, the existing denoising approaches generally focus on exploiting effective natural image priors. The utilization and analysis of the noise model are often ignored, although the noise model can provide complementary information to the denoising algorithms. In this paper, we propose a novel Flow-based joint Image and NOise model (FINO) that distinctly decouples the image and noise in the latent space and losslessly reconstructs them via a series of invertible transformations. We further present a variable swapping strategy to align structural information in images and a noise correlation matrix to constrain the noise based on spatially minimized correlation information. Experimental results demonstrate FINO's capacity to remove both synthetic additive white Gaussian noise (AWGN) and real noise. Furthermore, the generalization of FINO to the removal of spatially variant noise and noise with inaccurate estimation surpasses that of the popular and state-of-the-art methods by large margins.
Probabilistic Contrastive Learning for Domain Adaptation
Authors: Junjie Li, Yixin Zhang, Zilei Wang, Keyu Tu, Saihui Hou
Date: 2021-11-11 00:00:00
Tasks: Contrastive Learning, Domain Adaptation, Image Classification, Representation Learning, Semantic Segmentation, Semi-supervised Domain Adaptation, Semi-Supervised Image Classification, Unsupervised Domain Adaptation

Contrastive learning can largely enhance the feature discriminability in a self-supervised manner and has achieved remarkable success for various visual tasks. However, it is undesirably observed that the standard contrastive paradigm (features+$\ell_{2}$ normalization) only brings little help for domain adaptation. In this work, we delve into this phenomenon and find that the main reason is due to the class weights (weights of the final fully connected layer) which are vital for the recognition yet ignored in the optimization. To tackle this issue, we propose a simple yet powerful Probabilistic Contrastive Learning (PCL), which does not only assist in extracting discriminative features but also enforces them to be clustered around the class weights. Specifically, we break the standard contrastive paradigm by removing $\ell_{2}$ normalization and replacing the features with probabilities. In this way, PCL can enforce the probability to approximate the one-hot form, thereby reducing the deviation between the features and class weights. Benefiting from the conciseness, PCL can be well generalized to different settings. In this work, we conduct extensive experiments on five tasks and observe consistent performance gains, i.e., Unsupervised Domain Adaptation (UDA), Semi-Supervised Domain Adaptation (SSDA), Semi-Supervised Learning (SSL), UDA Detection, and UDA Semantic Segmentation. Notably, for UDA Semantic Segmentation on SYNTHIA, PCL surpasses the sophisticated CPSL-D by $>\!2\%$ in terms of mean IoU with a much smaller training cost (PCL: 1*3090, 5 days v.s. CPSL-D: 4*V100, 11 days). Code is available at https://github.com/ljjcoder/Probabilistic-Contrastive-Learning.
csBoundary: City-scale Road-boundary Detection in Aerial Images for High-definition Maps
Authors: Zhenhua Xu, Yuxuan Liu, Lu Gan, Xiangcheng Hu, Yuxiang Sun, Ming Liu, Lujia Wang
Date: 2021-11-11 00:00:00
Tasks: Autonomous Driving, Boundary Detection, Semantic Segmentation

High-Definition (HD) maps can provide precise geometric and semantic information of static traffic environments for autonomous driving. Road-boundary is one of the most important information contained in HD maps since it distinguishes between road areas and off-road areas, which can guide vehicles to drive within road areas. But it is labor-intensive to annotate road boundaries for HD maps at the city scale. To enable automatic HD map annotation, current work uses semantic segmentation or iterative graph growing for road-boundary detection. However, the former could not ensure topological correctness since it works at the pixel level, while the latter suffers from inefficiency and drifting issues. To provide a solution to the aforementioned problems, in this letter, we propose a novel system termed csBoundary to automatically detect road boundaries at the city scale for HD map annotation. Our network takes as input an aerial image patch, and directly infers the continuous road-boundary graph (i.e., vertices and edges) from this image. To generate the city-scale road-boundary graph, we stitch the obtained graphs from all the image patches. Our csBoundary is evaluated and compared on a public benchmark dataset. The results demonstrate our superiority. The accompanied demonstration video is available at our project page \url{https://sites.google.com/view/csboundary/}.
Synthetic Document Generator for Annotation-free Layout Recognition
Authors: Natraj Raman, Sameena Shah, Manuela Veloso
Date: 2021-11-11 00:00:00

Analyzing the layout of a document to identify headers, sections, tables, figures etc. is critical to understanding its content. Deep learning based approaches for detecting the layout structure of document images have been promising. However, these methods require a large number of annotated examples during training, which are both expensive and time consuming to obtain. We describe here a synthetic document generator that automatically produces realistic documents with labels for spatial positions, extents and categories of the layout elements. The proposed generative process treats every physical component of a document as a random variable and models their intrinsic dependencies using a Bayesian Network graph. Our hierarchical formulation using stochastic templates allow parameter sharing between documents for retaining broad themes and yet the distributional characteristics produces visually unique samples, thereby capturing complex and diverse layouts. We empirically illustrate that a deep layout detection model trained purely on the synthetic documents can match the performance of a model that uses real documents.
Traffic4cast -- Large-scale Traffic Prediction using 3DResNet and Sparse-UNet
Authors: Bo wang, Reza Mohajerpoor, Chen Cai, Inhi Kim, Hai L. Vu
Date: 2021-11-10 00:00:00
Tasks: Traffic Prediction

The IARAI competition Traffic4cast 2021 aims to predict short-term city-wide high-resolution traffic states given the static and dynamic traffic information obtained previously. The aim is to build a machine learning model for predicting the normalized average traffic speed and flow of the subregions of multiple large-scale cities using historical data points. The model is supposed to be generic, in a way that it can be applied to new cities. By considering spatiotemporal feature learning and modeling efficiency, we explore 3DResNet and Sparse-UNet approaches for the tasks in this competition. The 3DResNet based models use 3D convolution to learn the spatiotemporal features and apply sequential convolutional layers to enhance the temporal relationship of the outputs. The Sparse-UNet model uses sparse convolutions as the backbone for spatiotemporal feature learning. Since the latter algorithm mainly focuses on non-zero data points of the inputs, it dramatically reduces the computation time, while maintaining a competitive accuracy. Our results show that both of the proposed models achieve much better performance than the baseline algorithms. The codes and pretrained models are available at https://github.com/resuly/Traffic4Cast-2021.
Self-Supervised Real-time Video Stabilization
Authors: Jinsoo Choi, Jaesik Park, In So Kweon
Date: 2021-11-10 00:00:00
Tasks: Optical Flow Estimation, Video Stabilization

Videos are a popular media form, where online video streaming has recently gathered much popularity. In this work, we propose a novel method of real-time video stabilization - transforming a shaky video to a stabilized video as if it were stabilized via gimbals in real-time. Our framework is trainable in a self-supervised manner, which does not require data captured with special hardware setups (i.e., two cameras on a stereo rig or additional motion sensors). Our framework consists of a transformation estimator between given frames for global stability adjustments, followed by scene parallax reduction module via spatially smoothed optical flow for further stability. Then, a margin inpainting module fills in the missing margin regions created during stabilization to reduce the amount of post-cropping. These sequential steps reduce distortion and margin cropping to a minimum while enhancing stability. Hence, our approach outperforms state-of-the-art real-time video stabilization methods as well as offline methods that require camera trajectory optimization. Our method procedure takes approximately 24.3 ms yielding 41 fps regardless of resolution (e.g., 480p or 1080p).
SUPER-Net: Trustworthy Medical Image Segmentation with Uncertainty Propagation in Encoder-Decoder Networks
Authors: Giuseppina Carannante, Dimah Dera, Nidhal C. Bouaynaya, Hassan M. Fathallah-Shaykh, Ghulam Rasool
Date: 2021-11-10 00:00:00
Tasks: Decoder, Image Segmentation, Medical Image Segmentation, Segmentation, Semantic Segmentation, Uncertainty Quantification

Deep Learning (DL) holds great promise in reshaping the healthcare industry owing to its precision, efficiency, and objectivity. However, the brittleness of DL models to noisy and out-of-distribution inputs is ailing their deployment in the clinic. Most models produce point estimates without further information about model uncertainty or confidence. This paper introduces a new Bayesian DL framework for uncertainty quantification in segmentation neural networks: SUPER-Net: trustworthy medical image Segmentation with Uncertainty Propagation in Encoder-decodeR Networks. SUPER-Net analytically propagates, using Taylor series approximations, the first two moments (mean and covariance) of the posterior distribution of the model parameters across the nonlinear layers. In particular, SUPER-Net simultaneously learns the mean and covariance without expensive post-hoc Monte Carlo sampling or model ensembling. The output consists of two simultaneous maps: the segmented image and its pixelwise uncertainty map, which corresponds to the covariance matrix of the predictive distribution. We conduct an extensive evaluation of SUPER-Net on medical image segmentation of Magnetic Resonances Imaging and Computed Tomography scans under various noisy and adversarial conditions. Our experiments on multiple benchmark datasets demonstrate that SUPER-Net is more robust to noise and adversarial attacks than state-of-the-art segmentation models. Moreover, the uncertainty map of the proposed SUPER-Net associates low confidence (or equivalently high uncertainty) to patches in the test input images that are corrupted with noise, artifacts, or adversarial attacks. Perhaps more importantly, the model exhibits the ability of self-assessment of its segmentation decisions, notably when making erroneous predictions due to noise or adversarial examples.