pwc-archive/papers-with-abstracts huggingface dataset:
Class-agnostic Object Detection with Multi-modal Transformer
Authors: Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, Ming-Hsuan Yang
Date: 2021-11-22 00:00:00
Tasks: Class-agnostic Object Detection, Object, object-detection, Object Detection, Object Proposal Generation, Open World Object Detection

What constitutes an object? This has been a long-standing question in computer vision. Towards this goal, numerous learning-free and learning-based approaches have been developed to score objectness. However, they generally do not scale well across new domains and novel objects. In this paper, we advocate that existing methods lack a top-down supervision signal governed by human-understandable semantics. For the first time in literature, we demonstrate that Multi-modal Vision Transformers (MViT) trained with aligned image-text pairs can effectively bridge this gap. Our extensive experiments across various domains and novel objects show the state-of-the-art performance of MViTs to localize generic objects in images. Based on the observation that existing MViTs do not include multi-scale feature processing and usually require longer training schedules, we develop an efficient MViT architecture using multi-scale deformable attention and late vision-language fusion. We show the significance of MViT proposals in a diverse range of applications including open-world object detection, salient and camouflage object detection, supervised and self-supervised detection tasks. Further, MViTs can adaptively generate proposals given a specific language query and thus offer enhanced interactability. Code: \url{https://git.io/J1HPY}.
Benchmarking Detection Transfer Learning with Vision Transformers
Authors: Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, Ross Girshick
Date: 2021-11-22 00:00:00
Tasks: Benchmarking, object-detection, Object Detection, Self-Supervised Learning, Transfer Learning

Object detection is a central downstream task used to test if pre-trained network parameters confer benefits, such as improved accuracy or training speed. The complexity of object detection methods can make this benchmarking non-trivial when new architectures, such as Vision Transformer (ViT) models, arrive. These difficulties (e.g., architectural incompatibility, slow training, high memory consumption, unknown training formulae, etc.) have prevented recent studies from benchmarking detection transfer learning with standard ViT models. In this paper, we present training techniques that overcome these challenges, enabling the use of standard ViT models as the backbone of Mask R-CNN. These tools facilitate the primary goal of our study: we compare five ViT initializations, including recent state-of-the-art self-supervised learning methods, supervised initialization, and a strong random initialization baseline. Our results show that recent masking-based unsupervised learning methods may, for the first time, provide convincing transfer learning improvements on COCO, increasing box AP up to 4% (absolute) over supervised and prior self-supervised pre-training methods. Moreover, these masking-based initializations scale better, with the improvement growing as model size increases.
Neural Fields in Visual Computing and Beyond
Authors: Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, Srinath Sridhar
Date: 2021-11-22 00:00:00
Tasks: 3D Reconstruction, Image Animation, Pose Estimation

Recent advances in machine learning have created increasing interest in solving visual computing problems using a class of coordinate-based neural networks that parametrize physical properties of scenes or objects across space and time. These methods, which we call neural fields, have seen successful application in the synthesis of 3D shapes and image, animation of human bodies, 3D reconstruction, and pose estimation. However, due to rapid progress in a short time, many papers exist but a comprehensive review and formulation of the problem has not yet emerged. In this report, we address this limitation by providing context, mathematical grounding, and an extensive review of literature on neural fields. This report covers research along two dimensions. In Part I, we focus on techniques in neural fields by identifying common components of neural field methods, including different representations, architectures, forward mapping, and generalization methods. In Part II, we focus on applications of neural fields to different problems in visual computing, and beyond (e.g., robotics, audio). Our review shows the breadth of topics already covered in visual computing, both historically and in current incarnations, demonstrating the improved quality, flexibility, and capability brought by neural fields methods. Finally, we present a companion website that contributes a living version of this review that can be continually updated by the community.
FAZSeg: A New User-Friendly Software for Quantification of the Foveal Avascular Zone
Authors: V. K. Viekash, Janarthanam Jothi Balaji, Vasudevan Lakshminarayanan
Date: 2021-11-22 00:00:00

Various ocular diseases and high myopia influence the anatomical reference point Foveal Avascular Zone (FAZ) dimensions. Therefore, it is important to segment and quantify the FAZs dimensions accurately. To the best of our knowledge, there is no automated tool or algorithms available to segment the FAZ's deep retinal layer. The paper describes a new open-access software with a user-friendly Graphical User Interface (GUI) and compares the results with the ground truth (manual segmentation).
Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Tasks
Authors: Linus Ericsson, Henry Gouk, Timothy M. Hospedales
Date: 2021-11-22 00:00:00
Tasks: Data Augmentation, Representation Learning, Self-Supervised Learning

Self-supervised learning is a powerful paradigm for representation learning on unlabelled images. A wealth of effective new methods based on instance matching rely on data-augmentation to drive learning, and these have reached a rough agreement on an augmentation scheme that optimises popular recognition benchmarks. However, there is strong reason to suspect that different tasks in computer vision require features to encode different (in)variances, and therefore likely require different augmentation strategies. In this paper, we measure the invariances learned by contrastive methods and confirm that they do learn invariance to the augmentations used and further show that this invariance largely transfers to related real-world changes in pose and lighting. We show that learned invariances strongly affect downstream task performance and confirm that different downstream tasks benefit from polar opposite (in)variances, leading to performance loss when the standard augmentation strategy is used. Finally, we demonstrate that a simple fusion of representations with complementary invariances ensures wide transferability to all the diverse downstream tasks considered.
Lebanon Solar Rooftop Potential Assessment using Buildings Segmentation from Aerial Images
Authors: Hasan Nasrallah, Abed Ellatif Samhat, Yilei Shi, Xiaoxiang Zhu, Ghaleb Faour, Ali J. Ghandour
Date: 2021-11-22 00:00:00
Tasks: Instance Segmentation, Semantic Segmentation

Estimating solar rooftop potential at a national level is a fundamental building block for every country to utilize solar power efficiently. Solar rooftop potential assessment relies on several features such as building geometry, location, and surrounding facilities. Hence, national-level approximations that do not take these factors into deep consideration are often inaccurate. This paper introduces Lebanon's first comprehensive footprint and solar rooftop potential maps using deep learning-based instance segmentation to extract buildings' footprints from satellite images. A photovoltaic panels placement algorithm that considers the morphology of each roof is proposed. We show that the average rooftop's solar potential can fulfill the yearly electric needs of a single-family residence while using only 5% of the roof surface. The usage of 50% of a residential apartment rooftop area would achieve energy security for up to 8 households. We also compute the average and total solar rooftop potential per district to localize regions corresponding to the highest and lowest solar rooftop potential yield. Factors such as size, ground coverage ratio and PV_out are carefully investigated for each district. Baalbeck district yielded the highest total solar rooftop potential despite its low built-up area. While, Beirut capital city has the highest average solar rooftop potential due to its extremely populated urban nature. Reported results and analysis reveal solar rooftop potential urban patterns and provides policymakers and key stakeholders with tangible insights. Lebanon's total solar rooftop potential is about 28.1 TWh/year, two times larger than the national energy consumption in 2019.
4D iterative reconstruction of brain fMRI in the moving fetus
Authors: Athena Taymourtash, Hamza Kebiri, Sébastien Tourbier, Ernst Schwartz, Karl-Heinz Nenning, Roxane Licandro, Daniel Sobotka, Hélène Lajous, Priscille de Dumast, Meritxell Bach Cuadra, Georg Langs
Date: 2021-11-22 00:00:00
Tasks: Functional Connectivity, Time Series, Time Series Analysis

Resting-state functional Magnetic Resonance Imaging (fMRI) is a powerful imaging technique for studying functional development of the brain in utero. However, unpredictable and excessive movement of fetuses has limited clinical application since it causes substantial signal fluctuations which can systematically alter observed patterns of functional connectivity. Previous studies have focused on the accurate estimation of the motion parameters in case of large fetal head movement and used a 3D single step interpolation approach at each timepoint to recover motion-free fMRI images. This does not guarantee that the reconstructed image corresponds to the minimum error representation of fMRI time series given the acquired data. Here, we propose a novel technique based on four dimensional iterative reconstruction of the scattered slices acquired during fetal fMRI. The accuracy of the proposed method was quantitatively evaluated on a group of real clinical fMRI fetuses. The results indicate improvements of reconstruction quality compared to the conventional 3D interpolation approach.
Adversarial Examples on Segmentation Models Can be Easy to Transfer
Authors: Jindong Gu, Hengshuang Zhao, Volker Tresp, Philip Torr
Date: 2021-11-22 00:00:00
Tasks: Adversarial Robustness, Attribute, Classification, image-classification, Image Classification, Object Recognition, Segmentation, Semantic Segmentation

Deep neural network-based image classification can be misled by adversarial examples with small and quasi-imperceptible perturbations. Furthermore, the adversarial examples created on one classification model can also fool another different model. The transferability of the adversarial examples has recently attracted a growing interest since it makes black-box attacks on classification models feasible. As an extension of classification, semantic segmentation has also received much attention towards its adversarial robustness. However, the transferability of adversarial examples on segmentation models has not been systematically studied. In this work, we intensively study this topic. First, we explore the overfitting phenomenon of adversarial examples on classification and segmentation models. In contrast to the observation made on classification models that the transferability is limited by overfitting to the source model, we find that the adversarial examples on segmentations do not always overfit the source models. Even when no overfitting is presented, the transferability of adversarial examples is limited. We attribute the limitation to the architectural traits of segmentation models, i.e., multi-scale object recognition. Then, we propose a simple and effective method, dubbed dynamic scaling, to overcome the limitation. The high transferability achieved by our method shows that, in contrast to the observations in previous work, adversarial examples on a segmentation model can be easy to transfer to other segmentation models. Our analysis and proposals are supported by extensive experiments.
FFNB: Forgetting-Free Neural Blocks for Deep Continual Visual Learning
Authors: Hichem Sahbi, Haoming Zhan
Date: 2021-11-22 00:00:00
Tasks: Continual Learning, Memorization

Deep neural networks (DNNs) have recently achieved a great success in computer vision and several related fields. Despite such progress, current neural architectures still suffer from catastrophic interference (a.k.a. forgetting) which obstructs DNNs to learn continually. While several state-of-the-art methods have been proposed to mitigate forgetting, these existing solutions are either highly rigid (as regularization) or time/memory demanding (as replay). An intermediate class of methods, based on dynamic networks, has been proposed in the literature and provides a reasonable balance between task memorization and computational footprint. In this paper, we devise a dynamic network architecture for continual learning based on a novel forgetting-free neural block (FFNB). Training FFNB features on new tasks is achieved using a novel procedure that constrains the underlying parameters in the null-space of the previous tasks, while training classifier parameters equates to Fisher discriminant analysis. The latter provides an effective incremental process which is also optimal from a Bayesian perspective. The trained features and classifiers are further enhanced using an incremental "end-to-end" fine-tuning. Extensive experiments, conducted on different challenging classification problems, show the high effectiveness of the proposed method.
ShufaNet: Classification method for calligraphers who have reached the professional level
Authors: Ge Yunfei, Diao Changyu, Li Min, Yu Ruohan, Qiu Linshan, Xu Duanqing
Date: 2021-11-22 00:00:00
Tasks: Classification, Few-Shot Learning, Metric Learning

The authenticity of calligraphy is significant but difficult task in the realm of art, where the key problem is the few-shot classification of calligraphy. We propose a novel method, ShufaNet ("Shufa" is the pinyin of Chinese calligraphy), to classify Chinese calligraphers' styles based on metric learning in the case of few-shot, whose classification accuracy exceeds the level of students majoring in calligraphy. We present a new network architecture, including the unique expression of the style of handwriting fonts called ShufaLoss and the calligraphy category information as prior knowledge. Meanwhile, we modify the spatial attention module and create ShufaAttention for handwriting fonts based on the traditional Chinese nine Palace thought. For the training of the model, we build a calligraphers' data set. Our method achieved 65% accuracy rate in our data set for few-shot learning, surpassing resNet and other mainstream CNNs. Meanwhile, we conducted battle for calligraphy major students, and finally surpassed them. This is the first attempt of deep learning in the field of calligrapher classification, and we expect to provide ideas for subsequent research.
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion
Authors: Arthur Douillard, Alexandre Ramé, Guillaume Couairon, Matthieu Cord
Date: 2021-11-22 00:00:00
Proceeding: CVPR 2022 1
Tasks: Class Incremental Learning, Continual Learning, Decoder, Incremental Learning

Deep network architectures struggle to continually learn new tasks without forgetting the previous tasks. A recent trend indicates that dynamic architectures based on an expansion of the parameters can reduce catastrophic forgetting efficiently in continual learning. However, existing approaches often require a task identifier at test-time, need complex tuning to balance the growing number of parameters, and barely share any information across tasks. As a result, they struggle to scale to a large number of tasks without significant overhead. In this paper, we propose a transformer architecture based on a dedicated encoder/decoder framework. Critically, the encoder and decoder are shared among all tasks. Through a dynamic expansion of special tokens, we specialize each forward of our decoder network on a task distribution. Our strategy scales to a large number of tasks while having negligible memory and time overheads due to strict control of the parameters expansion. Moreover, this efficient strategy doesn't need any hyperparameter tuning to control the network's expansion. Our model reaches excellent results on CIFAR100 and state-of-the-art performances on the large-scale ImageNet100 and ImageNet1000 while having less parameters than concurrent dynamic frameworks.
SSR: An Efficient and Robust Framework for Learning with Unknown Label Noise
Authors: Chen Feng, Georgios Tzimiropoulos, Ioannis Patras
Date: 2021-11-22 00:00:00
Tasks: Image Classification, Learning with noisy labels, Self-Supervised Learning, Transfer Learning

Despite the large progress in supervised learning with neural networks, there are significant challenges in obtaining high-quality, large-scale and accurately labelled datasets. In such a context, how to learn in the presence of noisy labels has received more and more attention. As a relatively complex problem, in order to achieve good results, current approaches often integrate components from several fields, such as supervised learning, semi-supervised learning, transfer learning and resulting in complicated methods. Furthermore, they often make multiple assumptions about the type of noise of the data. This affects the model robustness and limits its performance under different noise conditions. In this paper, we consider a novel problem setting, Learning with Unknown Label Noise}(LULN), that is, learning when both the degree and the type of noise are unknown. Under this setting, unlike previous methods that often introduce multiple assumptions and lead to complex solutions, we propose a simple, efficient and robust framework named Sample Selection and Relabelling(SSR), that with a minimal number of hyperparameters achieves SOTA results in various conditions. At the heart of our method is a sample selection and relabelling mechanism based on a non-parametric KNN classifier~(NPK) $g_q$ and a parametric model classifier~(PMC) $g_p$, respectively, to select the clean samples and gradually relabel the noisy samples. Without bells and whistles, such as model co-training, self-supervised pre-training and semi-supervised learning, and with robustness concerning the settings of its few hyper-parameters, our method significantly surpasses previous methods on both CIFAR10/CIFAR100 with synthetic noise and real-world noisy datasets such as WebVision, Clothing1M and ANIMAL-10N. Code is available at https://github.com/MrChenFeng/SSR_BMVC2022.
Point Cloud Color Constancy
Authors: Xiaoyan Xing, Yanlin Qian, Sibo Feng, Yuhan Dong, Jiri Matas
Date: 2021-11-22 00:00:00
Proceeding: CVPR 2022 1
Tasks: Color Constancy

In this paper, we present Point Cloud Color Constancy, in short PCCC, an illumination chromaticity estimation algorithm exploiting a point cloud. We leverage the depth information captured by the time-of-flight (ToF) sensor mounted rigidly with the RGB sensor, and form a 6D cloud where each point contains the coordinates and RGB intensities, noted as (x,y,z,r,g,b). PCCC applies the PointNet architecture to the color constancy problem, deriving the illumination vector point-wise and then making a global decision about the global illumination chromaticity. On two popular RGB-D datasets, which we extend with illumination information, as well as on a novel benchmark, PCCC obtains lower error than the state-of-the-art algorithms. Our method is simple and fast, requiring merely 16*16-size input and reaching speed over 500 fps, including the cost of building the point cloud and net inference.
Automated cross-sectional view selection in CT angiography of aortic dissections with uncertainty awareness and retrospective clinical annotations
Authors: Antonio Pepe, Jan Egger, Marina Codari, Martin J. Willemink, Christina Gsaxner, Jianning Li, Peter M. Roth, Gabriel Mistelbauer, Dieter Schmalstieg, Dominik Fleischmann
Date: 2021-11-22 00:00:00
Tasks: Uncertainty Quantification

Objective: Surveillance imaging of chronic aortic diseases, such as dissections, relies on obtaining and comparing cross-sectional diameter measurements at predefined aortic landmarks, over time. Due to a lack of robust tools, the orientation of the cross-sectional planes is defined manually by highly trained operators. We show how manual annotations routinely collected in a clinic can be efficiently used to ease this task, despite the presence of a non-negligible interoperator variability in the measurements. Impact: Ill-posed but repetitive imaging tasks can be eased or automated by leveraging imperfect, retrospective clinical annotations. Methodology: In this work, we combine convolutional neural networks and uncertainty quantification methods to predict the orientation of such cross-sectional planes. We use clinical data randomly processed by 11 operators for training, and test on a smaller set processed by 3 independent operators to assess interoperator variability. Results: Our analysis shows that manual selection of cross-sectional planes is characterized by 95% limits of agreement (LOA) of $10.6^\circ$ and $21.4^\circ$ per angle. Our method showed to decrease static error by $3.57^\circ$ ($40.2$%) and $4.11^\circ$ ($32.8$%) against state of the art and LOA by $5.4^\circ$ ($49.0$%) and $16.0^\circ$ ($74.6$%) against manual processing. Conclusion: This suggests that pre-existing annotations can be an inexpensive resource in clinics to ease ill-posed and repetitive tasks like cross-section extraction for surveillance of aortic dissections.
MiNet: A Convolutional Neural Network for Identifying and Categorising Minerals
Authors: Emmanuel Asiedu Brempong, Millicent Agangiba, Daniel Aikins
Date: 2021-11-22 00:00:00
Tasks: image-classification, Image Classification

Identification of minerals in the field is a task that is wrought with many challenges. Traditional approaches are prone to errors where there is no enough experience and expertise. Several existing techniques mainly make use of features of the minerals under a microscope and tend to favour a manual feature extraction pipeline. Deep learning methods can help overcome some of these hurdles and provide simple and effective ways to identify minerals. In this paper, we present an algorithm for identifying minerals from hand specimen images. Using a Convolutional Neural Network (CNN), we develop a single-label image classification model to identify and categorise seven classes of minerals. Experiments conducted using real-world datasets show that the model achieves an accuracy of 90.75%.
Nanorobot queue: Cooperative treatment of cancer based on team member communication and image processing
Authors: Xinyu Zhou
Date: 2021-11-22 00:00:00
Tasks: image-classification, Image Classification

Although nanorobots have been used as clinical prescriptions for work such as gastroscopy, and even photoacoustic tomography technology has been proposed to control nanorobots to deliver drugs at designated delivery points in real time, and there are cases of eliminating "superbacteria" in blood through nanorobots, most technologies are immature, either with low efficiency or low accuracy, Either it can not be mass produced, so the most effective way to treat cancer diseases at this stage is through chemotherapy and radiotherapy. Patients are suffering and can not be cured. Therefore, this paper proposes an ideal model of a treatment method that can completely cure cancer, a cooperative treatment method based on nano robot queue through team member communication and computer vision image classification (target detection).