pwc-archive/papers-with-abstracts huggingface dataset:
Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study
Authors: Ziyuan Qin, Huahui Yi, Qicheng Lao, Kang Li
Date: 2022-09-30 00:00:00

The large-scale pre-trained vision language models (VLM) have shown remarkable domain transfer capability on natural images. However, it remains unknown whether this capability can also apply to the medical image domain. This paper thoroughly studies the knowledge transferability of pre-trained VLMs to the medical domain, where we show that well-designed medical prompts are the key to elicit knowledge from pre-trained VLMs. We demonstrate that by prompting with expressive attributes that are shared between domains, the VLM can carry the knowledge across domains and improve its generalization. This mechanism empowers VLMs to recognize novel objects with fewer or without image samples. Furthermore, to avoid the laborious manual designing process, we develop three approaches for automatic generation of medical prompts, which can inject expert-level medical knowledge and image-specific information into the prompts for fine-grained grounding. We conduct extensive experiments on thirteen different medical datasets across various modalities, showing that our well-designed prompts greatly improve the zero-shot performance compared to the default prompts, and our fine-tuned models surpass the supervised models by a significant margin.
Sphere-Guided Training of Neural Implicit Surfaces
Authors: Andreea Dogaru, Andrei Timotei Ardelean, Savva Ignatyev, Egor Zakharov, Evgeny Burnaev
Date: 2022-09-30 00:00:00
Proceeding: CVPR 2023 1
Tasks: 3D Reconstruction, Multi-View 3D Reconstruction, Surface Reconstruction

In recent years, neural distance functions trained via volumetric ray marching have been widely adopted for multi-view 3D reconstruction. These methods, however, apply the ray marching procedure for the entire scene volume, leading to reduced sampling efficiency and, as a result, lower reconstruction quality in the areas of high-frequency details. In this work, we address this problem via joint training of the implicit function and our new coarse sphere-based surface reconstruction. We use the coarse representation to efficiently exclude the empty volume of the scene from the volumetric ray marching procedure without additional forward passes of the neural surface network, which leads to an increased fidelity of the reconstructions compared to the base systems. We evaluate our approach by incorporating it into the training procedures of several implicit surface modeling methods and observe uniform improvements across both synthetic and real-world datasets. Our codebase can be accessed via the project page: https://andreeadogaru.github.io/SphereGuided
Learning Second Order Local Anomaly for General Face Forgery Detection
Authors: Jianwei Fei, Yunshu Dai, Peipeng Yu, Tianrun Shen, Zhihua Xia, Jian Weng
Date: 2022-09-30 00:00:00
Proceeding: CVPR 2022 1

In this work, we propose a novel method to improve the generalization ability of CNN-based face forgery detectors. Our method considers the feature anomalies of forged faces caused by the prevalent blending operations in face forgery algorithms. Specifically, we propose a weakly supervised Second Order Local Anomaly (SOLA) learning module to mine anomalies in local regions using deep feature maps. SOLA first decomposes the neighborhood of local features by different directions and distances and then calculates the first and second order local anomaly maps which provide more general forgery traces for the classifier. We also propose a Local Enhancement Module (LEM) to improve the discrimination between local features of real and forged regions, so as to ensure accuracy in calculating anomalies. Besides, an improved Adaptive Spatial Rich Model (ASRM) is introduced to help mine subtle noise features via learnable high pass filters. With neither pixel level annotations nor external synthetic data, our method using a simple ResNet18 backbone achieves competitive performances compared with state-of-the-art works when evaluated on unseen forgeries.
Impact of Face Image Quality Estimation on Presentation Attack Detection
Authors: Carlos Aravena, Diego Pasmino, Juan E. Tapia, Christoph Busch
Date: 2022-09-30 00:00:00
Tasks: Face Image Quality, Face Image Quality Assessment, Face Recognition, Image Quality Assessment, Image Quality Estimation

Non-referential face image quality assessment methods have gained popularity as a pre-filtering step on face recognition systems. In most of them, the quality score is usually designed with face matching in mind. However, a small amount of work has been done on measuring their impact and usefulness on Presentation Attack Detection (PAD). In this paper, we study the effect of quality assessment methods on filtering bona fide and attack samples, their impact on PAD systems, and how the performance of such systems is improved when training on a filtered (by quality) dataset. On a Vision Transformer PAD algorithm, a reduction of 20% of the training dataset by removing lower quality samples allowed us to improve the BPCER by 3% in a cross-dataset test.
Point Cloud Quality Assessment using 3D Saliency Maps
Authors: Zhengyu Wang, Yujie Zhang, Qi Yang, Yiling Xu, Jun Sun, Shan Liu
Date: 2022-09-30 00:00:00
Tasks: Point Cloud Quality Assessment, Saliency Detection

Point cloud quality assessment (PCQA) has become an appealing research field in recent days. Considering the importance of saliency detection in quality assessment, we propose an effective full-reference PCQA metric which makes the first attempt to utilize the saliency information to facilitate quality prediction, called point cloud quality assessment using 3D saliency maps (PQSM). Specifically, we first propose a projection-based point cloud saliency map generation method, in which depth information is introduced to better reflect the geometric characteristics of point clouds. Then, we construct point cloud local neighborhoods to derive three structural descriptors to indicate the geometry, color and saliency discrepancies. Finally, a saliency-based pooling strategy is proposed to generate the final quality score. Extensive experiments are performed on four independent PCQA databases. The results demonstrate that the proposed PQSM shows competitive performances compared to multiple state-of-the-art PCQA metrics.
Reliable Face Morphing Attack Detection in On-The-Fly Border Control Scenario with Variation in Image Resolution and Capture Distance
Authors: Jag Mohan Singh, Raghavendra Ramachandra
Date: 2022-09-30 00:00:00
Tasks: Face Morphing Attack Detection, Face Recognition, MORPH

Face Recognition Systems (FRS) are vulnerable to various attacks performed directly and indirectly. Among these attacks, face morphing attacks are highly potential in deceiving automatic FRS and human observers and indicate a severe security threat, especially in the border control scenario. This work presents a face morphing attack detection, especially in the On-The-Fly (OTF) Automatic Border Control (ABC) scenario. We present a novel Differential-MAD (D-MAD) algorithm based on the spherical interpolation and hierarchical fusion of deep features computed from six different pre-trained deep Convolutional Neural Networks (CNNs). Extensive experiments are carried out on the newly generated face morphing dataset (SCFace-Morph) based on the publicly available SCFace dataset by considering the real-life scenario of Automatic Border Control (ABC) gates. Experimental protocols are designed to benchmark the proposed and state-of-the-art (SOTA) D-MAD techniques for different camera resolutions and capture distances. Obtained results have indicated the superior performance of the proposed D-MAD method compared to the existing methods.
Two-headed eye-segmentation approach for biometric identification
Authors: Wiktor Lazarski, Maciej Zieba, Tanguy Jeanneau, Tobias Zillig, Christian Brendel
Date: 2022-09-30 00:00:00
Tasks: Person Identification, Vocal Bursts Valence Prediction

Iris-based identification systems are among the most popular approaches for person identification. Such systems require good-quality segmentation modules that ideally identify the regions for different eye components. This paper introduces the new two-headed architecture, where the eye components and eyelashes are segmented using two separate decoding modules. Moreover, we investigate various training scenarios by adopting different training losses. Thanks to the two-headed approach, we were also able to examine the quality of the model with the convex prior, which enforces the convexity of the segmented shapes. We conducted an extensive evaluation of various learning scenarios on real-life conditions high-resolution near-infrared iris images.
Melanoma Skin Cancer and Nevus Mole Classification using Intensity Value Estimation with Convolutional Neural Network
Authors: N. I. Md. Ashafuddula, Rafiqul Islam
Date: 2022-09-30 00:00:00
Tasks: Specificity

Melanoma skin cancer is one of the most dangerous and life-threatening cancer. Exposure to ultraviolet rays may damage the skin cell's DNA, which causes melanoma skin cancer. However, it is difficult to detect and classify melanoma and nevus mole at the immature stages. In this work, an automatic deep learning system is developed based on the intensity value estimation with a convolutional neural network model (CNN) to detect and classify melanoma and nevus mole more accurately. Since intensity levels are the most distinctive features for object or region of interest identification, the high-intensity pixel values are selected from the extracted lesion images. Incorporating those high-intensity features into the CNN improves the overall performance of the proposed model than the state-of-the-art methods for detecting melanoma skin cancer. To evaluate the system, we used 5-fold cross-validation. Experimental results show that a superior percentage of accuracy (92.58%), sensitivity (93.76%), specificity (91.56%), and precision (90.68%) are achieved.
Road Network Deterioration Monitoring Using Aerial Images and Computer Vision
Authors: Nicolas Parra-A, Vladimir Vargas-Calderón, Herbert Vinck-Posada, Nicanor Vinck
Date: 2022-09-30 00:00:00

Road maintenance is an essential process for guaranteeing the quality of transportation in any city. A crucial step towards effective road maintenance is the ability to update the inventory of the road network. We present a proof of concept of a protocol for maintaining said inventory based on the use of unmanned aerial vehicles to quickly collect images which are processed by a computer vision program that automatically identifies potholes and their severity. Our protocol aims to provide information to local governments to prioritise the road network maintenance budget, and to be able to detect early stages of road deterioration so as to minimise maintenance expenditure.
Semi-Supervised Domain Generalization for Cardiac Magnetic Resonance Image Segmentation with High Quality Pseudo Labels
Authors: Wanqin Ma, Huifeng Yao, Yiqun Lin, Jiarong Guo, Xiaomeng Li
Date: 2022-09-30 00:00:00
Tasks: Domain Generalization, Image Segmentation, Segmentation, Semantic Segmentation, Semi-Supervised Domain Generalization

Developing a deep learning method for medical segmentation tasks heavily relies on a large amount of labeled data. However, the annotations require professional knowledge and are limited in number. Recently, semi-supervised learning has demonstrated great potential in medical segmentation tasks. Most existing methods related to cardiac magnetic resonance images only focus on regular images with similar domains and high image quality. A semi-supervised domain generalization method was developed in [2], which enhances the quality of pseudo labels on varied datasets. In this paper, we follow the strategy in [2] and present a domain generalization method for semi-supervised medical segmentation. Our main goal is to improve the quality of pseudo labels under extreme MRI Analysis with various domains. We perform Fourier transformation on input images to learn low-level statistics and cross-domain information. Then we feed the augmented images as input to the double cross pseudo supervision networks to calculate the variance among pseudo labels. We evaluate our method on the CMRxMotion dataset [1]. With only partially labeled data and without domain labels, our approach consistently generates accurate segmentation results of cardiac magnetic resonance images with different respiratory motions. Code is available at: https://github.com/MAWanqin2002/STACOM2022Ma
Entropy-driven Unsupervised Keypoint Representation Learning in Videos
Authors: Ali Younes, Simone Schaub-Meyer, Georgia Chalvatzaki
Date: 2022-09-30 00:00:00
Tasks: Representation Learning

Extracting informative representations from videos is fundamental for effectively learning various downstream tasks. We present a novel approach for unsupervised learning of meaningful representations from videos, leveraging the concept of image spatial entropy (ISE) that quantifies the per-pixel information in an image. We argue that \textit{local entropy} of pixel neighborhoods and their temporal evolution create valuable intrinsic supervisory signals for learning prominent features. Building on this idea, we abstract visual features into a concise representation of keypoints that act as dynamic information transmitters, and design a deep learning model that learns, purely unsupervised, spatially and temporally consistent representations \textit{directly} from video frames. Two original information-theoretic losses, computed from local entropy, guide our model to discover consistent keypoint representations; a loss that maximizes the spatial information covered by the keypoints and a loss that optimizes the keypoints' information transportation over time. We compare our keypoint representation to strong baselines for various downstream tasks, \eg, learning object dynamics. Our empirical results show superior performance for our information-driven keypoints that resolve challenges like attendance to static and dynamic objects or objects abruptly entering and leaving the scene.
Rethinking the Learning Paradigm for Facial Expression Recognition
Authors: Weijie Wang, Nicu Sebe, Bruno Lepri
Date: 2022-09-30 00:00:00
Tasks: Facial Expression Recognition, Facial Expression Recognition (FER)

Due to the subjective crowdsourcing annotations and the inherent inter-class similarity of facial expressions, the real-world Facial Expression Recognition (FER) datasets usually exhibit ambiguous annotation. To simplify the learning paradigm, most previous methods convert ambiguous annotation results into precise one-hot annotations and train FER models in an end-to-end supervised manner. In this paper, we rethink the existing training paradigm and propose that it is better to use weakly supervised strategies to train FER models with original ambiguous annotation.
Semi-Supervised Single-View 3D Reconstruction via Prototype Shape Priors
Authors: Zhen Xing, Hengduo Li, Zuxuan Wu, Yu-Gang Jiang
Date: 2022-09-30 00:00:00
Tasks: 3D Reconstruction, image-classification, Image Classification, Object Reconstruction, Semi-Supervised Image Classification, Single-View 3D Reconstruction

The performance of existing single-view 3D reconstruction methods heavily relies on large-scale 3D annotations. However, such annotations are tedious and expensive to collect. Semi-supervised learning serves as an alternative way to mitigate the need for manual labels, but remains unexplored in 3D reconstruction. Inspired by the recent success of semi-supervised image classification tasks, we propose SSP3D, a semi-supervised framework for 3D reconstruction. In particular, we introduce an attention-guided prototype shape prior module for guiding realistic object reconstruction. We further introduce a discriminator-guided module to incentivize better shape generation, as well as a regularizer to tolerate noisy training samples. On the ShapeNet benchmark, the proposed approach outperforms previous supervised methods by clear margins under various labeling ratios, (i.e., 1%, 5% , 10% and 20%). Moreover, our approach also performs well when transferring to real-world Pix3D datasets under labeling ratios of 10%. We also demonstrate our method could transfer to novel categories with few novel supervised data. Experiments on the popular ShapeNet dataset show that our method outperforms the zero-shot baseline by over 12% and we also perform rigorous ablations and analysis to validate our approach.
DELAD: Deep Landweber-guided deconvolution with Hessian and sparse prior
Authors: Tomas Chobola, Anton Theileis, Jan Taucher, Tingying Peng
Date: 2022-09-30 00:00:00
Tasks: Benchmarking, Blind Image Deblurring, Deblurring, Image Deblurring, Image Deconvolution, Image Reconstruction

We present a model for non-blind image deconvolution that incorporates the classic iterative method into a deep learning application. Instead of using large over-parameterised generative networks to create sharp picture representations, we build our network based on the iterative Landweber deconvolution algorithm, which is integrated with trainable convolutional layers to enhance the recovered image structures and details. Additional to the data fidelity term, we also add Hessian and sparse constraints as regularization terms to improve the image reconstruction quality. Our proposed model is \textit{self-supervised} and converges to a solution based purely on the input blurred image and respective blur kernel without the requirement of any pre-training. We evaluate our technique using standard computer vision benchmarking datasets as well as real microscope images obtained by our enhanced depth-of-field (EDOF) underwater microscope, demonstrating the capabilities of our model in a real-world application. The quantitative results demonstrate that our approach is competitive with state-of-the-art non-blind image deblurring methods despite having a fraction of the parameters and not being pre-trained, demonstrating the efficiency and efficacy of embedding a classic deconvolution approach inside a deep network.
NBV-SC: Next Best View Planning based on Shape Completion for Fruit Mapping and Reconstruction
Authors: Rohit Menon, Tobias Zaenker, Nils Dengler, Maren Bennewitz
Date: 2022-09-30 00:00:00
Tasks: Image Segmentation, Semantic Segmentation

Active perception for fruit mapping and harvesting is a difficult task since occlusions occur frequently and the location as well as size of fruits change over time. State-of-the-art viewpoint planning approaches utilize computationally expensive ray casting operations to find good viewpoints aiming at maximizing information gain and covering the fruits in the scene. In this paper, we present a novel viewpoint planning approach that explicitly uses information about the predicted fruit shapes to compute targeted viewpoints that observe as yet unobserved parts of the fruits. Furthermore, we formulate the concept of viewpoint dissimilarity to reduce the sampling space for more efficient selection of useful, dissimilar viewpoints. Our simulation experiments with a UR5e arm equipped with an RGB-D sensor provide a quantitative demonstration of the efficacy of our iterative next best view planning method based on shape completion. In comparative experiments with a state-of-the-art viewpoint planner, we demonstrate improvement not only in the estimation of the fruit sizes, but also in their reconstruction, while significantly reducing the planning time. Finally, we show the viability of our approach for mapping sweet peppers plants with a real robotic system in a commercial glasshouse.
Inharmonious Region Localization by Magnifying Domain Discrepancy
Authors: Jing Liang, Li Niu, Penghao Wu, Fengjun Guo, Teng Long
Date: 2022-09-30 00:00:00
Tasks: Image Harmonization

Inharmonious region localization aims to localize the region in a synthetic image which is incompatible with surrounding background. The inharmony issue is mainly attributed to the color and illumination inconsistency produced by image editing techniques. In this work, we tend to transform the input image to another color space to magnify the domain discrepancy between inharmonious region and background, so that the model can identify the inharmonious region more easily. To this end, we present a novel framework consisting of a color mapping module and an inharmonious region localization network, in which the former is equipped with a novel domain discrepancy magnification loss and the latter could be an arbitrary localization network. Extensive experiments on image harmonization dataset show the superiority of our designed framework. Our code is available at https://github.com/bcmi/MadisNet-Inharmonious-Region-Localization.
Towards End-to-end Handwritten Document Recognition
Authors: Denis Coquenet
Date: 2022-09-30 00:00:00
Tasks: Handwriting Recognition, Handwritten Document Recognition, Handwritten Text Recognition

Handwritten text recognition has been widely studied in the last decades for its numerous applications. Nowadays, the state-of-the-art approach consists in a three-step process. The document is segmented into text lines, which are then ordered and recognized. However, this three-step approach has many drawbacks. The three steps are treated independently whereas they are closely related. Errors accumulate from one step to the other. The ordering step is based on heuristic rules which prevent its use for documents with a complex layouts or for heterogeneous documents. The need for additional physical segmentation annotations for training the segmentation stage is inherent to this approach. In this thesis, we propose to tackle these issues by performing the handwritten text recognition of whole document in an end-to-end way. To this aim, we gradually increase the difficulty of the recognition task, moving from isolated lines to paragraphs, and then to whole documents. We proposed an approach at the line level, based on a fully convolutional network, in order to design a first generic feature extraction step for the handwriting recognition task. Based on this preliminary work, we studied two different approaches to recognize handwritten paragraphs. We reached state-of-the-art results at paragraph level on the RIMES 2011, IAM and READ 2016 datasets and outperformed the line-level state of the art on these datasets. We finally proposed the first end-to-end approach dedicated to the recognition of both text and layout, at document level. Characters and layout tokens are sequentially predicted following a learned reading order. We proposed two new metrics we used to evaluate this task on the RIMES 2009 and READ 2016 dataset, at page level and double-page level.
SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation
Authors: Rita Ramos, Bruno Martins, Desmond Elliott, Yova Kementchedjhieva
Date: 2022-09-30 00:00:00
Proceeding: CVPR 2023 1
Tasks: Decoder, Image Captioning, Retrieval

Recent advances in image captioning have focused on scaling the data and model size, substantially increasing the cost of pre-training and finetuning. As an alternative to large models, we present SmallCap, which generates a caption conditioned on an input image and related captions retrieved from a datastore. Our model is lightweight and fast to train, as the only learned parameters are in newly introduced cross-attention layers between a pre-trained CLIP encoder and GPT-2 decoder. SmallCap can transfer to new domains without additional finetuning and can exploit large-scale data in a training-free fashion since the contents of the datastore can be readily replaced. Our experiments show that SmallCap, trained only on COCO, has competitive performance on this benchmark, and also transfers to other domains without retraining, solely through retrieval from target-domain data. Further improvement is achieved through the training-free exploitation of diverse human-labeled and web data, which proves to be effective for a range of domains, including the nocaps benchmark, designed to test generalization to unseen visual concepts.