pwc-archive/papers-with-abstracts huggingface dataset:
SemFormer: Semantic Guided Activation Transformer for Weakly Supervised Semantic Segmentation
Authors: Junliang Chen, Xiaodong Zhao, Cheng Luo, Linlin Shen
Date: 2022-10-26 00:00:00
Tasks: Semantic Segmentation, Weakly supervised Semantic Segmentation, Weakly-Supervised Semantic Segmentation

Recent mainstream weakly supervised semantic segmentation (WSSS) approaches are mainly based on Class Activation Map (CAM) generated by a CNN (Convolutional Neural Network) based image classifier. In this paper, we propose a novel transformer-based framework, named Semantic Guided Activation Transformer (SemFormer), for WSSS. We design a transformer-based Class-Aware AutoEncoder (CAAE) to extract the class embeddings for the input image and learn class semantics for all classes of the dataset. The class embeddings and learned class semantics are then used to guide the generation of activation maps with four losses, i.e., class-foreground, class-background, activation suppression, and activation complementation loss. Experimental results show that our SemFormer achieves \textbf{74.3}\% mIoU and surpasses many recent mainstream WSSS approaches by a large margin on PASCAL VOC 2012 dataset. Code will be available at \url{https://github.com/JLChen-C/SemFormer}.
Analyzing Deep Learning Representations of Point Clouds for Real-Time In-Vehicle LiDAR Perception
Authors: Marc Uecker, Tobias Fleck, Marcel Pflugfelder, J. Marius Zöllner
Date: 2022-10-26 00:00:00
Tasks: Autonomous Vehicles, Computational Efficiency, Semantic Segmentation

LiDAR sensors are an integral part of modern autonomous vehicles as they provide an accurate, high-resolution 3D representation of the vehicle's surroundings. However, it is computationally difficult to make use of the ever-increasing amounts of data from multiple high-resolution LiDAR sensors. As frame-rates, point cloud sizes and sensor resolutions increase, real-time processing of these point clouds must still extract semantics from this increasingly precise picture of the vehicle's environment. One deciding factor of the run-time performance and accuracy of deep neural networks operating on these point clouds is the underlying data representation and the way it is computed. In this work, we examine the relationship between the computational representations used in neural networks and their performance characteristics. To this end, we propose a novel computational taxonomy of LiDAR point cloud representations used in modern deep neural networks for 3D point cloud processing. Using this taxonomy, we perform a structured analysis of different families of approaches. Thereby, we uncover common advantages and limitations in terms of computational efficiency, memory requirements, and representational capacity as measured by semantic segmentation performance. Finally, we provide some insights and guidance for future developments in neural point cloud processing methods.
Automatic Diagnosis of Myocarditis Disease in Cardiac MRI Modality using Deep Transformers and Explainable Artificial Intelligence
Authors: Mahboobeh Jafari, Afshin Shoeibi, Navid Ghassemi, Jonathan Heras, Sai Ho Ling, Amin Beheshti, Yu-Dong Zhang, Shui-Hua Wang, Roohallah Alizadehsani, Juan M. Gorriz, U. Rajendra Acharya, Hamid Alinejad Rokny
Date: 2022-10-26 00:00:00
Tasks: Data Augmentation, Denoising, Explainable artificial intelligence

Myocarditis is a significant cardiovascular disease (CVD) that poses a threat to the health of many individuals by causing damage to the myocardium. The occurrence of microbes and viruses, including the likes of HIV, plays a crucial role in the development of myocarditis disease (MCD). The images produced during cardiac magnetic resonance imaging (CMRI) scans are low contrast, which can make it challenging to diagnose cardiovascular diseases. In other hand, checking numerous CMRI slices for each CVD patient can be a challenging task for medical doctors. To overcome the existing challenges, researchers have suggested the use of artificial intelligence (AI)-based computer-aided diagnosis systems (CADS). The presented paper outlines a CADS for the detection of MCD from CMR images, utilizing deep learning (DL) methods. The proposed CADS consists of several steps, including dataset, preprocessing, feature extraction, classification, and post-processing. First, the Z-Alizadeh dataset was selected for the experiments. Subsequently, the CMR images underwent various preprocessing steps, including denoising, resizing, as well as data augmentation (DA) via CutMix and MixUp techniques. In the following, the most current deep pre-trained and transformer models are used for feature extraction and classification on the CMR images. The findings of our study reveal that transformer models exhibit superior performance in detecting MCD as opposed to pre-trained architectures. In terms of DL architectures, the Turbulence Neural Transformer (TNT) model exhibited impressive accuracy, reaching 99.73% utilizing a 10-fold cross-validation approach. Additionally, to pinpoint areas of suspicion for MCD in CMRI images, the Explainable-based Grad Cam method was employed.
A novel filter based on three variables mutual information for dimensionality reduction and classification of hyperspectral images
Authors: Asma Elmaizi, Elkebir Sarhrouni, Ahmed Hammouch, Chafik Nacir
Date: 2022-10-26 00:00:00
Tasks: Classification, Classification Of Hyperspectral Images, Dimensionality Reduction

The high dimensionality of hyperspectral images (HSI) that contains more than hundred bands (images) for the same region called Ground Truth Map, often imposes a heavy computational burden for image processing and complicates the learning process. In fact, the removal of irrelevant, noisy and redundant bands helps increase the classification accuracy. Band selection filter based on "Mutual Information" is a common technique for dimensionality reduction. In this paper, a categorization of dimensionality reduction methods according to the evaluation process is presented. Moreover, a new filter approach based on three variables mutual information is developed in order to measure band correlation for classification, it considers not only bands relevance but also bands interaction. The proposed approach is compared to a reproduced filter algorithm based on mutual information. Experimental results on HSI AVIRIS 92AV3C have shown that the proposed approach is very competitive, effective and outperforms the reproduced filter strategy performance. Keywords - Hyperspectral images, Classification, band Selection, Three variables Mutual Information, information gain.
End-to-end Tracking with a Multi-query Transformer
Authors: Bruno Korbar, Andrew Zisserman
Date: 2022-10-26 00:00:00
Tasks: Multiple Object Tracking, Object, Object Tracking

Multiple-object tracking (MOT) is a challenging task that requires simultaneous reasoning about location, appearance, and identity of the objects in the scene over time. Our aim in this paper is to move beyond tracking-by-detection approaches, that perform well on datasets where the object classes are known, to class-agnostic tracking that performs well also for unknown object classes.To this end, we make the following three contributions: first, we introduce {\em semantic detector queries} that enable an object to be localized by specifying its approximate position, or its appearance, or both; second, we use these queries within an auto-regressive framework for tracking, and propose a multi-query tracking transformer (\textit{MQT}) model for simultaneous tracking and appearance-based re-identification (reID) based on the transformer architecture with deformable attention. This formulation allows the tracker to operate in a class-agnostic manner, and the model can be trained end-to-end; finally, we demonstrate that \textit{MQT} performs competitively on standard MOT benchmarks, outperforms all baselines on generalised-MOT, and generalises well to a much harder tracking problems such as tracking any object on the TAO dataset.
A Stronger Baseline For Automatic Pfirrmann Grading Of Lumbar Spine MRI Using Deep Learning
Authors: Narasimharao Kowlagi, Huy Hoang Nguyen, Terence McSweeney, Simo Saarakkala, Juhani määttä, Jaro Karppinen, Aleksei Tiulpin
Date: 2022-10-26 00:00:00
Tasks: Semantic Segmentation

This paper addresses the challenge of grading visual features in lumbar spine MRI using Deep Learning. Such a method is essential for the automatic quantification of structural changes in the spine, which is valuable for understanding low back pain. Multiple recent studies investigated different architecture designs, and the most recent success has been attributed to the use of transformer architectures. In this work, we argue that with a well-tuned three-stage pipeline comprising semantic segmentation, localization, and classification, convolutional networks outperform the state-of-the-art approaches. We conducted an ablation study of the existing methods in a population cohort, and report performance generalization across various subgroups. Our code is publicly available to advance research on disc degeneration and low back pain.
Compressed Sensing MRI Reconstruction Regularized by VAEs with Structured Image Covariance
Authors: Margaret Duff, Ivor J. A. Simpson, Matthias J. Ehrhardt, Neill D. F. Campbell
Date: 2022-10-26 00:00:00
Tasks: compressed sensing, MRI Reconstruction

Objective: This paper investigates how generative models, trained on ground-truth images, can be used \changes{as} priors for inverse problems, penalizing reconstructions far from images the generator can produce. The aim is that learned regularization will provide complex data-driven priors to inverse problems while still retaining the control and insight of a variational regularization method. Moreover, unsupervised learning, without paired training data, allows the learned regularizer to remain flexible to changes in the forward problem such as noise level, sampling pattern or coil sensitivities in MRI. Approach: We utilize variational autoencoders (VAEs) that generate not only an image but also a covariance uncertainty matrix for each image. The covariance can model changing uncertainty dependencies caused by structure in the image, such as edges or objects, and provides a new distance metric from the manifold of learned images. Main results: We evaluate these novel generative regularizers on retrospectively sub-sampled real-valued MRI measurements from the fastMRI dataset. We compare our proposed learned regularization against other unlearned regularization approaches and unsupervised and supervised deep learning methods. Significance: Our results show that the proposed method is competitive with other state-of-the-art methods and behaves consistently with changing sampling patterns and noise levels.
Towards the Detection of Diffusion Model Deepfakes
Authors: Jonas Ricker, Simon Damm, Thorsten Holz, Asja Fischer
Date: 2022-10-26 00:00:00
Tasks: Attribute, Image Generation, model

In the course of the past few years, diffusion models (DMs) have reached an unprecedented level of visual quality. However, relatively little attention has been paid to the detection of DM-generated images, which is critical to prevent adverse impacts on our society. In contrast, generative adversarial networks (GANs), have been extensively studied from a forensic perspective. In this work, we therefore take the natural next step to evaluate whether previous methods can be used to detect images generated by DMs. Our experiments yield two key findings: (1) state-of-the-art GAN detectors are unable to reliably distinguish real from DM-generated images, but (2) re-training them on DM-generated images allows for almost perfect detection, which remarkably even generalizes to GANs. Together with a feature space analysis, our results lead to the hypothesis that DMs produce fewer detectable artifacts and are thus more difficult to detect compared to GANs. One possible reason for this is the absence of grid-like frequency artifacts in DM-generated images, which are a known weakness of GANs. However, we make the interesting observation that diffusion models tend to underestimate high frequencies, which we attribute to the learning objective.
FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralization
Authors: Junyang Wang, Yi Zhang, Jitao Sang
Date: 2022-10-26 00:00:00
Tasks: Attribute, Fairness, Image Retrieval, Retrieval

The Vision-Language Pre-training (VLP) models like CLIP have gained popularity in recent years. However, many works found that the social biases hidden in CLIP easily manifest in downstream tasks, especially in image retrieval, which can have harmful effects on human society. In this work, we propose FairCLIP to eliminate the social bias in CLIP-based image retrieval without damaging the retrieval performance achieving the compatibility between the debiasing effect and the retrieval performance. FairCLIP is divided into two steps: Attribute Prototype Learning (APL) and Representation Neutralization (RN). In the first step, we extract the concepts needed for debiasing in CLIP. We use the query with learnable word vector prefixes as the extraction structure. In the second step, we first divide the attributes into target and bias attributes. By analysis, we find that both attributes have an impact on the bias. Therefore, we try to eliminate the bias by using Re-Representation Matrix (RRM) to achieve the neutralization of the representation. We compare the debiasing effect and retrieval performance with other methods, and experiments demonstrate that FairCLIP can achieve the best compatibility. Although FairCLIP is used to eliminate bias in image retrieval, it achieves the neutralization of the representation which is common to all CLIP downstream tasks. This means that FairCLIP can be applied as a general debiasing method for other fairness issues related to CLIP.
Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering
Authors: Qingyi Si, Yuanxin Liu, Zheng Lin, Peng Fu, Weiping Wang
Date: 2022-10-26 00:00:00
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA)

Despite the excellent performance of vision-language pre-trained models (VLPs) on conventional VQA task, they still suffer from two problems: First, VLPs tend to rely on language biases in datasets and fail to generalize to out-of-distribution (OOD) data. Second, they are inefficient in terms of memory footprint and computation. Although promising progress has been made in both problems, most existing works tackle them independently. To facilitate the application of VLP to VQA tasks, it is imperative to jointly study VLP compression and OOD robustness, which, however, has not yet been explored. This paper investigates whether a VLP can be compressed and debiased simultaneously by searching sparse and robust subnetworks. To this end, we systematically study the design of a training and compression pipeline to search the subnetworks, as well as the assignment of sparsity to different modality-specific modules. Our experiments involve 3 VLPs, 2 compression methods, 4 training methods, 2 datasets and a range of sparsity levels and random seeds. Our results show that there indeed exist sparse and robust subnetworks, which are competitive with the debiased full VLP and clearly outperform the debiasing SoTAs with fewer parameters on OOD datasets VQA-CP v2 and VQA-VS. The codes can be found at https://github.com/PhoebusSi/Compress-Robust-VQA.
RGB-T Semantic Segmentation with Location, Activation, and Sharpening
Authors: Gongyang Li, Yike Wang, Zhi Liu, Xinpeng Zhang, Dan Zeng
Date: 2022-10-26 00:00:00
Tasks: Decoder, Scene Understanding, Segmentation, Semantic Segmentation, Thermal Image Segmentation

Semantic segmentation is important for scene understanding. To address the scenes of adverse illumination conditions of natural images, thermal infrared (TIR) images are introduced. Most existing RGB-T semantic segmentation methods follow three cross-modal fusion paradigms, i.e. encoder fusion, decoder fusion, and feature fusion. Some methods, unfortunately, ignore the properties of RGB and TIR features or the properties of features at different levels. In this paper, we propose a novel feature fusion-based network for RGB-T semantic segmentation, named \emph{LASNet}, which follows three steps of location, activation, and sharpening. The highlight of LASNet is that we fully consider the characteristics of cross-modal features at different levels, and accordingly propose three specific modules for better segmentation. Concretely, we propose a Collaborative Location Module (CLM) for high-level semantic features, aiming to locate all potential objects. We propose a Complementary Activation Module for middle-level features, aiming to activate exact regions of different objects. We propose an Edge Sharpening Module (ESM) for low-level texture features, aiming to sharpen the edges of objects. Furthermore, in the training phase, we attach a location supervision and an edge supervision after CLM and ESM, respectively, and impose two semantic supervisions in the decoder part to facilitate network convergence. Experimental results on two public datasets demonstrate that the superiority of our LASNet over relevant state-of-the-art methods. The code and results of our method are available at https://github.com/MathLee/LASNet.
End-to-End Multimodal Representation Learning for Video Dialog
Authors: Huda Alamri, Anthony Bilic, Michael Hu, Apoorva Beedu, Irfan Essa
Date: 2022-10-26 00:00:00
Tasks: Representation Learning, Retrieval

Video-based dialog task is a challenging multimodal learning task that has received increasing attention over the past few years with state-of-the-art obtaining new performance records. This progress is largely powered by the adaptation of the more powerful transformer-based language encoders. Despite this progress, existing approaches do not effectively utilize visual features to help solve tasks. Recent studies show that state-of-the-art models are biased toward textual information rather than visual cues. In order to better leverage the available visual information, this study proposes a new framework that combines 3D-CNN network and transformer-based networks into a single visual encoder to extract more robust semantic representations from videos. The visual encoder is jointly trained end-to-end with other input modalities such as text and audio. Experiments on the AVSD task show significant improvement over baselines in both generative and retrieval tasks.
SimpleDG: Simple Domain Generalization Baseline without Bells and Whistles
Authors: Zhi Lv, Bo Lin, Siyuan Liang, Lihua Wang, Mochen Yu, Yao Tang, Jiajun Liang
Date: 2022-10-26 00:00:00
Tasks: Domain Generalization

We present a simple domain generalization baseline, which wins second place in both the common context generalization track and the hybrid context generalization track respectively in NICO CHALLENGE 2022. We verify the founding in recent literature, domainbed, that ERM is a strong baseline compared to recent state-of-the-art domain generalization methods and propose SimpleDG which includes several simple yet effective designs that further boost generalization performance. Code is available at https://github.com/megvii-research/SimpleDG
Reconstruction from edge image combined with color and gradient difference for industrial surface anomaly detection
Authors: Tongkun Liu, Bing Li, Zhuo Zhao, Xiao Du, Bingke Jiang, Leqi Geng
Date: 2022-10-26 00:00:00
Tasks: Anomaly Detection, Denoising

Reconstruction-based methods are widely explored in industrial visual anomaly detection. Such methods commonly require the model to well reconstruct the normal patterns but fail in the anomalies, and thus the anomalies can be detected by evaluating the reconstruction errors. However, in practice, it's usually difficult to control the generalization boundary of the model. The model with an overly strong generalization capability can even well reconstruct the abnormal regions, making them less distinguishable, while the model with a poor generalization capability can not reconstruct those changeable high-frequency components in the normal regions, which ultimately leads to false positives. To tackle the above issue, we propose a new reconstruction network where we reconstruct the original RGB image from its gray value edges (EdgRec). Specifically, this is achieved by an UNet-type denoising autoencoder with skip connections. The input edge and skip connections can well preserve the high-frequency information in the original image. Meanwhile, the proposed restoration task can force the network to memorize the normal low-frequency and color information. Besides, the denoising design can prevent the model from directly copying the original high-frequent components. To evaluate the anomalies, we further propose a new interpretable hand-crafted evaluation function that considers both the color and gradient differences. Our method achieves competitive results on the challenging benchmark MVTec AD (97.8\% for detection and 97.7\% for localization, AUROC). In addition, we conduct experiments on the MVTec 3D-AD dataset and show convincing results using RGB images only. Our code will be available at https://github.com/liutongkun/EdgRec.
TPFNet: A Novel Text In-painting Transformer for Text Removal
Authors: Onkar Susladkar, Dhruv Makwana, Gayatri Deshmukh, Sparsh Mittal, Sai Chandra Teja R, Rekha Singhal
Date: 2022-10-26 00:00:00
Tasks: Image Generation, Segmentation, Text Detection

Text erasure from an image is helpful for various tasks such as image editing and privacy preservation. In this paper, we present TPFNet, a novel one-stage (end-toend) network for text removal from images. Our network has two parts: feature synthesis and image generation. Since noise can be more effectively removed from low-resolution images, part 1 operates on low-resolution images. The output of part 1 is a low-resolution text-free image. Part 2 uses the features learned in part 1 to predict a high-resolution text-free image. In part 1, we use "pyramidal vision transformer" (PVT) as the encoder. Further, we use a novel multi-headed decoder that generates a high-pass filtered image and a segmentation map, in addition to a text-free image. The segmentation branch helps locate the text precisely, and the high-pass branch helps in learning the image structure. To precisely locate the text, TPFNet employs an adversarial loss that is conditional on the segmentation map rather than the input image. On Oxford, SCUT, and SCUT-EnsText datasets, our network outperforms recently proposed networks on nearly all the metrics. For example, on SCUT-EnsText dataset, TPFNet has a PSNR (higher is better) of 39.0 and text-detection precision (lower is better) of 21.1, compared to the best previous technique, which has a PSNR of 32.3 and precision of 53.2. The source code can be obtained from https://github.com/CandleLabAI/TPFNet
PredNAS: A Universal and Sample Efficient Neural Architecture Search Framework
Authors: Liuchun Yuan, Zehao Huang, Naiyan Wang
Date: 2022-10-26 00:00:00
Tasks: Neural Architecture Search

In this paper, we present a general and effective framework for Neural Architecture Search (NAS), named PredNAS. The motivation is that given a differentiable performance estimation function, we can directly optimize the architecture towards higher performance by simple gradient ascent. Specifically, we adopt a neural predictor as the performance predictor. Surprisingly, PredNAS can achieve state-of-the-art performances on NAS benchmarks with only a few training samples (less than 100). To validate the universality of our method, we also apply our method on large-scale tasks and compare our method with RegNet on ImageNet and YOLOX on MSCOCO. The results demonstrate that our PredNAS can explore novel architectures with competitive performances under specific computational complexity constraints.