pwc-archive/papers-with-abstracts huggingface dataset:
PoseTrans: A Simple Yet Effective Pose Transformation Augmentation for Human Pose Estimation
Authors: Wentao Jiang, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Si Liu
Date: 2022-08-16 00:00:00
Tasks: Data Augmentation, Diversity, Pose Estimation

Human pose estimation aims to accurately estimate a wide variety of human poses. However, existing datasets often follow a long-tailed distribution that unusual poses only occupy a small portion, which further leads to the lack of diversity of rare poses. These issues result in the inferior generalization ability of current pose estimators. In this paper, we present a simple yet effective data augmentation method, termed Pose Transformation (PoseTrans), to alleviate the aforementioned problems. Specifically, we propose Pose Transformation Module (PTM) to create new training samples that have diverse poses and adopt a pose discriminator to ensure the plausibility of the augmented poses. Besides, we propose Pose Clustering Module (PCM) to measure the pose rarity and select the "rarest" poses to help balance the long-tailed distribution. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our method, especially on rare poses. Also, our method is efficient and simple to implement, which can be easily integrated into the training pipeline of existing pose estimation models.
Introducing Intermediate Domains for Effective Self-Training during Test-Time
Authors: Robert A. Marsden, Mario Döbler, Bin Yang
Date: 2022-08-16 00:00:00
Tasks: Domain Adaptation, Scene Segmentation, Style Transfer, Test-time Adaptation, Unsupervised Domain Adaptation

Experiencing domain shifts during test-time is nearly inevitable in practice and likely results in a severe performance degradation. To overcome this issue, test-time adaptation continues to update the initial source model during deployment. A promising direction are methods based on self-training which have been shown to be well suited for gradual domain adaptation, since reliable pseudo-labels can be provided. In this work, we address two problems that exist when applying self-training in the setting of test-time adaptation. First, adapting a model to long test sequences that contain multiple domains can lead to error accumulation. Second, naturally, not all shifts are gradual in practice. To tackle these challenges, we introduce GTTA. By creating artificial intermediate domains that divide the current domain shift into a more gradual one, effective self-training through high quality pseudo-labels can be performed. To create the intermediate domains, we propose two independent variations: mixup and light-weight style transfer. We demonstrate the effectiveness of our approach on the continual and gradual corruption benchmarks, as well as ImageNet-R. To further investigate gradual shifts in the context of urban scene segmentation, we publish a new benchmark: CarlaTTA. It enables the exploration of several non-stationary domain shifts.
Rain Removal from Light Field Images with 4D Convolution and Multi-scale Gaussian Process
Authors: Tao Yan, Mingyue Li, Bin Li, Yang Yang, Rynson W. H. Lau
Date: 2022-08-16 00:00:00
Tasks: Depth Estimation, Rain Removal

Existing deraining methods focus mainly on a single input image. However, with just a single input image, it is extremely difficult to accurately detect and remove rain streaks, in order to restore a rain-free image. In contrast, a light field image (LFI) embeds abundant 3D structure and texture information of the target scene by recording the direction and position of each incident ray via a plenoptic camera. LFIs are becoming popular in the computer vision and graphics communities. However, making full use of the abundant information available from LFIs, such as 2D array of sub-views and the disparity map of each sub-view, for effective rain removal is still a challenging problem. In this paper, we propose a novel method, 4D-MGP-SRRNet, for rain streak removal from LFIs. Our method takes as input all sub-views of a rainy LFI. To make full use of the LFI, it adopts 4D convolutional layers to simultaneously process all sub-views of the LFI. In the pipeline, the rain detection network, MGPDNet, with a novel Multi-scale Self-guided Gaussian Process (MSGP) module is proposed to detect high-resolution rain streaks from all sub-views of the input LFI at multi-scales. Semi-supervised learning is introduced for MSGP to accurately detect rain streaks by training on both virtual-world rainy LFIs and real-world rainy LFIs at multi-scales via computing pseudo ground truths for real-world rain streaks. We then feed all sub-views subtracting the predicted rain streaks into a 4D convolution-based Depth Estimation Residual Network (DERNet) to estimate the depth maps, which are later converted into fog maps. Finally, all sub-views concatenated with the corresponding rain streaks and fog maps are fed into a powerful rainy LFI restoring model based on the adversarial recurrent neural network to progressively eliminate rain streaks and recover the rain-free LFI.
Local Low-light Image Enhancement via Region-Aware Normalization
Authors: Shihurong Yao, Yizhan Huang, Xiaogang Xu
Date: 2022-08-16 00:00:00
Tasks: Image Enhancement, Low-Light Image Enhancement

In the realm of Low-Light Image Enhancement (LLIE), existing research primarily focuses on enhancing images globally. However, many applications require local LLIE, where users are allowed to illuminate specific regions using an input mask, such as creating a protagonist stage or spotlight effect. However, this task has received limited attention currently. This paper aims to systematically define the requirements of local LLIE and proposes a novel strategy to convert current existing global LLIE methods into local versions. The image space is divided into three regions: Masked Area A be enlightened to achieve the desired lighting effects; Transition Area B is a smooth transition from the enlightened area (Area A) to the unchanged region (Area C). To achieve the task of local LLIE, we introduce Region-Aware Normalization for Local Enhancement, dubbed as RANLEN. RANLEN uses a dynamically designed mask-based normalization operation, which enhances an image in a spatially varying manner, ensuring that the enhancement results are consistent with the requirements specified by the input mask. Additionally, a set of region-aware loss terms is formulated to facilitate the learning of the local LLIE framework. Our strategy can be applied to existing global LLIE networks with varying structures. Extensive experiments demonstrate that our approach can produce the desired lighting effects compared to global LLIE, all the while offering controllable local enhancement with various mask shapes.
The LAM Dataset: A Novel Benchmark for Line-Level Handwritten Text Recognition
Authors: Silvia Cascianelli, Vittorio Pippi, Martin Maarand, Marcella Cornia, Lorenzo Baraldi, Christopher Kermorvant, Rita Cucchiara
Date: 2022-08-16 00:00:00
Tasks: Handwritten Text Recognition, HTR

Handwritten Text Recognition (HTR) is an open problem at the intersection of Computer Vision and Natural Language Processing. The main challenges, when dealing with historical manuscripts, are due to the preservation of the paper support, the variability of the handwriting -- even of the same author over a wide time-span -- and the scarcity of data from ancient, poorly represented languages. With the aim of fostering the research on this topic, in this paper we present the Ludovico Antonio Muratori (LAM) dataset, a large line-level HTR dataset of Italian ancient manuscripts edited by a single author over 60 years. The dataset comes in two configurations: a basic splitting and a date-based splitting which takes into account the age of the author. The first setting is intended to study HTR on ancient documents in Italian, while the second focuses on the ability of HTR systems to recognize text written by the same writer in time periods for which training data are not available. For both configurations, we analyze quantitative and qualitative characteristics, also with respect to other line-level HTR benchmarks, and present the recognition performance of state-of-the-art HTR architectures. The dataset is available for download at \url{https://aimagelab.ing.unimore.it/go/lam}.
FEC: Fast Euclidean Clustering for Point Cloud Segmentation
Authors: Yu Cao, Yancheng Wang, Yifei Xue, Huiqing Zhang, Yizhen Lao
Date: 2022-08-16 00:00:00
Tasks: Clustering, Instance Segmentation, Point Cloud Segmentation, Segmentation, Semantic Segmentation

Segmentation from point cloud data is essential in many applications such as remote sensing, mobile robots, or autonomous cars. However, the point clouds captured by the 3D range sensor are commonly sparse and unstructured, challenging efficient segmentation. In this paper, we present a fast solution to point cloud instance segmentation with small computational demands. To this end, we propose a novel fast Euclidean clustering (FEC) algorithm which applies a pointwise scheme over the clusterwise scheme used in existing works. Our approach is conceptually simple, easy to implement (40 lines in C++), and achieves two orders of magnitudes faster against the classical segmentation methods while producing high-quality results.
Matching Multiple Perspectives for Efficient Representation Learning
Authors: Omiros Pantazis, Mathew Salvaris
Date: 2022-08-16 00:00:00
Tasks: Object, Representation Learning, Self-Supervised Learning

Representation learning approaches typically rely on images of objects captured from a single perspective that are transformed using affine transformations. Additionally, self-supervised learning, a successful paradigm of representation learning, relies on instance discrimination and self-augmentations which cannot always bridge the gap between observations of the same object viewed from a different perspective. Viewing an object from multiple perspectives aids holistic understanding of an object which is particularly important in situations where data annotations are limited. In this paper, we present an approach that combines self-supervised learning with a multi-perspective matching technique and demonstrate its effectiveness on learning higher quality representations on data captured by a robotic vacuum with an embedded camera. We show that the availability of multiple views of the same object combined with a variety of self-supervised pretraining algorithms can lead to improved object classification performance without extra labels.
Does lossy image compression affect racial bias within face recognition?
Authors: Seyma Yucer, Matt Poyser, Noura Al Moubayed, Toby P. Breckon
Date: 2022-08-16 00:00:00
Tasks: Face Recognition, Image Compression

Yes - This study investigates the impact of commonplace lossy image compression on face recognition algorithms with regard to the racial characteristics of the subject. We adopt a recently proposed racial phenotype-based bias analysis methodology to measure the effect of varying levels of lossy compression across racial phenotype categories. Additionally, we determine the relationship between chroma-subsampling and race-related phenotypes for recognition performance. Prior work investigates the impact of lossy JPEG compression algorithm on contemporary face recognition performance. However, there is a gap in how this impact varies with different race-related inter-sectional groups and the cause of this impact. Via an extensive experimental setup, we demonstrate that common lossy image compression approaches have a more pronounced negative impact on facial recognition performance for specific racial phenotype categories such as darker skin tones (by up to 34.55\%). Furthermore, removing chroma-subsampling during compression improves the false matching rate (up to 15.95\%) across all phenotype categories affected by the compression, including darker skin tones, wide noses, big lips, and monolid eye categories. In addition, we outline the characteristics that may be attributable as the underlying cause of such phenomenon for lossy compression algorithms such as JPEG.
Uncertainty-guided Source-free Domain Adaptation
Authors: Subhankar Roy, Martin Trapp, Andrea Pilzer, Juho Kannala, Nicu Sebe, Elisa Ricci, Arno Solin
Date: 2022-08-16 00:00:00
Tasks: Domain Adaptation, Source-Free Domain Adaptation

Source-free domain adaptation (SFDA) aims to adapt a classifier to an unlabelled target data set by only using a pre-trained source model. However, the absence of the source data and the domain shift makes the predictions on the target data unreliable. We propose quantifying the uncertainty in the source model predictions and utilizing it to guide the target adaptation. For this, we construct a probabilistic source model by incorporating priors on the network parameters inducing a distribution over the model predictions. Uncertainties are estimated by employing a Laplace approximation and incorporated to identify target data points that do not lie in the source manifold and to down-weight them when maximizing the mutual information on the target data. Unlike recent works, our probabilistic treatment is computationally lightweight, decouples source training and target adaptation, and requires no specialized source training or changes of the model architecture. We show the advantages of uncertainty-guided SFDA over traditional SFDA in the closed-set and open-set settings and provide empirical evidence that our approach is more robust to strong domain shifts even without tuning.
Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis
Authors: Licai Sun, Zheng Lian, Bin Liu, JianHua Tao
Date: 2022-08-16 00:00:00
Tasks: Multimodal Sentiment Analysis, Representation Learning, Sentiment Analysis

With the proliferation of user-generated online videos, Multimodal Sentiment Analysis (MSA) has attracted increasing attention recently. Despite significant progress, there are still two major challenges on the way towards robust MSA: 1) inefficiency when modeling cross-modal interactions in unaligned multimodal data; and 2) vulnerability to random modality feature missing which typically occurs in realistic settings. In this paper, we propose a generic and unified framework to address them, named Efficient Multimodal Transformer with Dual-Level Feature Restoration (EMT-DLFR). Concretely, EMT employs utterance-level representations from each modality as the global multimodal context to interact with local unimodal features and mutually promote each other. It not only avoids the quadratic scaling cost of previous local-local cross-modal interaction methods but also leads to better performance. To improve model robustness in the incomplete modality setting, on the one hand, DLFR performs low-level feature reconstruction to implicitly encourage the model to learn semantic information from incomplete data. On the other hand, it innovatively regards complete and incomplete data as two different views of one sample and utilizes siamese representation learning to explicitly attract their high-level representations. Comprehensive experiments on three popular datasets demonstrate that our method achieves superior performance in both complete and incomplete modality settings.
Neural network fragile watermarking with no model performance degradation
Authors: Zhaoxia Yin, Heng Yin, Xinpeng Zhang
Date: 2022-08-16 00:00:00
Tasks: Data Poisoning

Deep neural networks are vulnerable to malicious fine-tuning attacks such as data poisoning and backdoor attacks. Therefore, in recent research, it is proposed how to detect malicious fine-tuning of neural network models. However, it usually negatively affects the performance of the protected model. Thus, we propose a novel neural network fragile watermarking with no model performance degradation. In the process of watermarking, we train a generative model with the specific loss function and secret key to generate triggers that are sensitive to the fine-tuning of the target classifier. In the process of verifying, we adopt the watermarked classifier to get labels of each fragile trigger. Then, malicious fine-tuning can be detected by comparing secret keys and labels. Experiments on classic datasets and classifiers show that the proposed method can effectively detect model malicious fine-tuning with no model performance degradation.
HVS-Inspired Signal Degradation Network for Just Noticeable Difference Estimation
Authors: Jian Jin, Yuan Xue, Xingxing Zhang, Lili Meng, Yao Zhao, Weisi Lin
Date: 2022-08-16 00:00:00

Significant improvement has been made on just noticeable difference (JND) modelling due to the development of deep neural networks, especially for the recently developed unsupervised-JND generation models. However, they have a major drawback that the generated JND is assessed in the real-world signal domain instead of in the perceptual domain in the human brain. There is an obvious difference when JND is assessed in such two domains since the visual signal in the real world is encoded before it is delivered into the brain with the human visual system (HVS). Hence, we propose an HVS-inspired signal degradation network for JND estimation. To achieve this, we carefully analyze the HVS perceptual process in JND subjective viewing to obtain relevant insights, and then design an HVS-inspired signal degradation (HVS-SD) network to represent the signal degradation in the HVS. On the one hand, the well learnt HVS-SD enables us to assess the JND in the perceptual domain. On the other hand, it provides more accurate prior information for better guiding JND generation. Additionally, considering the requirement that reasonable JND should not lead to visual attention shifting, a visual attention loss is proposed to control JND generation. Experimental results demonstrate that the proposed method achieves the SOTA performance for accurately estimating the redundancy of the HVS. Source code will be available at https://github.com/jianjin008/HVS-SD-JND.
Object Discovery via Contrastive Learning for Weakly Supervised Object Detection
Authors: Jinhwan Seo, Wonho Bae, Danica J. Sutherland, Junhyug Noh, Daijin Kim
Date: 2022-08-16 00:00:00
Tasks: Contrastive Learning, Object, Object Detection, Object Discovery, Weakly Supervised Object Detection

Weakly Supervised Object Detection (WSOD) is a task that detects objects in an image using a model trained only on image-level annotations. Current state-of-the-art models benefit from self-supervised instance-level supervision, but since weak supervision does not include count or location information, the most common ``argmax'' labeling method often ignores many instances of objects. To alleviate this issue, we propose a novel multiple instance labeling method called object discovery. We further introduce a new contrastive loss under weak supervision where no instance-level information is available for sampling, called weakly supervised contrastive loss (WSCL). WSCL aims to construct a credible similarity threshold for object discovery by leveraging consistent features for embedding vectors in the same class. As a result, we achieve new state-of-the-art results on MS-COCO 2014 and 2017 as well as PASCAL VOC 2012, and competitive results on PASCAL VOC 2007.
Prediction of Seismic Intensity Distributions Using Neural Networks
Authors: Koyu Mizutani, Haruki Mitarai, Kakeru Miyazaki, Ryugo Shimamura, Soichiro Kumano, Toshihiko Yamasaki
Date: 2022-08-16 00:00:00
Tasks: motion prediction, Prediction, regression

The ground motion prediction equation is commonly used to predict the seismic intensity distribution. However, it is not easy to apply this method to seismic distributions affected by underground plate structures, which are commonly known as abnormal seismic distributions. This study proposes a hybrid of regression and classification approaches using neural networks. The proposed model treats the distributions as 2-dimensional data like an image. Our method can accurately predict seismic intensity distributions, even abnormal distributions.
Self-supervised training of deep denoisers in multi-coil MRI considering noise correlations
Authors: Juhyung Park, Dongwon Park, Sooyeon Ji, Hyeong-Geol Shin, Se Young Chun, Jongho Lee
Date: 2022-08-16 00:00:00
Tasks: Denoising, Image Denoising

Deep learning-based denoising methods have shown powerful results for improving the signal-to-noise ratio of magnetic resonance (MR) images, mostly by leveraging supervised learning with clean ground truth. However, acquiring clean ground truth images is often expensive and time-consuming. Self supervised methods have been widely investigated to mitigate the dependency on clean images, but mostly rely on the suboptimal splitting of K-space measurements of an image to yield input and target images for ensuring statistical independence. In this study, we investigate an alternative self-supervised training method for deep denoisers in multi-coil MRI, dubbed Coil2Coil (C2C), that naturally split and combine the multi-coil data among phased array coils, generating two noise-corrupted images for training. This novel approach allows exploiting multi-coil redundancy, but the images are statistically correlated and may not have the same clean image. To mitigate these issues, we propose the methods to pproximately decorrelate the statistical dependence of these images and match the underlying clean images, thus enabling them to be used as the training pairs. For synthetic denoising experiments, C2C yielded the best performance against prior self-supervised methods, reporting outcome comparable even to supervised methods. For real-world denoising cases, C2C yielded consistent performance as synthetic cases, removing only noise structures.
Color Image Edge Detection using Multi-scale and Multi-directional Gabor filter
Authors: Yunhong Li, Yuandong Bi, Weichuan Zhang, Jie Ren, Jinni Chen
Date: 2022-08-16 00:00:00
Tasks: Edge Detection

In this paper, a color edge detection method is proposed where the multi-scale Gabor filter are used to obtain edges from input color images. The main advantage of the proposed method is that high edge detection accuracy is attained while maintaining good noise robustness. The proposed method consists of three aspects: First, the RGB color image is converted to CIE L*a*b* space because of its wide coloring area and uniform color distribution. Second, a set of Gabor filters are used to smooth the input images and the color edge strength maps are extracted, which are fused into a new ESM with the noise robustness and accurate edge extraction. Third, Embedding the fused ESM in the route of the Canny detector yields a noise-robust color edge detector. The results show that the proposed detector has the better experience in detection accuracy and noise-robustness.
SGM-Net: Semantic Guided Matting Net
Authors: Qing Song, Wenfeng Sun, Donghan Yang, Mengjie Hu, Chun Liu
Date: 2022-08-16 00:00:00
Tasks: Image Generation, Image Matting

Human matting refers to extracting human parts from natural images with high quality, including human detail information such as hair, glasses, hat, etc. This technology plays an essential role in image synthesis and visual effects in the film industry. When the green screen is not available, the existing human matting methods need the help of additional inputs (such as trimap, background image, etc.), or the model with high computational cost and complex network structure, which brings great difficulties to the application of human matting in practice. To alleviate such problems, most existing methods (such as MODNet) use multi-branches to pave the way for matting through segmentation, but these methods do not make full use of the image features and only utilize the prediction results of the network as guidance information. Therefore, we propose a module to generate foreground probability map and add it to MODNet to obtain Semantic Guided Matting Net (SGM-Net). Under the condition of only one image, we can realize the human matting task. We verify our method on the P3M-10k dataset. Compared with the benchmark, our method has significantly improved in various evaluation indicators.