pwc-archive/papers-with-abstracts huggingface dataset:
Saliency Guided Contrastive Learning on Scene Images
Authors: Meilin Chen, Yizhou Wang, Shixiang Tang, Feng Zhu, Haiyang Yang, Lei Bai, Rui Zhao, Donglian Qi, Wanli Ouyang
Date: 2023-02-22 00:00:00
Tasks: Contrastive Learning, Linear evaluation, Representation Learning, Self-Supervised Learning

Self-supervised learning holds promise in leveraging large numbers of unlabeled data. However, its success heavily relies on the highly-curated dataset, e.g., ImageNet, which still needs human cleaning. Directly learning representations from less-curated scene images is essential for pushing self-supervised learning to a higher level. Different from curated images which include simple and clear semantic information, scene images are more complex and mosaic because they often include complex scenes and multiple objects. Despite being feasible, recent works largely overlooked discovering the most discriminative regions for contrastive learning to object representations in scene images. In this work, we leverage the saliency map derived from the model's output during learning to highlight these discriminative regions and guide the whole contrastive learning. Specifically, the saliency map first guides the method to crop its discriminative regions as positive pairs and then reweighs the contrastive losses among different crops by its saliency scores. Our method significantly improves the performance of self-supervised learning on scene images by +1.1, +4.3, +2.2 Top1 accuracy in ImageNet linear evaluation, Semi-supervised learning with 1% and 10% ImageNet labels, respectively. We hope our insights on saliency maps can motivate future research on more general-purpose unsupervised representation learning from scene data.
Fusing Visual Appearance and Geometry for Multi-modality 6DoF Object Tracking
Authors: Manuel Stoiber, Mariam Elsayed, Anne E. Reichert, Florian Steidle, Dongheui Lee, Rudolph Triebel
Date: 2023-02-22 00:00:00
Tasks: 3D Object Tracking, 6D Pose Estimation, 6D Pose Estimation using RGBD, Object, Object Tracking

In many applications of advanced robotic manipulation, six degrees of freedom (6DoF) object pose estimates are continuously required. In this work, we develop a multi-modality tracker that fuses information from visual appearance and geometry to estimate object poses. The algorithm extends our previous method ICG, which uses geometry, to additionally consider surface appearance. In general, object surfaces contain local characteristics from text, graphics, and patterns, as well as global differences from distinct materials and colors. To incorporate this visual information, two modalities are developed. For local characteristics, keypoint features are used to minimize distances between points from keyframes and the current image. For global differences, a novel region approach is developed that considers multiple regions on the object surface. In addition, it allows the modeling of external geometries. Experiments on the YCB-Video and OPT datasets demonstrate that our approach ICG+ performs best on both datasets, outperforming both conventional and deep learning-based methods. At the same time, the algorithm is highly efficient and runs at more than 300 Hz. The source code of our tracker is publicly available.
Singular value decomposition based matrix surgery
Authors: Jehan Ghafuri, Sabah Jassim
Date: 2023-02-22 00:00:00

This paper aims to develop a simple procedure to reduce and control the condition number of random matrices, and investigate the effect on the persistent homology (PH) of point clouds of well- and ill-conditioned matrices. For a square matrix generated randomly using Gaussian/Uniform distribution, the SVD-Surgery procedure works by: (1) computing its singular value decomposition (SVD), (2) replacing the diagonal factor by changing a list of the smaller singular values by a convex linear combination of the entries in the list, and (3) compute the new matrix by reversing the SVD. Applying SVD-Surgery on a matrix often results in having different diagonal factor to those of the input matrix. The spatial distribution of random square matrices are known to be correlated to the distribution of their condition numbers. The persistent homology (PH) investigations, therefore, are focused on comparing the effect of SVD-Surgery on point clouds of large datasets of randomly generated well-conditioned and ill-conditioned matrices, as well as that of the point clouds formed by their inverses. This work is motivated by the desire to stabilise the impact of Deep Learning (DL) training on medical images in terms of the condition numbers of their sets of convolution filters as a mean of reducing overfitting and improving robustness against tolerable amounts of image noise. When applied to convolution filters during training, the SVD-Surgery acts as a spectral regularisation of the DL model without the need for learning extra parameters. We shall demonstrate that for several point clouds of sufficiently large convolution filters our simple strategy preserve filters norm and reduces the norm of its inverse depending on the chosen linear combination parameters. Moreover, our approach showed significant improvements towards the well-conditioning of matrices and stable topological behaviour.
Enhanced Face Authentication With Separate Loss Functions
Authors: Anh-Kiet Duong, Hoang-Lan Nguyen, Toan-Thinh Truong
Date: 2023-02-22 00:00:00
Tasks: Face Detection, Face Recognition

The overall objective of the main project is to propose and develop a system of facial authentication in unlocking phones or applications in phones using facial recognition. The system will include four separate architectures: face detection, face recognition, face spoofing, and classification of closed eyes. In which, we consider the problem of face recognition to be the most important, determining the true identity of the person standing in front of the screen with absolute accuracy is what facial recognition systems need to achieve. Along with the development of the face recognition problem, the problem of the anti-fake face is also gradually becoming popular and equally important. Our goal is to propose and develop two loss functions: LMCot and Double Loss. Then apply them to the face authentication process.
Structure Embedded Nucleus Classification for Histopathology Images
Authors: Wei Lou, Xiang Wan, Guanbin Li, Xiaoying Lou, Chenghang Li, Feng Gao, Haofeng Li
Date: 2023-02-22 00:00:00
Tasks: Classification, Graph Neural Network, Graph structure learning, Nuclei Classification

Nuclei classification provides valuable information for histopathology image analysis. However, the large variations in the appearance of different nuclei types cause difficulties in identifying nuclei. Most neural network based methods are affected by the local receptive field of convolutions, and pay less attention to the spatial distribution of nuclei or the irregular contour shape of a nucleus. In this paper, we first propose a novel polygon-structure feature learning mechanism that transforms a nucleus contour into a sequence of points sampled in order, and employ a recurrent neural network that aggregates the sequential change in distance between key points to obtain learnable shape features. Next, we convert a histopathology image into a graph structure with nuclei as nodes, and build a graph neural network to embed the spatial distribution of nuclei into their representations. To capture the correlations between the categories of nuclei and their surrounding tissue patterns, we further introduce edge features that are defined as the background textures between adjacent nuclei. Lastly, we integrate both polygon and graph structure learning mechanisms into a whole framework that can extract intra and inter-nucleus structural characteristics for nuclei classification. Experimental results show that the proposed framework achieves significant improvements compared to the state-of-the-art methods.
Gradient Adjusting Networks for Domain Inversion
Authors: Erez Sheffi, Michael Rotman, Lior Wolf
Date: 2023-02-22 00:00:00
Tasks: Image Generation

StyleGAN2 was demonstrated to be a powerful image generation engine that supports semantic editing. However, in order to manipulate a real-world image, one first needs to be able to retrieve its corresponding latent representation in StyleGAN's latent space that is decoded to an image as close as possible to the desired image. For many real-world images, a latent representation does not exist, which necessitates the tuning of the generator network. We present a per-image optimization method that tunes a StyleGAN2 generator such that it achieves a local edit to the generator's weights, resulting in almost perfect inversion, while still allowing image editing, by keeping the rest of the mapping between an input latent representation tensor and an output image relatively intact. The method is based on a one-shot training of a set of shallow update networks (aka. Gradient Modification Modules) that modify the layers of the generator. After training the Gradient Modification Modules, a modified generator is obtained by a single application of these networks to the original parameters, and the previous editing capabilities of the generator are maintained. Our experiments show a sizable gap in performance over the current state of the art in this very active domain. Our code is available at \url{https://github.com/sheffier/gani}.
Entity-Level Text-Guided Image Manipulation
Authors: Yikai Wang, Jianan Wang, Guansong Lu, Hang Xu, Zhenguo Li, Wei zhang, Yanwei Fu
Date: 2023-02-22 00:00:00
Tasks: Denoising, Image Manipulation

Existing text-guided image manipulation methods aim to modify the appearance of the image or to edit a few objects in a virtual or simple scenario, which is far from practical applications. In this work, we study a novel task on text-guided image manipulation on the entity level in the real world (eL-TGIM). The task imposes three basic requirements, (1) to edit the entity consistent with the text descriptions, (2) to preserve the entity-irrelevant regions, and (3) to merge the manipulated entity into the image naturally. To this end, we propose an elegant framework, dubbed as SeMani, forming the Semantic Manipulation of real-world images that can not only edit the appearance of entities but also generate new entities corresponding to the text guidance. To solve eL-TGIM, SeMani decomposes the task into two phases: the semantic alignment phase and the image manipulation phase. In the semantic alignment phase, SeMani incorporates a semantic alignment module to locate the entity-relevant region to be manipulated. In the image manipulation phase, SeMani adopts a generative model to synthesize new images conditioned on the entity-irrelevant regions and target text descriptions. We discuss and propose two popular generation processes that can be utilized in SeMani, the discrete auto-regressive generation with transformers and the continuous denoising generation with diffusion models, yielding SeMani-Trans and SeMani-Diff, respectively. We conduct extensive experiments on the real datasets CUB, Oxford, and COCO datasets to verify that SeMani can distinguish the entity-relevant and -irrelevant regions and achieve more precise and flexible manipulation in a zero-shot manner compared with baseline methods. Our codes and models will be released at https://github.com/Yikai-Wang/SeMani.
Vision-Based Estimation of Small Body Rotational State during the Approach Phase
Authors: Paolo Panicucci, Jérémy Lebreton, Roland Brochard, Emmanuel Zenou, Michel Delpech
Date: 2023-02-22 00:00:00

The heterogeneity of the small body population complicates the prediction of small body properties before the spacecraft's arrival. In the context of autonomous small body exploration, it is crucial to develop algorithms that estimate the small body characteristics before orbit insertion and close proximity operations. This paper develops a vision-based estimation of the small-body rotational state (i.e., the center of rotation and rotation axis direction) during the approach phase. In this mission phase, the spacecraft observes the rotating celestial body and tracks features in images. As feature tracks are the projection of the landmarks' circular movement, the possible rotation axes are computed. Then, the rotation axis solution is chosen among the possible candidates by exploiting feature motion and a heuristic approach. Finally, the center of rotation is estimated from the center of brightness. The algorithm is tested on more than 800 test cases with two different asteroids (i.e., Bennu and Itokawa), three different lighting conditions, and more than 100 different rotation axis orientations. Each test case is composed of about 250 synthetic images of the asteroid which are used to track features and determine the rotational state. Results show that the error between the true rotation axis and its estimation is below $10^{\circ}$ for $80\%$ of the considered test cases, implying that the proposed algorithm is a suitable method for autonomous small body characterization.
HDR image watermarking using saliency detection and quantization index modulation
Authors: Ahmed Khan, Minoru Kuribayashi, KokSheik Wong, Vishnu Monn Baskaran
Date: 2023-02-22 00:00:00
Tasks: object-detection, Object Detection, Quantization, Saliency Detection, Salient Object Detection

High-dynamic range (HDR) images are circulated rapidly over the internet with risks of being exploited for unauthorized usage. To protect these images, some HDR image based watermarking (HDR-IW) methods were put forward. However, they inherited the same problem faced by conventional IW methods for standard dynamic range (SDR) images, where only trade-offs among conflicting requirements are managed instead of simultaneous improvement. In this paper, a novel saliency (eye-catching object) detection based trade-off independent HDR-IW is proposed, to simultaneously improve robustness, imperceptibility and payload. First, the host image goes through our proposed salient object detection model to produce a saliency map, which is, in turn, exploited to segment the foreground and background of the host image. Next, the binary watermark is partitioned into the foregrounds and backgrounds using the same mask and scrambled using a random permutation algorithm. Finally, the watermark segments are embedded into selected bit-plane of the corresponding host segments using quantized indexed modulation. Experimental results suggest that the proposed work outperforms state-of-the-art methods in terms of improving the conflicting requirements.
Poisson Conjugate Prior for PHD Filtering based Track-Before-Detect Strategies in Radar Systems
Authors: Haiyi Mao, Cong Peng, Yue Liu, Jinping Tang, Hua Peng, Wei Yi
Date: 2023-02-22 00:00:00

A variety of filters with track-before-detect (TBD) strategies have been developed and applied to low signal-to-noise ratio (SNR) scenarios, including the probability hypothesis density (PHD) filter. Assumptions of the standard point measurement model based on detect-before-track (DBT) strategies are not suitable for the amplitude echo model based on TBD strategies. However, based on different models and unmatched assumptions, the measurement update formulas for DBT-PHD filter are just mechanically applied to existing TBD-PHD filters. In this paper, based on the Kullback-Leibler divergence minimization criterion, finite set statistics theory and rigorous Bayes rule, a principled closed-form solution of TBD-PHD filter is derived. Furthermore, we emphasize that PHD filter is conjugated to the Poisson prior based on TBD strategies. Next, a capping operation is devised to handle the divergence of target number estimation as SNR increases. Moreover, the sequential Monte Carlo implementations of dynamic and amplitude echo models are proposed for the radar system. Finally, Monte Carlo experiments exhibit good performance in Rayleigh noise and low SNR scenarios.
X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentation
Authors: Tom van Sonsbeek, Marcel Worring
Date: 2023-02-22 00:00:00
Tasks: Cross-Modal Retrieval, Retrieval

An important component of human analysis of medical images and their context is the ability to relate newly seen things to related instances in our memory. In this paper we mimic this ability by using multi-modal retrieval augmentation and apply it to several tasks in chest X-ray analysis. By retrieving similar images and/or radiology reports we expand and regularize the case at hand with additional knowledge, while maintaining factual knowledge consistency. The method consists of two components. First, vision and language modalities are aligned using a pre-trained CLIP model. To enforce that the retrieval focus will be on detailed disease-related content instead of global visual appearance it is fine-tuned using disease class information. Subsequently, we construct a non-parametric retrieval index, which reaches state-of-the-art retrieval levels. We use this index in our downstream tasks to augment image representations through multi-head attention for disease classification and report retrieval. We show that retrieval augmentation gives considerable improvements on these tasks. Our downstream report retrieval even shows to be competitive with dedicated report generation methods, paving the path for this method in medical imaging.
Steerable Equivariant Representation Learning
Authors: Sangnie Bhardwaj, Willie McClinton, Tongzhou Wang, Guillaume Lajoie, Chen Sun, Phillip Isola, Dilip Krishnan
Date: 2023-02-22 00:00:00
Tasks: Image Retrieval, object-detection, Object Detection, Out-of-Distribution Detection, Representation Learning, Retrieval, Transfer Learning

Pre-trained deep image representations are useful for post-training tasks such as classification through transfer learning, image retrieval, and object detection. Data augmentations are a crucial aspect of pre-training robust representations in both supervised and self-supervised settings. Data augmentations explicitly or implicitly promote invariance in the embedding space to the input image transformations. This invariance reduces generalization to those downstream tasks which rely on sensitivity to these particular data augmentations. In this paper, we propose a method of learning representations that are instead equivariant to data augmentations. We achieve this equivariance through the use of steerable representations. Our representations can be manipulated directly in embedding space via learned linear maps. We demonstrate that our resulting steerable and equivariant representations lead to better performance on transfer learning and robustness: e.g. we improve linear probe top-1 accuracy by between 1% to 3% for transfer; and ImageNet-C accuracy by upto 3.4%. We further show that the steerability of our representations provides significant speedup (nearly 50x) for test-time augmentations; by applying a large number of augmentations for out-of-distribution detection, we significantly improve OOD AUC on the ImageNet-C dataset over an invariant representation.
A Gradient Boosting Approach for Training Convolutional and Deep Neural Networks
Authors: Seyedsaman Emami, Gonzalo Martínez-Muñoz
Date: 2023-02-22 00:00:00
Tasks: Classification, image-classification, Image Classification

Deep learning has revolutionized the computer vision and image classification domains. In this context Convolutional Neural Networks (CNNs) based architectures are the most widely applied models. In this article, we introduced two procedures for training Convolutional Neural Networks (CNNs) and Deep Neural Network based on Gradient Boosting (GB), namely GB-CNN and GB-DNN. These models are trained to fit the gradient of the loss function or pseudo-residuals of previous models. At each iteration, the proposed method adds one dense layer to an exact copy of the previous deep NN model. The weights of the dense layers trained on previous iterations are frozen to prevent over-fitting, permitting the model to fit the new dense as well as to fine-tune the convolutional layers (for GB-CNN) while still utilizing the information already learned. Through extensive experimentation on different 2D-image classification and tabular datasets, the presented models show superior performance in terms of classification accuracy with respect to standard CNN and Deep-NN with the same architectures.
Human MotionFormer: Transferring Human Motions with Vision Transformers
Authors: Hongyu Liu, Xintong Han, ChengBin Jin, Lihui Qian, Huawei Wei, Zhe Lin, Faqiang Wang, Haoye Dong, Yibing Song, Jia Xu, Qifeng Chen
Date: 2023-02-22 00:00:00
Tasks: Decoder, Motion Synthesis

Human motion transfer aims to transfer motions from a target dynamic person to a source static one for motion synthesis. An accurate matching between the source person and the target motion in both large and subtle motion changes is vital for improving the transferred motion quality. In this paper, we propose Human MotionFormer, a hierarchical ViT framework that leverages global and local perceptions to capture large and subtle motion matching, respectively. It consists of two ViT encoders to extract input features (i.e., a target motion image and a source human image) and a ViT decoder with several cascaded blocks for feature matching and motion transfer. In each block, we set the target motion feature as Query and the source person as Key and Value, calculating the cross-attention maps to conduct a global feature matching. Further, we introduce a convolutional layer to improve the local perception after the global cross-attention computations. This matching process is implemented in both warping and generation branches to guide the motion transfer. During training, we propose a mutual learning loss to enable the co-supervision between warping and generation branches for better motion representations. Experiments show that our Human MotionFormer sets the new state-of-the-art performance both qualitatively and quantitatively. Project page: \url{https://github.com/KumapowerLIU/Human-MotionFormer}
View Consistency Aware Holistic Triangulation for 3D Human Pose Estimation
Authors: Xiaoyue Wan, Zhuo Chen, Xu Zhao
Date: 2023-02-22 00:00:00
Tasks: 3D Human Pose Estimation, 3D Reconstruction, Anatomy, Multi-view 3D Human Pose Estimation, Pose Estimation

The rapid development of multi-view 3D human pose estimation (HPE) is attributed to the maturation of monocular 2D HPE and the geometry of 3D reconstruction. However, 2D detection outliers in occluded views due to neglect of view consistency, and 3D implausible poses due to lack of pose coherence, remain challenges. To solve this, we introduce a Multi-View Fusion module to refine 2D results by establishing view correlations. Then, Holistic Triangulation is proposed to infer the whole pose as an entirety, and anatomy prior is injected to maintain the pose coherence and improve the plausibility. Anatomy prior is extracted by PCA whose input is skeletal structure features, which can factor out global context and joint-by-joint relationship from abstract to concrete. Benefiting from the closed-form solution, the whole framework is trained end-to-end. Our method outperforms the state of the art in both precision and plausibility which is assessed by a new metric.