pwc-archive/papers-with-abstracts huggingface dataset:
Unsupervised domain-adaptive person re-identification with multi-camera constraints
Authors: S. Takeuchi, F. Li, S. Iwasaki, J. Ning, G. Suzuki
Date: 2022-10-25 00:00:00
Tasks: Domain Adaptive Person Re-Identification, Person Re-Identification

Person re-identification is a key technology for analyzing video-based human behavior; however, its application is still challenging in practical situations due to the performance degradation for domains different from those in the training data. Here, we propose an environment-constrained adaptive network for reducing the domain gap. This network refines pseudo-labels estimated via a self-training scheme by imposing multi-camera constraints. The proposed method incorporates person-pair information without person identity labels obtained from the environment into the model training. In addition, we develop a method that appropriately selects a person from the pair that contributes to the performance improvement. We evaluate the performance of the network using public and private datasets and confirm the performance surpasses state-of-the-art methods in domains with overlapping camera views. To the best of our knowledge, this is the first study on domain-adaptive learning with multi-camera constraints that can be obtained in real environments.
A Comparative Study on Deep-Learning Methods for Dense Image Matching of Multi-angle and Multi-date Remote Sensing Stereo Images
Authors: Hessah Albanwan, Rongjun Qin
Date: 2022-10-25 00:00:00
Tasks: Stereo Matching

Deep learning (DL) stereo matching methods gained great attention in remote sensing satellite datasets. However, most of these existing studies conclude assessments based only on a few/single stereo images lacking a systematic evaluation on how robust DL methods are on satellite stereo images with varying radiometric and geometric configurations. This paper provides an evaluation of four DL stereo matching methods through hundreds of multi-date multi-site satellite stereo pairs with varying geometric configurations, against the traditional well-practiced Census-SGM (Semi-global matching), to comprehensively understand their accuracy, robustness, generalization capabilities, and their practical potential. The DL methods include a learning-based cost metric through convolutional neural networks (MC-CNN) followed by SGM, and three end-to-end (E2E) learning models using Geometry and Context Network (GCNet), Pyramid Stereo Matching Network (PSMNet), and LEAStereo. Our experiments show that E2E algorithms can achieve upper limits of geometric accuracies, while may not generalize well for unseen data. The learning-based cost metric and Census-SGM are rather robust and can consistently achieve acceptable results. All DL algorithms are robust to geometric configurations of stereo pairs and are less sensitive in comparison to the Census-SGM, while learning-based cost metrics can generalize on satellite images when trained on different datasets (airborne or ground-view).
On Fine-Tuned Deep Features for Unsupervised Domain Adaptation
Authors: Qian Wang, Toby P. Breckon
Date: 2022-10-25 00:00:00
Tasks: Domain Adaptation, Unsupervised Domain Adaptation

Prior feature transformation based approaches to Unsupervised Domain Adaptation (UDA) employ the deep features extracted by pre-trained deep models without fine-tuning them on the specific source or target domain data for a particular domain adaptation task. In contrast, end-to-end learning based approaches optimise the pre-trained backbones and the customised adaptation modules simultaneously to learn domain-invariant features for UDA. In this work, we explore the potential of combining fine-tuned features and feature transformation based UDA methods for improved domain adaptation performance. Specifically, we integrate the prevalent progressive pseudo-labelling techniques into the fine-tuning framework to extract fine-tuned features which are subsequently used in a state-of-the-art feature transformation based domain adaptation method SPL (Selective Pseudo-Labeling). Thorough experiments with multiple deep models including ResNet-50/101 and DeiT-small/base are conducted to demonstrate the combination of fine-tuned features and SPL can achieve state-of-the-art performance on several benchmark datasets.
Lafite2: Few-shot Text-to-Image Generation
Authors: Yufan Zhou, Chunyuan Li, Changyou Chen, Jianfeng Gao, Jinhui Xu
Date: 2022-10-25 00:00:00
Tasks: Image Generation, Retrieval, Text to Image Generation, Text-to-Image Generation

Text-to-image generation models have progressed considerably in recent years, which can now generate impressive realistic images from arbitrary text. Most of such models are trained on web-scale image-text paired datasets, which may not be affordable for many researchers. In this paper, we propose a novel method for pre-training text-to-image generation model on image-only datasets. It considers a retrieval-then-optimization procedure to synthesize pseudo text features: for a given image, relevant pseudo text features are first retrieved, then optimized for better alignment. The low requirement of the proposed method yields high flexibility and usability: it can be beneficial to a wide range of settings, including the few-shot, semi-supervised and fully-supervised learning; it can be applied on different models including generative adversarial networks (GANs) and diffusion models. Extensive experiments illustrate the effectiveness of the proposed method. On MS-COCO dataset, our GAN model obtains Fr\'echet Inception Distance (FID) of 6.78 which is the new state-of-the-art (SoTA) of GANs under fully-supervised setting. Our diffusion model obtains FID of 8.42 and 4.28 on zero-shot and supervised setting respectively, which are competitive to SoTA diffusion models with a much smaller model size.
Learning Explicit Object-Centric Representations with Vision Transformers
Authors: Oscar Vikström, Alexander Ilin
Date: 2022-10-25 00:00:00
Tasks: Object, Segmentation, Semantic Segmentation

With the recent successful adaptation of transformers to the vision domain, particularly when trained in a self-supervised fashion, it has been shown that vision transformers can learn impressive object-reasoning-like behaviour and features expressive for the task of object segmentation in images. In this paper, we build on the self-supervision task of masked autoencoding and explore its effectiveness for explicitly learning object-centric representations with transformers. To this end, we design an object-centric autoencoder using transformers only and train it end-to-end to reconstruct full images from unmasked patches. We show that the model efficiently learns to decompose simple scenes as measured by segmentation metrics on several multi-object benchmarks.
From colouring-in to pointillism: revisiting semantic segmentation supervision
Authors: Rodrigo Benenson, Vittorio Ferrari
Date: 2022-10-25 00:00:00
Tasks: Active Learning, Segmentation, Semantic Segmentation

The prevailing paradigm for producing semantic segmentation training data relies on densely labelling each pixel of each image in the training set, akin to colouring-in books. This approach becomes a bottleneck when scaling up in the number of images, classes, and annotators. Here we propose instead a pointillist approach for semantic segmentation annotation, where only point-wise yes/no questions are answered. We explore design alternatives for such an active learning approach, measure the speed and consistency of human annotators on this task, show that this strategy enables training good segmentation models, and that it is suitable for evaluating models at test time. As concrete proof of the scalability of our method, we collected and released 22.6M point labels over 4,171 classes on the Open Images dataset. Our results enable to rethink the semantic segmentation pipeline of annotation, training, and evaluation from a pointillism point of view.
GlassesGAN: Eyewear Personalization using Synthetic Appearance Discovery and Targeted Subspace Modeling
Authors: Richard Plesh, Peter Peer, Vitomir Štruc
Date: 2022-10-24 00:00:00
Proceeding: CVPR 2023 1

We present GlassesGAN, a novel image editing framework for custom design of glasses, that sets a new standard in terms of image quality, edit realism, and continuous multi-style edit capability. To facilitate the editing process with GlassesGAN, we propose a Targeted Subspace Modelling (TSM) procedure that, based on a novel mechanism for (synthetic) appearance discovery in the latent space of a pre-trained GAN generator, constructs an eyeglasses-specific (latent) subspace that the editing framework can utilize. Additionally, we also introduce an appearance-constrained subspace initialization (SI) technique that centers the latent representation of the given input image in the well-defined part of the constructed subspace to improve the reliability of the learned edits. We test GlassesGAN on two (diverse) high-resolution datasets (CelebA-HQ and SiblingsDB-HQf) and compare it to three state-of-the-art competitors, i.e., InterfaceGAN, GANSpace, and MaskGAN. The reported results show that GlassesGAN convincingly outperforms all competing techniques, while offering additional functionality (e.g., fine-grained multi-style editing) not available with any of the competitors. The source code will be made freely available.
Food Ingredients Recognition through Multi-label Learning
Authors: Rameez Ismail, Zhaorui Yuan
Date: 2022-10-24 00:00:00
Tasks: Decoder, Food Recognition, Multi-Label Learning

The ability to recognize various food-items in a generic food plate is a key determinant for an automated diet assessment system. This study motivates the need for automated diet assessment and proposes a framework to achieve this. Within this framework, we focus on one of the core functionalities to visually recognize various ingredients. To this end, we employed a deep multi-label learning approach and evaluated several state-of-the-art neural networks for their ability to detect an arbitrary number of ingredients in a dish image. The models evaluated in this work follow a definite meta-structure, consisting of an encoder and a decoder component. Two distinct decoding schemes, one based on global average pooling and the other on attention mechanism, are evaluated and benchmarked. Whereas for encoding, several well-known architectures, including DenseNet, EfficientNet, MobileNet, Inception and Xception, were employed. We present promising preliminary results for deep learning-based ingredients detection, using a challenging dataset, Nutrition5K, and establish a strong baseline for future explorations.
Atlas flow : compatible local structures on the manifold
Authors: Taejin Paik, Jaemin Park, Jung Ho Park
Date: 2022-10-24 00:00:00
Tasks: Topological Data Analysis

In this paper, we focus on the intersections of a manifold's local structures to analyze the global structure of a manifold. We obtain local regions on data manifolds such as the latent space of StyleGAN2, using Mapper, a tool from topological data analysis. We impose gluing compatibility conditions on overlapping local regions, which guarantee that the local structures can be glued together to the global structure of a manifold. We propose a novel generative flow model called Atlas flow that uses compatibility to reattach the local regions. Our model shows that the generating processes perform well on synthetic dataset samples of well-known manifolds with noise. Furthermore, we investigate the style vector manifold of StyleGAN2 using our model.
Drastically Reducing the Number of Trainable Parameters in Deep CNNs by Inter-layer Kernel-sharing
Authors: Alireza Azadbakht, Saeed Reza Kheradpisheh, Ismail Khalfaoui-Hassani, Timothée Masquelier
Date: 2022-10-23 00:00:00
Tasks: Edge-computing, image-classification, Image Classification, object-detection, Object Detection, Semantic Segmentation

Deep convolutional neural networks (DCNNs) have become the state-of-the-art (SOTA) approach for many computer vision tasks: image classification, object detection, semantic segmentation, etc. However, most SOTA networks are too large for edge computing. Here, we suggest a simple way to reduce the number of trainable parameters and thus the memory footprint: sharing kernels between multiple convolutional layers. Kernel-sharing is only possible between ``isomorphic" layers, i.e.layers having the same kernel size, input and output channels. This is typically the case inside each stage of a DCNN. Our experiments on CIFAR-10 and CIFAR-100, using the ConvMixer and SE-ResNet architectures show that the number of parameters of these models can drastically be reduced with minimal cost on accuracy. The resulting networks are appealing for certain edge computing applications that are subject to severe memory constraints, and even more interesting if leveraging "frozen weights" hardware accelerators. Kernel-sharing is also an efficient regularization method, which can reduce overfitting. The codes are publicly available at https://github.com/AlirezaAzadbakht/kernel-sharing.
Detection of Real-time DeepFakes in Video Conferencing with Active Probing and Corneal Reflection
Authors: Hui Guo, Xin Wang, Siwei Lyu
Date: 2022-10-21 00:00:00

The COVID pandemic has led to the wide adoption of online video calls in recent years. However, the increasing reliance on video calls provides opportunities for new impersonation attacks by fraudsters using the advanced real-time DeepFakes. Real-time DeepFakes pose new challenges to detection methods, which have to run in real-time as a video call is ongoing. In this paper, we describe a new active forensic method to detect real-time DeepFakes. Specifically, we authenticate video calls by displaying a distinct pattern on the screen and using the corneal reflection extracted from the images of the call participant's face. This pattern can be induced by a call participant displaying on a shared screen or directly integrated into the video-call client. In either case, no specialized imaging or lighting hardware is required. Through large-scale simulations, we evaluate the reliability of this approach under a range in a variety of real-world imaging scenarios.
Isomorphic mesh generation from point clouds with multilayer perceptrons
Authors: Shoko Miyauchi, Ken'ichi Morooka, Ryo Kurazume
Date: 2022-10-21 00:00:00

We propose a new neural network, called isomorphic mesh generator (iMG), which generates isomorphic meshes from point clouds containing noise and missing parts. Isomorphic meshes of arbitrary objects have a unified mesh structure even though the objects belong to different classes. This unified representation enables surface models to be handled by DNNs. Moreover, the unified mesh structure of isomorphic meshes enables the same process to be applied to all isomorphic meshes; although in the case of general mesh models, we need to consider the processes depending on their mesh structures. Therefore, the use of isomorphic meshes leads to efficient memory usage and calculation time compared with general mesh models. As iMG is a data-free method, preparing any point clouds as training data in advance is unnecessary, except a point cloud of the target object used as the input data of iMG. Additionally, iMG outputs an isomorphic mesh obtained by mapping a reference mesh to a given input point cloud. To estimate the mapping function stably, we introduce a step-by-step mapping strategy. This strategy achieves a flexible deformation while maintaining the structure of the reference mesh. From simulation and experiments using a mobile phone, we confirmed that iMG can generate isomorphic meshes of given objects reliably even when the input point cloud includes noise and missing parts.
Aligning MAGMA by Few-Shot Learning and Finetuning
Authors: Jean-Charles Layoun, Alexis Roger, Irina Rish
Date: 2022-10-18 00:00:00
Tasks: Few-Shot Learning, Image Captioning, Language Modeling, Language Modelling, Question Answering, Visual Question Answering, Visual Question Answering (VQA)

The goal of vision-language modeling is to allow models to tie language understanding with visual inputs. The aim of this paper is to evaluate and align the Visual Language Model (VLM) called Multimodal Augmentation of Generative Models through Adapter-based finetuning (MAGMA) with human values. MAGMA is a VLM that is capable of image captioning and visual question-answering. We will evaluate its alignment in three different scenarios. To begin, we assess MAGMA's out-of-the-box alignment through the checkpoint provided by Hugging Face. Then, we measure if few-shot learning manages to improve the results. Finally, we finetune the model on aligned examples and evaluate its behavior.
Commonsense Knowledge from Scene Graphs for Textual Environments
Authors: Tsunehiko Tanaka, Daiki Kimura, Michiaki Tatsubori
Date: 2022-10-19 00:00:00
Tasks: Common Sense Reasoning, text-based games

Text-based games are becoming commonly used in reinforcement learning as real-world simulation environments. They are usually imperfect information games, and their interactions are only in the textual modality. To challenge these games, it is effective to complement the missing information by providing knowledge outside the game, such as human common sense. However, such knowledge has only been available from textual information in previous works. In this paper, we investigate the advantage of employing commonsense reasoning obtained from visual datasets such as scene graph datasets. In general, images convey more comprehensive information compared with text for humans. This property enables to extract commonsense relationship knowledge more useful for acting effectively in a game. We compare the statistics of spatial relationships available in Visual Genome (a scene graph dataset) and ConceptNet (a text-based knowledge) to analyze the effectiveness of introducing scene graph datasets. We also conducted experiments on a text-based game task that requires commonsense reasoning. Our experimental results demonstrated that our proposed methods have higher and competitive performance than existing state-of-the-art methods.
Multi-Granularity Cross-Modality Representation Learning for Named Entity Recognition on Social Media
Authors: Peipei Liu, Gaosheng Wang, Hong Li, Jie Liu, Yimo Ren, Hongsong Zhu, Limin Sun
Date: 2022-10-19 00:00:00
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Representation Learning, Semantic correspondence

Named Entity Recognition (NER) on social media refers to discovering and classifying entities from unstructured free-form content, and it plays an important role for various applications such as intention understanding and user recommendation. With social media posts tending to be multimodal, Multimodal Named Entity Recognition (MNER) for the text with its accompanying image is attracting more and more attention since some textual components can only be understood in combination with visual information. However, there are two drawbacks in existing approaches: 1) Meanings of the text and its accompanying image do not match always, so the text information still plays a major role. However, social media posts are usually shorter and more informal compared with other normal contents, which easily causes incomplete semantic description and the data sparsity problem. 2) Although the visual representations of whole images or objects are already used, existing methods ignore either fine-grained semantic correspondence between objects in images and words in text or the objective fact that there are misleading objects or no objects in some images. In this work, we solve the above two problems by introducing the multi-granularity cross-modality representation learning. To resolve the first problem, we enhance the representation by semantic augmentation for each word in text. As for the second issue, we perform the cross-modality semantic interaction between text and vision at the different vision granularity to get the most effective multimodal guidance representation for every word. Experiments show that our proposed approach can achieve the SOTA or approximate SOTA performance on two benchmark datasets of tweets. The code, data and the best performing models are available at https://github.com/LiuPeiP-CS/IIE4MNER
Understanding Key Point Cloud Features for Development Three-dimensional Adversarial Attacks
Authors: Hanieh Naderi, Chinthaka Dinesh, Ivan V. Bajic, Shohreh Kasaei
Date: 2022-10-19 00:00:00
Tasks: 3D Point Cloud Classification, Autonomous Driving, Classification, Decision Making, feature selection, Point Cloud Classification

Adversarial attacks pose serious challenges for deep neural network (DNN)-based analysis of various input signals. In the case of three-dimensional point clouds, methods have been developed to identify points that play a key role in network decision, and these become crucial in generating existing adversarial attacks. For example, a saliency map approach is a popular method for identifying adversarial drop points, whose removal would significantly impact the network decision. This paper seeks to enhance the understanding of three-dimensional adversarial attacks by exploring which point cloud features are most important for predicting adversarial points. Specifically, Fourteen key point cloud features such as edge intensity and distance from the centroid are defined, and multiple linear regression is employed to assess their predictive power for adversarial points. Based on critical feature selection insights, a new attack method has been developed to evaluate whether the selected features can generate an attack successfully. Unlike traditional attack methods that rely on model-specific vulnerabilities, this approach focuses on the intrinsic characteristics of the point clouds themselves. It is demonstrated that these features can predict adversarial points across four different DNN architectures, Point Network (PointNet), PointNet++, Dynamic Graph Convolutional Neural Networks (DGCNN), and Point Convolutional Network (PointConv) outperforming random guessing and achieving results comparable to saliency map-based attacks. This study has important engineering applications, such as enhancing the security and robustness of three-dimensional point cloud-based systems in fields like robotics and autonomous driving.
MEEV: Body Mesh Estimation On Egocentric Video
Authors: Nicolas Monet, Dongyoon Wee
Date: 2022-10-21 00:00:00
Tasks: 3D human pose and shape estimation, 3D Human Pose Estimation, 3D Pose Estimation, Pose Estimation

This technical report introduces our solution, MEEV, proposed to the EgoBody Challenge at ECCV 2022. Captured from head-mounted devices, the dataset consists of human body shape and motion of interacting people. The EgoBody dataset has challenges such as occluded body or blurry image. In order to overcome the challenges, MEEV is designed to exploit multiscale features for rich spatial information. Besides, to overcome the limited size of dataset, the model is pre-trained with the dataset aggregated 2D and 3D pose estimation datasets. Achieving 82.30 for MPJPE and 92.93 for MPVPE, MEEV has won the EgoBody Challenge at ECCV 2022, which shows the effectiveness of the proposed method. The code is available at https://github.com/clovaai/meev