pwc-archive/papers-with-abstracts huggingface dataset:
Three-Stream Joint Network for Zero-Shot Sketch-Based Image Retrieval
Authors: Yu-Wei Zhan, Xin Luo, Yongxin Wang, Zhen-Duo Chen, Xin-Shun Xu
Date: 2022-04-12 00:00:00
Tasks: Image Retrieval, Retrieval, Sketch-Based Image Retrieval

The Zero-Shot Sketch-based Image Retrieval (ZS-SBIR) is a challenging task because of the large domain gap between sketches and natural images as well as the semantic inconsistency between seen and unseen categories. Previous literature bridges seen and unseen categories by semantic embedding, which requires prior knowledge of the exact class names and additional extraction efforts. And most works reduce domain gap by mapping sketches and natural images into a common high-level space using constructed sketch-image pairs, which ignore the unpaired information between images and sketches. To address these issues, in this paper, we propose a novel Three-Stream Joint Training Network (3JOIN) for the ZS-SBIR task. To narrow the domain differences between sketches and images, we extract edge maps for natural images and treat them as a bridge between images and sketches, which have similar content to images and similar style to sketches. For exploiting a sufficient combination of sketches, natural images, and edge maps, a novel three-stream joint training network is proposed. In addition, we use a teacher network to extract the implicit semantics of the samples without the aid of other semantics and transfer the learned knowledge to unseen classes. Extensive experiments conducted on two real-world datasets demonstrate the superiority of our proposed method.
How to Register a Live onto a Liver ? Partial Matching in the Space of Varifolds
Authors: Pierre-Louis Antonsanti, Thomas Benseghir, Vincent Jugnon, Mario Ghosn, Perrine Chassat, Irène Kaltenmark, Joan Glaunès
Date: 2022-04-12 00:00:00
Tasks: Anatomy, Computed Tomography (CT)

Partial shapes correspondences is a problem that often occurs in computer vision (occlusion, evolution in time...). In medical imaging, data may come from different modalities and be acquired under different conditions which leads to variations in shapes and topologies. In this paper we use an asymmetric data dissimilarity term applicable to various geometric shapes like sets of curves or surfaces, assessing the embedding of a shape into another one without relying on correspondences. It is designed as a data attachment for the Large Deformation Diffeomorphic Metric Mapping (LDDMM) framework, allowing to compute a meaningful deformation of one shape onto a subset of the other. We refine it in order to control the resulting non-rigid deformations and provide consistent deformations of the shapes along with their ambient space. We show that partial matching can be used for robust multi-modal liver registration between a Computed Tomography (CT) volume and a Cone Beam Computed Tomography (CBCT) volume. The 3D imaging of the patient CBCT at point of care that we call live is truncated while the CT pre-intervention provides a full visualization of the liver. The proposed method allows the truncated surfaces from CBCT to be aligned non-rigidly, yet realistically, with surfaces from CT with an average distance of 2.6mm(+/- 2.2). The generated deformations extend consistently to the liver volume, and are evaluated on points of interest for the physicians, with an average distance of 5.8mm (+/- 2.7) for vessels bifurcations and 5.13mm (+/- 2.5) for tumors landmarks. Such multi-modality volumes registrations would help the physicians in the perspective of navigating their tools in the patient's anatomy to locate structures that are hardly visible in the CBCT used during their procedures. Our code is available at https://github.com/plantonsanti/PartialMatchingVarifolds.
X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks
Authors: Zhaowei Cai, Gukyeong Kwon, Avinash Ravichandran, Erhan Bas, Zhuowen Tu, Rahul Bhotika, Stefano Soatto
Date: 2022-04-12 00:00:00

In this paper, we study the challenging instance-wise vision-language tasks, where the free-form language is required to align with the objects instead of the whole image. To address these tasks, we propose X-DETR, whose architecture has three major components: an object detector, a language encoder, and vision-language alignment. The vision and language streams are independent until the end and they are aligned using an efficient dot-product operation. The whole network is trained end-to-end, such that the detector is optimized for the vision-language tasks instead of an off-the-shelf component. To overcome the limited size of paired object-language annotations, we leverage other weak types of supervision to expand the knowledge coverage. This simple yet effective architecture of X-DETR shows good accuracy and fast speeds for multiple instance-wise vision-language tasks, e.g., 16.4 AP on LVIS detection of 1.2K categories at ~20 frames per second without using any LVIS annotation during training.
Continual Predictive Learning from Videos
Authors: Geng Chen, Wendong Zhang, Han Lu, Siyu Gao, Yunbo Wang, Mingsheng Long, Xiaokang Yang
Date: 2022-04-12 00:00:00
Proceeding: CVPR 2022 1
Tasks: Continual Learning, Prediction, Test-time Adaptation, Video Prediction

Predictive learning ideally builds the world model of physical processes in one or more given environments. Typical setups assume that we can collect data from all environments at all times. In practice, however, different prediction tasks may arrive sequentially so that the environments may change persistently throughout the training procedure. Can we develop predictive learning algorithms that can deal with more realistic, non-stationary physical environments? In this paper, we study a new continual learning problem in the context of video prediction, and observe that most existing methods suffer from severe catastrophic forgetting in this setup. To tackle this problem, we propose the continual predictive learning (CPL) approach, which learns a mixture world model via predictive experience replay and performs test-time adaptation with non-parametric task inference. We construct two new benchmarks based on RoboNet and KTH, in which different tasks correspond to different physical robotic environments or human actions. Our approach is shown to effectively mitigate forgetting and remarkably outperform the na\"ive combinations of previous art in video prediction and continual learning.
Regression or Classification? Reflection on BP prediction from PPG data using Deep Neural Networks in the scope of practical applications
Authors: Fabian Schrumpf, Paul Rudi Serdack, Mirco Fuchs
Date: 2022-04-12 00:00:00
Tasks: Classification, Diagnostic, regression

Photoplethysmographic (PPG) signals offer diagnostic potential beyond heart rate analysis or blood oxygen level monitoring. In the recent past, research focused extensively on non-invasive PPG-based approaches to blood pressure (BP) estimation. These approaches can be subdivided into regression and classification methods. The latter assign PPG signals to predefined BP intervals that represent clinically relevant ranges. The former predict systolic (SBP) and diastolic (DBP) BP as continuous variables and are of particular interest to the research community. However, the reported accuracies of BP regression methods vary widely among publications with some authors even questioning the feasibility of PPG-based BP regression altogether. In our work, we compare BP regression and classification approaches. We argue that BP classification might provide diagnostic value that is equivalent to regression in many clinically relevant scenarios while being similar or even superior in terms of performance. We compare several established neural architectures using publicly available PPG data for SBP regression and classification with and without personalization using subject-specific data. We found that classification and regression models perform similar before personalization. However, after personalization, the accuracy of classification based methods outperformed regression approaches. We conclude that BP classification might be preferable over BP regression in certain scenarios where a coarser segmentation of the BP range is sufficient.
Towards Open-Set Object Detection and Discovery
Authors: Jiyang Zheng, Weihao Li, Jie Hong, Lars Petersson, Nick Barnes
Date: 2022-04-12 00:00:00
Tasks: Incremental Learning, Object, object-detection, Object Detection

With the human pursuit of knowledge, open-set object detection (OSOD) has been designed to identify unknown objects in a dynamic world. However, an issue with the current setting is that all the predicted unknown objects share the same category as "unknown", which require incremental learning via a human-in-the-loop approach to label novel classes. In order to address this problem, we present a new task, namely Open-Set Object Detection and Discovery (OSODD). This new task aims to extend the ability of open-set object detectors to further discover the categories of unknown objects based on their visual appearance without human effort. We propose a two-stage method that first uses an open-set object detector to predict both known and unknown objects. Then, we study the representation of predicted objects in an unsupervised manner and discover new categories from the set of unknown objects. With this method, a detector is able to detect objects belonging to known classes and define novel categories for objects of unknown classes with minimal supervision. We show the performance of our model on the MS-COCO dataset under a thorough evaluation protocol. We hope that our work will promote further research towards a more robust real-world detection system.
HyperDet3D: Learning a Scene-conditioned 3D Object Detector
Authors: Yu Zheng, Yueqi Duan, Jiwen Lu, Jie zhou, Qi Tian
Date: 2022-04-12 00:00:00
Proceeding: CVPR 2022 1
Tasks: 3D Object Detection, Object, object-detection, Object Detection

A bathtub in a library, a sink in an office, a bed in a laundry room -- the counter-intuition suggests that scene provides important prior knowledge for 3D object detection, which instructs to eliminate the ambiguous detection of similar objects. In this paper, we propose HyperDet3D to explore scene-conditioned prior knowledge for 3D object detection. Existing methods strive for better representation of local elements and their relations without scene-conditioned knowledge, which may cause ambiguity merely based on the understanding of individual points and object candidates. Instead, HyperDet3D simultaneously learns scene-agnostic embeddings and scene-specific knowledge through scene-conditioned hypernetworks. More specifically, our HyperDet3D not only explores the sharable abstracts from various 3D scenes, but also adapts the detector to the given scene at test time. We propose a discriminative Multi-head Scene-specific Attention (MSA) module to dynamically control the layer parameters of the detector conditioned on the fusion of scene-conditioned knowledge. Our HyperDet3D achieves state-of-the-art results on the 3D object detection benchmark of the ScanNet and SUN RGB-D datasets. Moreover, through cross-dataset evaluation, we show the acquired scene-conditioned prior knowledge still takes effect when facing 3D scenes with domain gap.
On the Equity of Nuclear Norm Maximization in Unsupervised Domain Adaptation
Authors: Wenju Zhang, Xiang Zhang, Qing Liao, Long Lan, Mengzhu Wang, Wei Wang, Baoyun Peng, Zhengming Ding
Date: 2022-04-12 00:00:00
Tasks: Domain Adaptation, image-classification, Image Classification, Unsupervised Domain Adaptation

Nuclear norm maximization has shown the power to enhance the transferability of unsupervised domain adaptation model (UDA) in an empirical scheme. In this paper, we identify a new property termed equity, which indicates the balance degree of predicted classes, to demystify the efficacy of nuclear norm maximization for UDA theoretically. With this in mind, we offer a new discriminability-and-equity maximization paradigm built on squares loss, such that predictions are equalized explicitly. To verify its feasibility and flexibility, two new losses termed Class Weighted Squares Maximization (CWSM) and Normalized Squares Maximization (NSM), are proposed to maximize both predictive discriminability and equity, from the class level and the sample level, respectively. Importantly, we theoretically relate these two novel losses (i.e., CWSM and NSM) to the equity maximization under mild conditions, and empirically suggest the importance of the predictive equity in UDA. Moreover, it is very efficient to realize the equity constraints in both losses. Experiments of cross-domain image classification on three popular benchmark datasets show that both CWSM and NSM contribute to outperforming the corresponding counterparts.
SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection
Authors: Zhengyi Liu, Yacheng Tan, Qian He, Yun Xiao
Date: 2022-04-12 00:00:00
Tasks: Decoder, object-detection, Object Detection, RGB-T Salient Object Detection, Salient Object Detection

Convolutional neural networks (CNNs) are good at extracting contexture features within certain receptive fields, while transformers can model the global long-range dependency features. By absorbing the advantage of transformer and the merit of CNN, Swin Transformer shows strong feature representation ability. Based on it, we propose a cross-modality fusion model SwinNet for RGB-D and RGB-T salient object detection. It is driven by Swin Transformer to extract the hierarchical features, boosted by attention mechanism to bridge the gap between two modalities, and guided by edge information to sharp the contour of salient object. To be specific, two-stream Swin Transformer encoder first extracts multi-modality features, and then spatial alignment and channel re-calibration module is presented to optimize intra-level cross-modality features. To clarify the fuzzy boundary, edge-guided decoder achieves inter-level cross-modality fusion under the guidance of edge features. The proposed model outperforms the state-of-the-art models on RGB-D and RGB-T datasets, showing that it provides more insight into the cross-modality complementarity task.
Compact Model Training by Low-Rank Projection with Energy Transfer
Authors: Kailing Guo, Zhenquan Lin, Canyang Chen, Xiaofen Xing, Fang Liu, Xiangmin Xu
Date: 2022-04-12 00:00:00
Tasks: Low-rank compression

Low-rankness plays an important role in traditional machine learning, but is not so popular in deep learning. Most previous low-rank network compression methods compress networks by approximating pre-trained models and re-training. However, the optimal solution in the Euclidean space may be quite different from the one with low-rank constraint. A well-pre-trained model is not a good initialization for the model with low-rank constraints. Thus, the performance of a low-rank compressed network degrades significantly. Compared with other network compression methods such as pruning, low-rank methods attract less attention in recent years. In this paper, we devise a new training method, low-rank projection with energy transfer (LRPET), that trains low-rank compressed networks from scratch and achieves competitive performance. We propose to alternately perform stochastic gradient descent training and projection of each weight matrix onto the corresponding low-rank manifold. Compared to re-training on the compact model, this enables full utilization of model capacity since solution space is relaxed back to Euclidean space after projection. The matrix energy (the sum of squares of singular values) reduction caused by projection is compensated by energy transfer. We uniformly transfer the energy of the pruned singular values to the remaining ones. We theoretically show that energy transfer eases the trend of gradient vanishing caused by projection. In modern networks, a batch normalization (BN) layer can be merged into the previous convolution layer for inference, thereby influencing the optimal low-rank approximation of the previous layer. We propose BN rectification to cut off its effect on the optimal low-rank approximation, which further improves the performance.
DistPro: Searching A Fast Knowledge Distillation Process via Meta Optimization
Authors: Xueqing Deng, Dawei Sun, Shawn Newsam, Peng Wang
Date: 2022-04-12 00:00:00
Tasks: Knowledge Distillation, Meta-Learning

Recent Knowledge distillation (KD) studies show that different manually designed schemes impact the learned results significantly. Yet, in KD, automatically searching an optimal distillation scheme has not yet been well explored. In this paper, we propose DistPro, a novel framework which searches for an optimal KD process via differentiable meta-learning. Specifically, given a pair of student and teacher networks, DistPro first sets up a rich set of KD connection from the transmitting layers of the teacher to the receiving layers of the student, and in the meanwhile, various transforms are also proposed for comparing feature maps along its pathway for the distillation. Then, each combination of a connection and a transform choice (pathway) is associated with a stochastic weighting process which indicates its importance at every step during the distillation. In the searching stage, the process can be effectively learned through our proposed bi-level meta-optimization strategy. In the distillation stage, DistPro adopts the learned processes for knowledge distillation, which significantly improves the student accuracy especially when faster training is required. Lastly, we find the learned processes can be generalized between similar tasks and networks. In our experiments, DistPro produces state-of-the-art (SoTA) accuracy under varying number of learning epochs on popular datasets, i.e. CIFAR100 and ImageNet, which demonstrate the effectiveness of our framework.
Undoing the Damage of Label Shift for Cross-domain Semantic Segmentation
Authors: Yahao Liu, Jinhong Deng, Jiale Tao, Tong Chu, Lixin Duan, Wen Li
Date: 2022-04-12 00:00:00
Proceeding: CVPR 2022 1
Tasks: Semantic Segmentation

Existing works typically treat cross-domain semantic segmentation (CDSS) as a data distribution mismatch problem and focus on aligning the marginal distribution or conditional distribution. However, the label shift issue is unfortunately overlooked, which actually commonly exists in the CDSS task, and often causes a classifier bias in the learnt model. In this paper, we give an in-depth analysis and show that the damage of label shift can be overcome by aligning the data conditional distribution and correcting the posterior probability. To this end, we propose a novel approach to undo the damage of the label shift problem in CDSS. In implementation, we adopt class-level feature alignment for conditional distribution alignment, as well as two simple yet effective methods to rectify the classifier bias from source to target by remolding the classifier predictions. We conduct extensive experiments on the benchmark datasets of urban scenes, including GTA5 to Cityscapes and SYNTHIA to Cityscapes, where our proposed approach outperforms previous methods by a large margin. For instance, our model equipped with a self-training strategy reaches 59.3% mIoU on GTA5 to Cityscapes, pushing to a new state-of-the-art. The code will be available at https://github.com/manmanjun/Undoing UDA.
Towards Reliable Image Outpainting: Learning Structure-Aware Multimodal Fusion with Depth Guidance
Authors: Lei Zhang, Kang Liao, Chunyu Lin, Yao Zhao
Date: 2022-04-12 00:00:00
Tasks: Image Outpainting

Image outpainting technology generates visually plausible content regardless of authenticity, making it unreliable to be applied in practice. Thus, we propose a reliable image outpainting task, introducing the sparse depth from LiDARs to extrapolate authentic RGB scenes. The large field view of LiDARs allows it to serve for data enhancement and further multimodal tasks. Concretely, we propose a Depth-Guided Outpainting Network to model different feature representations of two modalities and learn the structure-aware cross-modal fusion. And two components are designed: 1) The Multimodal Learning Module produces unique depth and RGB feature representations from the perspectives of different modal characteristics. 2) The Depth Guidance Fusion Module leverages the complete depth modality to guide the establishment of RGB contents by progressive multimodal feature fusion. Furthermore, we specially design an additional constraint strategy consisting of Cross-modal Loss and Edge Loss to enhance ambiguous contours and expedite reliable content generation. Extensive experiments on KITTI and Waymo datasets demonstrate our superiority over the state-of-the-art method, quantitatively and qualitatively.
Content and Style Aware Generation of Text-line Images for Handwriting Recognition
Authors: Lei Kang, Pau Riba, Marçal Rusiñol, Alicia Fornés, Mauricio Villegas
Date: 2022-04-12 00:00:00
Tasks: Handwriting Recognition, Handwritten Text Recognition

Handwritten Text Recognition has achieved an impressive performance in public benchmarks. However, due to the high inter- and intra-class variability between handwriting styles, such recognizers need to be trained using huge volumes of manually labeled training data. To alleviate this labor-consuming problem, synthetic data produced with TrueType fonts has been often used in the training loop to gain volume and augment the handwriting style variability. However, there is a significant style bias between synthetic and real data which hinders the improvement of recognition performance. To deal with such limitations, we propose a generative method for handwritten text-line images, which is conditioned on both visual appearance and textual content. Our method is able to produce long text-line samples with diverse handwriting styles. Once properly trained, our method can also be adapted to new target data by only accessing unlabeled text-line images to mimic handwritten styles and produce images with any textual content. Extensive experiments have been done on making use of the generated samples to boost Handwritten Text Recognition performance. Both qualitative and quantitative results demonstrate that the proposed approach outperforms the current state of the art.
NightLab: A Dual-level Architecture with Hardness Detection for Segmentation at Night
Authors: Xueqing Deng, Peng Wang, Xiaochen Lian, Shawn Newsam
Date: 2022-04-12 00:00:00
Proceeding: CVPR 2022 1
Tasks: Segmentation, Self-Driving Cars, Semantic Segmentation

The semantic segmentation of nighttime scenes is a challenging problem that is key to impactful applications like self-driving cars. Yet, it has received little attention compared to its daytime counterpart. In this paper, we propose NightLab, a novel nighttime segmentation framework that leverages multiple deep learning models imbued with night-aware features to yield State-of-The-Art (SoTA) performance on multiple night segmentation benchmarks. Notably, NightLab contains models at two levels of granularity, i.e. image and regional, and each level is composed of light adaptation and segmentation modules. Given a nighttime image, the image level model provides an initial segmentation estimate while, in parallel, a hardness detection module identifies regions and their surrounding context that need further analysis. A regional level model focuses on these difficult regions to provide a significantly improved segmentation. All the models in NightLab are trained end-to-end using a set of proposed night-aware losses without handcrafted heuristics. Extensive experiments on the NightCity and BDD100K datasets show NightLab achieves SoTA performance compared to concurrent methods.