pwc-archive/papers-with-abstracts huggingface dataset:
Flexible Sampling for Long-tailed Skin Lesion Classification
Authors: Lie Ju, Yicheng Wu, Lin Wang, Zhen Yu, Xin Zhao, Xin Wang, Paul Bonnington, ZongYuan Ge
Date: 2022-04-07 00:00:00
Tasks: Classification, Lesion Classification, Skin Lesion Classification

Most of the medical tasks naturally exhibit a long-tailed distribution due to the complex patient-level conditions and the existence of rare diseases. Existing long-tailed learning methods usually treat each class equally to re-balance the long-tailed distribution. However, considering that some challenging classes may present diverse intra-class distributions, re-balancing all classes equally may lead to a significant performance drop. To address this, in this paper, we propose a curriculum learning-based framework called Flexible Sampling for the long-tailed skin lesion classification task. Specifically, we initially sample a subset of training data as anchor points based on the individual class prototypes. Then, these anchor points are used to pre-train an inference model to evaluate the per-class learning difficulty. Finally, we use a curriculum sampling module to dynamically query new samples from the rest training samples with the learning difficulty-aware sampling probability. We evaluated our model against several state-of-the-art methods on the ISIC dataset. The results with two long-tailed settings have demonstrated the superiority of our proposed training strategy, which achieves a new benchmark for long-tailed skin lesion classification.
Just-Noticeable-Difference Based Edge Map Quality Measure
Authors: Ijaz Ahmad, Seokjoo Shin
Date: 2022-04-07 00:00:00

The performance of an edge detector can be improved when assisted with an effective edge map quality measure. Several evaluation methods have been proposed resulting in different performance score for the same candidate edge map. However, an effective measure is the one that can be automated and which correlates with human judgement perceived quality of the edge map. Distance-based edge map measures are widely used for assessment of edge map quality. These methods consider distance and statistical properties of edge pixels to estimate a performance score. The existing methods can be automated; however, they lack perceptual features. This paper presents edge map quality measure based on Just-Noticeable-Difference (JND) feature of human visual system, to compensate the shortcomings of distance-based edge measures. For this purpose, we have designed constant stimulus experiment to measure the JND value for two spatial alternative. Experimental results show that JND based distance calculation outperforms existing distance-based measures according to subjective evaluation.
Exploring Cross-Domain Pretrained Model for Hyperspectral Image Classification
Authors: Hyungtae Lee, Sungmin Eum, Heesung Kwon
Date: 2022-04-07 00:00:00
Tasks: Classification, Hyperspectral Image Classification, image-classification, Image Classification

A pretrain-finetune strategy is widely used to reduce the overfitting that can occur when data is insufficient for CNN training. First few layers of a CNN pretrained on a large-scale RGB dataset are capable of acquiring general image characteristics which are remarkably effective in tasks targeted for different RGB datasets. However, when it comes down to hyperspectral domain where each domain has its unique spectral properties, the pretrain-finetune strategy no longer can be deployed in a conventional way while presenting three major issues: 1) inconsistent spectral characteristics among the domains (e.g., frequency range), 2) inconsistent number of data channels among the domains, and 3) absence of large-scale hyperspectral dataset. We seek to train a universal cross-domain model which can later be deployed for various spectral domains. To achieve, we physically furnish multiple inlets to the model while having a universal portion which is designed to handle the inconsistent spectral characteristics among different domains. Note that only the universal portion is used in the finetune process. This approach naturally enables the learning of our model on multiple domains simultaneously which acts as an effective workaround for the issue of the absence of large-scale dataset. We have carried out a study to extensively compare models that were trained using cross-domain approach with ones trained from scratch. Our approach was found to be superior both in accuracy and in training efficiency. In addition, we have verified that our approach effectively reduces the overfitting issue, enabling us to deepen the model up to 13 layers (from 9) without compromising the accuracy.
DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects
Authors: Priya Sundaresan, Rika Antonova, Jeannette Bohg
Date: 2022-04-07 00:00:00

Research in manipulation of deformable objects is typically conducted on a limited range of scenarios, because handling each scenario on hardware takes significant effort. Realistic simulators with support for various types of deformations and interactions have the potential to speed up experimentation with novel tasks and algorithms. However, for highly deformable objects it is challenging to align the output of a simulator with the behavior of real objects. Manual tuning is not intuitive, hence automated methods are needed. We view this alignment problem as a joint perception-inference challenge and demonstrate how to use recent neural network architectures to successfully perform simulation parameter inference from real point clouds. We analyze the performance of various architectures, comparing their data and training requirements. Furthermore, we propose to leverage differentiable point cloud sampling and differentiable simulation to significantly reduce the time to achieve the alignment. We employ an efficient way to propagate gradients from point clouds to simulated meshes and further through to the physical simulation parameters, such as mass and stiffness. Experiments with highly deformable objects show that our method can achieve comparable or better alignment with real object behavior, while reducing the time needed to achieve this by more than an order of magnitude. Videos and supplementary material are available at https://diffcloud.github.io.
AutoCOR: Autonomous Condylar Offset Ratio Calculator on TKA-Postoperative Lateral Knee X-ray
Authors: Gulsade Rabia Cakmak, Ibrahim Ethem Hamamci, Mehmet Kursat Yilmaz, Reda Alhajj, Ibrahim Azboy, Mehmet Kemal Ozdemir
Date: 2022-04-06 00:00:00

The postoperative range of motion is one of the crucial factors indicating the outcome of Total Knee Arthroplasty (TKA). Although the correlation between range of knee flexion and posterior condylar offset (PCO) is controversial in the literature, PCO maintains its importance on evaluation of TKA. Due to limitations on PCO measurement, two novel parameters, posterior condylar offset ratio (PCOR) and anterior condylar offset ratio (ACOR), were introduced. Nowadays, the calculation of PCOR and ACOR on plain lateral radiographs is done manually by orthopedic surgeons. In this regard, we developed a software, AutoCOR, to calculate PCOR and ACOR autonomously, utilizing unsupervised machine learning algorithm (k-means clustering) and digital image processing techniques. The software AutoCOR is capable of detecting the anterior/posterior edge points and anterior/posterior cortex of the femoral shaft on true postoperative lateral conventional radiographs. To test the algorithm, 50 postoperative true lateral radiographs from Istanbul Kosuyolu Medipol Hospital Database were used (32 patients). The mean PCOR was 0.984 (SD 0.235) in software results and 0.972 (SD 0.164) in ground truth values. It shows strong and significant correlation between software and ground truth values (Pearson r=0.845 p<0.0001). The mean ACOR was 0.107 (SD 0.092) in software results and 0.107 (SD 0.070) in ground truth values. It shows moderate and significant correlation between software and ground truth values (Spearman's rs=0.519 p=0.0001412). We suggest that AutoCOR is a useful tool that can be used in clinical practice.
UIGR: Unified Interactive Garment Retrieval
Authors: Xiao Han, Sen He, Li Zhang, Yi-Zhe Song, Tao Xiang
Date: 2022-04-06 00:00:00
Tasks: Retrieval

Interactive garment retrieval (IGR) aims to retrieve a target garment image based on a reference garment image along with user feedback on what to change on the reference garment. Two IGR tasks have been studied extensively: text-guided garment retrieval (TGR) and visually compatible garment retrieval (VCR). The user feedback for the former indicates what semantic attributes to change with the garment category preserved, while the category is the only thing to be changed explicitly for the latter, with an implicit requirement on style preservation. Despite the similarity between these two tasks and the practical need for an efficient system tackling both, they have never been unified and modeled jointly. In this paper, we propose a Unified Interactive Garment Retrieval (UIGR) framework to unify TGR and VCR. To this end, we first contribute a large-scale benchmark suited for both problems. We further propose a strong baseline architecture to integrate TGR and VCR in one model. Extensive experiments suggest that unifying two tasks in one framework is not only more efficient by requiring a single model only, it also leads to better performance. Code and datasets are available at https://github.com/BrandonHanx/CompFashion.
AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis
Authors: Zhiqin Chen, Kangxue Yin, Sanja Fidler
Date: 2022-04-06 00:00:00
Proceeding: CVPR 2022 1
Tasks: 3D Reconstruction, Single-View 3D Reconstruction, Texture Synthesis

In this paper, we address the problem of texture representation for 3D shapes for the challenging and underexplored tasks of texture transfer and synthesis. Previous works either apply spherical texture maps which may lead to large distortions, or use continuous texture fields that yield smooth outputs lacking details. We argue that the traditional way of representing textures with images and linking them to a 3D mesh via UV mapping is more desirable, since synthesizing 2D images is a well-studied problem. We propose AUV-Net which learns to embed 3D surfaces into a 2D aligned UV space, by mapping the corresponding semantic parts of different 3D shapes to the same location in the UV space. As a result, textures are aligned across objects, and can thus be easily synthesized by generative models of images. Texture alignment is learned in an unsupervised manner by a simple yet effective texture alignment module, taking inspiration from traditional works on linear subspace learning. The learned UV mapping and aligned texture representations enable a variety of applications including texture transfer, texture synthesis, and textured single view 3D reconstruction. We conduct experiments on multiple datasets to demonstrate the effectiveness of our method. Project page: https://nv-tlabs.github.io/AUV-NET.
Hierarchical Self-supervised Representation Learning for Movie Understanding
Authors: Fanyi Xiao, Kaustav Kundu, Joseph Tighe, Davide Modolo
Date: 2022-04-06 00:00:00
Proceeding: CVPR 2022 1
Tasks: Action Recognition, Contrastive Learning, Representation Learning

Most self-supervised video representation learning approaches focus on action recognition. In contrast, in this paper we focus on self-supervised video learning for movie understanding and propose a novel hierarchical self-supervised pretraining strategy that separately pretrains each level of our hierarchical movie understanding model (based on [37]). Specifically, we propose to pretrain the low-level video backbone using a contrastive learning objective, while pretrain the higher-level video contextualizer using an event mask prediction task, which enables the usage of different data sources for pretraining different levels of the hierarchy. We first show that our self-supervised pretraining strategies are effective and lead to improved performance on all tasks and metrics on VidSitu benchmark [37] (e.g., improving on semantic role prediction from 47% to 61% CIDEr scores). We further demonstrate the effectiveness of our contextualized event features on LVU tasks [54], both when used alone and when combined with instance features, showing their complementarity.
Audio-Visual Person-of-Interest DeepFake Detection
Authors: Davide Cozzolino, Alessandro Pianese, Matthias Nießner, Luisa Verdoliva
Date: 2022-04-06 00:00:00
Tasks: Contrastive Learning, DeepFake Detection, Face Swapping

Face manipulation technology is advancing very rapidly, and new methods are being proposed day by day. The aim of this work is to propose a deepfake detector that can cope with the wide variety of manipulation methods and scenarios encountered in the real world. Our key insight is that each person has specific characteristics that a synthetic generator likely cannot reproduce. Accordingly, we extract audio-visual features which characterize the identity of a person, and use them to create a person-of-interest (POI) deepfake detector. We leverage a contrastive learning paradigm to learn the moving-face and audio segment embeddings that are most discriminative for each identity. As a result, when the video and/or audio of a person is manipulated, its representation in the embedding space becomes inconsistent with the real identity, allowing reliable detection. Training is carried out exclusively on real talking-face video; thus, the detector does not depend on any specific manipulation method and yields the highest generalization ability. In addition, our method can detect both single-modality (audio-only, video-only) and multi-modality (audio-video) attacks, and is robust to low-quality or corrupted videos. Experiments on a wide variety of datasets confirm that our method ensures a SOTA performance, especially on low quality videos. Code is publicly available on-line at https://github.com/grip-unina/poi-forensics.
Instance Segmentation of Unlabeled Modalities via Cyclic Segmentation GAN
Authors: Leander Lauenburg, Zudi Lin, Ruihan Zhang, Márcia dos Santos, Siyu Huang, Ignacio Arganda-Carreras, Edward S. Boyden, Hanspeter Pfister, Donglai Wei
Date: 2022-04-06 00:00:00
Tasks: Generative Adversarial Network, Image Segmentation, Instance Segmentation, Segmentation, Semantic Segmentation, Translation

Instance segmentation for unlabeled imaging modalities is a challenging but essential task as collecting expert annotation can be expensive and time-consuming. Existing works segment a new modality by either deploying a pre-trained model optimized on diverse training data or conducting domain translation and image segmentation as two independent steps. In this work, we propose a novel Cyclic Segmentation Generative Adversarial Network (CySGAN) that conducts image translation and instance segmentation jointly using a unified framework. Besides the CycleGAN losses for image translation and supervised losses for the annotated source domain, we introduce additional self-supervised and segmentation-based adversarial objectives to improve the model performance by leveraging unlabeled target domain images. We benchmark our approach on the task of 3D neuronal nuclei segmentation with annotated electron microscopy (EM) images and unlabeled expansion microscopy (ExM) data. Our CySGAN outperforms both pretrained generalist models and the baselines that sequentially conduct image translation and segmentation. Our implementation and the newly collected, densely annotated ExM nuclei dataset, named NucExM, are available at https://connectomics-bazaar.github.io/proj/CySGAN/index.html.
OSCARS: An Outlier-Sensitive Content-Based Radiography Retrieval System
Authors: Xiaoyuan Guo, Jiali Duan, Saptarshi Purkayastha, Hari Trivedi, Judy Wawira Gichoya, Imon Banerjee
Date: 2022-04-06 00:00:00
Tasks: Metric Learning, Retrieval

Improving the retrieval relevance on noisy datasets is an emerging need for the curation of a large-scale clean dataset in the medical domain. While existing methods can be applied for class-wise retrieval (aka. inter-class), they cannot distinguish the granularity of likeness within the same class (aka. intra-class). The problem is exacerbated on medical external datasets, where noisy samples of the same class are treated equally during training. Our goal is to identify both intra/inter-class similarities for fine-grained retrieval. To achieve this, we propose an Outlier-Sensitive Content-based rAdiologhy Retrieval System (OSCARS), consisting of two steps. First, we train an outlier detector on a clean internal dataset in an unsupervised manner. Then we use the trained detector to generate the anomaly scores on the external dataset, whose distribution will be used to bin intra-class variations. Second, we propose a quadruplet (a, p, nintra, ninter) sampling strategy, where intra-class negatives nintra are sampled from bins of the same class other than the bin anchor a belongs to, while niner are randomly sampled from inter-classes. We suggest a weighted metric learning objective to balance the intra and inter-class feature learning. We experimented on two representative public radiography datasets. Experiments show the effectiveness of our approach. The training and evaluation code can be found in https://github.com/XiaoyuanGuo/oscars.
The Self-Optimal-Transport Feature Transform
Authors: Daniel Shalam, Simon Korman
Date: 2022-04-06 00:00:00
Tasks: Few-Shot Image Classification, Large-Scale Person Re-Identification, Person Re-Identification

The Self-Optimal-Transport (SOT) feature transform is designed to upgrade the set of features of a data instance to facilitate downstream matching or grouping related tasks. The transformed set encodes a rich representation of high order relations between the instance features. Distances between transformed features capture their direct original similarity and their third party agreement regarding similarity to other features in the set. A particular min-cost-max-flow fractional matching problem, whose entropy regularized version can be approximated by an optimal transport (OT) optimization, results in our transductive transform which is efficient, differentiable, equivariant, parameterless and probabilistically interpretable. Empirically, the transform is highly effective and flexible in its use, consistently improving networks it is inserted into, in a variety of tasks and training schemes. We demonstrate its merits through the problem of unsupervised clustering and its efficiency and wide applicability for few-shot-classification, with state-of-the-art results, and large-scale person re-identification.
Late multimodal fusion for image and audio music transcription
Authors: María Alfaro-Contreras, Jose J. Valero-Mas, José M. Iñesta, Jorge Calvo-Zaragoza
Date: 2022-04-06 00:00:00
Tasks: Information Retrieval, Music Information Retrieval, Music Transcription, Retrieval

Music transcription, which deals with the conversion of music sources into a structured digital format, is a key problem for Music Information Retrieval (MIR). When addressing this challenge in computational terms, the MIR community follows two lines of research: music documents, which is the case of Optical Music Recognition (OMR), or audio recordings, which is the case of Automatic Music Transcription (AMT). The different nature of the aforementioned input data has conditioned these fields to develop modality-specific frameworks. However, their recent definition in terms of sequence labeling tasks leads to a common output representation, which enables research on a combined paradigm. In this respect, multimodal image and audio music transcription comprises the challenge of effectively combining the information conveyed by image and audio modalities. In this work, we explore this question at a late-fusion level: we study four combination approaches in order to merge, for the first time, the hypotheses regarding end-to-end OMR and AMT systems in a lattice-based search space. The results obtained for a series of performance scenarios -- in which the corresponding single-modality models yield different error rates -- showed interesting benefits of these approaches. In addition, two of the four strategies considered significantly improve the corresponding unimodal standard recognition frameworks.
Thermal to Visible Image Synthesis under Atmospheric Turbulence
Authors: Kangfu Mei, Yiqun Mei, Vishal M. Patel
Date: 2022-04-06 00:00:00
Tasks: Face Verification, Image Generation, Translation

In many practical applications of long-range imaging such as biometrics and surveillance, thermal imagining modalities are often used to capture images in low-light and nighttime conditions. However, such imaging systems often suffer from atmospheric turbulence, which introduces severe blur and deformation artifacts to the captured images. Such an issue is unavoidable in long-range imaging and significantly decreases the face verification accuracy. In this paper, we first investigate the problem with a turbulence simulation method on real-world thermal images. An end-to-end reconstruction method is then proposed which can directly transform thermal images into visible-spectrum images by utilizing natural image priors based on a pre-trained StyleGAN2 network. Compared with the existing two-steps methods of consecutive turbulence mitigation and thermal to visible image translation, our method is demonstrated to be effective in terms of both the visual quality of the reconstructed results and face verification accuracy. Moreover, to the best of our knowledge, this is the first work that studies the problem of thermal to visible image translation under atmospheric turbulence.
Fusing finetuned models for better pretraining
Authors: Leshem Choshen, Elad Venezian, Noam Slonim, Yoav Katz
Date: 2022-04-06 00:00:00

Pretrained models are the standard starting point for training. This approach consistently outperforms the use of a random initialization. However, pretraining is a costly endeavour that few can undertake. In this paper, we create better base models at hardly any cost, by fusing multiple existing fine tuned models into one. Specifically, we fuse by averaging the weights of these models. We show that the fused model results surpass the pretrained model ones. We also show that fusing is often better than intertraining. We find that fusing is less dependent on the target task. Furthermore, weight decay nullifies intertraining effects but not those of fusing.
DSGN++: Exploiting Visual-Spatial Relation for Stereo-based 3D Detectors
Authors: Yilun Chen, Shijia Huang, Shu Liu, Bei Yu, Jiaya Jia
Date: 2022-04-06 00:00:00
Tasks: 3D geometry, 3D Object Detection, 3D Object Detection From Stereo Images, cross-modal alignment, Relation

Camera-based 3D object detectors are welcome due to their wider deployment and lower price than LiDAR sensors. We first revisit the prior stereo detector DSGN for its stereo volume construction ways for representing both 3D geometry and semantics. We polish the stereo modeling and propose the advanced version, DSGN++, aiming to enhance effective information flow throughout the 2D-to-3D pipeline in three main aspects. First, to effectively lift the 2D information to stereo volume, we propose depth-wise plane sweeping (DPS) that allows denser connections and extracts depth-guided features. Second, for grasping differently spaced features, we present a novel stereo volume -- Dual-view Stereo Volume (DSV) that integrates front-view and top-view features and reconstructs sub-voxel depth in the camera frustum. Third, as the foreground region becomes less dominant in 3D space, we propose a multi-modal data editing strategy -- Stereo-LiDAR Copy-Paste, which ensures cross-modal alignment and improves data efficiency. Without bells and whistles, extensive experiments in various modality setups on the popular KITTI benchmark show that our method consistently outperforms other camera-based 3D detectors for all categories. Code is available at https://github.com/chenyilun95/DSGN2.