pwc-archive/papers-with-abstracts huggingface dataset:
Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition
Authors: Youngjoon Jang, Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, Joon Son Chung, In So Kweon
Date: 2022-11-01 00:00:00
Tasks: Benchmarking, Disentanglement, Sign Language Recognition

The goal of this work is background-robust continuous sign language recognition. Most existing Continuous Sign Language Recognition (CSLR) benchmarks have fixed backgrounds and are filmed in studios with a static monochromatic background. However, signing is not limited only to studios in the real world. In order to analyze the robustness of CSLR models under background shifts, we first evaluate existing state-of-the-art CSLR models on diverse backgrounds. To synthesize the sign videos with a variety of backgrounds, we propose a pipeline to automatically generate a benchmark dataset utilizing existing CSLR benchmarks. Our newly constructed benchmark dataset consists of diverse scenes to simulate a real-world environment. We observe even the most recent CSLR method cannot recognize glosses well on our new dataset with changed backgrounds. In this regard, we also propose a simple yet effective training scheme including (1) background randomization and (2) feature disentanglement for CSLR models. The experimental results on our dataset demonstrate that our method generalizes well to other unseen background data with minimal additional training images.
A new filter for dimensionality reduction and classification of hyperspectral images using GLCM features and mutual information
Authors: Hasna Nhaila, Elkebir Sarhrouni, Ahmed Hammouch
Date: 2022-11-01 00:00:00
Tasks: Classification, Classification Of Hyperspectral Images, Dimensionality Reduction, feature selection

Dimensionality reduction is an important preprocessing step of the hyperspectral images classification (HSI), it is inevitable task. Some methods use feature selection or extraction algorithms based on spectral and spatial information. In this paper, we introduce a new methodology for dimensionality reduction and classification of HSI taking into account both spectral and spatial information based on mutual information. We characterise the spatial information by the texture features extracted from the grey level cooccurrence matrix (GLCM); we use Homogeneity, Contrast, Correlation and Energy. For classification, we use support vector machine (SVM). The experiments are performed on three well-known hyperspectral benchmark datasets. The proposed algorithm is compared with the state of the art methods. The obtained results of this fusion show that our method outperforms the other approaches by increasing the classification accuracy in a good timing. This method may be improved for more performance Keywords: hyperspectral images; classification; spectral and spatial features; grey level cooccurrence matrix; GLCM; mutual information; support vector machine; SVM.
Oracle-guided Contrastive Clustering
Authors: Mengdie Wang, Liyuan Shang, Suyun Zhao, Yiming Wang, Hong Chen, Cuiping Li, XiZhao Wang
Date: 2022-11-01 00:00:00
Tasks: Active Learning, Clustering, Contrastive Learning, Deep Clustering

Deep clustering aims to learn a clustering representation through deep architectures. Most of the existing methods usually conduct clustering with the unique goal of maximizing clustering performance, that ignores the personalized demand of clustering tasks.% and results in unguided clustering solutions. However, in real scenarios, oracles may tend to cluster unlabeled data by exploiting distinct criteria, such as distinct semantics (background, color, object, etc.), and then put forward personalized clustering tasks. To achieve task-aware clustering results, in this study, Oracle-guided Contrastive Clustering(OCC) is then proposed to cluster by interactively making pairwise ``same-cluster" queries to oracles with distinctive demands. Specifically, inspired by active learning, some informative instance pairs are queried, and evaluated by oracles whether the pairs are in the same cluster according to their desired orientation. And then these queried same-cluster pairs extend the set of positive instance pairs for contrastive learning, guiding OCC to extract orientation-aware feature representation. Accordingly, the query results, guided by oracles with distinctive demands, may drive the OCC's clustering results in a desired orientation. Theoretically, the clustering risk in an active learning manner is given with a tighter upper bound, that guarantees active queries to oracles do mitigate the clustering risk. Experimentally, extensive results verify that OCC can cluster accurately along the specific orientation and it substantially outperforms the SOTA clustering methods as well. To the best of our knowledge, it is the first deep framework to perform personalized clustering.
Galaxy classification: a deep learning approach for classifying Sloan Digital Sky Survey images
Authors: Sarvesh Gharat, Yogesh Dandawate
Date: 2022-11-01 00:00:00
Tasks: Classification

In recent decades, large-scale sky surveys such as Sloan Digital Sky Survey (SDSS) have resulted in generation of tremendous amount of data. The classification of this enormous amount of data by astronomers is time consuming. To simplify this process, in 2007 a volunteer-based citizen science project called Galaxy Zoo was introduced, which has reduced the time for classification by a good extent. However, in this modern era of deep learning, automating this classification task is highly beneficial as it reduces the time for classification. For the last few years, many algorithms have been proposed which happen to do a phenomenal job in classifying galaxies into multiple classes. But all these algorithms tend to classify galaxies into less than six classes. However, after considering the minute information which we know about galaxies, it is necessary to classify galaxies into more than eight classes. In this study, a neural network model is proposed so as to classify SDSS data into 10 classes from an extended Hubble Tuning Fork. Great care is given to disc edge and disc face galaxies, distinguishing between a variety of substructures and minute features which are associated with each class. The proposed model consists of convolution layers to extract features making this method fully automatic. The achieved test accuracy is 84.73 per cent which happens to be promising after considering such minute details in classes. Along with convolution layers, the proposed model has three more layers responsible for classification, which makes the algorithm consume less time.
Expansion of Visual Hints for Improved Generalization in Stereo Matching
Authors: Andrea Pilzer, Yuxin Hou, Niki Loppi, Arno Solin, Juho Kannala
Date: 2022-11-01 00:00:00
Tasks: Stereo Matching, Visual Odometry

We introduce visual hints expansion for guiding stereo matching to improve generalization. Our work is motivated by the robustness of Visual Inertial Odometry (VIO) in computer vision and robotics, where a sparse and unevenly distributed set of feature points characterizes a scene. To improve stereo matching, we propose to elevate 2D hints to 3D points. These sparse and unevenly distributed 3D visual hints are expanded using a 3D random geometric graph, which enhances the learning and inference process. We evaluate our proposal on multiple widely adopted benchmarks and show improved performance without access to additional sensors other than the image sequence. To highlight practical applicability and symbiosis with visual odometry, we demonstrate how our methods run on embedded hardware.
Seg&Struct: The Interplay Between Part Segmentation and Structure Inference for 3D Shape Parsing
Authors: Jeonghyun Kim, Kaichun Mo, Minhyuk Sung, Woontack Woo
Date: 2022-11-01 00:00:00
Tasks: Segmentation

We propose Seg&Struct, a supervised learning framework leveraging the interplay between part segmentation and structure inference and demonstrating their synergy in an integrated framework. Both part segmentation and structure inference have been extensively studied in the recent deep learning literature, while the supervisions used for each task have not been fully exploited to assist the other task. Namely, structure inference has been typically conducted with an autoencoder that does not leverage the point-to-part associations. Also, segmentation has been mostly performed without structural priors that tell the plausibility of the output segments. We present how these two tasks can be best combined while fully utilizing supervision to improve performance. Our framework first decomposes a raw input shape into part segments using an off-the-shelf algorithm, whose outputs are then mapped to nodes in a part hierarchy, establishing point-to-part associations. Following this, ours predicts the structural information, e.g., part bounding boxes and part relationships. Lastly, the segmentation is rectified by examining the confusion of part boundaries using the structure-based part features. Our experimental results based on the StructureNet and PartNet demonstrate that the interplay between the two tasks results in remarkable improvements in both tasks: 27.91% in structure inference and 0.5% in segmentation.
combined digital drone camera and optical channel parameters for air surveillance
Authors: Wamidh Jalil Mazher, hadeel Tariq Ibrahim
Date: 2022-11-01 00:00:00

Digital drone cameras with free-space optical (FSO) communication networks have been proposed to be promising for air surveillance. In the FSO channel, atmospheric turbulence (AT) degrades the signal. In this study, we combined the parameters of the digital drone camera and the optical channel to mitigate the AT effect. The digital drone camera parameters are indicated by the field of view and camera object distance to support this proposal. Meanwhile, the optical channel parameters, rather than the altitude, are denoted by the most critical parameter, which is the refractive index structure parameter used to characterize the effects of AT. Consequently, two lemmas are proposed and combined to present the optimum relationship between the digital drone camera and optical channel parameters. Therefore, the quality of the entire air surveillance system with a digital drone camera FSO is significantly improved. Furthermore, the analysis and optimization for practical cases were applied to support our findings. Finally, our results demonstrated that an impressive performance improvement of an air surveillance system of 17 dB is possible compared without optimization by combining digital drone camera and FSO parameters at a target outage transceiver probability of $10^-6$.
Universal Perturbation Attack on Differentiable No-Reference Image- and Video-Quality Metrics
Authors: Ekaterina Shumitskaya, Anastasia Antsiferova, Dmitriy Vatolin
Date: 2022-11-01 00:00:00
Tasks: Adversarial Attack, No-Reference Image Quality Assessment

Universal adversarial perturbation attacks are widely used to analyze image classifiers that employ convolutional neural networks. Nowadays, some attacks can deceive image- and video-quality metrics. So sustainability analysis of these metrics is important. Indeed, if an attack can confuse the metric, an attacker can easily increase quality scores. When developers of image- and video-algorithms can boost their scores through detached processing, algorithm comparisons are no longer fair. Inspired by the idea of universal adversarial perturbation for classifiers, we suggest a new method to attack differentiable no-reference quality metrics through universal perturbation. We applied this method to seven no-reference image- and video-quality metrics (PaQ-2-PiQ, Linearity, VSFA, MDTVSFA, KonCept512, Nima and SPAQ). For each one, we trained a universal perturbation that increases the respective scores. We also propose a method for assessing metric stability and identify the metrics that are the most vulnerable and the most resistant to our attack. The existence of successful universal perturbations appears to diminish the metric's ability to provide reliable scores. We therefore recommend our proposed method as an additional verification of metric reliability to complement traditional subjective tests and benchmarks.
Siamese Transition Masked Autoencoders as Uniform Unsupervised Visual Anomaly Detector
Authors: Haiming Yao, Xue Wang, Wenyong Yu
Date: 2022-11-01 00:00:00
Tasks: Anomaly Detection, Decoder

Unsupervised visual anomaly detection conveys practical significance in many scenarios and is a challenging task due to the unbounded definition of anomalies. Moreover, most previous methods are application-specific, and establishing a unified model for anomalies across application scenarios remains unsolved. This paper proposes a novel hybrid framework termed Siamese Transition Masked Autoencoders(ST-MAE) to handle various visual anomaly detection tasks uniformly via deep feature transition. Concretely, the proposed method first extracts hierarchical semantics features from a pre-trained deep convolutional neural network and then develops a feature decoupling strategy to split the deep features into two disjoint feature patch subsets. Leveraging the decoupled features, the ST-MAE is developed with the Siamese encoders that operate on each subset of feature patches and perform the latent representations transition of two subsets, along with a lightweight decoder that reconstructs the original feature from the transitioned latent representation. Finally, the anomalous attributes can be detected using the semantic deep feature residual. Our deep feature transition scheme yields a nontrivial and semantic self-supervisory task to extract prototypical normal patterns, which allows for learning uniform models that generalize well for different visual anomaly detection tasks. The extensive experiments conducted demonstrate that the proposed ST-MAE method can advance state-of-the-art performance on multiple benchmarks across application scenarios with a superior inference efficiency, which exhibits great potential to be the uniform model for unsupervised visual anomaly detection.
RGMIM: Region-Guided Masked Image Modeling for Learning Meaningful Representations from X-Ray Images
Authors: Guang Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
Date: 2022-11-01 00:00:00
Tasks: Self-Supervised Learning, valid

In this study, we propose a novel method called region-guided masked image modeling (RGMIM) for learning meaningful representations from X-ray images. Our method adopts a new masking strategy that utilizes organ mask information to identify valid regions for learning more meaningful representations. We conduct quantitative evaluations on an open lung X-ray image dataset as well as masking ratio hyperparameter studies. When using the entire training set, RGMIM outperformed other comparable methods, achieving a 0.962 lung disease detection accuracy. Specifically, RGMIM significantly improved performance in small data volumes, such as 5% and 10% of the training set compared to other methods. RGMIM can mask more valid regions, facilitating the learning of discriminative representations and the subsequent high-accuracy lung disease detection. RGMIM outperforms other state-of-the-art self-supervised learning methods in experiments, particularly when limited training data is used.
HDNet: Hierarchical Dynamic Network for Gait Recognition using Millimeter-Wave Radar
Authors: Yanyan Huang, Yong Wang, Kun Shi, Chaojie Gu, Yu Fu, Cheng Zhuo, Zhiguo Shi
Date: 2022-11-01 00:00:00
Tasks: Gait Recognition

Gait recognition is widely used in diversified practical applications. Currently, the most prevalent approach is to recognize human gait from RGB images, owing to the progress of computer vision technologies. Nevertheless, the perception capability of RGB cameras deteriorates in rough circumstances, and visual surveillance may cause privacy invasion. Due to the robustness and non-invasive feature of millimeter wave (mmWave) radar, radar-based gait recognition has attracted increasing attention in recent years. In this research, we propose a Hierarchical Dynamic Network (HDNet) for gait recognition using mmWave radar. In order to explore more dynamic information, we propose point flow as a novel point clouds descriptor. We also devise a dynamic frame sampling module to promote the efficiency of computation without deteriorating performance noticeably. To prove the superiority of our methods, we perform extensive experiments on two public mmWave radar-based gait recognition datasets, and the results demonstrate that our model is superior to existing state-of-the-art methods.
Exploring Structure-Wise Uncertainty for 3D Medical Image Segmentation
Authors: Anton Vasiliuk, Daria Frolova, Mikhail Belyaev, Boris Shirokikh
Date: 2022-11-01 00:00:00
Tasks: Image Segmentation, Medical Image Segmentation, Out of Distribution (OOD) Detection, Segmentation, Semantic Segmentation, Tumor Segmentation

When applying a Deep Learning model to medical images, it is crucial to estimate the model uncertainty. Voxel-wise uncertainty is a useful visual marker for human experts and could be used to improve the model's voxel-wise output, such as segmentation. Moreover, uncertainty provides a solid foundation for out-of-distribution (OOD) detection, improving the model performance on the image-wise level. However, one of the frequent tasks in medical imaging is the segmentation of distinct, local structures such as tumors or lesions. Here, the structure-wise uncertainty allows more precise operations than image-wise and more semantic-aware than voxel-wise. The way to produce uncertainty for individual structures remains poorly explored. We propose a framework to measure the structure-wise uncertainty and evaluate the impact of OOD data on the model performance. Thus, we identify the best UE method to improve the segmentation quality. The proposed framework is tested on three datasets with the tumor segmentation task: LIDC-IDRI, LiTS, and a private one with multiple brain metastases cases.
Self-supervised Character-to-Character Distillation for Text Recognition
Authors: Tongkun Guan, Wei Shen, Xue Yang, Qi Feng, Zekun Jiang, Xiaokang Yang
Date: 2022-11-01 00:00:00
Proceeding: ICCV 2023 1
Tasks: Data Augmentation, Representation Learning, Scene Text Recognition, Self-Learning, Self-Supervised Learning, self-supervised scene text recognition, SSIM, Super-Resolution, Text Segmentation

When handling complicated text images (e.g., irregular structures, low resolution, heavy occlusion, and uneven illumination), existing supervised text recognition methods are data-hungry. Although these methods employ large-scale synthetic text images to reduce the dependence on annotated real images, the domain gap still limits the recognition performance. Therefore, exploring the robust text feature representations on unlabeled real images by self-supervised learning is a good solution. However, existing self-supervised text recognition methods conduct sequence-to-sequence representation learning by roughly splitting the visual features along the horizontal axis, which limits the flexibility of the augmentations, as large geometric-based augmentations may lead to sequence-to-sequence feature inconsistency. Motivated by this, we propose a novel self-supervised Character-to-Character Distillation method, CCD, which enables versatile augmentations to facilitate general text representation learning. Specifically, we delineate the character structures of unlabeled real images by designing a self-supervised character segmentation module. Following this, CCD easily enriches the diversity of local characters while keeping their pairwise alignment under flexible augmentations, using the transformation matrix between two augmented views from images. Experiments demonstrate that CCD achieves state-of-the-art results, with average performance gains of 1.38% in text recognition, 1.7% in text segmentation, 0.24 dB (PSNR) and 0.0321 (SSIM) in text super-resolution. Code is available at https://github.com/TongkunGuan/CCD.
Training Vision-Language Models with Less Bimodal Supervision
Authors: Elad Segal, Ben Bogin, Jonathan Berant
Date: 2022-11-01 00:00:00
Tasks: Language Modelling

Standard practice in pretraining multimodal models, such as vision-language models, is to rely on pairs of aligned inputs from both modalities, for example, aligned image-text pairs. However, such pairs can be difficult to obtain in low-resource settings and for some modality pairs (e.g., structured tables and images). In this work, we investigate the extent to which we can reduce the reliance on such parallel data, which we term \emph{bimodal supervision}, and use models that are pretrained on each modality independently. We experiment with a high-performing vision-language model, and analyze the effect of bimodal supervision on three vision-language tasks. We find that on simpler tasks, such as VQAv2 and GQA, one can eliminate bimodal supervision completely, suffering only a minor loss in performance. Conversely, for NLVR2, which requires more complex reasoning, training without bimodal supervision leads to random performance. Nevertheless, using only 5\% of the bimodal data (142K images along with their captions), or leveraging weak supervision in the form of a list of machine-generated labels for each image, leads to only a moderate degradation compared to using 3M image-text pairs: 74\%$\rightarrow$$\sim$70\%. Our code is available at https://github.com/eladsegal/less-bimodal-sup.
Angular upsampling in diffusion MRI using contextual HemiHex sub-sampling in q-space
Authors: Abrar Faiyaz, Md Nasir Uddin, Giovanni Schifitto
Date: 2022-11-01 00:00:00
Tasks: Diffusion  MRI, regression

Artificial Intelligence (Deep Learning(DL)/ Machine Learning(ML)) techniques are widely being used to address and overcome all kinds of ill-posed problems in medical imaging which was or in fact is seemingly impossible. Reducing gradient directions but harnessing high angular resolution(HAR) diffusion data in MR that retains clinical features is an important and challenging problem in the field. While the DL/ML approaches are promising, it is important to incorporate relevant context for the data to ensure that maximum prior information is provided for the AI model to infer the posterior. In this paper, we introduce HemiHex (HH) subsampling to suggestively address training data sampling on q-space geometry, followed by a nearest neighbor regression training on the HH-samples to finally upsample the dMRI data. Earlier studies has tried to use regression for up-sampling dMRI data but yields performance issues as it fails to provide structured geometrical measures for inference. Our proposed approach is a geometrically optimized regression technique which infers the unknown q-space thus addressing the limitations in the earlier studies.
Pixel-Wise Contrastive Distillation
Authors: Junqiang Huang, Zichao Guo
Date: 2022-11-01 00:00:00
Proceeding: ICCV 2023 1
Tasks: Prediction

We present a simple but effective pixel-level self-supervised distillation framework friendly to dense prediction tasks. Our method, called Pixel-Wise Contrastive Distillation (PCD), distills knowledge by attracting the corresponding pixels from student's and teacher's output feature maps. PCD includes a novel design called SpatialAdaptor which ``reshapes'' a part of the teacher network while preserving the distribution of its output features. Our ablation experiments suggest that this reshaping behavior enables more informative pixel-to-pixel distillation. Moreover, we utilize a plug-in multi-head self-attention module that explicitly relates the pixels of student's feature maps to enhance the effective receptive field, leading to a more competitive student. PCD \textbf{outperforms} previous self-supervised distillation methods on various dense prediction tasks. A backbone of \mbox{ResNet-18-FPN} distilled by PCD achieves $37.4$ AP$^\text{bbox}$ and $34.0$ AP$^\text{mask}$ on COCO dataset using the detector of \mbox{Mask R-CNN}. We hope our study will inspire future research on how to pre-train a small model friendly to dense prediction tasks in a self-supervised fashion.