pwc-archive/papers-with-abstracts huggingface dataset:
STAIR: Learning Sparse Text and Image Representation in Grounded Tokens
Authors: Chen Chen, BoWen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexander Toshev, Jonathon Shlens, Ruoming Pang, Yinfei Yang
Date: 2023-01-30 00:00:00
Tasks: Information Retrieval, Retrieval, Text Retrieval

Image and text retrieval is one of the foundational tasks in the vision and language domain with multiple real-world applications. State-of-the-art approaches, e.g. CLIP, ALIGN, represent images and texts as dense embeddings and calculate the similarity in the dense embedding space as the matching score. On the other hand, sparse semantic features like bag-of-words models are more interpretable, but believed to suffer from inferior accuracy than dense representations. In this work, we show that it is possible to build a sparse semantic representation that is as powerful as, or even better than, dense presentations. We extend the CLIP model and build a sparse text and image representation (STAIR), where the image and text are mapped to a sparse token space. Each token in the space is a (sub-)word in the vocabulary, which is not only interpretable but also easy to integrate with existing information retrieval systems. STAIR model significantly outperforms a CLIP model with +$4.9\%$ and +$4.3\%$ absolute Recall@1 improvement on COCO-5k text$\rightarrow$image and image$\rightarrow$text retrieval respectively. It also achieved better performance on both of ImageNet zero-shot and linear probing compared to CLIP.
DELTA: degradation-free fully test-time adaptation
Authors: Bowen Zhao, Chen Chen, Shu-Tao Xia
Date: 2023-01-30 00:00:00
Tasks: Self-Learning, Test-time Adaptation

Fully test-time adaptation aims at adapting a pre-trained model to the test stream during real-time inference, which is urgently required when the test distribution differs from the training distribution. Several efforts have been devoted to improving adaptation performance. However, we find that two unfavorable defects are concealed in the prevalent adaptation methodologies like test-time batch normalization (BN) and self-learning. First, we reveal that the normalization statistics in test-time BN are completely affected by the currently received test samples, resulting in inaccurate estimates. Second, we show that during test-time adaptation, the parameter update is biased towards some dominant classes. In addition to the extensively studied test stream with independent and class-balanced samples, we further observe that the defects can be exacerbated in more complicated test environments, such as (time) dependent or class-imbalanced data. We observe that previous approaches work well in certain scenarios while show performance degradation in others due to their faults. In this paper, we provide a plug-in solution called DELTA for Degradation-freE fuLly Test-time Adaptation, which consists of two components: (i) Test-time Batch Renormalization (TBR), introduced to improve the estimated normalization statistics. (ii) Dynamic Online re-weighTing (DOT), designed to address the class bias within optimization. We investigate various test-time adaptation methods on three commonly used datasets with four scenarios, and a newly introduced real-world dataset. DELTA can help them deal with all scenarios simultaneously, leading to SOTA performance.
FedFA: Federated Feature Augmentation
Authors: Tianfei Zhou, Ender Konukoglu
Date: 2023-01-30 00:00:00
Tasks: Federated Learning

Federated learning is a distributed paradigm that allows multiple parties to collaboratively train deep models without exchanging the raw data. However, the data distribution among clients is naturally non-i.i.d., which leads to severe degradation of the learnt model. The primary goal of this paper is to develop a robust federated learning algorithm to address feature shift in clients' samples, which can be caused by various factors, e.g., acquisition differences in medical imaging. To reach this goal, we propose FedFA to tackle federated learning from a distinct perspective of federated feature augmentation. FedFA is based on a major insight that each client's data distribution can be characterized by statistics (i.e., mean and standard deviation) of latent features; and it is likely to manipulate these local statistics globally, i.e., based on information in the entire federation, to let clients have a better sense of the underlying distribution and therefore alleviate local data bias. Based on this insight, we propose to augment each local feature statistic probabilistically based on a normal distribution, whose mean is the original statistic and variance quantifies the augmentation scope. Key to our approach is the determination of a meaningful Gaussian variance, which is accomplished by taking into account not only biased data of each individual client, but also underlying feature statistics characterized by all participating clients. We offer both theoretical and empirical justifications to verify the effectiveness of FedFA. Our code is available at https://github.com/tfzhou/FedFA.
Human Vision Based 3D Point Cloud Semantic Segmentation of Large-Scale Outdoor Scene
Authors: Sunghwan Yoo, Yeongjeong Jeong, Maryam Jameela, Gunho Sohn
Date: 2023-01-30 00:00:00
Tasks: 3D Semantic Segmentation, Philosophy, Point Cloud Segmentation, Semantic Segmentation

This paper proposes EyeNet, a novel semantic segmentation network for point clouds that addresses the critical yet often overlooked parameter of coverage area size. Inspired by human peripheral vision, EyeNet overcomes the limitations of conventional networks by introducing a simple but efficient multi-contour input and a parallel processing network with connection blocks between parallel streams. The proposed approach effectively addresses the challenges of dense point clouds, as demonstrated by our ablation studies and state-of-the-art performance on Large-Scale Outdoor datasets.
Factors that affect Camera based Self-Monitoring of Vitals in the Wild
Authors: Nikhil S. Narayan, Shashanka B. R., Rohit Damodaran, Dr. Chandrashekhar Jayaram, Dr. M. A. Kareem, Dr. Mamta P., Dr. Saravanan K. R., Dr. Monu Krishnan, Dr. Raja Indana
Date: 2023-01-30 00:00:00
Tasks: Position

The reliability of the results of self monitoring of the vitals in the wild using medical devices or wearables or camera based smart phone solutions is subject to variabilities such as position of placement, hardware of the device and environmental factors. In this first of its kind study, we demonstrate that this variability in self monitoring of Blood Pressure (BP), Blood oxygen saturation level (SpO2) and Heart rate (HR) is statistically significant (p<0.05) on 203 healthy subjects by quantifying positional and hardware variability. We also establish the existence of this variability in camera based solutions for self-monitoring of vitals in smart phones and thus prove that the use of camera based smart phone solutions is similar to the use of medical devices or wearables for self-monitoring in the wild.
Data-driven soiling detection in PV modules
Authors: Alexandros Kalimeris, Ioannis Psarros, Giorgos Giannopoulos, Manolis Terrovitis, George Papastefanatos, Gregory Kotsis
Date: 2023-01-30 00:00:00
Tasks: Time Series, Time Series Analysis

Soiling is the accumulation of dirt in solar panels which leads to a decreasing trend in solar energy yield and may be the cause of vast revenue losses. The effect of soiling can be reduced by washing the panels, which is, however, a procedure of non-negligible cost. Moreover, soiling monitoring systems are often unreliable or very costly. We study the problem of estimating the soiling ratio in photo-voltaic (PV) modules, i.e., the ratio of the real power output to the power output that would be produced if solar panels were clean. A key advantage of our algorithms is that they estimate soiling, without needing to train on labelled data, i.e., periods of explicitly monitoring the soiling in each park, and without relying on generic analytical formulas which do not take into account the peculiarities of each installation. We consider as input a time series comprising a minimum set of measurements, that are available to most PV park operators. Our experimental evaluation shows that we significantly outperform current state-of-the-art methods for estimating soiling ratio.
PromptMix: Text-to-image diffusion models enhance the performance of lightweight networks
Authors: Arian Bakhtiarnia, Qi Zhang, Alexandros Iosifidis
Date: 2023-01-30 00:00:00
Tasks: Crowd Counting, Data Augmentation, Image Captioning, Synthetic Data Generation

Many deep learning tasks require annotations that are too time consuming for human operators, resulting in small dataset sizes. This is especially true for dense regression problems such as crowd counting which requires the location of every person in the image to be annotated. Techniques such as data augmentation and synthetic data generation based on simulations can help in such cases. In this paper, we introduce PromptMix, a method for artificially boosting the size of existing datasets, that can be used to improve the performance of lightweight networks. First, synthetic images are generated in an end-to-end data-driven manner, where text prompts are extracted from existing datasets via an image captioning deep network, and subsequently introduced to text-to-image diffusion models. The generated images are then annotated using one or more high-performing deep networks, and mixed with the real dataset for training the lightweight network. By extensive experiments on five datasets and two tasks, we show that PromptMix can significantly increase the performance of lightweight networks by up to 26%.
Half of an image is enough for quality assessment
Authors: Junyong You, Yuan Lin, Jari Korhonen
Date: 2023-01-30 00:00:00
Tasks: Image Quality Assessment

Deep networks have demonstrated promising results in the field of Image Quality Assessment (IQA). However, there has been limited research on understanding how deep models in IQA work. This study introduces a novel positional masked transformer for IQA and provides insights into the contribution of different regions of an image towards its overall quality. Results indicate that half of an image may play a trivial role in determining image quality, while the other half is critical. This observation is extended to several other CNN-based IQA models, revealing that half of the image regions can significantly impact the overall image quality. To further enhance our understanding, three semantic measures (saliency, frequency, and objectness) were derived and found to have high correlation with the importance of image regions in IQA.
M3FAS: An Accurate and Robust MultiModal Mobile Face Anti-Spoofing System
Authors: Chenqi Kong, Kexin Zheng, Yibing Liu, Shiqi Wang, Anderson Rocha, Haoliang Li
Date: 2023-01-30 00:00:00
Tasks: Face Anti-Spoofing, Face Recognition

Face presentation attacks (FPA), also known as face spoofing, have brought increasing concerns to the public through various malicious applications, such as financial fraud and privacy leakage. Therefore, safeguarding face recognition systems against FPA is of utmost importance. Although existing learning-based face anti-spoofing (FAS) models can achieve outstanding detection performance, they lack generalization capability and suffer significant performance drops in unforeseen environments. Many methodologies seek to use auxiliary modality data (e.g., depth and infrared maps) during the presentation attack detection (PAD) to address this limitation. However, these methods can be limited since (1) they require specific sensors such as depth and infrared cameras for data capture, which are rarely available on commodity mobile devices, and (2) they cannot work properly in practical scenarios when either modality is missing or of poor quality. In this paper, we devise an accurate and robust MultiModal Mobile Face Anti-Spoofing system named M3FAS to overcome the issues above. The primary innovation of this work lies in the following aspects: (1) To achieve robust PAD, our system combines visual and auditory modalities using three commonly available sensors: camera, speaker, and microphone; (2) We design a novel two-branch neural network with three hierarchical feature aggregation modules to perform cross-modal feature fusion; (3). We propose a multi-head training strategy, allowing the model to output predictions from the vision, acoustic, and fusion heads, resulting in a more flexible PAD. Extensive experiments have demonstrated the accuracy, robustness, and flexibility of M3FAS under various challenging experimental settings. The source code and dataset are available at: https://github.com/ChenqiKONG/M3FAS/
Eye Image-based Algorithms to Estimate Percentage Closure of Eye and Saccadic Ratio for Alertness Detection
Authors: Supratim Gupta
Date: 2023-01-30 00:00:00
Tasks: Position

The current research work has developed two novel algorithms for image-based measurement of Percentage Closure of Eyes-PERCLOS and Saccadic Ratio-SR. The PERCLOS is estimated by correlation filter-based technique. An innovative combination of gray scale and Near Infrared sensitive camera with passive NIR illuminator helps to achieve higher accuracy than the existing art. Two novel techniques have been developed for the detection of iris centre and eye corners. We propose an index called Form Factor to find the iris position. The saccadic velocity profile can be estimated from the temporal information of the iris positions using standard tracking algorithm such as Extended Kalman filter. Experimental results indicate that the estimation of both SR and PERCLOS can predict the level of alertness of an operator from onset of diminished alertness to fatigue.
Reliable Federated Disentangling Network for Non-IID Domain Feature
Authors: Meng Wang, Kai Yu, Chun-Mei Feng, Yiming Qian, Ke Zou, Lianyu Wang, Rick Siow Mong Goh, Yong liu, Huazhu Fu
Date: 2023-01-30 00:00:00
Tasks: Federated Learning

Federated learning (FL), as an effective decentralized distributed learning approach, enables multiple institutions to jointly train a model without sharing their local data. However, the domain feature shift caused by different acquisition devices/clients substantially degrades the performance of the FL model. Furthermore, most existing FL approaches aim to improve accuracy without considering reliability (e.g., confidence or uncertainty). The predictions are thus unreliable when deployed in safety-critical applications. Therefore, aiming at improving the performance of FL in non-Domain feature issues while enabling the model more reliable. In this paper, we propose a novel reliable federated disentangling network, termed RFedDis, which utilizes feature disentangling to enable the ability to capture the global domain-invariant cross-client representation and preserve local client-specific feature learning. Meanwhile, to effectively integrate the decoupled features, an uncertainty-aware decision fusion is also introduced to guide the network for dynamically integrating the decoupled features at the evidence level, while producing a reliable prediction with an estimated uncertainty. To the best of our knowledge, our proposed RFedDis is the first work to develop an FL approach based on evidential uncertainty combined with feature disentangling, which enhances the performance and reliability of FL in non-IID domain features. Extensive experimental results show that our proposed RFedDis provides outstanding performance with a high degree of reliability as compared to other state-of-the-art FL approaches.
Rendering the Directional TSDF for Tracking and Multi-Sensor Registration with Point-To-Plane Scale ICP
Authors: Malte Splietker, Sven Behnke
Date: 2023-01-30 00:00:00

Dense real-time tracking and mapping from RGB-D images is an important tool for many robotic applications, such as navigation and manipulation. The recently presented Directional Truncated Signed Distance Function (DTSDF) is an augmentation of the regular TSDF that shows potential for more coherent maps and improved tracking performance. In this work, we present methods for rendering depth- and color images from the DTSDF, making it a true drop-in replacement for the regular TSDF in established trackers. We evaluate the algorithm on well-established datasets and observe that our method improves tracking performance and increases re-usability of mapped scenes. Furthermore, we add color integration which notably improves color-correctness at adjacent surfaces. Our novel formulation of combined ICP with frame-to-keyframe photometric error minimization further improves tracking results. Lastly, we introduce Sim3 point-to-plane ICP for refining pose priors in a multi-sensor scenario with different scale factors.
RGB Arabic Alphabets Sign Language Dataset
Authors: Muhammad Al-Barham, Adham Alsharkawi, Musa Al-Yaman, Mohammad Al-Fetyani, Ashraf Elnagar, Ahmad Abu SaAleek, Mohammad Al-Odat
Date: 2023-01-30 00:00:00

This paper introduces the RGB Arabic Alphabet Sign Language (AASL) dataset. AASL comprises 7,856 raw and fully labelled RGB images of the Arabic sign language alphabets, which to our best knowledge is the first publicly available RGB dataset. The dataset is aimed to help those interested in developing real-life Arabic sign language classification models. AASL was collected from more than 200 participants and with different settings such as lighting, background, image orientation, image size, and image resolution. Experts in the field supervised, validated and filtered the collected images to ensure a high-quality dataset. AASL is made available to the public on Kaggle.
PointSmile: Point Self-supervised Learning via Curriculum Mutual Information
Authors: Xin Li, Mingqiang Wei, Songcan Chen
Date: 2023-01-30 00:00:00
Tasks: Data Augmentation, Self-Supervised Learning

Self-supervised learning is attracting wide attention in point cloud processing. However, it is still not well-solved to gain discriminative and transferable features of point clouds for efficient training on downstream tasks, due to their natural sparsity and irregularity. We propose PointSmile, a reconstruction-free self-supervised learning paradigm by maximizing curriculum mutual information (CMI) across the replicas of point cloud objects. From the perspective of how-and-what-to-learn, PointSmile is designed to imitate human curriculum learning, i.e., starting with an easy curriculum and gradually increasing the difficulty of that curriculum. To solve "how-to-learn", we introduce curriculum data augmentation (CDA) of point clouds. CDA encourages PointSmile to learn from easy samples to hard ones, such that the latent space can be dynamically affected to create better embeddings. To solve "what-to-learn", we propose to maximize both feature- and class-wise CMI, for better extracting discriminative features of point clouds. Unlike most of existing methods, PointSmile does not require a pretext task, nor does it require cross-modal data to yield rich latent representations. We demonstrate the effectiveness and robustness of PointSmile in downstream tasks including object classification and segmentation. Extensive results show that our PointSmile outperforms existing self-supervised methods, and compares favorably with popular fully-supervised methods on various standard architectures.
FractalAD: A simple industrial anomaly detection method using fractal anomaly generation and backbone knowledge distillation
Authors: Xuan Xia, Weijie Lv, Xing He, Nan Li, Chuanqi Liu, Ning Ding
Date: 2023-01-30 00:00:00
Tasks: Anomaly Detection, Knowledge Distillation, Semantic Segmentation

Although industrial anomaly detection (AD) technology has made significant progress in recent years, generating realistic anomalies and learning priors of normal remain challenging tasks. In this study, we propose an end-to-end industrial anomaly detection method called FractalAD. Training samples are obtained by synthesizing fractal images and patches from normal samples. This fractal anomaly generation method is designed to sample the full morphology of anomalies. Moreover, we designed a backbone knowledge distillation structure to extract prior knowledge contained in normal samples. The differences between a teacher and a student model are converted into anomaly attention using a cosine similarity attention module. The proposed method enables an end-to-end semantic segmentation network to be used for anomaly detection without adding any trainable parameters to the backbone and segmentation head, and has obvious advantages over other methods in training and inference speed.. The results of ablation studies confirmed the effectiveness of fractal anomaly generation and backbone knowledge distillation. The results of performance experiments showed that FractalAD achieved competitive results on the MVTec AD dataset and MVTec 3D-AD dataset compared with other state-of-the-art anomaly detection methods.
Robust Meta Learning for Image based tasks
Authors: Penghao Jiang, Xin Ke, Zifeng Wang, Chunxi Li
Date: 2023-01-30 00:00:00
Tasks: Meta-Learning

A machine learning model that generalizes well should obtain low errors on unseen test examples. Thus, if we learn an optimal model in training data, it could have better generalization performance in testing tasks. However, learning such a model is not possible in standard machine learning frameworks as the distribution of the test data is unknown. To tackle this challenge, we propose a novel robust meta-learning method, which is more robust to the image-based testing tasks which is unknown and has distribution shifts with training tasks. Our robust meta-learning method can provide robust optimal models even when data from each distribution are scarce. In experiments, we demonstrate that our algorithm not only has better generalization performance but also robust to different unknown testing tasks.
Edge-guided Multi-domain RGB-to-TIR image Translation for Training Vision Tasks with Challenging Labels
Authors: Dong-Guw Lee, Myung-Hwan Jeon, Younggun Cho, Ayoung Kim
Date: 2023-01-30 00:00:00
Tasks: object-detection, Object Detection, Optical Flow Estimation, Translation

The insufficient number of annotated thermal infrared (TIR) image datasets not only hinders TIR image-based deep learning networks to have comparable performances to that of RGB but it also limits the supervised learning of TIR image-based tasks with challenging labels. As a remedy, we propose a modified multidomain RGB to TIR image translation model focused on edge preservation to employ annotated RGB images with challenging labels. Our proposed method not only preserves key details in the original image but also leverages the optimal TIR style code to portray accurate TIR characteristics in the translated image, when applied on both synthetic and real world RGB images. Using our translation model, we have enabled the supervised learning of deep TIR image-based optical flow estimation and object detection that ameliorated in deep TIR optical flow estimation by reduction in end point error by 56.5\% on average and the best object detection mAP of 23.9\% respectively. Our code and supplementary materials are available at https://github.com/rpmsnu/sRGB-TIR.