pwc-archive/papers-with-abstracts huggingface dataset:
A Generic Image Retrieval Method for Date Estimation of Historical Document Collections
Authors: Adrià Molina, Lluis Gomez, Oriol Ramos Terrades, Josep Lladós
Date: 2022-04-08 00:00:00
Tasks: Image Retrieval, Retrieval

Date estimation of historical document images is a challenging problem, with several contributions in the literature that lack of the ability to generalize from one dataset to others. This paper presents a robust date estimation system based in a retrieval approach that generalizes well in front of heterogeneous collections. we use a ranking loss function named smooth-nDCG to train a Convolutional Neural Network that learns an ordination of documents for each problem. One of the main usages of the presented approach is as a tool for historical contextual retrieval. It means that scholars could perform comparative analysis of historical images from big datasets in terms of the period where they were produced. We provide experimental evaluation on different types of documents from real datasets of manuscript and newspaper images.
Engagement Detection with Multi-Task Training in E-Learning Environments
Authors: Onur Copur, Mert Nakıp, Simone Scardapane, Jürgen Slowack
Date: 2022-04-08 00:00:00
Tasks: Emotion Recognition, Triplet

Recognition of user interaction, in particular engagement detection, became highly crucial for online working and learning environments, especially during the COVID-19 outbreak. Such recognition and detection systems significantly improve the user experience and efficiency by providing valuable feedback. In this paper, we propose a novel Engagement Detection with Multi-Task Training (ED-MTT) system which minimizes mean squared error and triplet loss together to determine the engagement level of students in an e-learning environment. The performance of this system is evaluated and compared against the state-of-the-art on a publicly available dataset as well as videos collected from real-life scenarios. The results show that ED-MTT achieves 6% lower MSE than the best state-of-the-art performance with highly acceptable training time and lightweight feature extraction.
Multimodal Quasi-AutoRegression: Forecasting the visual popularity of new fashion products
Authors: Stefanos I. Papadopoulos, Christos Koutlis, Symeon Papadopoulos, Ioannis Kompatsiaris
Date: 2022-04-08 00:00:00
Tasks: Image Captioning, image-classification, Image Classification, New Product Sales Forecasting, Popularity Forecasting, Time Series, Time Series Analysis

Estimating the preferences of consumers is of utmost importance for the fashion industry as appropriately leveraging this information can be beneficial in terms of profit. Trend detection in fashion is a challenging task due to the fast pace of change in the fashion industry. Moreover, forecasting the visual popularity of new garment designs is even more demanding due to lack of historical data. To this end, we propose MuQAR, a Multimodal Quasi-AutoRegressive deep learning architecture that combines two modules: (1) a multi-modal multi-layer perceptron processing categorical, visual and textual features of the product and (2) a quasi-autoregressive neural network modelling the "target" time series of the product's attributes along with the "exogenous" time series of all other attributes. We utilize computer vision, image classification and image captioning, for automatically extracting visual features and textual descriptions from the images of new products. Product design in fashion is initially expressed visually and these features represent the products' unique characteristics without interfering with the creative process of its designers by requiring additional inputs (e.g manually written texts). We employ the product's target attributes time series as a proxy of temporal popularity patterns, mitigating the lack of historical data, while exogenous time series help capture trends among interrelated attributes. We perform an extensive ablation analysis on two large scale image fashion datasets, Mallzee and SHIFT15m to assess the adequacy of MuQAR and also use the Amazon Reviews: Home and Kitchen dataset to assess generalisability to other domains. A comparative study on the VISUELLE dataset, shows that MuQAR is capable of competing and surpassing the domain's current state of the art by 4.65% and 4.8% in terms of WAPE and MAE respectively.
Points to Patches: Enabling the Use of Self-Attention for 3D Shape Recognition
Authors: Axel Berg, Magnus Oskarsson, Mark O'Connor
Date: 2022-04-08 00:00:00
Tasks: 3D Feature Matching, 3D Point Cloud Classification, 3D Shape Recognition, Point Cloud Registration

While the Transformer architecture has become ubiquitous in the machine learning field, its adaptation to 3D shape recognition is non-trivial. Due to its quadratic computational complexity, the self-attention operator quickly becomes inefficient as the set of input points grows larger. Furthermore, we find that the attention mechanism struggles to find useful connections between individual points on a global scale. In order to alleviate these problems, we propose a two-stage Point Transformer-in-Transformer (Point-TnT) approach which combines local and global attention mechanisms, enabling both individual points and patches of points to attend to each other effectively. Experiments on shape classification show that such an approach provides more useful features for downstream tasks than the baseline Transformer, while also being more computationally efficient. In addition, we also extend our method to feature matching for scene reconstruction, showing that it can be used in conjunction with existing scene reconstruction pipelines.
Probabilistic Representations for Video Contrastive Learning
Authors: Jungin Park, Jiyoung Lee, Ig-Jae Kim, Kwanghoon Sohn
Date: 2022-04-08 00:00:00
Proceeding: CVPR 2022 1
Tasks: Action Recognition, Contrastive Learning, Representation Learning, Retrieval, Video Retrieval

This paper presents Probabilistic Video Contrastive Learning, a self-supervised representation learning method that bridges contrastive learning with probabilistic representation. We hypothesize that the clips composing the video have different distributions in short-term duration, but can represent the complicated and sophisticated video distribution through combination in a common embedding space. Thus, the proposed method represents video clips as normal distributions and combines them into a Mixture of Gaussians to model the whole video distribution. By sampling embeddings from the whole video distribution, we can circumvent the careful sampling strategy or transformations to generate augmented views of the clips, unlike previous deterministic methods that have mainly focused on such sample generation strategies for contrastive learning. We further propose a stochastic contrastive loss to learn proper video distributions and handle the inherent uncertainty from the nature of the raw video. Experimental results verify that our probabilistic embedding stands as a state-of-the-art video representation learning for action recognition and video retrieval on the most popular benchmarks, including UCF101 and HMDB51.
Study of a committee of neural networks for biometric hand-geometry recognition
Authors: Marcos Faundez-Zanuy
Date: 2022-04-08 00:00:00

This Paper studies different committees of neural networks for biometric pattern recognition. We use the neural nets as classifiers for identification and verification purposes. We show that a committee of nets can improve the recognition rates when compared with a multi-start initialization algo-rithm that just picks up the neural net which offers the best performance. On the other hand, we found that there is no strong correlation between identifi-cation and verification applications using the same classifier.
Does Robustness on ImageNet Transfer to Downstream Tasks?
Authors: Yutaro Yamada, Mayu Otani
Date: 2022-04-08 00:00:00
Proceeding: CVPR 2022 1
Tasks: Classification, image-classification, Image Classification, object-detection, Object Detection, Segmentation, Semantic Segmentation, Transfer Learning

As clean ImageNet accuracy nears its ceiling, the research community is increasingly more concerned about robust accuracy under distributional shifts. While a variety of methods have been proposed to robustify neural networks, these techniques often target models trained on ImageNet classification. At the same time, it is a common practice to use ImageNet pretrained backbones for downstream tasks such as object detection, semantic segmentation, and image classification from different domains. This raises a question: Can these robust image classifiers transfer robustness to downstream tasks? For object detection and semantic segmentation, we find that a vanilla Swin Transformer, a variant of Vision Transformer tailored for dense prediction tasks, transfers robustness better than Convolutional Neural Networks that are trained to be robust to the corrupted version of ImageNet. For CIFAR10 classification, we find that models that are robustified for ImageNet do not retain robustness when fully fine-tuned. These findings suggest that current robustification techniques tend to emphasize ImageNet evaluations. Moreover, network architecture is a strong source of robustness when we consider transfer learning.
Deep Hyperspectral-Depth Reconstruction Using Single Color-Dot Projection
Authors: Chunyu Li, Yusuke Monno, Masatoshi Okutomi
Date: 2022-04-08 00:00:00
Proceeding: CVPR 2022 1

Depth reconstruction and hyperspectral reflectance reconstruction are two active research topics in computer vision and image processing. Conventionally, these two topics have been studied separately using independent imaging setups and there is no existing method which can acquire depth and spectral reflectance simultaneously in one shot without using special hardware. In this paper, we propose a novel single-shot hyperspectral-depth reconstruction method using an off-the-shelf RGB camera and projector. Our method is based on a single color-dot projection, which simultaneously acts as structured light for depth reconstruction and spatially-varying color illuminations for hyperspectral reflectance reconstruction. To jointly reconstruct the depth and the hyperspectral reflectance from a single color-dot image, we propose a novel end-to-end network architecture that effectively incorporates a geometric color-dot pattern loss and a photometric hyperspectral reflectance loss. Through the experiments, we demonstrate that our hyperspectral-depth reconstruction method outperforms the combination of an existing state-of-the-art single-shot hyperspectral reflectance reconstruction method and depth reconstruction method.
Biometric identification by means of hand geometry and a neural net classifier
Authors: Marcos Faundez-Zanuy, Guillermo Mar Navarro Mérida
Date: 2022-04-08 00:00:00

This Paper describes a hand geometry biometric identification system. We have acquired a database of 22 people using a conventional document scanner. The experimental section consists of a study about the discrimination capability of different extracted features, and the identification rate using different classifiers based on neural networks.
A Survey of Supernet Optimization and its Applications: Spatial and Temporal Optimization for Neural Architecture Search
Authors: Stephen Cha, Taehyeon Kim, Hayeon Lee, Se-Young Yun
Date: 2022-04-08 00:00:00
Tasks: Domain Generalization, Image Classification, Neural Architecture Search, object-detection, Object Detection, Semantic Segmentation, Survey

This survey focuses on categorizing and evaluating the methods of supernet optimization in the field of Neural Architecture Search (NAS). Supernet optimization involves training a single, over-parameterized network that encompasses the search space of all possible network architectures. The survey analyses supernet optimization methods based on their approaches to spatial and temporal optimization. Spatial optimization relates to optimizing the architecture and parameters of the supernet and its subnets, while temporal optimization deals with improving the efficiency of selecting architectures from the supernet. The benefits, limitations, and potential applications of these methods in various tasks and settings, including transferability, domain generalization, and Transformer models, are also discussed.
CD$^2$-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learning
Authors: Yiqing Shen, Yuyin Zhou, Lequan Yu
Date: 2022-04-08 00:00:00
Tasks: Federated Learning, Medical Image Analysis

Federated learning (FL) is a distributed learning paradigm that enables multiple clients to collaboratively learn a shared global model. Despite the recent progress, it remains challenging to deal with heterogeneous data clients, as the discrepant data distributions usually prevent the global model from delivering good generalization ability on each participating client. In this paper, we propose CD^2-pFed, a novel Cyclic Distillation-guided Channel Decoupling framework, to personalize the global model in FL, under various settings of data heterogeneity. Different from previous works which establish layer-wise personalization to overcome the non-IID data across different clients, we make the first attempt at channel-wise assignment for model personalization, referred to as channel decoupling. To further facilitate the collaboration between private and shared weights, we propose a novel cyclic distillation scheme to impose a consistent regularization between the local and global model representations during the federation. Guided by the cyclical distillation, our channel decoupling framework can deliver more accurate and generalized results for different kinds of heterogeneity, such as feature skew, label distribution skew, and concept shift. Comprehensive experiments on four benchmarks, including natural image and medical image analysis tasks, demonstrate the consistent effectiveness of our method on both local and external validations.
Spatial Transformer Network on Skeleton-based Gait Recognition
Authors: Cun Zhang, Xing-Peng Chen, Guo-Qiang Han, Xiang-Jie Liu
Date: 2022-04-08 00:00:00
Tasks: Gait Recognition

Skeleton-based gait recognition models usually suffer from the robustness problem, as the Rank-1 accuracy varies from 90\% in normal walking cases to 70\% in walking with coats cases. In this work, we propose a state-of-the-art robust skeleton-based gait recognition model called Gait-TR, which is based on the combination of spatial transformer frameworks and temporal convolutional networks. Gait-TR achieves substantial improvements over other skeleton-based gait models with higher accuracy and better robustness on the well-known gait dataset CASIA-B. Particularly in walking with coats cases, Gait-TR get a 90\% Rank-1 gait recognition accuracy rate, which is higher than the best result of silhouette-based models, which usually have higher accuracy than the silhouette-based gait recognition models. Moreover, our experiment on CASIA-B shows that the spatial transformer can extract gait features from the human skeleton better than the widely used graph convolutional network.
Controllable Missingness from Uncontrollable Missingness: Joint Learning Measurement Policy and Imputation
Authors: Seongwook Yoon, Jaehyun Kim, Heejeong Lim, Sanghoon Sull
Date: 2022-04-08 00:00:00
Tasks: Imputation

Due to the cost or interference of measurement, we need to control measurement system. Assuming that each variable can be measured sequentially, there exists optimal policy choosing next measurement for the former observations. Though optimal measurement policy is actually dependent on the goal of measurement, we mainly focus on retrieving complete data, so called as imputation. Also, we adapt the imputation method to missingness varying with measurement policy. However, learning measurement policy and imputation requires complete data which is impossible to be observed, unfortunately. To tackle this problem, we propose a data generation method and joint learning algorithm. The main idea is that 1) the data generation method is inherited by imputation method, and 2) the adaptation of imputation encourages measurement policy to learn more than individual learning. We implemented some variations of proposed algorithm for two different datasets and various missing rates. From the experimental results, we demonstrate that our algorithm is generally applicable and outperforms baseline methods.
Frequency Selective Augmentation for Video Representation Learning
Authors: Jinhyung Kim, Taeoh Kim, Minho Shim, Dongyoon Han, Dongyoon Wee, Junmo Kim
Date: 2022-04-08 00:00:00
Tasks: Action Localization, Action Recognition, Data Augmentation, Representation Learning, Self-Supervised Learning, Temporal Action Localization

Recent self-supervised video representation learning methods focus on maximizing the similarity between multiple augmented views from the same video and largely rely on the quality of generated views. However, most existing methods lack a mechanism to prevent representation learning from bias towards static information in the video. In this paper, we propose frequency augmentation (FreqAug), a spatio-temporal data augmentation method in the frequency domain for video representation learning. FreqAug stochastically removes specific frequency components from the video so that learned representation captures essential features more from the remaining information for various downstream tasks. Specifically, FreqAug pushes the model to focus more on dynamic features rather than static features in the video via dropping spatial or temporal low-frequency components. To verify the generality of the proposed method, we experiment with FreqAug on multiple self-supervised learning frameworks along with standard augmentations. Transferring the improved representation to five video action recognition and two temporal action localization downstream tasks shows consistent improvements over baselines.
Multi-scale temporal network for continuous sign language recognition
Authors: Qidan Zhu, Jing Li, Fei Yuan, Quan Gan
Date: 2022-04-08 00:00:00
Tasks: Sign Language Recognition

Continuous Sign Language Recognition (CSLR) is a challenging research task due to the lack of accurate annotation on the temporal sequence of sign language data. The recent popular usage is a hybrid model based on "CNN + RNN" for CSLR. However, when extracting temporal features in these works, most of the methods using a fixed temporal receptive field and cannot extract the temporal features well for each sign language word. In order to obtain more accurate temporal features, this paper proposes a multi-scale temporal network (MSTNet). The network mainly consists of three parts. The Resnet and two fully connected (FC) layers constitute the frame-wise feature extraction part. The time-wise feature extraction part performs temporal feature learning by first extracting temporal receptive field features of different scales using the proposed multi-scale temporal block (MST-block) to improve the temporal modeling capability, and then further encoding the temporal features of different scales by the transformers module to obtain more accurate temporal features. Finally, the proposed multi-level Connectionist Temporal Classification (CTC) loss part is used for training to obtain recognition results. The multi-level CTC loss enables better learning and updating of the shallow network parameters in CNN, and the method has no parameter increase and can be flexibly embedded in other models. Experimental results on two publicly available datasets demonstrate that our method can effectively extract sign language features in an end-to-end manner without any prior knowledge, improving the accuracy of CSLR and achieving competitive results.
Prediction of COVID-19 using chest X-ray images
Authors: Narayana Darapaneni, Suma Maram, Harpreet Singh, Syed Subhani, Mandeep Kour, Sathish Nagam, Anwesh Reddy Paduri
Date: 2022-04-08 00:00:00

COVID-19, also known as Novel Coronavirus Disease, is a highly contagious disease that first surfaced in China in late 2019. SARS-CoV-2 is a coronavirus that belongs to the vast family of coronaviruses that causes this disease. The sickness originally appeared in Wuhan, China in December 2019 and quickly spread to over 213 nations, becoming a global pandemic. Fever, dry cough, and tiredness are the most typical COVID-19 symptoms. Aches, pains, and difficulty breathing are some of the other symptoms that patients may face. The majority of these symptoms are indicators of respiratory infections and lung abnormalities, which radiologists can identify. Chest x-rays of COVID-19 patients seem similar, with patchy and hazy lungs rather than clear and healthy lungs. On x-rays, however, pneumonia and other chronic lung disorders can resemble COVID-19. Trained radiologists must be able to distinguish between COVID-19 and an illness that is less contagious. Our AI algorithm seeks to give doctors a quantitative estimate of the risk of deterioration. So that patients at high risk of deterioration can be triaged and treated efficiently. The method could be particularly useful in pandemic hotspots when screening upon admission is important for allocating limited resources like hospital beds.
From 2D Images to 3D Model:Weakly Supervised Multi-View Face Reconstruction with Deep Fusion
Authors: Weiguang Zhao, Chaolong Yang, Jianan Ye, Rui Zhang, Yuyao Yan, Xi Yang, Bin Dong, Amir Hussain, Kaizhu Huang
Date: 2022-04-08 00:00:00
Tasks: 3D Face Reconstruction, Face Model, Face Reconstruction, Weakly-supervised Learning

While weakly supervised multi-view face reconstruction (MVR) is garnering increased attention, one critical issue still remains open: how to effectively interact and fuse multiple image information to reconstruct high-precision 3D models. In this regard, we propose a novel pipeline called Deep Fusion MVR (DF-MVR) to explore the feature correspondences between multi-view images and reconstruct high-precision 3D faces. Specifically, we present a novel multi-view feature fusion backbone that utilizes face masks to align features from multiple encoders and integrates one multi-layer attention mechanism to enhance feature interaction and fusion, resulting in one unified facial representation. Additionally, we develop one concise face mask mechanism that facilitates multi-view feature fusion and facial reconstruction by identifying common areas and guiding the network's focus on critical facial features (e.g., eyes, brows, nose, and mouth). Experiments on Pixel-Face and Bosphorus datasets indicate the superiority of our model. Without 3D annotation, DF-MVR achieves 5.2% and 3.0% RMSE improvement over the existing weakly supervised MVRs respectively on Pixel-Face and Bosphorus dataset. Code will be available publicly at https://github.com/weiguangzhao/DF_MVR.