pwc-archive/papers-with-abstracts huggingface dataset:
ILSH: The Imperial Light-Stage Head Dataset for Human Head View Synthesis
Authors: Jiali Zheng, Youngkyoon Jang, Athanasios Papaioannou, Christos Kampouris, Rolandos Alexandros Potamias, Foivos Paraperas Papantoniou, Efstathios Galanakis, Ales Leonardis, Stefanos Zafeiriou
Date: 2023-10-06 00:00:00
Tasks: 4k, Neural Rendering

This paper introduces the Imperial Light-Stage Head (ILSH) dataset, a novel light-stage-captured human head dataset designed to support view synthesis academic challenges for human heads. The ILSH dataset is intended to facilitate diverse approaches, such as scene-specific or generic neural rendering, multiple-view geometry, 3D vision, and computer graphics, to further advance the development of photo-realistic human avatars. This paper details the setup of a light-stage specifically designed to capture high-resolution (4K) human head images and describes the process of addressing challenges (preprocessing, ethical issues) in collecting high-quality data. In addition to the data collection, we address the split of the dataset into train, validation, and test sets. Our goal is to design and support a fair view synthesis challenge task for this novel dataset, such that a similar level of performance can be maintained and expected when using the test set, as when using the validation set. The ILSH dataset consists of 52 subjects captured using 24 cameras with all 82 lighting sources turned on, resulting in a total of 1,248 close-up head images, border masks, and camera pose pairs.
Diffusion Models as Masked Audio-Video Learners
Authors: Elvis Nunez, Yanzi Jin, Mohammad Rastegari, Sachin Mehta, Maxwell Horton
Date: 2023-10-05 00:00:00
Tasks: Audio Classification, Contrastive Learning

Over the past several years, the synchronization between audio and visual signals has been leveraged to learn richer audio-visual representations. Aided by the large availability of unlabeled videos, many unsupervised training frameworks have demonstrated impressive results in various downstream audio and video tasks. Recently, Masked Audio-Video Learners (MAViL) has emerged as a state-of-the-art audio-video pre-training framework. MAViL couples contrastive learning with masked autoencoding to jointly reconstruct audio spectrograms and video frames by fusing information from both modalities. In this paper, we study the potential synergy between diffusion models and MAViL, seeking to derive mutual benefits from these two frameworks. The incorporation of diffusion into MAViL, combined with various training efficiency methodologies that include the utilization of a masking ratio curriculum and adaptive batch sizing, results in a notable 32% reduction in pre-training Floating-Point Operations (FLOPS) and an 18% decrease in pre-training wall clock time. Crucially, this enhanced efficiency does not compromise the model's performance in downstream audio-classification tasks when compared to MAViL's performance.
Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representation
Authors: Kashu Yamazaki, Taisei Hanyu, Khoa Vo, Thang Pham, Minh Tran, Gianfranco Doretto, Anh Nguyen, Ngan Le
Date: 2023-10-05 00:00:00
Tasks: 3D Scene Reconstruction

Precise 3D environmental mapping is pivotal in robotics. Existing methods often rely on predefined concepts during training or are time-intensive when generating semantic maps. This paper presents Open-Fusion, a groundbreaking approach for real-time open-vocabulary 3D mapping and queryable scene representation using RGB-D data. Open-Fusion harnesses the power of a pre-trained vision-language foundation model (VLFM) for open-set semantic comprehension and employs the Truncated Signed Distance Function (TSDF) for swift 3D scene reconstruction. By leveraging the VLFM, we extract region-based embeddings and their associated confidence maps. These are then integrated with 3D knowledge from TSDF using an enhanced Hungarian-based feature-matching mechanism. Notably, Open-Fusion delivers outstanding annotation-free 3D segmentation for open-vocabulary without necessitating additional 3D training. Benchmark tests on the ScanNet dataset against leading zero-shot methods highlight Open-Fusion's superiority. Furthermore, it seamlessly combines the strengths of region-based VLFM and TSDF, facilitating real-time 3D scene comprehension that includes object concepts and open-world semantics. We encourage the readers to view the demos on our project page: https://uark-aicv.github.io/OpenFusion
Coloring Deep CNN Layers with Activation Hue Loss
Authors: Louis-François Bouchard, Mohsen Ben Lazreg, Matthew Toews
Date: 2023-10-05 00:00:00

This paper proposes a novel hue-like angular parameter to model the structure of deep convolutional neural network (CNN) activation space, referred to as the {\em activation hue}, for the purpose of regularizing models for more effective learning. The activation hue generalizes the notion of color hue angle in standard 3-channel RGB intensity space to $N$-channel activation space. A series of observations based on nearest neighbor indexing of activation vectors with pre-trained networks indicate that class-informative activations are concentrated about an angle $\theta$ in both the $(x,y)$ image plane and in multi-channel activation space. A regularization term in the form of hue-like angular $\theta$ labels is proposed to complement standard one-hot loss. Training from scratch using combined one-hot + activation hue loss improves classification performance modestly for a wide variety of classification tasks, including ImageNet.
TWICE Dataset: Digital Twin of Test Scenarios in a Controlled Environment
Authors: Leonardo Novicki Neto, Fabio Reway, Yuri Poledna, Maikol Funk Drechsler, Eduardo Parente Ribeiro, Werner Huber, Christian Icking
Date: 2023-10-05 00:00:00
Tasks: Autonomous Vehicles

Ensuring the safe and reliable operation of autonomous vehicles under adverse weather remains a significant challenge. To address this, we have developed a comprehensive dataset composed of sensor data acquired in a real test track and reproduced in the laboratory for the same test scenarios. The provided dataset includes camera, radar, LiDAR, inertial measurement unit (IMU), and GPS data recorded under adverse weather conditions (rainy, night-time, and snowy conditions). We recorded test scenarios using objects of interest such as car, cyclist, truck and pedestrian -- some of which are inspired by EURONCAP (European New Car Assessment Programme). The sensor data generated in the laboratory is acquired by the execution of simulation-based tests in hardware-in-the-loop environment with the digital twin of each real test scenario. The dataset contains more than 2 hours of recording, which totals more than 280GB of data. Therefore, it is a valuable resource for researchers in the field of autonomous vehicles to test and improve their algorithms in adverse weather conditions, as well as explore the simulation-to-reality gap. The dataset is available for download at: https://twicedataset.github.io/site/
Characterizing the Features of Mitotic Figures Using a Conditional Diffusion Probabilistic Model
Authors: Cagla Deniz Bahadir, Benjamin Liechty, David J. Pisapia, Mert R. Sabuncu
Date: 2023-10-05 00:00:00

Mitotic figure detection in histology images is a hard-to-define, yet clinically significant task, where labels are generated with pathologist interpretations and where there is no ``gold-standard'' independent ground-truth. However, it is well-established that these interpretation based labels are often unreliable, in part, due to differences in expertise levels and human subjectivity. In this paper, our goal is to shed light on the inherent uncertainty of mitosis labels and characterize the mitotic figure classification task in a human interpretable manner. We train a probabilistic diffusion model to synthesize patches of cell nuclei for a given mitosis label condition. Using this model, we can then generate a sequence of synthetic images that correspond to the same nucleus transitioning into the mitotic state. This allows us to identify different image features associated with mitosis, such as cytoplasm granularity, nuclear density, nuclear irregularity and high contrast between the nucleus and the cell body. Our approach offers a new tool for pathologists to interpret and communicate the features driving the decision to recognize a mitotic figure.
FNOSeg3D: Resolution-Robust 3D Image Segmentation with Fourier Neural Operator
Authors: Ken C. L. Wong, Hongzhi Wang, Tanveer Syeda-Mahmood
Date: 2023-10-05 00:00:00
Tasks: Image Segmentation, Medical Image Segmentation, Semantic Segmentation, Super-Resolution

Due to the computational complexity of 3D medical image segmentation, training with downsampled images is a common remedy for out-of-memory errors in deep learning. Nevertheless, as standard spatial convolution is sensitive to variations in image resolution, the accuracy of a convolutional neural network trained with downsampled images can be suboptimal when applied on the original resolution. To address this limitation, we introduce FNOSeg3D, a 3D segmentation model robust to training image resolution based on the Fourier neural operator (FNO). The FNO is a deep learning framework for learning mappings between functions in partial differential equations, which has the appealing properties of zero-shot super-resolution and global receptive field. We improve the FNO by reducing its parameter requirement and enhancing its learning capability through residual connections and deep supervision, and these result in our FNOSeg3D model which is parameter efficient and resolution robust. When tested on the BraTS'19 dataset, it achieved superior robustness to training image resolution than other tested models with less than 1% of their model parameters.
Consistency Regularization Improves Placenta Segmentation in Fetal EPI MRI Time Series
Authors: Yingcheng Liu, Neerav Karani, Neel Dey, S. Mazdak Abulnaga, Junshen Xu, P. Ellen Grant, Esra Abaci Turk, Polina Golland
Date: 2023-10-05 00:00:00
Tasks: Decision Making, Placenta Segmentation, Segmentation, Time Series

The placenta plays a crucial role in fetal development. Automated 3D placenta segmentation from fetal EPI MRI holds promise for advancing prenatal care. This paper proposes an effective semi-supervised learning method for improving placenta segmentation in fetal EPI MRI time series. We employ consistency regularization loss that promotes consistency under spatial transformation of the same image and temporal consistency across nearby images in a time series. The experimental results show that the method improves the overall segmentation accuracy and provides better performance for outliers and hard samples. The evaluation also indicates that our method improves the temporal coherency of the prediction, which could lead to more accurate computation of temporal placental biomarkers. This work contributes to the study of the placenta and prenatal clinical decision-making. Code is available at https://github.com/firstmover/cr-seg.
Integrating Audio-Visual Features for Multimodal Deepfake Detection
Authors: Sneha Muppalla, Shan Jia, Siwei Lyu
Date: 2023-10-05 00:00:00
Tasks: Binary Classification, DeepFake Detection, Face Swapping

Deepfakes are AI-generated media in which an image or video has been digitally modified. The advancements made in deepfake technology have led to privacy and security issues. Most deepfake detection techniques rely on the detection of a single modality. Existing methods for audio-visual detection do not always surpass that of the analysis based on single modalities. Therefore, this paper proposes an audio-visual-based method for deepfake detection, which integrates fine-grained deepfake identification with binary classification. We categorize the samples into four types by combining labels specific to each single modality. This method enhances the detection under intra-domain and cross-domain testing.
WLST: Weak Labels Guided Self-training for Weakly-supervised Domain Adaptation on 3D Object Detection
Authors: Tsung-Lin Tsou, Tsung-Han Wu, Winston H. Hsu
Date: 2023-10-05 00:00:00
Tasks: 3D Object Detection, Domain Adaptation, object-detection, Object Detection, Unsupervised Domain Adaptation

In the field of domain adaptation (DA) on 3D object detection, most of the work is dedicated to unsupervised domain adaptation (UDA). Yet, without any target annotations, the performance gap between the UDA approaches and the fully-supervised approach is still noticeable, which is impractical for real-world applications. On the other hand, weakly-supervised domain adaptation (WDA) is an underexplored yet practical task that only requires few labeling effort on the target domain. To improve the DA performance in a cost-effective way, we propose a general weak labels guided self-training framework, WLST, designed for WDA on 3D object detection. By incorporating autolabeler, which can generate 3D pseudo labels from 2D bounding boxes, into the existing self-training pipeline, our method is able to generate more robust and consistent pseudo labels that would benefit the training process on the target domain. Extensive experiments demonstrate the effectiveness, robustness, and detector-agnosticism of our WLST framework. Notably, it outperforms previous state-of-the-art methods on all evaluation tasks.
LightCDNet: Lightweight Change Detection Network Based on VHR Images
Authors: Yuanjun Xing, Jiawei Jiang, Jun Xiang, Enping Yan, Yabin Song, Dengkui Mo
Date: 2023-08-11 00:00:00
Proceeding: IEEE Geoscience and Remote Sensing Letters 2023 8
Tasks: Building change detection for remote sensing images, Change Detection, Decoder

Lightweight change detection models are essential for industrial applications and edge devices. Reducing the model size while maintaining high accuracy is a key challenge in developing lightweight change detection models. However, many existing methods oversimplify the model architecture, leading to a loss of information and reduced performance. Therefore, developing a lightweight model that can effectively preserve the input information is a challenging problem. To address this challenge, we propose LightCDNet, a novel lightweight change detection model that effectively preserves the input information. LightCDNet consists of an early fusion backbone network and a pyramid decoder for end-to-end change detection. The core component of LightCDNet is the Deep Supervised Fusion Module (DSFM), which guides the early fusion of primary features to improve performance. We evaluated LightCDNet on the LEVIR-CD dataset and found that it achieved comparable or better performance than state-of-the-art models while being 10–117 times smaller in size.
Deep learning for diagnosis and classification of faults in industrial rotating machinery
Authors: Roberto M. Souza, Erick G.S. Nascimento *, Ubatan A. Miranda, Wenisten J.D. Silva, Herman A. Lepikson
Date: 2021-03-01 00:00:00
Proceeding: Computers & Industrial Engineering 2021 3
Tasks: Fault Diagnosis

Application of deep-learning techniques has been increasing, which redefines state-of-the-art technology, espe­
cially in industrial applications such as fault diagnosis and classification. Therefore, implementing a system that
can automatically detect faults at an early stage and recommend stopping of a machine to avoid unsafe condi­
tions in the process and environment has become possible. This paper proposes the use of Predictive Maintenance
model with Convolutional Neural Network (PdM-CNN), to classify automatically rotating equipment faults and
advise when maintenance actions should be taken. This work uses data from only one vibration sensor installed
on the motor-drive end bearing, which is the most common layout present in the industry. This work was
developed under controlled ambient varying rotational speeds, load levels and severities, in order to verify
whether it is possible to build a model capable of classifying such faults in rotating machinery using only one set
of vibration sensors. The results showed that the accuracies of the PdM-CNN model were of 99.58% and 97.3%,
when applied to two different publicly available databases. This demonstrates the model’s ability to accurately
detect and classify faults in industrial rotating machinery. With this model companies can improve the financial
performance of their rotating machine monitoring through reducing sensor acquisition costs for fault identifi­
cation and classification problems, easing their way towards the digital transformation required for the fourth
industrial revolution.
Learnable Extended Activation Function (LEAF) for Deep Neural Networks
Authors: Yevgeniy, Bodyanskiy and Serhii, Kostiuk
Date: 2023-09-30 00:00:00
Proceeding: International Journal of Computing 2023 9
Tasks: Activation Function Synthesis, image-classification, Image Classification, Learning Theory, Neural Architecture Search

This paper introduces Learnable Extended Activation Function (LEAF) - an adaptive activation function that combines the properties of squashing functions and rectifier units. Depending on the target architecture and data processing task, LEAF adapts its form during training to achieve lower loss values and improve the training results. While not suffering from the &quot;vanishing gradient&quot; effect, LEAF can directly replace SiLU, ReLU, Sigmoid, Tanh, Swish, and AHAF in feed-forward, recurrent, and many other neural network architectures. The training process for LEAF features a two-stage approach when the activation function parameters update before the synaptic weights. The experimental evaluation in the image classification task shows the superior performance of LEAF compared to the non-adaptive alternatives. Particularly, LEAF-asTanh provides 7% better classification accuracy than hyperbolic tangents on the CIFAR-10 dataset. As empirically examined, LEAF-as-SiLU and LEAF-as-Sigmoid in convolutional networks tend to &quot;evolve&quot; into SiLU-like forms. The proposed activation function and the corresponding training algorithm are relatively simple from the computational standpoint and easily apply to existing deep neural networks.
Mode-constrained Model-based Reinforcement Learning via Gaussian Processes
Authors: Aidan Scannell, Carl Henrik Ek, Arthur Richards
Date: 2023-07-25 00:00:00
Proceeding: Proceedings of The 26th International Conference on Artificial Intelligence and Statistics 2023 7
Tasks: Gaussian Processes, Model-based Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Model-based reinforcement learning (RL) algorithms do not typically consider environments with multiple dynamic modes, where it is beneficial to avoid inoperable or undesirable modes. We present a model-based RL algorithm that constrains training to a single dynamic mode with high probability. This is a difficult problem because the mode constraint is a hidden variable associated with the environment’s dynamics. As such, it is 1) unknown a priori and 2) we do not observe its output from the environment, so cannot learn it with supervised learning. We present a nonparametric dynamic model which learns the mode constraint alongside the dynamic modes. Importantly, it learns latent structure that our planning scheme leverages to 1) enforce the mode constraint with high probability, and 2) escape local optima induced by the mode constraint. We validate our method by showing that it can solve a simulated quadcopter navigation task whilst providing a level of constraint satisfaction both during and after training.
Spatial Attention-Based 3D Graph Convolutional Neural Network for Sign Language Recognition
Authors: Muneer Al-Hammadi, Mohamed A. Bencherif, Mansour Alsulaiman, Ghulam Muhammad, Mohamed Amine Mekhtiche, Wadood Abdul, Yousef A. Alohali, Tareq S. Alrayes, Hassan Mathkour, Mohammed Faisal, Mohammed Algabri, Hamdi Altaheri, Taha Alfakihm, Hamid Ghaleb
Date: 2022-06-16 00:00:00
Proceeding: MDPI Sensors 2022 6
Tasks: Graph Neural Network, Sign Language Recognition

Sign language is the main channel for hearing-impaired people to communicate with others. It is a visual language that conveys highly structured components of manual and non-manual parameters such that it needs a lot of effort to master by hearing people. Sign language recognition aims to facilitate this mastering difficulty and bridge the communication gap between hearing-impaired people and others. This study presents an efficient architecture for sign language recognition based on a convolutional graph neural network (GCN). The presented architecture consists of a few separable 3DGCN layers, which are enhanced by a spatial attention mechanism. The limited number of layers in the proposed architecture enables it to avoid the common over-smoothing problem in deep graph neural networks. Furthermore, the attention mechanism enhances the spatial context representation of the gestures. The proposed architecture is evaluated on different datasets and shows outstanding results.
Sign Language Recognition Without Frame-Sequencing Constraints: A Proof of Concept on the Argentinian Sign Language
Authors: Franco Ronchetti, Facundo Quiroga, César Estrebou, Laura Lanzarini, Alejandro Rosete
Date: 2016-10-14 00:00:00
Proceeding: IBERAMIA 2016 10
Tasks: Dynamic Time Warping, Sign Language Recognition

Automatic sign language recognition (SLR) is an important topic within the areas of human-computer interaction and machine learning. On the one hand, it poses a complex challenge that requires the intervention of various knowledge areas, such as video processing, image processing, intelligent systems and linguistics. On the other hand, robust recognition of sign language could assist in the translation process and the integration of hearing-impaired people, as well as the teaching of sign language for the hearing population.

SLR systems usually employ Hidden Markov Models, Dynamic Time Warping or similar models to recognize signs. Such techniques exploit the sequential ordering of frames to reduce the number of hypothesis. This paper presents a general probabilistic model for sign classification that combines sub-classifiers based on different types of features such as position, movement and handshape. The model employs a bag-of-words approach in all classification steps, to explore the hypothesis that ordering is not essential for recognition. The proposed model achieved an accuracy rate of 97 % on an Argentinian Sign Language dataset containing 64 classes of signs and 3200 samples, providing some evidence that indeed recognition without ordering is possible.
A Study of Convolutional Architectures for Handshape Recognition applied to Sign Language
Authors: Quiroga Facundo, Antonio Ramiro, Ronchetti Franco, LanzariniLaura Cristina, Rosete Alejandro
Date: 2017-10-01 00:00:00
Proceeding: CACIC 2017 10
Tasks: Hand Gesture Recognition, Sign Language Recognition, Transfer Learning

Convolutional Neural Networks have been providing a performance boost in many areas in the last few years, but their performance for Handshape Recognition in the context of Sign Language Recognition has not been thoroughly studied. We evaluated several convolutional architectures in order to determine their applicability for this problem.
Using the LSA16 and RWTH-PHOENIX-Weather handshape datasets, we performed experiments with the LeNet, VGG16, ResNet-34 and All Convolutional architectures, as well as Inception with normal training and via transfer learning, and compared them to the state of the art in these datasets. We included experiments with a feedforward neural network as a baseline. We also explored various preprocessing schemes to analyze their impact on the recognition.
We determined that while all models perform reasonably well on both datasets (with performance similar to hand-engineered methods), VGG16 produced the best results, closely followed by the traditional LeNet architecture.
Also, pre-segmenting the hands from the background provided a big boost to accuracy.