pwc-archive/papers-with-abstracts huggingface dataset:
A polar prediction model for learning to represent visual transformations
Date: 2023-03-06 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Motion Compensation, Prediction, Representation Learning

All organisms make temporal predictions, and their evolutionary fitness level depends on the accuracy of these predictions. In the context of visual perception, the motions of both the observer and objects in the scene structure the dynamics of sensory signals, allowing for partial prediction of future signals based on past ones. Here, we propose a self-supervised representation-learning framework that extracts and exploits the regularities of natural videos to compute accurate predictions. We motivate the polar architecture by appealing to the Fourier shift theorem and its group-theoretic generalization, and we optimize its parameters on next-frame prediction. Through controlled experiments, we demonstrate that this approach can discover the representation of simple transformation groups acting in data. When trained on natural video datasets, our framework achieves better prediction performance than traditional motion compensation and rivals conventional deep networks, while maintaining interpretability and speed. Furthermore, the polar computations can be restructured into components resembling normalized simple and direction-selective complex cell models of primate V1 neurons. Thus, polar prediction offers a principled framework for understanding how the visual system represents sensory inputs in a form that simplifies temporal prediction.
Hyperspectral Compressive Wavefront Sensing
Authors: Sunny Howard, Jannik Esslinger, Robin H. W. Wang, Peter Norreys, Andreas Doepp
Date: 2023-03-06 00:00:00
Tasks: compressed sensing, Rolling Shutter Correction

Presented is a novel way to combine snapshot compressive imaging and lateral shearing interferometry in order to capture the spatio-spectral phase of an ultrashort laser pulse in a single shot. A deep unrolling algorithm is utilised for the snapshot compressive imaging reconstruction due to its parameter efficiency and superior speed relative to other methods, potentially allowing for online reconstruction. The algorithm's regularisation term is represented using neural network with 3D convolutional layers, to exploit the spatio-spectral correlations that exist in laser wavefronts. Compressed sensing is not typically applied to modulated signals, but we demonstrate its success here. Furthermore, we train a neural network to predict the wavefronts from a lateral shearing interferogram in terms of Zernike polynomials, which again increases the speed of our technique without sacrificing fidelity. This method is supported with simulation-based results. While applied to the example of lateral shearing interferometry, the methods presented here are generally applicable to a wide range of signals, including Shack-Hartmann-type sensors. The results may be of interest beyond the context of laser wavefront characterization, including within quantitative phase imaging.
FSVVD: A Dataset of Full Scene Volumetric Video
Authors: Kaiyuan Hu, Yili Jin, Haowen Yang, Junhua Liu, Fangxin Wang
Date: 2023-03-07 00:00:00

Recent years have witnessed a rapid development of immersive multimedia which bridges the gap between the real world and virtual space. Volumetric videos, as an emerging representative 3D video paradigm that empowers extended reality, stand out to provide unprecedented immersive and interactive video watching experience. Despite the tremendous potential, the research towards 3D volumetric video is still in its infancy, relying on sufficient and complete datasets for further exploration. However, existing related volumetric video datasets mostly only include a single object, lacking details about the scene and the interaction between them. In this paper, we focus on the current most widely used data format, point cloud, and for the first time release a full-scene volumetric video dataset that includes multiple people and their daily activities interacting with the external environments. Comprehensive dataset description and analysis are conducted, with potential usage of this dataset. The dataset and additional tools can be accessed via the following website: https://cuhksz-inml.github.io/full_scene_volumetric_video_dataset/.
Deep Learning for Inertial Positioning: A Survey
Authors: Changhao Chen, Xianfei Pan
Date: 2023-03-07 00:00:00
Tasks: Deep Learning, Sensor Fusion, Survey

Inertial sensors are widely utilized in smartphones, drones, robots, and IoT devices, playing a crucial role in enabling ubiquitous and reliable localization. Inertial sensor-based positioning is essential in various applications, including personal navigation, location-based security, and human-device interaction. However, low-cost MEMS inertial sensors' measurements are inevitably corrupted by various error sources, leading to unbounded drifts when integrated doubly in traditional inertial navigation algorithms, subjecting inertial positioning to the problem of error drifts. In recent years, with the rapid increase in sensor data and computational power, deep learning techniques have been developed, sparking significant research into addressing the problem of inertial positioning. Relevant literature in this field spans across mobile computing, robotics, and machine learning. In this article, we provide a comprehensive review of deep learning-based inertial positioning and its applications in tracking pedestrians, drones, vehicles, and robots. We connect efforts from different fields and discuss how deep learning can be applied to address issues such as sensor calibration, positioning error drift reduction, and multi-sensor fusion. This article aims to attract readers from various backgrounds, including researchers and practitioners interested in the potential of deep learning-based techniques to solve inertial positioning problems. Our review demonstrates the exciting possibilities that deep learning brings to the table and provides a roadmap for future research in this field.
Patched Diffusion Models for Unsupervised Anomaly Detection in Brain MRI
Authors: Finn Behrendt, Debayan Bhattacharya, Julia Krüger, Roland Opfer, Alexander Schlaefer
Date: 2023-03-07 00:00:00
Tasks: Anatomy, Anomaly Detection, Diversity, Unsupervised Anomaly Detection

The use of supervised deep learning techniques to detect pathologies in brain MRI scans can be challenging due to the diversity of brain anatomy and the need for annotated data sets. An alternative approach is to use unsupervised anomaly detection, which only requires sample-level labels of healthy brains to create a reference representation. This reference representation can then be compared to unhealthy brain anatomy in a pixel-wise manner to identify abnormalities. To accomplish this, generative models are needed to create anatomically consistent MRI scans of healthy brains. While recent diffusion models have shown promise in this task, accurately generating the complex structure of the human brain remains a challenge. In this paper, we propose a method that reformulates the generation task of diffusion models as a patch-based estimation of healthy brain anatomy, using spatial context to guide and improve reconstruction. We evaluate our approach on data of tumors and multiple sclerosis lesions and demonstrate a relative improvement of 25.1% compared to existing baselines.
External Camera-based Mobile Robot Pose Estimation for Collaborative Perception with Smart Edge Sensors
Authors: Simon Bultmann, Raphael Memmesheimer, Sven Behnke
Date: 2023-03-07 00:00:00
Tasks: Pose Estimation, Robot Pose Estimation

We present an approach for estimating a mobile robot's pose w.r.t. the allocentric coordinates of a network of static cameras using multi-view RGB images. The images are processed online, locally on smart edge sensors by deep neural networks to detect the robot and estimate 2D keypoints defined at distinctive positions of the 3D robot model. Robot keypoint detections are synchronized and fused on a central backend, where the robot's pose is estimated via multi-view minimization of reprojection errors. Through the pose estimation from external cameras, the robot's localization can be initialized in an allocentric map from a completely unknown state (kidnapped robot problem) and robustly tracked over time. We conduct a series of experiments evaluating the accuracy and robustness of the camera-based pose estimation compared to the robot's internal navigation stack, showing that our camera-based method achieves pose errors below 3 cm and 1{\deg} and does not drift over time, as the robot is localized allocentrically. With the robot's pose precisely estimated, its observations can be fused into the allocentric scene model. We show a real-world application, where observations from mobile robot and static smart edge sensors are fused to collaboratively build a 3D semantic map of a $\sim$240 m$^2$ indoor environment.
VOCALExplore: Pay-as-You-Go Video Data Exploration and Model Building [Technical Report]
Authors: Maureen Daum, Enhao Zhang, Dong He, Stephen Mussmann, Brandon Haynes, Ranjay Krishna, Magdalena Balazinska
Date: 2023-03-07 00:00:00
Tasks: feature selection

We introduce VOCALExplore, a system designed to support users in building domain-specific models over video datasets. VOCALExplore supports interactive labeling sessions and trains models using user-supplied labels. VOCALExplore maximizes model quality by automatically deciding how to select samples based on observed skew in the collected labels. It also selects the optimal video representations to use when training models by casting feature selection as a rising bandit problem. Finally, VOCALExplore implements optimizations to achieve low latency without sacrificing model performance. We demonstrate that VOCALExplore achieves close to the best possible model quality given candidate acquisition functions and feature extractors, and it does so with low visible latency (~1 second per iteration) and no expensive preprocessing.
NEPHELE: A Neural Platform for Highly Realistic Cloud Radiance Rendering
Authors: Haimin Luo, Siyuan Zhang, Fuqiang Zhao, Haotian Jing, Penghao Wang, Zhenxiao Yu, Dongxue Yan, Junran Ding, Boyuan Zhang, Qiang Hu, Shu Yin, Lan Xu, Jingyi Yu
Date: 2023-03-07 00:00:00
Tasks: GPU, NeRF, Neural Rendering

We have recently seen tremendous progress in neural rendering (NR) advances, i.e., NeRF, for photo-real free-view synthesis. Yet, as a local technique based on a single computer/GPU, even the best-engineered Instant-NGP or i-NGP cannot reach real-time performance when rendering at a high resolution, and often requires huge local computing resources. In this paper, we resort to cloud rendering and present NEPHELE, a neural platform for highly realistic cloud radiance rendering. In stark contrast with existing NR approaches, our NEPHELE allows for more powerful rendering capabilities by combining multiple remote GPUs and facilitates collaboration by allowing multiple people to view the same NeRF scene simultaneously. We introduce i-NOLF to employ opacity light fields for ultra-fast neural radiance rendering in a one-query-per-ray manner. We further resemble the Lumigraph with geometry proxies for fast ray querying and subsequently employ a small MLP to model the local opacity lumishperes for high-quality rendering. We also adopt Perfect Spatial Hashing in i-NOLF to enhance cache coherence. As a result, our i-NOLF achieves an order of magnitude performance gain in terms of efficiency than i-NGP, especially for the multi-user multi-viewpoint setting under cloud rendering scenarios. We further tailor a task scheduler accompanied by our i-NOLF representation and demonstrate the advance of our methodological design through a comprehensive cloud platform, consisting of a series of cooperated modules, i.e., render farms, task assigner, frame composer, and detailed streaming strategies. Using such a cloud platform compatible with neural rendering, we further showcase the capabilities of our cloud radiance rendering through a series of applications, ranging from cloud VR/AR rendering.
TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Prediction
Authors: Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, Luc van Gool
Date: 2023-03-07 00:00:00
Tasks: Autonomous Driving, Autonomous Vehicles, Model-based Reinforcement Learning, motion prediction

Data-driven simulation has become a favorable way to train and test autonomous driving algorithms. The idea of replacing the actual environment with a learned simulator has also been explored in model-based reinforcement learning in the context of world models. In this work, we show data-driven traffic simulation can be formulated as a world model. We present TrafficBots, a multi-agent policy built upon motion prediction and end-to-end driving, and based on TrafficBots we obtain a world model tailored for the planning module of autonomous vehicles. Existing data-driven traffic simulators are lacking configurability and scalability. To generate configurable behaviors, for each agent we introduce a destination as navigational information, and a time-invariant latent personality that specifies the behavioral style. To improve the scalability, we present a new scheme of positional encoding for angles, allowing all agents to share the same vectorized context and the use of an architecture based on dot-product attention. As a result, we can simulate all traffic participants seen in dense urban scenarios. Experiments on the Waymo open motion dataset show TrafficBots can simulate realistic multi-agent behaviors and achieve good performance on the motion prediction task.
How Auditing Methodologies Can Impact Our Understanding of YouTube's Recommendation Systems
Authors: Sarmad Chandio, Daniyal Pirwani Dar, Rishab Nithyanand
Date: 2023-03-06 00:00:00
Tasks: Recommendation Systems, valid

Data generated by audits of social media websites have formed the basis of our understanding of the biases presented in algorithmic content recommendation systems. As legislators around the world are beginning to consider regulating the algorithmic systems that drive online platforms, it is critical to ensure the correctness of these inferred biases. However, as we will show in this paper, doing so is a challenging task for a variety of reasons related to the complexity of configuration parameters associated with the audits that gather data from a specific platform. Focusing specifically on YouTube, we show that conducting audits to make inferences about YouTube's recommendation systems is more methodologically challenging than one might expect. There are many methodological decisions that need to be considered in order to obtain scientifically valid results, and each of these decisions incur costs. For example, should an auditor use (expensive to obtain) logged-in YouTube accounts while gathering recommendations from the algorithm to obtain more accurate inferences? We explore the impact of this and many other decisions and make some startling discoveries about the methodological choices that impact YouTube's recommendations. Taken all together, our research suggests auditing configuration compromises that YouTube auditors and researchers can use to reduce audit overhead, both economically and computationally, without sacrificing accuracy of their inferences. Similarly, we also identify several configuration parameters that have a significant impact on the accuracy of measured inferences and should be carefully considered.
Impulse-Noise-Tolerant Data-Selective LMS Algorithm
Authors: Ying-Ren Chien, Chih-Hsiang Yu
Date: 2021-08-02 00:00:00
Proceeding: IEICE TRANSACTIONS on Fundamentals of Electronics, Communications and Computer Sciences 2021 8

Exponential growth in data volumes has promoted widespread interest in data-selective adaptive algorithms. In a pioneering work, Diniz developed the data-selective least mean square (DS-LMS) algorithm, which is able to reduce specific quantities of computation data without compromising performance. Note however that the existing framework fails to consider the issue of impulse noise (IN), which can greatly undermine the benefits of reduced computation. In this letter, we present an error-based IN detection algorithm for implementation in conjunction with the DS-LMS algorithm. Numerical evaluations confirm the effectiveness of our proposed IN-tolerant DS-LMS algorithm.
Deep Radial Embedding for Visual Sequence Learning
Authors: Yuecong Min, Peiqi Jiao, Yanan Li, Xiaotao Wang, Lei Lei, Xiujuan Chai, Xilin Chen
Date: 2022-11-11 00:00:00
Proceeding: European Conference on Computer Vision 2022 11
Tasks: ARC, Sign Language Recognition

Connectionist Temporal Classification (CTC) is a popular
objective function in sequence recognition, which provides supervision
for unsegmented sequence data through aligning sequence and its corresponding labeling iteratively. The blank class of CTC plays a crucial
role in the alignment process and is often considered responsible for the
peaky behavior of CTC. In this study, we propose an objective function
named RadialCTC that constrains sequence features on a hypersphere
while retaining the iterative alignment mechanism of CTC. The learned
features of each non-blank class are distributed on a radial arc from the
center of the blank class, which provides a clear geometric interpretation
and makes the alignment process more efficient. Besides, RadialCTC can
control the peaky behavior by simply modifying the logit of the blank
class. Experimental results of recognition and localization demonstrate
the effectiveness of RadialCTC on two sequence recognition applications.
Real-world image dehazing with improved joint enhancement and exposure fusion
Authors: Nur Huseyin KAPLAN
Date: 2023-12-09 00:00:00
Proceeding: Journal of Visual Communication and Image Representation 2023 12
Tasks: Image Dehazing, Single Image Dehazing

In this work, a single image dehazing method that improves the haze removal capacity of the Joint Contrast Enhancement and Exposure Fusion (CEEF) method with Smoothing-Sharpening Image Filter (SSIF) is presented. In this method, the hazy image is first sharpened with SSIF to obtain a sharper image. In this way, the difference between haze and objects is amplified. Then, the AHE procedure in CEEF is replaced by CLAHE to obtain an enhanced CEEF. The enhanced CEEF is applied to the filtering result to obtain the final dehazed image. Observations demonstrate that the proposed method obtains enhanced results while reducing the amount of haze. The visual and quantitative comparisons between the proposed method and state-of-the-art dehazing methods show that the proposed method has better dehazing performance and has a 50% improvement in terms of the FADE metric compared to the closest result.
Structured Kernel Estimation for Photon-Limited Deconvolution
Authors: Yash Sanghvi, Zhiyuan Mao, Stanley H. Chan
Date: 2023-03-06 00:00:00
Proceeding: CVPR 2023 1
Tasks: Image Restoration

Images taken in a low light condition with the presence of camera shake suffer from motion blur and photon shot noise. While state-of-the-art image restoration networks show promising results, they are largely limited to well-illuminated scenes and their performance drops significantly when photon shot noise is strong. In this paper, we propose a new blur estimation technique customized for photon-limited conditions. The proposed method employs a gradient-based backpropagation method to estimate the blur kernel. By modeling the blur kernel using a low-dimensional representation with the key points on the motion trajectory, we significantly reduce the search space and improve the regularity of the kernel estimation problem. When plugged into an iterative framework, our novel low-dimensional representation provides improved kernel estimates and hence significantly better deconvolution performance when compared to end-to-end trained neural networks. The source code and pretrained models are available at \url{https://github.com/sanghviyashiitb/structured-kernel-cvpr23}
Grasping Student: semi-supervised learning for robotic manipulation
Authors: Piotr Krzywicki, Krzysztof Ciebiera, Rafał Michaluk, Inga Maziarz, Marek Cygan
Date: 2023-03-08 00:00:00

Gathering real-world data from the robot quickly becomes a bottleneck when constructing a robot learning system for grasping. In this work, we design a semi-supervised grasping system that, on top of a small sample of robot experience, takes advantage of images of products to be picked, which are collected without any interactions with the robot. We validate our findings both in the simulation and in the real world. In the regime of a small number of robot training samples, taking advantage of the unlabeled data allows us to achieve performance at the level of 10-fold bigger dataset size used by the baseline. The code and datasets used in the paper will be released at https://github.com/nomagiclab/grasping-student.
Adaptive Search-and-Training for Robust and Efficient Network Pruning
Authors: Lu Xiaotong, Dong Weisheng, Li Xin, Wu Jinjian, Li Leida, Shi Guangming
Date: 2023-02-01 00:00:00
Proceeding: PAMI 2023 2
Tasks: Knowledge Distillation, Network Pruning, Neural Architecture Search

Both network pruning and neural architecture search (NAS) can be interpreted as techniques to automate the design and optimization of artificial neural networks. In this paper, we challenge the conventional wisdom of training before pruning by proposing a joint search-and-training approach to learn a compact network directly from scratch. Using pruning as a search strategy, we advocate three new insights for network engineering: 1) to formulate adaptive search as a cold start strategy to find a compact subnetwork on the coarse scale; and 2) to automatically learn the threshold for network pruning; 3) to offer flexibility to choose between efficiency and robustness . More specifically, we propose an adaptive search algorithm in the cold start by exploiting the randomness and flexibility of filter pruning. The weights associated with the network filters will be updated by ThreshNet, a flexible coarse-to-fine pruning method inspired by reinforcement learning. In addition, we introduce a robust pruning strategy leveraging the technique of knowledge distillation through a teacher-student network. Extensive experiments on ResNet and VGGNet have shown that our proposed method can achieve a better balance in terms of efficiency and accuracy and notable advantages over current state-of-the-art pruning methods in several popular datasets, including CIFAR10, CIFAR100, and ImageNet.