pwc-archive/papers-with-abstracts huggingface dataset:
WorldSimBench: Towards Video Generation Models as World Simulators
Authors: Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, Lei Bai, Wanli Ouyang, Ruimao Zhang
Date: 2024-10-23 00:00:00
Tasks: Autonomous Driving, Robot Manipulation, Video Generation

Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues to hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use to train a Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulators. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation-aware video can be accurately translated into the correct control signals in dynamic environments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence.
CLEAR: Character Unlearning in Textual and Visual Modalities
Authors: Alexey Dontsov, Dmitrii Korzh, Alexey Zhavoronkin, Boris Mikheev, Denis Bobkov, Aibek Alanov, Oleg Y. Rogov, Ivan Oseledets, Elena Tutubalina
Date: 2024-10-23 00:00:00
Tasks: Machine Unlearning

Machine Unlearning (MU) is critical for enhancing privacy and security in deep learning models, particularly in large multimodal language models (MLLMs), by removing specific private or hazardous information. While MU has made significant progress in textual and visual modalities, multimodal unlearning (MMU) remains significantly underexplored, partially due to the absence of a suitable open-source benchmark. To address this, we introduce CLEAR, a new benchmark designed to evaluate MMU methods. CLEAR contains 200 fictitious individuals and 3,700 images linked with corresponding question-answer pairs, enabling a thorough evaluation across modalities. We assess 10 MU methods, adapting them for MMU, and highlight new challenges specific to multimodal forgetting. The dataset is available at https://huggingface.co/datasets/therem/CLEAR
In-Pixel Foreground and Contrast Enhancement Circuits with Customizable Mapping
Authors: Md Rahatul Islam Udoy, Md Mazharul Islam, Elijah Johnson, Ahmedullah Aziz
Date: 2024-10-23 00:00:00

This paper presents an innovative in-pixel contrast enhancement circuit that performs image processing directly within the pixel circuit. The circuit can be tuned for different modes of operation. In foreground enhancement mode, it suppresses low-intensity background pixels to nearly zero, isolating the foreground for better object visibility. In contrast enhancement mode, it improves overall image contrast. The contrast enhancement function is customizable both during the design phase and in real-time, allowing the circuit to adapt to specific applications and varying lighting conditions. A model of the designed pixel circuit is developed and applied to a full pixel array, demonstrating significant improvements in image quality. Simulations performed in HSPICE show a nearly 6x increase in Michelson Contrast Ratio (CR) in the foreground enhancement mode. The simulation results indicate its potential for real-time, adaptive contrast enhancement across various imaging environments.
Real time anomalies detection on video
Authors: Fabien Poirier
Date: 2024-10-23 00:00:00

Nowadays, many places use security cameras. Unfortunately, when an incident occurs, these technologies are used to show past events. So it can be considered as a deterrence tool than a detection tool. In this article, we will propose a deep learning approach trying to solve this problematic. This approach uses convolutional models (CNN) to extract relevant characteristics linked to the video images, theses characteristics will form times series to be analyzed by LSTM / GRU models.
Scalable Ranked Preference Optimization for Text-to-Image Generation
Authors: Shyamgopal Karthik, Huseyin Coskun, Zeynep Akata, Sergey Tulyakov, Jian Ren, Anil Kag
Date: 2024-10-23 00:00:00
Tasks: Image Generation, Text to Image Generation, Text-to-Image Generation

Direct Preference Optimization (DPO) has emerged as a powerful approach to align text-to-image (T2I) models with human feedback. Unfortunately, successful application of DPO to T2I models requires a huge amount of resources to collect and label large-scale datasets, e.g., millions of generated paired images annotated with human preferences. In addition, these human preference datasets can get outdated quickly as the rapid improvements of T2I models lead to higher quality images. In this work, we investigate a scalable approach for collecting large-scale and fully synthetic datasets for DPO training. Specifically, the preferences for paired images are generated using a pre-trained reward function, eliminating the need for involving humans in the annotation process, greatly improving the dataset collection efficiency. Moreover, we demonstrate that such datasets allow averaging predictions across multiple models and collecting ranked preferences as opposed to pairwise preferences. Furthermore, we introduce RankDPO to enhance DPO-based methods using the ranking feedback. Applying RankDPO on SDXL and SD3-Medium models with our synthetically generated preference dataset "Syn-Pic" improves both prompt-following (on benchmarks like T2I-Compbench, GenEval, and DPG-Bench) and visual quality (through user studies). This pipeline presents a practical and scalable solution to develop better preference datasets to enhance the performance of text-to-image models.
Characterization of the multiplicity of solutions for camera pose given two vertically-aligned landmarks and accelerometer
Authors: Alexander R. Pruss
Date: 2024-10-23 00:00:00
Tasks: Position

We consider the problem of recovering the position and orientation of a camera equipped with an accelerometer from sensor images of two labeled landmarks whose positions in a coordinate system aligned in a known way with gravity are known. This a variant on the much studied P$n$P problem of recovering camera position and orientation from $n$ points without any gravitational data. It is proved that in three types of singular cases there are infinitely many solutions, in another type of case there is one, and in a final type of case there are two. A precise characterization of each type of case. In particular, there is always a unique solution in the practically interesting case where the two landmarks are at the same altitude and the camera is at a different altitude. This case is studied by numerical simulation and an implementation on a consumer cellphone. It is also proved that if the two landmarks are unlabeled, then apart from the same singular cases, there are still always one or two solutions.
A Pipeline for Segmenting and Structuring RGB-D Data for Robotics Applications
Authors: Zhiwu Zheng, Lauren Mentzer, Berk Iskender, Michael Price, Colm Prendergast, Audren Cloitre
Date: 2024-10-23 00:00:00

We introduce a novel pipeline for segmenting and structuring color and depth (RGB-D) data. Existing processing pipelines for RGB-D data have focused on extracting geometric information alone. This approach precludes the development of more advanced robotic navigation and manipulation algorithms, which benefit from a semantic understanding of their environment. Our pipeline can segment RGB-D data into accurate semantic masks. These masks are then used to fuse raw captured point clouds into semantically separated point clouds. We store this information using the Universal Scene Description (USD) file format, a format suitable for easy querying by downstream robotics algorithms, human-friendly visualization, and robotics simulation.
Robust Two-View Geometry Estimation with Implicit Differentiation
Authors: Vladislav Pyatov, Iaroslav Koshelev, Stamatis Lefkimmiatis
Date: 2024-10-23 00:00:00
Tasks: Camera Pose Estimation, Pose Estimation

We present a novel two-view geometry estimation framework which is based on a differentiable robust loss function fitting. We propose to treat the robust fundamental matrix estimation as an implicit layer, which allows us to avoid backpropagation through time and significantly improves the numerical stability. To take full advantage of the information from the feature matching stage we incorporate learnable weights that depend on the matching confidences. In this way our solution brings together feature extraction, matching and two-view geometry estimation in a unified end-to-end trainable pipeline. We evaluate our approach on the camera pose estimation task in both outdoor and indoor scenarios. The experiments on several datasets show that the proposed method outperforms both classic and learning-based state-of-the-art methods by a large margin. The project webpage is available at: https://github.com/VladPyatov/ihls
A Wavelet Diffusion GAN for Image Super-Resolution
Authors: Lorenzo Aloisi, Luigi Sigillo, Aurelio Uncini, Danilo Comminiello
Date: 2024-10-23 00:00:00
Tasks: Dimensionality Reduction, Image Generation, Image Super-Resolution, Image-to-Image Translation, Super-Resolution, Text to Image Generation, Text-to-Image Generation

In recent years, diffusion models have emerged as a superior alternative to generative adversarial networks (GANs) for high-fidelity image generation, with wide applications in text-to-image generation, image-to-image translation, and super-resolution. However, their real-time feasibility is hindered by slow training and inference speeds. This study addresses this challenge by proposing a wavelet-based conditional Diffusion GAN scheme for Single-Image Super-Resolution (SISR). Our approach utilizes the diffusion GAN paradigm to reduce the timesteps required by the reverse diffusion process and the Discrete Wavelet Transform (DWT) to achieve dimensionality reduction, decreasing training and inference times significantly. The results of an experimental validation on the CelebA-HQ dataset confirm the effectiveness of our proposed scheme. Our approach outperforms other state-of-the-art methodologies successfully ensuring high-fidelity output while overcoming inherent drawbacks associated with diffusion models in time-sensitive applications.
Medical Imaging Complexity and its Effects on GAN Performance
Authors: William Cagas, Chan Ko, Blake Hsiao, Shryuk Grandhi, Rishi Bhattacharya, Kevin Zhu, Michael Lam
Date: 2024-10-23 00:00:00
Tasks: Image Generation

The proliferation of machine learning models in diverse clinical applications has led to a growing need for high-fidelity, medical image training data. Such data is often scarce due to cost constraints and privacy concerns. Alleviating this burden, medical image synthesis via generative adversarial networks (GANs) emerged as a powerful method for synthetically generating photo-realistic images based on existing sets of real medical images. However, the exact image set size required to efficiently train such a GAN is unclear. In this work, we experimentally establish benchmarks that measure the relationship between a sample dataset size and the fidelity of the generated images, given the dataset's distribution of image complexities. We analyze statistical metrics based on delentropy, an image complexity measure rooted in Shannon's entropy in information theory. For our pipeline, we conduct experiments with two state-of-the-art GANs, StyleGAN 3 and SPADE-GAN, trained on multiple medical imaging datasets with variable sample sizes. Across both GANs, general performance improved with increasing training set size but suffered with increasing complexity.
VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points
Authors: Linus Franke, Laura Fink, Marc Stamminger
Date: 2024-10-23 00:00:00
Tasks: 3DGS, NeRF, Novel View Synthesis

Recent advances in novel view synthesis have demonstrated impressive results in fast photorealistic scene rendering through differentiable point rendering, either via Gaussian Splatting (3DGS) [Kerbl and Kopanas et al. 2023] or neural point rendering [Aliev et al. 2020]. Unfortunately, these directions require either a large number of small Gaussians or expensive per-pixel post-processing for reconstructing fine details, which negatively impacts rendering performance. To meet the high performance demands of virtual reality (VR) systems, primitive or pixel counts therefore must be kept low, affecting visual quality. In this paper, we propose a novel hybrid approach based on foveated rendering as a promising solution that combines the strengths of both point rendering directions regarding performance sweet spots. Analyzing the compatibility with the human visual system, we find that using a low-detailed, few primitive smooth Gaussian representation for the periphery is cheap to compute and meets the perceptual demands of peripheral vision. For the fovea only, we use neural points with a convolutional neural network for the small pixel footprint, which provides sharp, detailed output within the rendering budget. This combination also allows for synergistic method accelerations with point occlusion culling and reducing the demands on the neural network. Our evaluation confirms that our approach increases sharpness and details compared to a standard VR-ready 3DGS configuration, and participants of a user study overwhelmingly preferred our method. Our system meets the necessary performance requirements for real-time VR interactions, ultimately enhancing the user's immersive experience. The project page can be found at: https://lfranke.github.io/vr_splatting
Gaze-Assisted Medical Image Segmentation
Authors: Leila Khaertdinova, Ilya Pershin, Tatiana Shmykova, Bulat Ibragimov
Date: 2024-10-23 00:00:00
Tasks: Diagnostic, Image Segmentation, Medical Image Segmentation, Segmentation, Semantic Segmentation, Semi-supervised Medical Image Segmentation

The annotation of patient organs is a crucial part of various diagnostic and treatment procedures, such as radiotherapy planning. Manual annotation is extremely time-consuming, while its automation using modern image analysis techniques has not yet reached levels sufficient for clinical adoption. This paper investigates the idea of semi-supervised medical image segmentation using human gaze as interactive input for segmentation correction. In particular, we fine-tuned the Segment Anything Model in Medical Images (MedSAM), a public solution that uses various prompt types as additional input for semi-automated segmentation correction. We used human gaze data from reading abdominal images as a prompt for fine-tuning MedSAM. The model was validated on a public WORD database, which consists of 120 CT scans of 16 abdominal organs. The results of the gaze-assisted MedSAM were shown to be superior to the results of the state-of-the-art segmentation models. In particular, the average Dice coefficient for 16 abdominal organs was 85.8%, 86.7%, 81.7%, and 90.5% for nnUNetV2, ResUNet, original MedSAM, and our gaze-assisted MedSAM model, respectively.
A utility-based spatial analysis of residential street-level conditions; A case study of Rotterdam
Authors: Sander van Cranenburgh, Francisco Garrido-Valenzuela
Date: 2024-10-23 00:00:00
Tasks: Discrete Choice Models

Residential location choices are traditionally modelled using factors related to accessibility and socioeconomic environments, neglecting the importance of local street-level conditions. Arguably, this neglect is due to data practices. Today, however, street-level images -- which are highly effective at encoding street-level conditions -- are widely available. Additionally, recent advances in discrete choice models incorporating computer vision capabilities offer opportunities to integrate street-level conditions into residential location choice analysis. This study leverages these developments to investigate the spatial distribution of utility derived from street-level conditions in residential location choices on a city-wide scale. In our case study of Rotterdam, the Netherlands, we find that the utility derived from street-level conditions varies significantly on a highly localised scale, with conditions rapidly changing even within neighbourhoods. Our results also reveal that the high real-estate prices in the city centre cannot be attributed to attractive street-level conditions. Furthermore, whereas the city centre is characterised by relatively unattractive residential street-level conditions, neighbourhoods in the southern part of the city -- often perceived as problematic -- exhibit surprisingly appealing street-level environments. The methodological contribution of this paper is that it advances the discrete choice models incorporating computer vision capabilities by introducing a semantic regularisation layer to the model. Thereby, it adds explainability and eliminates the need for a separate pipeline to extract information from images, streamlining the analysis. As such, this paper's findings and methodological advancements pave the way for further studies to explore integrating street-level conditions in urban planning.
CASCRNet: An Atrous Spatial Pyramid Pooling and Shared Channel Residual based Network for Capsule Endoscopy
Authors: K V Srinanda, M Manvith Prabhu, Shyam Lal
Date: 2024-10-23 00:00:00

This manuscript summarizes work on the Capsule Vision Challenge 2024 by MISAHUB. To address the multi-class disease classification task, which is challenging due to the complexity and imbalance in the Capsule Vision challenge dataset, this paper proposes CASCRNet (Capsule endoscopy-Aspp-SCR-Network), a parameter-efficient and novel model that uses Shared Channel Residual (SCR) blocks and Atrous Spatial Pyramid Pooling (ASPP) blocks. Further, the performance of the proposed model is compared with other well-known approaches. The experimental results yield that proposed model provides better disease classification results. The proposed model was successful in classifying diseases with an F1 Score of 78.5% and a Mean AUC of 98.3%, which is promising given its compact architecture.
Blendify -- Python rendering framework for Blender
Authors: Vladimir Guzov, Ilya A. Petrov, Gerard Pons-Moll
Date: 2024-10-23 00:00:00
Tasks: 10-shot image generation

With the rapid growth of the volume of research fields like computer vision and computer graphics, researchers require effective and user-friendly rendering tools to visualize results. While advanced tools like Blender offer powerful capabilities, they also require a significant effort to master. This technical report introduces Blendify, a lightweight Python-based framework that seamlessly integrates with Blender, providing a high-level API for scene creation and rendering. Blendify reduces the complexity of working with Blender's native API by automating object creation, handling the colors and material linking, and implementing features such as shadow-catcher objects while maintaining support for high-quality ray-tracing rendering output. With a focus on usability Blendify enables efficient and flexible rendering workflow for rendering in common computer vision and computer graphics use cases. The code is available at https://github.com/ptrvilya/blendify
Few-shot NeRF by Adaptive Rendering Loss Regularization
Authors: Qingshan Xu, Xuanyu Yi, Jianyao Xu, Wenbing Tao, Yew-Soon Ong, Hanwang Zhang
Date: 2024-10-23 00:00:00
Tasks: NeRF, Novel View Synthesis

Novel view synthesis with sparse inputs poses great challenges to Neural Radiance Field (NeRF). Recent works demonstrate that the frequency regularization of Positional Encoding (PE) can achieve promising results for few-shot NeRF. In this work, we reveal that there exists an inconsistency between the frequency regularization of PE and rendering loss. This prevents few-shot NeRF from synthesizing higher-quality novel views. To mitigate this inconsistency, we propose Adaptive Rendering loss regularization for few-shot NeRF, dubbed AR-NeRF. Specifically, we present a two-phase rendering supervision and an adaptive rendering loss weight learning strategy to align the frequency relationship between PE and 2D-pixel supervision. In this way, AR-NeRF can learn global structures better in the early training phase and adaptively learn local details throughout the training process. Extensive experiments show that our AR-NeRF achieves state-of-the-art performance on different datasets, including object-level and complex scenes.
From Real Artifacts to Virtual Reference: A Robust Framework for Translating Endoscopic Images
Authors: Junyang Wu, Fangfang Xie, Jiayuan Sun, Yun Gu, Guang-Zhong Yang
Date: 2024-10-15 00:00:00
Tasks: Contrastive Learning, Denoising, Domain Adaptation, Medical Image Analysis, Pose Estimation, Style Transfer, Translation

Domain adaptation, which bridges the distributions across different modalities, plays a crucial role in multimodal medical image analysis. In endoscopic imaging, combining pre-operative data with intra-operative imaging is important for surgical planning and navigation. However, existing domain adaptation methods are hampered by distribution shift caused by in vivo artifacts, necessitating robust techniques for aligning noisy and artifact abundant patient endoscopic videos with clean virtual images reconstructed from pre-operative tomographic data for pose estimation during intraoperative guidance. This paper presents an artifact-resilient image translation method and an associated benchmark for this purpose. The method incorporates a novel ``local-global'' translation framework and a noise-resilient feature extraction strategy. For the former, it decouples the image translation process into a local step for feature denoising, and a global step for global style transfer. For feature extraction, a new contrastive learning strategy is proposed, which can extract noise-resilient features for establishing robust correspondence across domains. Detailed validation on both public and in-house clinical datasets has been conducted, demonstrating significantly improved performance compared to the current state-of-the-art.