pwc-archive/papers-with-abstracts huggingface dataset:
Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment
Authors: Ziang Yan, Zhilin Li, Yinan He, Chenting Wang, Kunchang Li, Xinhao Li, Xiangyu Zeng, Zilei Wang, Yali Wang, Yu Qiao, LiMin Wang, Yi Wang
Date: 2024-12-26 00:00:00
Proceeding: CVPR 2025 1

Current multimodal large language models (MLLMs) struggle with fine-grained or precise understanding of visuals though they give comprehensive perception and reasoning in a spectrum of vision applications. Recent studies either develop tool-using or unify specific visual tasks into the autoregressive framework, often at the expense of overall multimodal performance. To address this issue and enhance MLLMs with visual tasks in a scalable fashion, we propose Task Preference Optimization (TPO), a novel method that utilizes differentiable task preferences derived from typical fine-grained visual tasks. TPO introduces learnable task tokens that establish connections between multiple task-specific heads and the MLLM. By leveraging rich visual labels during training, TPO significantly enhances the MLLM's multimodal capabilities and task-specific performance. Through multi-task co-training within TPO, we observe synergistic benefits that elevate individual task performance beyond what is achievable through single-task training methodologies. Our instantiation of this approach with VideoChat and LLaVA demonstrates an overall 14.6% improvement in multimodal performance compared to baseline models. Additionally, MLLM-TPO demonstrates robust zero-shot capabilities across various tasks, performing comparably to state-of-the-art supervised models. The code will be released at https://github.com/OpenGVLab/TPO
Experimental Study of RCS Diversity with Novel No-divergent OAM Beams
Authors: Yufei Zhao, Yong Liang Guan, Dong Chen, Afkar Mohamed Ismail, Xiaoyan Ma, Xiaobei Liu, Chau Yuen
Date: 2024-12-25 00:00:00
Tasks: Diversity

This research proposes a novel approach utilizing Orbital Angular Momentum (OAM) beams to enhance Radar Cross Section (RCS) diversity for target detection in future transportation systems. Unlike conventional OAM beams with hollow-shaped divergence patterns, the new proposed OAM beams provide uniform illumination across the target without a central energy void, but keep the inherent phase gradient of vortex property. We utilize waveguide slot antennas to generate four different modes of these novel OAM beams at X-band frequency. Furthermore, these different mode OAM beams are used to illuminate metal models, and the resulting RCS is compared with that obtained using plane waves. The findings reveal that the novel OAM beams produce significant azimuthal RCS diversity, providing a new approach for the detection of weak and small targets.This study not only reveals the RCS diversity phenomenon based on novel OAM beams of different modes but also addresses the issue of energy divergence that hinders traditional OAM beams in long-range detection applications.
RIS-Assisted Simultaneous Legitimate Monitoring and Jamming for Industrial Wireless Networks
Authors: Likang Zhang, Qinghe Du, Yijing Ren, Xiao Tang, Maged Elkashlan, Zhu Han
Date: 2024-12-25 00:00:00

In this paper, we study reconfigurable intelligent surface (RIS)-assisted simultaneous legitimate monitoring and jamming techniques for industrial environments, so that egitimate monitor (LM) and legitimate jammers (LJs) can sustainably monitor and interfere with suspicious communications with minimum transmission power. Specifically, we propose a Block Coordinate Descent-Particle Swarm Optimization (BCD-PSO) based scheme to optimize RIS's phase shift matrix and minimize LJs' transmission power, while successfully jamming and stably monitoring unauthorized communications. Simulation results demonstrate that the proposed BCD-PSO can enhance the performances in terms of monitoring, resource utilization and robustness. Moreover, we effectively exam the best deployment of RIS towards diverse objectives.
State-of-the-Art Underwater Vehicles and Technologies Enabling Smart Ocean: Survey and Classifications
Authors: Jiajie Xu, Xabier Irigoien, Mohamed-Slim Alouini
Date: 2024-12-24 00:00:00

The exploration and sustainable use of marine environments have become increasingly critical as oceans cover over 70% of surface of Earth. This paper provides a comprehensive survey and classification of state-of-the-art underwater vehicles (UVs) and supporting technologies essential for enabling a smart ocean. We categorize UVs into several types, including remotely operated vehicles (ROVs), autonomous underwater vehicles (AUVs), hybrid underwater vehicles (HUVs), unmanned surface vehicles (USVs), and underwater bionic vehicles (UBVs). These technologies are fundamental in a wide range of applications, such as environmental monitoring, deep-sea exploration, defense, and underwater infrastructure inspection. Additionally, the paper explores advancements in underwater communication technologies, namely acoustic, optical, and hybrid systems, as well as key support facilities, including submerged buoys, underwater docking stations, and wearable underwater localization systems. By classifying the vehicles and analyzing their technological capabilities and limitations, this work aims to guide future developments in underwater exploration and monitoring, addressing challenges such as energy efficiency, communication limitations, and environmental adaptability. The paper concludes by discussing the integration of artificial intelligence and machine learning in enhancing the autonomy and operational efficiency of these systems, paving the way for the realization of a fully interconnected and sustainable Smart Ocean.
Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries
Authors: Roberto Amoroso, Gengyuan Zhang, Rajat Koner, Lorenzo Baraldi, Rita Cucchiara, Volker Tresp
Date: 2024-12-26 00:00:00
Tasks: Question Answering, Video Question Answering, Video Understanding

Video Question Answering (Video QA) is a challenging video understanding task that requires models to comprehend entire videos, identify the most relevant information based on contextual cues from a given question, and reason accurately to provide answers. Recent advancements in Multimodal Large Language Models (MLLMs) have transformed video QA by leveraging their exceptional commonsense reasoning capabilities. This progress is largely driven by the effective alignment between visual data and the language space of MLLMs. However, for video QA, an additional space-time alignment poses a considerable challenge for extracting question-relevant information across frames. In this work, we investigate diverse temporal modeling techniques to integrate with MLLMs, aiming to achieve question-guided temporal modeling that leverages pre-trained visual and textual alignment in MLLMs. We propose T-Former, a novel temporal modeling method that creates a question-guided temporal bridge between frame-wise visual perception and the reasoning capabilities of LLMs. Our evaluation across multiple video QA benchmarks demonstrates that T-Former competes favorably with existing temporal modeling approaches and aligns with recent advancements in video QA.
Manga Generation via Layout-controllable Diffusion
Authors: Siyu Chen, Dengjie Li, Zenghao Bao, Yao Zhou, Lingfeng Tan, Yujie Zhong, Zheng Zhao
Date: 2024-12-26 00:00:00
Tasks: Semantic correspondence

Generating comics through text is widely studied. However, there are few studies on generating multi-panel Manga (Japanese comics) solely based on plain text. Japanese manga contains multiple panels on a single page, with characteristics such as coherence in storytelling, reasonable and diverse page layouts, consistency in characters, and semantic correspondence between panel drawings and panel scripts. Therefore, generating manga poses a significant challenge. This paper presents the manga generation task and constructs the Manga109Story dataset for studying manga generation solely from plain text. Additionally, we propose MangaDiffusion to facilitate the intra-panel and inter-panel information interaction during the manga generation process. The results show that our method particularly ensures the number of panels, reasonable and diverse page layouts. Based on our approach, there is potential to converting a large amount of textual stories into more engaging manga readings, leading to significant application prospects.
When SAM2 Meets Video Shadow and Mirror Detection
Authors: Leiping Jie
Date: 2024-12-26 00:00:00
Tasks: Image Segmentation, Mirror Detection, Segmentation, Semantic Segmentation, Shadow Detection, Video Segmentation, Video Semantic Segmentation, Video Shadow Detection

As the successor to the Segment Anything Model (SAM), the Segment Anything Model 2 (SAM2) not only improves performance in image segmentation but also extends its capabilities to video segmentation. However, its effectiveness in segmenting rare objects that seldom appear in videos remains underexplored. In this study, we evaluate SAM2 on three distinct video segmentation tasks: Video Shadow Detection (VSD) and Video Mirror Detection (VMD). Specifically, we use ground truth point or mask prompts to initialize the first frame and then predict corresponding masks for subsequent frames. Experimental results show that SAM2's performance on these tasks is suboptimal, especially when point prompts are used, both quantitatively and qualitatively. Code is available at \url{https://github.com/LeipingJie/SAM2Video}
3D Learnable Supertoken Transformer for LiDAR Point Cloud Scene Segmentation
Authors: Dening Lu, Jun Zhou, Kyle Gao, Linlin Xu, Jonathan Li
Date: 2024-05-23 00:00:00
Tasks: Clustering, Scene Segmentation

3D Transformers have achieved great success in point cloud understanding and representation. However, there is still considerable scope for further development in effective and efficient Transformers for large-scale LiDAR point cloud scene segmentation. This paper proposes a novel 3D Transformer framework, named 3D Learnable Supertoken Transformer (3DLST). The key contributions are summarized as follows. Firstly, we introduce the first Dynamic Supertoken Optimization (DSO) block for efficient token clustering and aggregating, where the learnable supertoken definition avoids the time-consuming pre-processing of traditional superpoint generation. Since the learnable supertokens can be dynamically optimized by multi-level deep features during network learning, they are tailored to the semantic homogeneity-aware token clustering. Secondly, an efficient Cross-Attention-guided Upsampling (CAU) block is proposed for token reconstruction from optimized supertokens. Thirdly, the 3DLST is equipped with a novel W-net architecture instead of the common U-net design, which is more suitable for Transformer-based feature learning. The SOTA performance on three challenging LiDAR datasets (airborne MultiSpectral LiDAR (MS-LiDAR) (89.3% of the average F1 score), DALES (80.2% of mIoU), and Toronto-3D dataset (80.4% of mIoU)) demonstrate the superiority of 3DLST and its strong adaptability to various LiDAR point cloud data (airborne MS-LiDAR, aerial LiDAR, and vehicle-mounted LiDAR data). Furthermore, 3DLST also achieves satisfactory results in terms of algorithm efficiency, which is up to 5x faster than previous best-performing methods.
SeaMo: A Season-Aware Multimodal Foundation Model for Remote Sensing
Authors: Xuyang Li, Chenyu Li, Gemine Vivone, Danfeng Hong
Date: 2024-12-26 00:00:00
Tasks: Earth Observation

Remote Sensing (RS) data encapsulates rich multi-dimensional information essential for Earth observation. Its vast volume, diverse sources, and temporal continuity make it particularly well-suited for developing large Visual Foundation Models (VFMs). These models serve as powerful feature extractors, leveraging extensive RS data for pretraining and subsequent fine-tuning in various geoscientific applications. However, existing VFMs in the RS domain often concentrate on specific image characteristics, neglecting the full season-aware potential of RS data. To bridge this gap, we introduce SeaMo, a novel VFM that effectively integrates multimodal and multi-seasonal RS information. SeaMo leverages a masked image modeling framework to fully exploit the spatial, spectral, and seasonal dimensions of RS data. Specifically, we employ unaligned spatial region selection to capture spatial heterogeneity, incorporate multi-source inputs for enhanced multimodal integration, and introduce temporal-multimodal fusion blocks to assimilate seasonal variations effectively. By explicitly modeling the complex, season-dependent attributes of RS data, SeaMo enhances generalization, robustness, and adaptability across geoscientific tasks. Extensive experiments and ablation studies demonstrate its superior performance, underscoring its potential as a foundational model for Earth observation.
Completion as Enhancement: A Degradation-Aware Selective Image Guided Network for Depth Completion
Authors: Zhiqiang Yan, Zhengxue Wang, Kun Wang, Jun Li, Jian Yang
Date: 2024-12-26 00:00:00
Proceeding: CVPR 2025 1
Tasks: Depth Completion, Mamba

In this paper, we introduce the Selective Image Guided Network (SigNet), a novel degradation-aware framework that transforms depth completion into depth enhancement for the first time. Moving beyond direct completion using convolutional neural networks (CNNs), SigNet initially densifies sparse depth data through non-CNN densification tools to obtain coarse yet dense depth. This approach eliminates the mismatch and ambiguity caused by direct convolution over irregularly sampled sparse data. Subsequently, SigNet redefines completion as enhancement, establishing a self-supervised degradation bridge between the coarse depth and the targeted dense depth for effective RGB-D fusion. To achieve this, SigNet leverages the implicit degradation to adaptively select high-frequency components (e.g., edges) of RGB data to compensate for the coarse depth. This degradation is further integrated into a multi-modal conditional Mamba, dynamically generating the state parameters to enable efficient global high-frequency information interaction. We conduct extensive experiments on the NYUv2, DIML, SUN RGBD, and TOFDC datasets, demonstrating the state-of-the-art (SOTA) performance of SigNet.
Transformer-Based Wireless Capsule Endoscopy Bleeding Tissue Detection and Classification
Authors: Basit Alawode, Shibani Hamza, Adarsh Ghimire, Divya Velayudhan
Date: 2024-12-26 00:00:00
Tasks: Classification, Decoder

Informed by the success of the transformer model in various computer vision tasks, we design an end-to-end trainable model for the automatic detection and classification of bleeding and non-bleeding frames extracted from Wireless Capsule Endoscopy (WCE) videos. Based on the DETR model, our model uses the Resnet50 for feature extraction, the transformer encoder-decoder for bleeding and non-bleeding region detection, and a feedforward neural network for classification. Trained in an end-to-end approach on the Auto-WCEBleedGen Version 1 challenge training set, our model performs both detection and classification tasks as a single unit. Our model achieves an accuracy, recall, and F1-score classification percentage score of 98.28, 96.79, and 98.37 respectively, on the Auto-WCEBleedGen version 1 validation set. Further, we record an average precision (AP @ 0.5), mean-average precision (mAP) of 0.7447 and 0.7328 detection results. This earned us a 3rd place position in the challenge. Our code is publicly available via https://github.com/BasitAlawode/WCEBleedGen.
NADER: Neural Architecture Design via Multi-Agent Collaboration
Authors: Zekang Yang, Wang Zeng, Sheng Jin, Chen Qian, Ping Luo, Wentao Liu
Date: 2024-12-26 00:00:00
Proceeding: CVPR 2025 1
Tasks: Neural Architecture Search

Designing effective neural architectures poses a significant challenge in deep learning. While Neural Architecture Search (NAS) automates the search for optimal architectures, existing methods are often constrained by predetermined search spaces and may miss critical neural architectures. In this paper, we introduce NADER (Neural Architecture Design via multi-agEnt collaboRation), a novel framework that formulates neural architecture design (NAD) as a LLM-based multi-agent collaboration problem. NADER employs a team of specialized agents to enhance a base architecture through iterative modification. Current LLM-based NAD methods typically operate independently, lacking the ability to learn from past experiences, which results in repeated mistakes and inefficient exploration. To address this issue, we propose the Reflector, which effectively learns from immediate feedback and long-term experiences. Additionally, unlike previous LLM-based methods that use code to represent neural architectures, we utilize a graph-based representation. This approach allows agents to focus on design aspects without being distracted by coding. We demonstrate the effectiveness of NADER in discovering high-performing architectures beyond predetermined search spaces through extensive experiments on benchmark tasks, showcasing its advantages over state-of-the-art methods. The codes will be released soon.
An End-to-End Depth-Based Pipeline for Selfie Image Rectification
Authors: Ahmed Alhawwary, Phong Nguyen-Ha, Janne Mustaniemi, Janne Heikkilä
Date: 2024-12-26 00:00:00
Tasks: Depth Estimation, Hallucination

Portraits or selfie images taken from a close distance typically suffer from perspective distortion. In this paper, we propose an end-to-end deep learning-based rectification pipeline to mitigate the effects of perspective distortion. We learn to predict the facial depth by training a deep CNN. The estimated depth is utilized to adjust the camera-to-subject distance by moving the camera farther, increasing the camera focal length, and reprojecting the 3D image features to the new perspective. The reprojected features are then fed to an inpainting module to fill in the missing pixels. We leverage a differentiable renderer to enable end-to-end training of our depth estimation and feature extraction nets to improve the rectified outputs. To boost the results of the inpainting module, we incorporate an auxiliary module to predict the horizontal movement of the camera which decreases the area that requires hallucination of challenging face parts such as ears. Unlike previous works, we process the full-frame input image at once without cropping the subject's face and processing it separately from the rest of the body, eliminating the need for complex post-processing steps to attach the face back to the subject's body. To train our network, we utilize the popular game engine Unreal Engine to generate a large synthetic face dataset containing various subjects, head poses, expressions, eyewear, clothes, and lighting. Quantitative and qualitative results show that our rectification pipeline outperforms previous methods, and produces comparable results with a time-consuming 3D GAN-based method while being more than 260 times faster.
Multi-Head Attention Driven Dynamic Visual-Semantic Embedding for Enhanced Image-Text Matching
Authors: Wenjing Chen
Date: 2024-12-26 00:00:00
Tasks: Image-text matching, Text Matching, Text Retrieval

With the rapid development of multimodal learning, the image-text matching task, as a bridge connecting vision and language, has become increasingly important. Based on existing research, this study proposes an innovative visual semantic embedding model, Multi-Headed Consensus-Aware Visual-Semantic Embedding (MH-CVSE). This model introduces a multi-head self-attention mechanism based on the consensus-aware visual semantic embedding model (CVSE) to capture information in multiple subspaces in parallel, significantly enhancing the model's ability to understand and represent the complex relationship between images and texts. In addition, we adopt a parameterized feature fusion strategy to flexibly integrate feature information at different levels, further improving the model's expressive power. In terms of loss function design, the MH-CVSE model adopts a dynamic weight adjustment strategy to dynamically adjust the weight according to the loss value itself, so that the model can better balance the contribution of different loss terms during training. At the same time, we introduce a cosine annealing learning rate strategy to help the model converge more stably in the later stages of training. Extensive experimental verification on the Flickr30k dataset shows that the MH-CVSE model achieves better performance than previous methods in both bidirectional image and text retrieval tasks, fully demonstrating its effectiveness and superiority.
Revisiting Monocular 3D Object Detection from Scene-Level Depth Retargeting to Instance-Level Spatial Refinement
Authors: Qiude Zhang, Chunyu Lin, Zhijie Shen, Nie Lang, Yao Zhao
Date: 2024-12-26 00:00:00
Tasks: 3D Object Detection, Depth Estimation, Monocular 3D Object Detection, Monocular Depth Estimation, Object, object-detection, Object Detection

Monocular 3D object detection is challenging due to the lack of accurate depth. However, existing depth-assisted solutions still exhibit inferior performance, whose reason is universally acknowledged as the unsatisfactory accuracy of monocular depth estimation models. In this paper, we revisit monocular 3D object detection from the depth perspective and formulate an additional issue as the limited 3D structure-aware capability of existing depth representations (\textit{e.g.}, depth one-hot encoding or depth distribution). To address this issue, we propose a novel depth-adapted monocular 3D object detection network, termed \textbf{RD3D}, that mainly comprises a Scene-Level Depth Retargeting (SDR) module and an Instance-Level Spatial Refinement (ISR) module. The former incorporates the scene-level perception of 3D structures, retargeting traditional depth representations to a new formulation: \textbf{Depth Thickness Field}. The latter refines the voxel spatial representation with the guidance of instances, eliminating the ambiguity of 3D occupation and thus improving detection accuracy. Extensive experiments on the KITTI and Waymo datasets demonstrate our superiority to existing state-of-the-art (SoTA) methods and the universality when equipped with different depth estimation models. The code will be available.