pwc-archive/papers-with-abstracts huggingface dataset:
Visual Motif Identification: Elaboration of a Curated Comparative Dataset and Classification Methods
Authors: Adam Phillips, Daniel Grandes Rodriguez, Miriam Sánchez-Manzano, Alan Salvadó, Manuel Garin, Gloria Haro, Coloma Ballester
Date: 2024-10-21 00:00:00

In cinema, visual motifs are recurrent iconographic compositions that carry artistic or aesthetic significance. Their use throughout the history of visual arts and media is interesting to researchers and filmmakers alike. Our goal in this work is to recognise and classify these motifs by proposing a new machine learning model that uses a custom dataset to that end. We show how features extracted from a CLIP model can be leveraged by using a shallow network and an appropriate loss to classify images into 20 different motifs, with surprisingly good results: an $F_1$-score of 0.91 on our test set. We also present several ablation studies justifying the input features, architecture and hyperparameters used.
LiOn-XA: Unsupervised Domain Adaptation via LiDAR-Only Cross-Modal Adversarial Training
Authors: Thomas Kreutz, Jens Lemke, Max Mühlhäuser, Alejandro Sanchez Guinea
Date: 2024-10-21 00:00:00
Tasks: Domain Adaptation, Semantic Segmentation, Unsupervised Domain Adaptation

In this paper, we propose LiOn-XA, an unsupervised domain adaptation (UDA) approach that combines LiDAR-Only Cross-Modal (X) learning with Adversarial training for 3D LiDAR point cloud semantic segmentation to bridge the domain gap arising from environmental and sensor setup changes. Unlike existing works that exploit multiple data modalities like point clouds and RGB image data, we address UDA in scenarios where RGB images might not be available and show that two distinct LiDAR data representations can learn from each other for UDA. More specifically, we leverage 3D voxelized point clouds to preserve important geometric structure in combination with 2D projection-based range images that provide information such as object orientations or surfaces. To further align the feature space between both domains, we apply adversarial training using both features and predictions of both 2D and 3D neural networks. Our experiments on 3 real-to-real adaptation scenarios demonstrate the effectiveness of our approach, achieving new state-of-the-art performance when compared to previous uni- and multi-model UDA methods. Our source code is publicly available at https://github.com/JensLe97/lion-xa.
FusionLungNet: Multi-scale Fusion Convolution with Refinement Network for Lung CT Image Segmentation
Authors: Sadjad Rezvani, Mansoor Fateh, Yeganeh Jalali, Amirreza Fateh
Date: 2024-10-21 00:00:00
Tasks: Image Reconstruction, Image Segmentation, Segmentation, Semantic Segmentation, SSIM

Early detection of lung cancer is crucial as it increases the chances of successful treatment. Automatic lung image segmentation assists doctors in identifying diseases such as lung cancer, COVID-19, and respiratory disorders. However, lung segmentation is challenging due to overlapping features like vascular and bronchial structures, along with pixel-level fusion of brightness, color, and texture. New lung segmentation methods face difficulties in identifying long-range relationships between image components, reliance on convolution operations that may not capture all critical features, and the complex structures of the lungs. Furthermore, semantic gaps between feature maps can hinder the integration of relevant information, reducing model accuracy. Skip connections can also limit the decoder's access to complete information, resulting in partial information loss during encoding. To overcome these challenges, we propose a hybrid approach using the FusionLungNet network, which has a multi-level structure with key components, including the ResNet-50 encoder, Channel-wise Aggregation Attention (CAA) module, Multi-scale Feature Fusion (MFF) block, self refinement (SR) module, and multiple decoders. The refinement sub-network uses convolutional neural networks for image post-processing to improve quality. Our method employs a combination of loss functions, including SSIM, IOU, and focal loss, to optimize image reconstruction quality. We created and publicly released a new dataset for lung segmentation called LungSegDB, including 1800 CT images from the LIDC-IDRI dataset (dataset version 1) and 700 images from the Chest CT Cancer Images from Kaggle dataset (dataset version 2). Our method achieved an IOU score of 98.04, outperforming existing methods and demonstrating significant improvements in segmentation accuracy. https://github.com/sadjadrz/FusionLungNet
Assisted Physical Interaction: Autonomous Aerial Robots with Neural Network Detection, Navigation, and Safety Layers
Authors: Andrea Berra, Viswa Narayanan Sankaranarayanan, Achilleas Santi Seisa, Julien Mellet, Udayanga G. W. K. N. Gamage, Sumeet Gajanan Satpute, Fabio Ruggiero, Vincenzo Lippiello, Silvia Tolu, Matteo Fumagalli, George Nikolakopoulos, Miguel Ángel Trujillo Soto, Guillermo Heredia
Date: 2024-10-21 00:00:00
Tasks: Edge-computing, Pose Estimation

The paper introduces a novel framework for safe and autonomous aerial physical interaction in industrial settings. It comprises two main components: a neural network-based target detection system enhanced with edge computing for reduced onboard computational load, and a control barrier function (CBF)-based controller for safe and precise maneuvering. The target detection system is trained on a dataset under challenging visual conditions and evaluated for accuracy across various unseen data with changing lighting conditions. Depth features are utilized for target pose estimation, with the entire detection framework offloaded into low-latency edge computing. The CBF-based controller enables the UAV to converge safely to the target for precise contact. Simulated evaluations of both the controller and target detection are presented, alongside an analysis of real-world detection performance.
Generalizing Motion Planners with Mixture of Experts for Autonomous Driving
Authors: Qiao Sun, Huimin Wang, Jiahao Zhan, Fan Nie, Xin Wen, Leimeng Xu, Kun Zhan, Peng Jia, Xianpeng Lang, Hang Zhao
Date: 2024-10-21 00:00:00
Tasks: Autonomous Driving, Data Augmentation, Decoder, Mixture-of-Experts

Large real-world driving datasets have sparked significant research into various aspects of data-driven motion planners for autonomous driving. These include data augmentation, model architecture, reward design, training strategies, and planner pipelines. These planners promise better generalizations on complicated and few-shot cases than previous methods. However, experiment results show that many of these approaches produce limited generalization abilities in planning performance due to overly complex designs or training paradigms. In this paper, we review and benchmark previous methods focusing on generalizations. The experimental results indicate that as models are appropriately scaled, many design elements become redundant. We introduce StateTransformer-2 (STR2), a scalable, decoder-only motion planner that uses a Vision Transformer (ViT) encoder and a mixture-of-experts (MoE) causal Transformer architecture. The MoE backbone addresses modality collapse and reward balancing by expert routing during training. Extensive experiments on the NuPlan dataset show that our method generalizes better than previous approaches across different test sets and closed-loop simulations. Furthermore, we assess its scalability on billions of real-world urban driving scenarios, demonstrating consistent accuracy improvements as both data and model size grow.
Improving Instance Optimization in Deformable Image Registration with Gradient Projection
Authors: Yi Zhang, Yidong Zhao, Qian Tao
Date: 2024-10-21 00:00:00
Tasks: Image Registration

Deformable image registration is inherently a multi-objective optimization (MOO) problem, requiring a delicate balance between image similarity and deformation regularity. These conflicting objectives often lead to poor optimization outcomes, such as being trapped in unsatisfactory local minima or experiencing slow convergence. Deep learning methods have recently gained popularity in this domain due to their efficiency in processing large datasets and achieving high accuracy. However, they often underperform during test time compared to traditional optimization techniques, which further explore iterative, instance-specific gradient-based optimization. This performance gap is more pronounced when a distribution shift between training and test data exists. To address this issue, we focus on the instance optimization (IO) paradigm, which involves additional optimization for test-time instances based on a pre-trained model. IO effectively combines the generalization capabilities of deep learning with the fine-tuning advantages of instance-specific optimization. Within this framework, we emphasize the use of gradient projection to mitigate conflicting updates in MOO. This technique projects conflicting gradients into a common space, better aligning the dual objectives and enhancing optimization stability. We validate our method using a state-of-the-art foundation model on the 3D Brain inter-subject registration task (LUMIR) from the Learn2Reg 2024 Challenge. Our results show significant improvements over standard gradient descent, leading to more accurate and reliable registration results.
How Important are Data Augmentations to Close the Domain Gap for Object Detection in Orbit?
Authors: Maximilian Ulmer, Leonard Klüpfel, Maximilian Durner, Rudolph Triebel
Date: 2024-10-21 00:00:00
Tasks: 2D Object Detection, Domain Generalization, Hyperparameter Optimization, object-detection, Object Detection

We investigate the efficacy of data augmentations to close the domain gap in spaceborne computer vision, crucial for autonomous operations like on-orbit servicing. As the use of computer vision in space increases, challenges such as hostile illumination and low signal-to-noise ratios significantly hinder performance. While learning-based algorithms show promising results, their adoption is limited by the need for extensive annotated training data and the domain gap that arises from differences between synthesized and real-world imagery. This study explores domain generalization in terms of data augmentations -- classical color and geometric transformations, corruptions, and noise -- to enhance model performance across the domain gap. To this end, we conduct an large scale experiment using a hyperparameter optimization pipeline that samples hundreds of different configurations and searches for the best set to bridge the domain gap. As a reference task, we use 2D object detection and evaluate on the SPEED+ dataset that contains real hardware-in-the-loop satellite images in its test set. Moreover, we evaluate four popular object detectors, including Mask R-CNN, Faster R-CNN, YOLO-v7, and the open set detector GroundingDINO, and highlight their trade-offs between performance, inference speed, and training time. Our results underscore the vital role of data augmentations in bridging the domain gap, improving model performance, robustness, and reliability for critical space applications. As a result, we propose two novel data augmentations specifically developed to emulate the visual effects observed in orbital imagery. We conclude by recommending the most effective augmentations for advancing computer vision in challenging orbital environments. Code for training detectors and hyperparameter search will be made publicly available.
Ship Segmentation and Georeferencing from Static Oblique View Images
Authors: Borja Carrillo-Perez, Sarah Barnes, Maurice Stephan
Date: 2022-04-01 00:00:00
Proceeding: Sensors 2022 4
Tasks: Instance Segmentation, Segmentation, Semantic Segmentation

Camera systems support the rapid assessment of ship traffic at ports, allowing for a better perspective of the maritime situation. However, optimal ship monitoring requires a level of automation that allows personnel to keep track of relevant variables in the maritime situation in an understandable and visualisable format. It therefore becomes important to have real-time recognition of ships present at the infrastructure, with their class and geographic position presented to the maritime situational awareness operator. This work presents a novel dataset, ShipSG, for the segmentation and georeferencing of ships in maritime monitoring scenes with a static oblique view. Moreover, an exploration of four instance segmentation methods, with a focus on robust (Mask-RCNN, DetectoRS) and real-time performances (YOLACT, Centermask-Lite) and their generalisation to other existing maritime datasets, is shown. Lastly, a method for georeferencing ship masks is proposed. This includes an automatic calculation of the pixel of the segmented ship to be georeferenced and the use of a homography to transform this pixel to geographic coordinates. DetectoRS provided the highest ship segmentation mAP of 0.747. The fastest segmentation method was Centermask-Lite, with 40.96 FPS. The accuracy of our georeferencing method was (22 ± 10) m for ships detected within a 400 m range, and (53 ± 24) m for ships over 400 m away from the camera.
Deep Learning Enhanced Road Traffic Analysis: Scalable Vehicle Detection and Velocity Estimation Using PlanetScope Imagery
Authors: Maciej Adamiak, Yulia Grinblat, Julian Psotta, Nir Fulman, Himshikhar Mazumdar, Shiyu Tang, Alexander Zipf
Date: 2024-10-04 00:00:00
Tasks: vehicle detection

This paper presents a method for detecting and estimating vehicle speeds using PlanetScope SuperDove satellite imagery, offering a scalable solution for global vehicle traffic monitoring. Conventional methods such as stationary sensors and mobile systems like UAVs are limited in coverage and constrained by high costs and legal restrictions. Satellite-based approaches provide broad spatial coverage but face challenges, including high costs, low frame rates, and difficulty detecting small vehicles in high-resolution imagery. We propose a Keypoint R-CNN model to track vehicle trajectories across RGB bands, leveraging band timing differences to estimate speed. Validation is performed using drone footage and GPS data covering highways in Germany and Poland. Our model achieved a Mean Average Precision of 0.53 and velocity estimation errors of approximately 3.4 m/s compared to GPS data. Results from drone comparison reveal underestimations, with average speeds of 112.85 km/h for satellite data versus 131.83 km/h from drone footage. While challenges remain with high-speed accuracy, this approach demonstrates the potential for scalable, daily traffic monitoring across vast areas, providing valuable insights into global traffic dynamics.
Elements of disinformation theory: cyber engagement via increasing adversary information consumption
Authors: Travis Cuvelier, Sean Ha, Maretta Morovitz
Date: 2024-10-18 00:00:00

We consider the case where an adversary is conducting a surveillance campaign against a networked control system (NCS), and take the perspective of a defender/control system operator who has successfully isolated the cyber intruder. To better understand the adversary's intentions and to drive up their operating costs, the defender directs the adversary towards a ``honeypot" that emulates a real control system and without actual connections to a physical plant. We propose a strategy for adversary engagement within the ``honey" control system to increase the adversary's costs of information processing. We assume that, based on an understanding of the adversary's control theoretic goals, cyber threat intelligence (CTI) provides the defender knowledge of the adversary's preferences for information acquisition. We use this knowledge to spoof sensor readings to maximize the amount of information the adversary consumes while making it (information theoretically) difficult for the adversary to detect that they are being spoofed. We discuss the case of imperfect versus perfect threat intelligence and perform a numerical comparison.
Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace
Authors: Jinluan Yang, Anke Tang, Didi Zhu, Zhengyu Chen, Li Shen, Fei Wu
Date: 2024-10-17 00:00:00
Tasks: Meta-Learning

Model merging has gained significant attention as a cost-effective approach to integrate multiple single-task fine-tuned models into a unified one that can perform well on multiple tasks. However, existing model merging techniques primarily focus on resolving conflicts between task-specific models, they often overlook potential security threats, particularly the risk of backdoor attacks in the open-source model ecosystem. In this paper, we first investigate the vulnerabilities of existing model merging methods to backdoor attacks, identifying two critical challenges: backdoor succession and backdoor transfer. To address these issues, we propose a novel Defense-Aware Merging (DAM) approach that simultaneously mitigates task interference and backdoor vulnerabilities. Specifically, DAM employs a meta-learning-based optimization method with dual masks to identify a shared and safety-aware subspace for model merging. These masks are alternately optimized: the Task-Shared mask identifies common beneficial parameters across tasks, aiming to preserve task-specific knowledge while reducing interference, while the Backdoor-Detection mask isolates potentially harmful parameters to neutralize security threats. This dual-mask design allows us to carefully balance the preservation of useful knowledge and the removal of potential vulnerabilities. Compared to existing merging methods, DAM achieves a more favorable balance between performance and security, reducing the attack success rate by 2-10 percentage points while sacrificing only about 1% in accuracy. Furthermore, DAM exhibits robust performance and broad applicability across various types of backdoor attacks and the number of compromised models involved in the merging process. Our codes and models are available at https://github.com/Yangjinluan/DAM.
CoreGuard: Safeguarding Foundational Capabilities of LLMs Against Model Stealing in Edge Deployment
Authors: Qinfeng Li, Yangfan Xie, Tianyu Du, Zhiqiang Shen, Zhenghan Qin, Hao Peng, Xinkui Zhao, Xianwei Zhu, Jianwei Yin, Xuhong Zhang
Date: 2024-10-16 00:00:00
Tasks: CPU, GPU

Proprietary large language models (LLMs) demonstrate exceptional generalization ability across various tasks. Additionally, deploying LLMs on edge devices is trending for efficiency and privacy reasons. However, edge deployment of proprietary LLMs introduces new security threats: attackers who obtain an edge-deployed LLM can easily use it as a base model for various tasks due to its high generalization ability, which we call foundational capability stealing. Unfortunately, existing model protection mechanisms are often task-specific and fail to protect general-purpose LLMs, as they mainly focus on protecting task-related parameters using trusted execution environments (TEEs). Although some recent TEE-based methods are able to protect the overall model parameters in a computation-efficient way, they still suffer from prohibitive communication costs between TEE and CPU/GPU, making it impractical to deploy for edge LLMs. To protect the foundational capabilities of edge LLMs, we propose CoreGuard, a computation- and communication-efficient model protection approach against model stealing on edge devices. The core component of CoreGuard is a lightweight and propagative authorization module residing in TEE. Extensive experiments show that CoreGuard achieves the same security protection as the black-box security guarantees with negligible overhead.
SoK: Prompt Hacking of Large Language Models
Authors: Baha Rababah, Shang, Wu, Matthew Kwiatkowski, Carson Leung, Cuneyt Gurcan Akcora
Date: 2024-10-16 00:00:00
Tasks: Binary Classification, Diagnostic

The safety and robustness of large language models (LLMs) based applications remain critical challenges in artificial intelligence. Among the key threats to these applications are prompt hacking attacks, which can significantly undermine the security and reliability of LLM-based systems. In this work, we offer a comprehensive and systematic overview of three distinct types of prompt hacking: jailbreaking, leaking, and injection, addressing the nuances that differentiate them despite their overlapping characteristics. To enhance the evaluation of LLM-based applications, we propose a novel framework that categorizes LLM responses into five distinct classes, moving beyond the traditional binary classification. This approach provides more granular insights into the AI's behavior, improving diagnostic precision and enabling more targeted enhancements to the system's safety and robustness.
BeniFul: Backdoor Defense via Middle Feature Analysis for Deep Neural Networks
Authors: Xinfu Li, Junying Zhang, Xindi Ma
Date: 2024-10-15 00:00:00
Tasks: backdoor defense

Backdoor defenses have recently become important in resisting backdoor attacks in deep neural networks (DNNs), where attackers implant backdoors into the DNN model by injecting backdoor samples into the training dataset. Although there are many defense methods to achieve backdoor detection for DNN inputs and backdoor elimination for DNN models, they still have not presented a clear explanation of the relationship between these two missions. In this paper, we use the features from the middle layer of the DNN model to analyze the difference between backdoor and benign samples and propose Backdoor Consistency, which indicates that at least one backdoor exists in the DNN model if the backdoor trigger is detected exactly on input. By analyzing the middle features, we design an effective and comprehensive backdoor defense method named BeniFul, which consists of two parts: a gray-box backdoor input detection and a white-box backdoor elimination. Specifically, we use the reconstruction distance from the Variational Auto-Encoder and model inference results to implement backdoor input detection and a feature distance loss to achieve backdoor elimination. Experimental results on CIFAR-10 and Tiny ImageNet against five state-of-the-art attacks demonstrate that our BeniFul exhibits a great defense capability in backdoor input detection and backdoor elimination.
A Formal Framework for Assessing and Mitigating Emergent Security Risks in Generative AI Models: Bridging Theory and Dynamic Risk Mitigation
Authors: Aviral Srivastava, Sourav Panda
Date: 2024-10-15 00:00:00
Tasks: Anomaly Detection, Red Teaming

As generative AI systems, including large language models (LLMs) and diffusion models, advance rapidly, their growing adoption has led to new and complex security risks often overlooked in traditional AI risk assessment frameworks. This paper introduces a novel formal framework for categorizing and mitigating these emergent security risks by integrating adaptive, real-time monitoring, and dynamic risk mitigation strategies tailored to generative models' unique vulnerabilities. We identify previously under-explored risks, including latent space exploitation, multi-modal cross-attack vectors, and feedback-loop-induced model degradation. Our framework employs a layered approach, incorporating anomaly detection, continuous red-teaming, and real-time adversarial simulation to mitigate these risks. We focus on formal verification methods to ensure model robustness and scalability in the face of evolving threats. Though theoretical, this work sets the stage for future empirical validation by establishing a detailed methodology and metrics for evaluating the performance of risk mitigation strategies in generative AI systems. This framework addresses existing gaps in AI safety, offering a comprehensive road map for future research and implementation.