pwc-archive/papers-with-abstracts huggingface dataset:
FollowNet: A Comprehensive Benchmark for Car-Following Behavior Modeling
Authors: Xianda Chen, Meixin Zhu, Kehua Chen, Pengqin Wang, Hongliang Lu, Hui Zhong, Xu Han, Yinhai Wang
Date: 2023-05-25 00:00:00
Tasks: Autonomous Vehicles, object-detection, Object Detection

Car-following is a control process in which a following vehicle (FV) adjusts its acceleration to keep a safe distance from the lead vehicle (LV). Recently, there has been a booming of data-driven models that enable more accurate modeling of car-following through real-world driving datasets. Although there are several public datasets available, their formats are not always consistent, making it challenging to determine the state-of-the-art models and how well a new model performs compared to existing ones. In contrast, research fields such as image recognition and object detection have benchmark datasets like ImageNet, Microsoft COCO, and KITTI. To address this gap and promote the development of microscopic traffic flow modeling, we establish a public benchmark dataset for car-following behavior modeling. The benchmark consists of more than 80K car-following events extracted from five public driving datasets using the same criteria. These events cover diverse situations including different road types, various weather conditions, and mixed traffic flows with autonomous vehicles. Moreover, to give an overview of current progress in car-following modeling, we implemented and tested representative baseline models with the benchmark. Results show that the deep deterministic policy gradient (DDPG) based model performs competitively with a lower MSE for spacing compared to traditional intelligent driver model (IDM) and Gazis-Herman-Rothery (GHR) models, and a smaller collision rate compared to fully connected neural network (NN) and long short-term memory (LSTM) models in most datasets. The established benchmark will provide researchers with consistent data formats and metrics for cross-comparing different car-following models, promoting the development of more accurate models. We open-source our dataset and implementation code in https://github.com/HKUST-DRIVE-AI-LAB/FollowNet.
DiffusionShield: A Watermark for Copyright Protection against Generative Diffusion Models
Authors: Yingqian Cui, Jie Ren, Han Xu, Pengfei He, Hui Liu, Lichao Sun, Yue Xing, Jiliang Tang
Date: 2023-05-25 00:00:00

Recently, Generative Diffusion Models (GDMs) have showcased their remarkable capabilities in learning and generating images. A large community of GDMs has naturally emerged, further promoting the diversified applications of GDMs in various fields. However, this unrestricted proliferation has raised serious concerns about copyright protection. For example, artists including painters and photographers are becoming increasingly concerned that GDMs could effortlessly replicate their unique creative works without authorization. In response to these challenges, we introduce a novel watermarking scheme, DiffusionShield, tailored for GDMs. DiffusionShield protects images from copyright infringement by GDMs through encoding the ownership information into an imperceptible watermark and injecting it into the images. Its watermark can be easily learned by GDMs and will be reproduced in their generated images. By detecting the watermark from generated images, copyright infringement can be exposed with evidence. Benefiting from the uniformity of the watermarks and the joint optimization method, DiffusionShield ensures low distortion of the original image, high watermark detection performance, and the ability to embed lengthy messages. We conduct rigorous and comprehensive experiments to show the effectiveness of DiffusionShield in defending against infringement by GDMs and its superiority over traditional watermarking methods. The code for DiffusionShield is accessible in https://github.com/Yingqiancui/DiffusionShield.
Interactive Fashion Content Generation Using LLMs and Latent Diffusion Models
Authors: Krishna Sri Ipsit Mantri, Nevasini Sasikumar
Date: 2023-05-15 00:00:00
Tasks: Image Generation, Text to Image Generation, Text-to-Image Generation, Virtual Try-on

Fashionable image generation aims to synthesize images of diverse fashion prevalent around the globe, helping fashion designers in real-time visualization by giving them a basic customized structure of how a specific design preference would look in real life and what further improvements can be made for enhanced customer satisfaction. Moreover, users can alone interact and generate fashionable images by just giving a few simple prompts. Recently, diffusion models have gained popularity as generative models owing to their flexibility and generation of realistic images from Gaussian noise. Latent diffusion models are a type of generative model that use diffusion processes to model the generation of complex data, such as images, audio, or text. They are called "latent" because they learn a hidden representation, or latent variable, of the data that captures its underlying structure. We propose a method exploiting the equivalence between diffusion models and energy-based models (EBMs) and suggesting ways to compose multiple probability distributions. We describe a pipeline on how our method can be used specifically for new fashionable outfit generation and virtual try-on using LLM-guided text-to-image generation. Our results indicate that using an LLM to refine the prompts to the latent diffusion model assists in generating globally creative and culturally diversified fashion styles and reducing bias.
Sequential Graph Neural Networks for Source Code Vulnerability Identification
Authors: Ammar Ahmed, Anwar Said, Mudassir Shabbir, Xenofon Koutsoukos
Date: 2023-05-23 00:00:00
Tasks: Graph Classification, Graph Neural Network

Vulnerability identification constitutes a task of high importance for cyber security. It is quite helpful for locating and fixing vulnerable functions in large applications. However, this task is rather challenging owing to the absence of reliable and adequately managed datasets and learning models. Existing solutions typically rely on human expertise to annotate datasets or specify features, which is prone to error. In addition, the learning models have a high rate of false positives. To bridge this gap, in this paper, we present a properly curated C/C++ source code vulnerability dataset, denoted as CVEFunctionGraphEmbeddings (CVEFGE), to aid in developing models. CVEFGE is automatically crawled from the CVE database, which contains authentic and publicly disclosed source code vulnerabilities. We also propose a learning framework based on graph neural networks, denoted SEquential Graph Neural Network (SEGNN) for learning a large number of code semantic representations. SEGNN consists of a sequential learning module, graph convolution, pooling, and fully connected layers. Our evaluations on two datasets and four baseline methods in a graph classification setting demonstrate state-of-the-art results.
Anomaly Detection in Satellite Videos using Diffusion Models
Authors: Akash Awasthi, Son Ly, Jaer Nizam, Samira Zare, Videet Mehta, Safwan Ahmed, Keshav Shah, Ramakrishna Nemani, Saurabh Prasad, Hien Van Nguyen
Date: 2023-05-25 00:00:00
Tasks: Anomaly Detection, Management

The definition of anomaly detection is the identification of an unexpected event. Real-time detection of extreme events such as wildfires, cyclones, or floods using satellite data has become crucial for disaster management. Although several earth-observing satellites provide information about disasters, satellites in the geostationary orbit provide data at intervals as frequent as every minute, effectively creating a video from space. There are many techniques that have been proposed to identify anomalies in surveillance videos; however, the available datasets do not have dynamic behavior, so we discuss an anomaly framework that can work on very high-frequency datasets to find very fast-moving anomalies. In this work, we present a diffusion model which does not need any motion component to capture the fast-moving anomalies and outperforms the other baseline methods.
SIVD: Dataset of Iranian Vehicles for Real-Time Multi-Camera Video Tracking and Recognition
Authors: Farbod Siahkali, Seyed Amirmahdi Alavi, Mehdi Tale Masouleh
Date: 2023-02-22 00:00:00
Proceeding: ICSPIS 2023 2
Tasks: Object Tracking, Vehicle Re-Identification

In this paper, a new publicly available 1 web-Scraped Iranian Vehicle Dataset (SIVD) for simultaneous real-time vehicle tracking and recognition is proposed. The datasets provided for Iranian cars in the literature have two fundamental problems. First, the lack of images from different angles, and second, the small number of classes compared to the dispersion of car models in the real world. Therefore, for the purposes of this paper, Iranian vehicle images from car sales websites are collected, and the SIVD dataset is proposed which contains 29 classes and 36,705 images. This paper aims at developing a classification network for Iranian vehicle recognition and implement a real-time tracking system using the YOLOv5 network to perform real-time vehicle model recognition and tracking tasks simultaneously. The ResNet50 achieved an accuracy of 99.29%, the highest among the investigated classification networks.
Deep Saliency Mapping for 3D Meshes and Applications
Authors: Stavros Nousias, Gerasimos Arvanitis, Aris Lalos, Konstantinos Moustakas
Date: 2023-02-06 00:00:00
Proceeding: ACM Transactions on Multimedia Computing, Communications, and Applications 2023 2
Tasks: Saliency Prediction

Nowadays, three-dimensional (3D) meshes are widely used in various applications in different areas (e.g., industry, education, entertainment and safety). The 3D models are captured with multiple RGB-D sensors, and the sampled geometric manifolds are processed, compressed, simplified, stored, and transmitted to be reconstructed in a virtual space. These low-level processing applications require the accurate representation of the 3D models that can be achieved through saliency estimation mechanisms that identify specific areas of the 3D model representing surface patches of importance. Therefore, saliency maps guide the selection of feature locations facilitating the prioritization of 3D manifold segments and attributing to vertices more bits during compression or lower decimation probability during simplification, since compression and simplification are counterparts of the same process. In this work, we present a novel deep saliency mapping approach applied to 3D meshes, emphasizing decreasing the execution time of the saliency map estimation, especially when compared with the corresponding time by other relevant approaches. Our method utilizes baseline 3D importance maps to train convolutional neural networks. Furthermore, we present applications that utilize the extracted saliency, namely feature-aware multiscale compression and simplification frameworks.
Superpixelwise Low-Rank Approximation-Based Partial Label Learning for Hyperspectral Image Classification
Authors: Shujun Yang; Yu Zhang; Yao Ding; Danfeng Hong
Date: 2023-05-25 00:00:00
Proceeding: Journal(GRSL) 2023 5
Tasks: Hyperspectral Image Classification, image-classification, Image Classification, Partial Label Learning, valid

Insufficient prior knowledge of a captured hyperspectral image (HSI) scene may lead the experts or the automatic labeling systems to offer incorrect labels or ambiguous labels (i.e., assigning each training sample to a group of candidate labels, among which only one of them is valid; this is also known as partial label learning) during the labeling process. Accordingly, how to learn from such data with ambiguous labels is a problem of great practical importance. In this letter, we propose a novel superpixelwise low-rank approximation (LRA)-based partial label learning method, namely SLAP, which is the first to take into account partial label learning in HSI classification. SLAP is mainly composed of two phases: disambiguating the training labels and acquiring the predictive model. Specifically, in the first phase, we propose a superpixelwise LRA-based model, preparing the affinity graph for the subsequent label propagation process while extracting the discriminative representation to enhance the following classification task of the second phase. Then to disambiguate the training labels, label propagation propagates the labeling information via the affinity graph of training pixels. In the second phase, we take advantage of the resulting disambiguated training labels and the discriminative representations to enhance the classification performance. The extensive experiments validate the advantage of the proposed SLAP method over state-of-the-art methods.
Bilevel Entropy based Mechanism Design for Balancing Meta in Video Games
Authors: Sumedh Pendurkar, Chris Chow, Luo Jie, Guni Sharon
Date: 2023-05-27 00:00:00
Proceeding: Autonomous Agents and Multi Agent Systems (AAMAS) 2023 5

We address a mechanism design problem where the goal of the
designer is to maximize the entropy of a player’s mixed strategy at
a Nash equilibrium. This objective is of special relevance to video
games where game designers wish to diversify the players’ interaction with the game. To solve this design problem, we propose a
bi-level alternating optimization technique that (1) approximates
the mixed strategy Nash equilibrium using a Nash Monte-Carlo
reinforcement learning approach and (2) applies a gradient-free optimization technique (Covariance-Matrix Adaptation Evolutionary
Strategy) to maximize the entropy of the mixed strategy obtained in
level (1). The experimental results show that our approach achieves
comparable results to the state-of-the-art approach on three benchmark domains “Rock-Paper-Scissors-Fire-Water”, “Workshop Warfare” and “Pokemon Video Game Championship”. Next, we show
that, unlike previous state-of-the-art approaches, the computational
complexity of our proposed approach scales significantly better in
larger combinatorial strategy spaces.
A Three-Stage Anomaly Detection Framework for Traffic Videos
Authors: Junzhou Chen; Jiancheng Wang; Jiajun Pu; Ronghui Zhang
Date: 2022-06-05 00:00:00
Proceeding: journal 2022 6
Tasks: Anomaly Detection

As reported by the United Nations in 2021, road accidents cause 1.3 million deaths and 50 million injuries worldwide each year. Detecting traffic anomalies timely and taking immediate emergency response and rescue measures are essential to reduce casualties, economic losses, and traffic congestion. This paper proposed a three-stage method for video-based traffic anomaly detection. In the first stage, the ViVit network is employed as a feature extractor to capture the spatiotemporal features from the input video. In the second stage, the class and patch tokens are fed separately to the segment-level and video-level traffic anomaly detectors. In the third stage, we finished the construction of the entire composite traffic anomaly detection framework by fusing outputs of two traffic anomaly detectors above with different granularity. Experimental evaluation demonstrates that the proposed method outperforms the SOTA method with 2.07% AUC on the TAD testing overall set and 1.43% AUC on the TAD testing anomaly subset. This work provides a new reference for traffic anomaly detection research.
Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features
Authors: Amin Ullah, Jamil Ahmad, Khan Muhammad, Muhammad Sajjad, Sung Wook Baik
Date: 2017-11-28 00:00:00
Proceeding: IEEE Access 2017 11
Tasks: Action Recognition, Activity Recognition, speech-recognition, Speech Recognition

Recurrent neural network (RNN) and long short-term memory (LSTM) have achieved great success in processing sequential multimedia data and yielded the state-of-the-art results in speech recognition, digital signal processing, video processing, and text data analysis. In this paper, we propose a novel action recognition method by processing the video data using convolutional neural network (CNN) and deep bidirectional LSTM (DB-LSTM) network. First, deep features are extracted from every sixth frame of the videos, which helps reduce the redundancy and complexity. Next, the sequential information among frame features is learnt using DB-LSTM network, where multiple layers are stacked together in both forward pass and backward pass of DB-LSTM to increase its depth. The proposed method is capable of learning long term sequences and can process lengthy videos by analyzing features for a certain time interval. Experimental results show significant improvements in action recognition using the proposed method on three benchmark data sets including UCF-101, YouTube 11 Actions, and HMDB51 compared with the state-of-the-art action recognition methods.
Neural FIM for learning Fisher Information Metrics from point cloud data
Authors: Oluwadamilola Fasina, Guillaume Huguet, Alexander Tong, Yanlei Zhang, Guy Wolf, Maximilian Nickel, Ian Adelstein, Smita Krishnaswamy
Date: 2023-06-01 00:00:00

Although data diffusion embeddings are ubiquitous in unsupervised learning and have proven to be a viable technique for uncovering the underlying intrinsic geometry of data, diffusion embeddings are inherently limited due to their discrete nature. To this end, we propose neural FIM, a method for computing the Fisher information metric (FIM) from point cloud data - allowing for a continuous manifold model for the data. Neural FIM creates an extensible metric space from discrete point cloud data such that information from the metric can inform us of manifold characteristics such as volume and geodesics. We demonstrate Neural FIM's utility in selecting parameters for the PHATE visualization method as well as its ability to obtain information pertaining to local volume illuminating branching points and cluster centers embeddings of a toy dataset and two single-cell datasets of IPSC reprogramming and PBMCs (immune cells).
Carefully Blending Adversarial Training, Purification, and Aggregation Improves Adversarial Robustness
Authors: Emanuele Ballarin, Alessio Ansuini, Luca Bortolussi
Date: 2023-05-25 00:00:00
Tasks: Adversarial Defense, Adversarial Purification, Adversarial Robustness, image-classification, Image Classification, Robust classification

In this work, we propose a novel adversarial defence mechanism for image classification - CARSO - blending the paradigms of adversarial training and adversarial purification in a synergistic robustness-enhancing way. The method builds upon an adversarially-trained classifier, and learns to map its internal representation associated with a potentially perturbed input onto a distribution of tentative clean reconstructions. Multiple samples from such distribution are classified by the same adversarially-trained model, and a carefully chosen aggregation of its outputs finally constitutes the robust prediction of interest. Experimental evaluation by a well-established benchmark of strong adaptive attacks, across different image datasets, shows that CARSO is able to defend itself against adaptive end-to-end white-box attacks devised for stochastic defences. Paying a modest clean accuracy toll, our method improves by a significant margin the state-of-the-art for Cifar-10, Cifar-100, and TinyImageNet-200 $\ell_\infty$ robust classification accuracy against AutoAttack. Code, and instructions to obtain pre-trained models are available at: https://github.com/emaballarin/CARSO .
ModeT: Learning Deformable Image Registration via Motion Decomposition Transformer
Authors: Haiqiao Wang, Dong Ni, Yi Wang
Date: 2023-06-09 00:00:00
Tasks: Image Registration, Medical Image Registration

The Transformer structures have been widely used in computer vision and have recently made an impact in the area of medical image registration. However, the use of Transformer in most registration networks is straightforward. These networks often merely use the attention mechanism to boost the feature learning as the segmentation networks do, but do not sufficiently design to be adapted for the registration task. In this paper, we propose a novel motion decomposition Transformer (ModeT) to explicitly model multiple motion modalities by fully exploiting the intrinsic capability of the Transformer structure for deformation estimation. The proposed ModeT naturally transforms the multi-head neighborhood attention relationship into the multi-coordinate relationship to model multiple motion modes. Then the competitive weighting module (CWM) fuses multiple deformation sub-fields to generate the resulting deformation field. Extensive experiments on two public brain magnetic resonance imaging (MRI) datasets show that our method outperforms current state-of-the-art registration networks and Transformers, demonstrating the potential of our ModeT for the challenging non-rigid deformation estimation problem. The benchmarks and our code are publicly available at https://github.com/ZAX130/SmileCode.
Realistic Saliency Guided Image Enhancement
Authors: S. Mahdi H. Miangoleh, Zoya Bylinskii, Eric Kee, Eli Shechtman, Yağız Aksoy
Date: 2023-06-09 00:00:00
Proceeding: CVPR 2023 1
Tasks: Image Enhancement

Common editing operations performed by professional photographers include the cleanup operations: de-emphasizing distracting elements and enhancing subjects. These edits are challenging, requiring a delicate balance between manipulating the viewer's attention while maintaining photo realism. While recent approaches can boast successful examples of attention attenuation or amplification, most of them also suffer from frequent unrealistic edits. We propose a realism loss for saliency-guided image enhancement to maintain high realism across varying image types, while attenuating distractors and amplifying objects of interest. Evaluations with professional photographers confirm that we achieve the dual objective of realism and effectiveness, and outperform the recent approaches on their own datasets, while requiring a smaller memory footprint and runtime. We thus offer a viable solution for automating image enhancement and photo cleanup operations.
DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particles
Authors: Tal Daniel, Aviv Tamar
Date: 2023-06-09 00:00:00
Tasks: Object, Position, Unconditional Video Generation, Video Generation, Video Prediction

We propose a new object-centric video prediction algorithm based on the deep latent particle (DLP) representation. In comparison to existing slot- or patch-based representations, DLPs model the scene using a set of keypoints with learned parameters for properties such as position and size, and are both efficient and interpretable. Our method, deep dynamic latent particles (DDLP), yields state-of-the-art object-centric video prediction results on several challenging datasets. The interpretable nature of DDLP allows us to perform ``what-if'' generation -- predict the consequence of changing properties of objects in the initial frames, and DLP's compact structure enables efficient diffusion-based unconditional video generation. Videos, code and pre-trained models are available: https://taldatech.github.io/ddlp-web
Prodigy: An Expeditiously Adaptive Parameter-Free Learner
Authors: Konstantin Mishchenko, Aaron Defazio
Date: 2023-06-09 00:00:00

We consider the problem of estimating the learning rate in adaptive methods, such as AdaGrad and Adam. We propose Prodigy, an algorithm that provably estimates the distance to the solution $D$, which is needed to set the learning rate optimally. At its core, Prodigy is a modification of the D-Adaptation method for learning-rate-free learning. It improves upon the convergence rate of D-Adaptation by a factor of $O(\sqrt{\log(D/d_0)})$, where $d_0$ is the initial estimate of $D$. We test Prodigy on 12 common logistic-regression benchmark datasets, VGG11 and ResNet-50 training on CIFAR10, ViT training on Imagenet, LSTM training on IWSLT14, DLRM training on Criteo dataset, VarNet on Knee MRI dataset, as well as RoBERTa and GPT transformer training on BookWiki. Our experimental results show that our approach consistently outperforms D-Adaptation and reaches test accuracy values close to that of hand-tuned Adam.