pwc-archive/papers-with-abstracts huggingface dataset:
STIGCN: spatial–temporal interaction‑aware graph convolution network for pedestrian trajectory prediction
Authors: Wangxing Chen, Haifeng Sang, Jinyu Wang, Zishan Zhao
Date: 2023-12-30 00:00:00
Proceeding: The Journal of Supercomputing 2023 12
Tasks: Autonomous Driving, Pedestrian Trajectory Prediction, Prediction, Robot Navigation, Trajectory Prediction

Accurately predicting the future trajectory of pedestrians is critical for tasks such as autonomous driving and robot navigation. Previous methods for pedestrian trajectory prediction dealt with social interaction and pedestrian movement factors either concurrently or sequentially, neglecting the link between them. Therefore, a Spatial-Temporal Interaction-aware Graph Convolution Network (STIGCN) is proposed for pedestrian trajectory prediction. STIGCN considers the correlation between social interaction and pedestrian movement factors to achieve more accurate interaction modeling. Specifically, we first constructed spatial and temporal graphs to model social interactions and movement factors. Then, we designed the spatial-temporal interaction-aware learning to utilize the spatial interaction features of each moment to assist the temporal interaction modeling and utilize the temporal interaction features of each pedestrian to assist the spatial interaction modeling, resulting in more accurate interaction modeling. Finally, a Time-Extrapolator Pyramid Convolution Neural Network (TEP-CNN) is designed to jointly estimate the two-dimensional Gaussian distribution parameters of future trajectories by combining the prediction features from multiple layers. Experimental results on two benchmark pedestrian trajectory prediction datasets show that our proposed method outperforms existing methods in terms of average displacement error and final displacement error and achieves more accurate predictions for pedestrian motions such as convergence and encounter.
Uni-MIS: United Multiple Intent Spoken Language Understanding via Multi-View Intent-Slot Interaction
Authors: Shangjian Yin, Peijie Huang, Yuhong Xu
Date: 2024-03-24 00:00:00
Proceeding: Proceedings of the AAAI Conference on Artificial Intelligence 2024 3
Tasks: Intent Detection, slot-filling, Slot Filling, Spoken Language Understanding

So far, multi-intent spoken language understanding (SLU) has become a research hotspot in the field of natural language processing (NLP) due to its ability to recognize and extract multiple intents expressed and annotate corresponding sequence slot tags within a single utterance. Previous research has primarily concentrated on the token-level intent-slot interaction to model joint intent detection and slot filling, which resulted in a failure to fully utilize anisotropic intent-guiding information during joint training. In this work, we present a novel architecture by modeling the multi-intent SLU as a multi-view intent-slot interaction. The architecture resolves the kernel bottleneck of unified multi-intent SLU by effectively modeling the intent-slot relations with utterance, chunk, and token-level interaction. We further develop a neural framework, namely Uni-MIS, in which the unified multi-intent SLU is modeled as a three-view intent-slot interaction fusion to better capture the interaction information after special encoding. A chunk-level intent detection decoder is used to sufficiently capture the multi-intent, and an adaptive intent-slot graph network is used to capture the fine-grained intent information to guide final slot filling. We perform extensive experiments on two widely used benchmark datasets for multi-intent SLU, where our model bets on all the current strong baselines, pushing the state-of-the-art performance of unified multi-intent SLU. Additionally, the ChatGPT benchmark that we have developed demonstrates that there is a considerable amount of potential research value in the field of multi-intent SLU.
FishSegSSL: A Semi-Supervised Semantic Segmentation Framework for Fish-Eye Images
Authors: Sneha Paul, Zachary Patterson, Nizar Bouguila
Date: 2024-03-15 00:00:00
Proceeding: Journal of Imaging 2024 3
Tasks: Autonomous Driving, Image Segmentation, Pseudo Label, Pseudo Label Filtering, Segmentation, Semantic Segmentation, Semi-Supervised Semantic Segmentation

The application of large field-of-view (FoV) cameras equipped with fish-eye lenses brings notable advantages to various real-world computer vision applications, including autonomous driving. While deep learning has proven successful in conventional computer vision applications using regular perspective images, its potential in fish-eye camera contexts remains largely unexplored due to limited datasets for fully supervised learning. Semi-supervised learning comes as a potential solution to manage this challenge. In this study, we explore and benchmark two popular semi-supervised methods from the perspective image domain for fish-eye image segmentation. We further introduce FishSegSSL, a novel fish-eye image segmentation framework featuring three semi-supervised components: pseudo-label filtering, dynamic confidence thresholding, and robust strong augmentation. Evaluation on the WoodScape dataset, collected from vehicle-mounted fish-eye cameras, demonstrates that our proposed method enhances the model’s performance by up to 10.49% over fully supervised methods using the same amount of labeled data. Our method also improves the existing image segmentation methods by 2.34%. To the best of our knowledge, this is the first work on semi-supervised semantic segmentation on fish-eye images. Additionally, we conduct a comprehensive ablation study and sensitivity analysis to showcase the efficacy of each proposed method in this research.
CrossMoCo: Multi-modal Momentum Contrastive Learning for Point Cloud
Authors: Sneha Paul, Zachary Patterson, Nizar Bouguila
Date: 2023-06-08 00:00:00
Proceeding: 20th Conference on Robots and Vision (CRV) 2023 6
Tasks: 3D Object Classification, 3D Point Cloud Classification, 3D Point Cloud Linear Classification, Contrastive Learning, Few-Shot 3D Point Cloud Classification, Few-Shot Learning, Self-Supervised Learning

The point cloud is a 3D geometric data that lacks a specific structure and is permutation-invariant. The applications of point clouds have gained significant attention recently in the field of vision tasks. However, most existing works on point clouds utilize supervised learning on large labelled data, which are costly and laborious to collect. To this end, unsupervised learning, for example, self-supervised learning, has shown promising performance in various tasks of 2D computer vision and holds the potential in 3D computer vision applications. In this study, we introduce a novel selfsupervised method called CrossMoCo, which learns the representations of unlabelled point cloud data in a multi-modal setup that also utilizes the 2D rendered images of the point clouds. CrossMoCo outperforms existing methods on multimodal self-supervised learning on point cloud by introducing two new concepts: momentum contrastive learning with more negative samples and multiple-view intra-modal contrastive learning. The first component learns from an online encoder and a momentum encoder with a large number of negative samples, which provides consistent learning signals. The second component enforces consistency between different views of the samples of the same modality, thereby improving multimodal representation. We conduct extensive studies on two popular benchmark datasets (ModelNet40 and ScanObjectNN) for linear classification and few-shot learning tasks. Our results demonstrate that CrossMoCo achieves superior performance over existing methods for both tasks on both datasets, achieving up to 4.36% improvement on linear classification and up to 9.2% on few-shot tasks. Our code is available at https://github.com/snehaputul/CrossMoCo.
DualMLP: a two-stream fusion model for 3D point cloud classification
Authors: Sneha Paul, Zachary Patterson, Nizar Bouguila
Date: 2023-10-10 00:00:00
Proceeding: The Visual Computer 2023 10
Tasks: 3D Point Cloud Classification, Point Cloud Classification, Scene Understanding

In this paper, we present DualMLP, a novel 3D model that introduces the idea of a two-stream network for existing 3D models to handle the trade-off between the number of points and the computational overhead. Existing works on point clouds use a small subset of points sampled from the entire 3D object as input. Although increasing the number of input points can enhance scene understanding, it also incurs a higher computational cost for existing networks. To tackle this challenge, we propose a novel architecture called DualMLP, which effectively mitigates the linear increase in computational expense as the number of input points grows. While we evaluate this concept on PointMLP and demonstrate its effectiveness, the idea can be applied to other existing models with minimal adjustments. DualMLP consists of two branches: DenseNet and SparseNet. The SparseNet, a relatively larger network, samples a small number of points from the complete point cloud, while the DenseNet, a lightweight network, takes in a larger number of points as input. Extensive experiments on the ScanObjectNN and ModelNet40 datasets demonstrate the effectiveness of the proposed model, achieving a 1.00% and 0.81% improvement over PointMLP for ScanObjectNN and ModelNet40 while being computationally efficient than the original PointMLP. To ensure the reproducibility of our experimental results, the code for this work is publicly available at https://github.com/snehaputul/DualMLP.
Semantic Segmentation Using Transfer Learning on Fisheye Images
Authors: Sneha Paul, Zachary Patterson, Nizar Bouguila
Date: 2023-12-17 00:00:00
Proceeding: International Conference on Machine Learning and Applications (ICMLA) 2023 12
Tasks: Image Segmentation, Segmentation, Semantic Segmentation, Transfer Learning

While semantic segmentation has been extensively studied in the realm of regular perspective images, its application to fisheye images remains relatively unexplored. Existing literature on fisheye semantic segmentation mostly revolves around multi-task or multi-modal models, which are computationally intensive. This motivated us to assess the performance of current segmentation methods, specifically on fisheye images. Surprisingly, we discover that these methods do not yield satisfactory results when directly trained on fisheye datasets using a fully supervised approach. This can be attributed to the fact that the models are not designed to handle fisheye images, and the available fisheye datasets are not sufficiently large to effectively train complex models. To overcome these challenges, we propose a novel training method by employing Transfer Learning (TL) on existing semantic segmentation models that concentrate on a single task and modality. To achieve this, we investigate six different fine-tuning configurations using the WoodScape fisheye image segmentation dataset. Furthermore, we introduce a pre-training stage that learns from perspective images by applying a fisheye transformation before employing transfer learning. As a result, our proposed training pipeline demonstrates a remarkable 18.29% improvement in mean Intersection over Union (mIoU) compared to directly adopting the best existing segmentation methods for fish eye images.
trajdata: A Unified Interface to Multiple Human Trajectory Datasets
Authors: Boris Ivanovic, Guanyu Song, Igor Gilitschenski, Marco Pavone
Date: 2023-07-26 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Autonomous Vehicles, Motion Forecasting, Trajectory Forecasting

The field of trajectory forecasting has grown significantly in recent years, partially owing to the release of numerous large-scale, real-world human trajectory datasets for autonomous vehicles (AVs) and pedestrian motion tracking. While such datasets have been a boon for the community, they each use custom and unique data formats and APIs, making it cumbersome for researchers to train and evaluate methods across multiple datasets. To remedy this, we present trajdata: a unified interface to multiple human trajectory datasets. At its core, trajdata provides a simple, uniform, and efficient representation and API for trajectory and map data. As a demonstration of its capabilities, in this work we conduct a comprehensive empirical evaluation of existing trajectory datasets, providing users with a rich understanding of the data underpinning much of current pedestrian and AV motion forecasting research, and proposing suggestions for future datasets from these insights. trajdata is permissively licensed (Apache 2.0) and can be accessed online at https://github.com/NVlabs/trajdata
VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representation
Authors: Zekun Qi, Muzhou Yu, Runpei Dong, Kaisheng Ma
Date: 2023-07-28 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: 3D Generation, 8k

Conditional 3D generation is undergoing a significant advancement, enabling the free creation of 3D content from inputs such as text or 2D images. However, previous approaches have suffered from low inference efficiency, limited generation categories, and restricted downstream applications. In this work, we revisit the impact of different 3D representations on generation quality and efficiency. We propose a progressive generation method through Voxel-Point Progressive Representation (VPP). VPP leverages structured voxel representation in the proposed Voxel Semantic Generator and the sparsity of unstructured point representation in the Point Upsampler, enabling efficient generation of multi-category objects. VPP can generate high-quality 8K point clouds within 0.2 seconds. Additionally, the masked generation Transformer allows for various 3D downstream tasks, such as generation, editing, completion, and pre-training. Extensive experiments demonstrate that VPP efficiently generates high-fidelity and diverse 3D shapes across different categories, while also exhibiting excellent representation transfer performance. Codes will be released at \url{https://github.com/qizekun/VPP}.
Heterogeneous Network Based Contrastive Learning Method for PolSAR Land Cover Classification
Authors: JianFeng Cai, Yue Ma, Zhixi Feng, Shuyuan Yang
Date: 2024-03-29 00:00:00
Tasks: Contrastive Learning, image-classification, Image Classification, Land Cover Classification, Superpixels

Polarimetric synthetic aperture radar (PolSAR) image interpretation is widely used in various fields. Recently, deep learning has made significant progress in PolSAR image classification. Supervised learning (SL) requires a large amount of labeled PolSAR data with high quality to achieve better performance, however, manually labeled data is insufficient. This causes the SL to fail into overfitting and degrades its generalization performance. Furthermore, the scattering confusion problem is also a significant challenge that attracts more attention. To solve these problems, this article proposes a Heterogeneous Network based Contrastive Learning method(HCLNet). It aims to learn high-level representation from unlabeled PolSAR data for few-shot classification according to multi-features and superpixels. Beyond the conventional CL, HCLNet introduces the heterogeneous architecture for the first time to utilize heterogeneous PolSAR features better. And it develops two easy-to-use plugins to narrow the domain gap between optics and PolSAR, including feature filter and superpixel-based instance discrimination, which the former is used to enhance the complementarity of multi-features, and the latter is used to increase the diversity of negative samples. Experiments demonstrate the superiority of HCLNet on three widely used PolSAR benchmark datasets compared with state-of-the-art methods. Ablation studies also verify the importance of each component. Besides, this work has implications for how to efficiently utilize the multi-features of PolSAR data to learn better high-level representation in CL and how to construct networks suitable for PolSAR data better.
Mol-AIR: Molecular Reinforcement Learning with Adaptive Intrinsic Rewards for Goal-directed Molecular Generation
Authors: Jinyeong Park, Jaegyoon Ahn, Jonghwan Choi, Jibum Kim
Date: 2024-03-29 00:00:00
Tasks: Drug Discovery, reinforcement-learning, Reinforcement Learning

Optimizing techniques for discovering molecular structures with desired properties is crucial in artificial intelligence(AI)-based drug discovery. Combining deep generative models with reinforcement learning has emerged as an effective strategy for generating molecules with specific properties. Despite its potential, this approach is ineffective in exploring the vast chemical space and optimizing particular chemical properties. To overcome these limitations, we present Mol-AIR, a reinforcement learning-based framework using adaptive intrinsic rewards for effective goal-directed molecular generation. Mol-AIR leverages the strengths of both history-based and learning-based intrinsic rewards by exploiting random distillation network and counting-based strategies. In benchmark tests, Mol-AIR demonstrates superior performance over existing approaches in generating molecules with desired properties without any prior knowledge, including penalized LogP, QED, and celecoxib similarity. We believe that Mol-AIR represents a significant advancement in drug discovery, offering a more efficient path to discovering novel therapeutics.
UltraLight VM-UNet: Parallel Vision Mamba Significantly Reduces Parameters for Skin Lesion Segmentation
Authors: Renkai Wu, Yinghao Liu, Pengchen Liang, Qing Chang
Date: 2024-03-29 00:00:00
Tasks: Image Segmentation, Lesion Segmentation, Mamba, Medical Image Segmentation, Segmentation, Semantic Segmentation, Skin Lesion Segmentation, State Space Models

Traditionally for improving the segmentation performance of models, most approaches prefer to use adding more complex modules. And this is not suitable for the medical field, especially for mobile medical devices, where computationally loaded models are not suitable for real clinical environments due to computational resource constraints. Recently, state-space models (SSMs), represented by Mamba, have become a strong competitor to traditional CNNs and Transformers. In this paper, we deeply explore the key elements of parameter influence in Mamba and propose an UltraLight Vision Mamba UNet (UltraLight VM-UNet) based on this. Specifically, we propose a method for processing features in parallel Vision Mamba, named PVM Layer, which achieves excellent performance with the lowest computational load while keeping the overall number of processing channels constant. We conducted comparisons and ablation experiments with several state-of-the-art lightweight models on three skin lesion public datasets and demonstrated that the UltraLight VM-UNet exhibits the same strong performance competitiveness with parameters of only 0.049M and GFLOPs of 0.060. In addition, this study deeply explores the key elements of parameter influence in Mamba, which will lay a theoretical foundation for Mamba to possibly become a new mainstream module for lightweighting in the future. The code is available from https://github.com/wurenkai/UltraLight-VM-UNet .
ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning
Authors: Beomyoung Kim, Joonsang Yu, Sung Ju Hwang
Date: 2024-03-29 00:00:00
Proceeding: CVPR 2024 1
Tasks: Continual Learning, Continual Panoptic Segmentation, Instance Segmentation, Knowledge Distillation, Panoptic Segmentation, Segmentation, Semantic Segmentation, Visual Prompt Tuning

Panoptic segmentation, combining semantic and instance segmentation, stands as a cutting-edge computer vision task. Despite recent progress with deep learning models, the dynamic nature of real-world applications necessitates continual learning, where models adapt to new classes (plasticity) over time without forgetting old ones (catastrophic forgetting). Current continual segmentation methods often rely on distillation strategies like knowledge distillation and pseudo-labeling, which are effective but result in increased training complexity and computational overhead. In this paper, we introduce a novel and efficient method for continual panoptic segmentation based on Visual Prompt Tuning, dubbed ECLIPSE. Our approach involves freezing the base model parameters and fine-tuning only a small set of prompt embeddings, addressing both catastrophic forgetting and plasticity and significantly reducing the trainable parameters. To mitigate inherent challenges such as error propagation and semantic drift in continual segmentation, we propose logit manipulation to effectively leverage common knowledge across the classes. Experiments on ADE20K continual panoptic segmentation benchmark demonstrate the superiority of ECLIPSE, notably its robustness against catastrophic forgetting and its reasonable plasticity, achieving a new state-of-the-art. The code is available at https://github.com/clovaai/ECLIPSE.
FABind+: Enhancing Molecular Docking through Improved Pocket Prediction and Pose Generation
Authors: Kaiyuan Gao, Qizhi Pei, Gongbo Zhang, Jinhua Zhu, Kun He, Lijun Wu
Date: 2024-03-29 00:00:00
Tasks: Blind Docking, Drug Discovery, Molecular Docking

Molecular docking is a pivotal process in drug discovery. While traditional techniques rely on extensive sampling and simulation governed by physical principles, these methods are often slow and costly. The advent of deep learning-based approaches has shown significant promise, offering increases in both accuracy and efficiency. Building upon the foundational work of FABind, a model designed with a focus on speed and accuracy, we present FABind+, an enhanced iteration that largely boosts the performance of its predecessor. We identify pocket prediction as a critical bottleneck in molecular docking and propose a novel methodology that significantly refines pocket prediction, thereby streamlining the docking process. Furthermore, we introduce modifications to the docking module to enhance its pose generation capabilities. In an effort to bridge the gap with conventional sampling/generative methods, we incorporate a simple yet effective sampling technique coupled with a confidence model, requiring only minor adjustments to the regression framework of FABind. Experimental results and analysis reveal that FABind+ remarkably outperforms the original FABind, achieves competitive state-of-the-art performance, and delivers insightful modeling strategies. This demonstrates FABind+ represents a substantial step forward in molecular docking and drug discovery. Our code is in https://github.com/QizhiPei/FABind.
Unsolvable Problem Detection: Evaluating Trustworthiness of Vision Language Models
Authors: Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, Kiyoharu Aizawa
Date: 2024-03-29 00:00:00
Tasks: Question Answering, Visual Question Answering, Visual Question Answering (VQA)

This paper introduces a novel and significant challenge for Vision Language Models (VLMs), termed Unsolvable Problem Detection (UPD). UPD examines the VLM's ability to withhold answers when faced with unsolvable problems in the context of Visual Question Answering (VQA) tasks. UPD encompasses three distinct settings: Absent Answer Detection (AAD), Incompatible Answer Set Detection (IASD), and Incompatible Visual Question Detection (IVQD). To deeply investigate the UPD problem, extensive experiments indicate that most VLMs, including GPT-4V and LLaVA-Next-34B, struggle with our benchmarks to varying extents, highlighting significant room for the improvements. To address UPD, we explore both training-free and training-based solutions, offering new insights into their effectiveness and limitations. We hope our insights, together with future efforts within the proposed UPD settings, will enhance the broader understanding and development of more practical and reliable VLMs.