pwc-archive/papers-with-abstracts huggingface dataset:
CalibRBEV: Multi-Camera Calibration via ReversedBird's-eye-view Representations for Autonomous Driving.
Authors: Wenlong Liao, Sunyuan Qiang, Xianfei Li, Xiaolei Chen, Haoyu Wang, Yanyan Liang, Junchi Yan, Tao He, Pai Peng
Date: 2024-07-21 00:00:00
Proceeding: ACM Multimedia, 2024 2024 7
Tasks: Autonomous Driving, Camera Calibration

Camera calibration consists of determining the intrinsic and extrinsic parameters of an imaging system, which forms the fundamentalbasis for various computer vision tasks and applications, e.g., robotics and autonomous driving (AD). However, prevailing cameracalibration models pose a time-consuming and labor-intensive offboard process particularly in mass production settings, while simultaneously lacking exploration of real-world autonomous drivingscenarios. To this end, in this paper, inspired by recent advancements in bird’s-eye-view (BEV) perception models, we proposes anovel automatic multi-camera Calibration method via ReversedBEV representations for autonomous driving, termed CalibRBEV.Specifically, the proposed CalibRBEV model primarily comprisestwo stages. Initially, we innovatively reverse the BEV perceptionpipeline, reconstructing bounding boxes through an attention autoencoder module to fully extract the latent reversed BEV representations. Subsequently, the obtained representations from encoderare interacted with the surrounding multi-view image features forfurther refinement and calibration parameters prediction. Extensiveexperimental results on nuScenes and Waymo datasets validate theeffectiveness of our proposed model.
The Decoupling Concept Bottleneck Model
Authors: Rui Zhang, Xingbo Du, Junchi Yan, Shihua Zhang
Date: 2024-11-01 00:00:00
Proceeding: IEEE Transactions on Pattern Analysis and Machine Intelligence 2024 11
Tasks: model, Mutual Information Estimation

The Concept Bottleneck Model (CBM) is an interpretable neural network that leverages high-level concepts to explainmodel decisions and conduct human-machine interaction. However, in real-world scenarios, the deficiency of informative concepts canimpede the model’s interpretability and subsequent interventions. This paper proves that insufficient concept information can lead to aninherent dilemma of concept and label distortions in CBM. To address this challenge, we propose the Decoupling Concept BottleneckModel (DCBM), which comprises two phases: 1) DCBM for prediction and interpretation, which decouples heterogeneous informationinto explicit and implicit concepts while maintaining high label and concept accuracy, and 2) DCBM for human-machine interaction,which automatically corrects labels and traces wrong concepts via mutual information estimation. The construction of the interactionsystem can be formulated as a light min-max optimization problem. Extensive experiments expose the success of alleviatingconcept/label distortions, especially when concepts are insufficient. In particular, we propose the Concept Contribution Score (CCS) toquantify the interpretability of DCBM. Numerical results demonstrate that CCS can be guaranteed by the Jensen-Shannon divergenceconstraint in DCBM. Moreover, DCBM expresses two effective human-machine interactions, including forward intervention andbackward rectification, to further promote concept/label accuracy via interaction with human experts.
Learning Dynamic Graph for Overtaking Strategy in Autonomous Driving
Authors: Xuemin Hu, Yanfang Liu, Bo Tang, Junchi Yan, Long Chen
Date: 2023-06-27 00:00:00
Proceeding: IEEE Transactions on Intelligent Transportation Systems, 2023 2023 6
Tasks: Autonomous Driving, Graph Learning, Imitation Learning

Automatic overtaking is a challenging task for self-driving vehicles. Traditional rule-based methods for overtaking in autonomous driving heavily rely on many predefined rules and are difficult to apply in complex driving scenarios. Learning-based methods usually use convolutional networks, recurrent networks, and multilayer perceptrons, etc., to extract features from environments, but they fail to effectively represent geometric and interactive information among traffic participants. Classic graph convolutional networks (GCNs) have the ability of represent graph-structural information but are limited to stable relationship representation due to the fixed adjacency matrix when applied in autonomous driving. In this paper, we propose a novel dynamic graph learning method based on a graph convolutional network with a trainable adjacency matrix (TAM-GCN) to enable the learning of dynamic relationships among different nodes in an ever-changing driving scene. In addition, we develop a planning method for overtaking strategy in autonomous driving, where the proposed TAM-GCN is used to extract the spatial graph-structural features, select appropriate overtaking time, and generate efficient overtaking actions. The proposed model is trained using the imitation learning method. We conduct comprehensive experiments in both closed-loop and open-loop testing in the CARLA simulator and compare our method with state-of-the-art methods. Experimental results demonstrate the proposed method achieves better accuracy, safety and overtaking performance than existing methods.
Opendenselane: A Dense Lidar-Based Dataset for HD Map Construction
Authors: Xiaolei Chen, Wenlong Liao, Bin Liu, Junchi Yan, Tao He
Date: 2022-06-18 00:00:00
Proceeding: IEEE International Conference on Multimedia and Expo (ICME), 2022 2022 6
Tasks: Autonomous Driving

In autonomous driving system, High-Definition (HD) map is an important basis for localization, perception and planning tasks. For the construction of HD map, land marking detection is the first step. Recent studies of land marking detection are mainly based on camera image data, while LiDAR-based land marking detection is rarely studied. The main reason is that there are few datasets specially developed for land marking detection. In this paper, a new LiDAR-based land marking dataset named OpenDenseLane is developed for the construction of HD map, and is released to support the academic research. OpenDenseLane contains 1,709 scenarios with 57,227 frames, and each frame includes two types of LiDAR point cloud, camera image and localization data. The LiDAR data in our dataset are dense point clouds to reduce the impact of sparse point distribution. OpenDense-Lane provides abundant annotations of ground signs, such as lane line, crosswalk and turn arrow. Experiments are conducted on the proposed dataset and the results of land marking detection and HD map construction are analysed. Open-DenseLane will be released at https://github.com/Thinklab-SJTU/OpenDenseLane.
Trajectory Prediction from EGO View: A Coordinate Transform and Tail-Light Event Driven Approach
Authors: Wenlong Liao, Xiaolei Chen, Weipeng Zhang, Huanxi Liu, Junchi Yan, Yingxin Lou, Tao Xue, Shugi Mei
Date: 2022-06-18 00:00:00
Proceeding: IEEE International Conference on Multimedia and Expo (ICME) 2022 6
Tasks: Prediction, Trajectory Prediction

Trajectory prediction plays an important role in modern au-tonomous driving system. The multi-modal characteristics of trajectory prediction makes it difficult to accurately predict the driving intention and future trajectory of vehicles, espe-cially in the complex conditions, such as lane changing and road intersection. To improve the prediction performance, ad-ditional features are needed. High-definition (HD) map fea-ture and tail light feature are employed in this paper, which are fused with trajectory feature to assist vehicle trajectory pre-diction. A prediction model based on temporal convolution network (TCN) and graph convolution network (GCN) is constructed with corresponding loss functions. Experiments are carried out on our simulation dataset and the results show the effectiveness of the proposed feature fusion method as well as the prediction model.
Towards Capturing the Temporal Dynamics for Trajectory Prediction: a Coarse-to-Fine Approach
Authors: Xiaosong Jia, Li Chen, Penghao Wu, Jia Zeng, Junchi Yan, Hongyang Li, Yu Qiao
Date: 2023-05-06 00:00:00
Proceeding: Annual Conference on Robot Learning 2023 5
Tasks: Autonomous Driving, Decoder, Inductive Bias, Trajectory Prediction

Trajectory prediction is one of the basic tasks in the autonomous driving field, which aims to predict the future position of other agents around the ego vehicle so that a safe yet efficient driving plan could be generated in the downstream module. Recently, deep learning based methods dominate the field. State-of-the-art (SOTA) methods usually follow an encoder-decoder paradigm. Specifically, the encoder is responsible for extracting information from agents' history states and HD-Map and providing a representation vector for each agent. Taking these vectors as input, the decoder predicts multi-step future positions for each agent, which is usually accomplished by a single multi-layer perceptron (MLP) to directly output a Tx2 tensor. Though models with adoptation of MLP decoder have dominated the leaderboard of multiple datasets, `the elephant in the room is that the temporal correlation among future time-steps is ignored since there is no direct relation among output neurons of a MLP. In this work, we examine this design choice and investigate several ways to apply the temporal inductive bias into the generation of future trajectories on top of a SOTA encoder. We find that simply using autoregressive RNN to generate future positions would lead to significant performance drop even with techniques such as history highway and teacher forcing. Instead, taking scratch trajectories generated by MLP as input, an additional refinement module based on structures with temporal prior such as RNN or 1D-CNN could remarkably boost the accuracy. Furthermore, we examine several objective functions to  emphasize the temporal priors. By the combination of aforementioned techniques to introduce the temporal prior, we improve the top-ranked method's performance by a large margin and achieve SOTA result on the Waymo Open Motion Challenge.
Dynamic Modeling of Hand-Object Interactions via Tactile Sensing.
Authors: Qiang Zhang, Yunzhu Li, Yiyue Luo, Wan Shou, Michael Foshey, Junchi Yan, Joshua B. Tenenbaum, Wojciech Matusik, Antonio Torralba
Date: 2021-09-27 00:00:00
Proceeding: International Conference on Intelligent Robots and Systems (IROS) 2021 9
Tasks: Contrastive Learning, Imitation Learning, Object

Tactile sensing is critical for humans to perform everyday tasks. While significant progress has been made in analyzing object grasping from vision, it remains unclear how we can utilize tactile sensing to reason about and model the dynamics of hand-object interactions. In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model, which can then be used on its own during the test time. The tactile model aims to predict the 3d locations of both the hand and the object purely from the touch data by combining a predictive model and a contrastive learning module. This framework can reason about the interaction patterns from the tactile data, hallucinate the changes in the environment, estimate the uncertainty of the prediction, and generalize to unseen objects. We also provide detailed ablation studies regarding different system designs as well as visualizations of the predicted trajectories. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing, which opens the door for future applications in activity learning, human-computer interactions, and imitation learning for robotics.
Exploring GLU Expansion Ratios: A Study of Structured Pruning in LLaMA-3.2 Models
Authors: Pere Martra.
Date: 2024-12-26 00:00:00
Proceeding: OSF Preprints 2024 12
Tasks: Computational Efficiency, Network Pruning

Large language models with GLU architectures are typically designed with significant expansion ratios in their MLP layers, where output dimensions are several times larger than input dimensions. While various pruning techniques have been proposed to reduce model size, the relationship between this expansion capacity and model performance has remained unexplored. This paper presents a systematic investigation of GLU expansion ratios as a key metric for model pruning, using Llama-3.2 models (1B and 3B variants) as case studies. 

Our findings reveal that models with an expansion ratio of 140% consistently outperform others, achieving a balance between redundancy reduction, task-specific performance, and environmental sustainability. For instance, Llama-3.2-1B at 40% pruning and Llama-3.2-3B at 10% pruning surpassed their respective baselines in multiple benchmarks, including BoolQ, IFEval, and MUSR. 

Beyond performance, this study underscores significant environmental benefits. Specifically, pruning the 3B model to 140% expansion with just 10% pruning achieved a remarkable 50% reduction in CO2 emissions, showcasing the potential of pruning to enhance computational efficiency while maintaining robust performance. 

Future research should explore extending these findings to other GLU-based architectures, such as Mistral, Qwen, and Microsoft Phi, to validate their broader applicability. This study provides a novel perspective on sustainable AI development, effectively bridging the goals of performance optimization and environmental responsibility.
Toward Adaptive Reasoning in Large Language Models with Thought Rollback
Authors: Chen, Sijia and Li, Baochun
Date: 2024-07-21 00:00:00
Proceeding: International Conference on Machine Learning (ICML) 2024 7
Tasks: Arithmetic Reasoning, Math, Mathematical Reasoning

Large language models (LLMs) have been routinely used to solve various tasks using step-by-step reasoning. However, the structure of intermediate reasoning steps, or \emph{thoughts}, is rigid and unidirectional, such as chains, trees, or acyclic-directed graphs. Consequently, the resulting inflexible and forward-only reasoning may not address challenging tasks and fail when the LLM frequently gives false responses, i.e., ``hallucinations''. This paper proposes a new reasoning framework, called \emph{Thought Rollback} (TR), allowing LLMs to adaptively build thought structure while maintaining effective reasoning toward problem-solving under ``hallucinations''. The core mechanism of TR is \emph{rolling back thoughts}, which allows LLMs to perform error analysis on thoughts, and thus roll back to any previously mistaken thought for revision. Subsequently, by including such trial-and-error in the prompt to guide the LLM, each rollback leads to one more reliable reasoning path. Therefore, starting with a simple prompt without human annotations, LLM with TR adaptively and gradually explores thoughts for a correct solution. Comprehensive experiments on mathematical problems and multi-task reasoning demonstrate the state-of-the-art performance of TR in terms of problem-solving rate and interaction cost. For instance, the solving rate of GPT-4 with TR outperforms the current best by $9\%$ on the \texttt{MATH} dataset. The source code is available under the folder \textit{examples/ThoughtRollback} of \url{https://github.com/iQua/llmpebase}.
LININ: Logic Integrated Neural Inference Network for Explanatory Visual Question Answering
Authors: Dizhan Xue, Shengsheng Qian, Quan Fang, Changsheng Xu
Date: 2024-12-24 00:00:00
Proceeding: IEEE Transactions on Multimedia 2024 12
Tasks: Explanatory Visual Question Answering, Multimodal Reasoning, Question Answering, Visual Question Answering, Visual Question Answering (VQA)

Explanatory Visual Question Answering (EVQA) is a recently proposed multimodal reasoning task consisting of answering the visual question and generating multimodal explanations for the reasoning processes. Unlike traditional Visual
Question Answering (VQA) task that only aims at predicting answers for visual questions, EVQA also aims to generate userfriendly explanations to improve the explainability and credibility of reasoning models. To date, existing methods for VQA and EVQA ignore the prompt in the question and enforce the model to predict the probabilities of all answers. Moreover, existing EVQA methods ignore the complex relationships among question words, visual regions, and explanation tokens. Therefore, in this work, we propose a Logic Integrated Neural Inference Network (LININ) to restrict the range of candidate answers based on first-order-logic (FOL) and capture cross-modal relationships to generate rational explanations. Firstly, we design a FOLbased question analysis program to fetch a small number of candidate answers. Secondly, we utilize a multimodal transformer encoder to extract visual and question features, and conduct the prediction on candidate answers. Finally, we design a multimodal explanation transformer to construct cross-modal relationships and generate rational explanations. Comprehensive experiments on benchmark datasets demonstrate the superiority of LININ compared with the state-of-the-art methods for EVQA.
Few-Shot Multimodal Explanation for Visual Question Answering
Authors: Dizhan Xue, Shengsheng Qian, Changsheng Xu
Date: 2024-10-28 00:00:00
Proceeding: ACM MM 2024 10
Tasks: Explainable artificial intelligence, Explainable Artificial Intelligence (XAI), FS-MEVQA, Question Answering, Visual Grounding, Visual Question Answering, Visual Question Answering (VQA)

A key object in eXplainable Artificial Intelligence (XAI) is to create intelligent systems capable of reasoning and explaining real-world data to facilitate reliable decision-making. Recent studies have acknowledged the importance of providing user-friendly and verifiable explanations to facilitate trustworthy Visual Question Answering (VQA) systems. This paper aims to promote explainable VQA from both data and method perspectives. First, we propose a new Standard Multimodal Explanation (SME) dataset and a new Few-Shot Multimodal Explanation for VQA (FS-MEVQA) task, which aims to generate the multimodal explanation of the underlying reasoning process for solving visual questions with few training samples. Our SME dataset includes 1,028,230 samples composed of questions, images, answers, and multimodal explanations, which can facilitate research in both traditional MEVQA and FS-MEVQA. To the best of our knowledge, this is the first large-scale dataset with joint language-vision explanations based on standard English and additional visual grounding tokens. Second, we propose a training-free Multimodal Explaining Agent (MEAgent) method based on an LLM agent with multimodal open-world tools to infer answers and generate multimodal explanations for visual questions. Our MEAgent can learn multimodal explanation from merely N(=16) training samples and leverage open-world abilities to perform FS-MEVQA on test samples. Comprehensive experimental results evaluated by language quality metrics, visual detection metric, and visual attribution metrics on our SME dataset indicate the superiority of our method for FS-MEVQA. Our code and data are available at https://github.com/LivXue/FS-MEVQA.
CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era
Authors: Yanlin Feng, Simone Papicchio, Sajjadur Rahman
Date: 2024-12-24 00:00:00
Tasks: Knowledge Base Question Answering, Knowledge Graphs, Question Answering, Retrieval, text-to-Cypher

Retrieval from graph data is crucial for augmenting large language models (LLM) with both open-domain knowledge and private enterprise data, and it is also a key component in the recent GraphRAG system (edge et al., 2024). Despite decades of research on knowledge graphs and knowledge base question answering, leading LLM frameworks (e.g. Langchain and LlamaIndex) have only minimal support for retrieval from modern encyclopedic knowledge graphs like Wikidata. In this paper, we analyze the root cause and suggest that modern RDF knowledge graphs (e.g. Wikidata, Freebase) are less efficient for LLMs due to overly large schemas that far exceed the typical LLM context window, use of resource identifiers, overlapping relation types and lack of normalization. As a solution, we propose property graph views on top of the underlying RDF graph that can be efficiently queried by LLMs using Cypher. We instantiated this idea on Wikidata and introduced CypherBench, the first benchmark with 11 large-scale, multi-domain property graphs with 7.8 million entities and over 10,000 questions. To achieve this, we tackled several key challenges, including developing an RDF-to-property graph conversion engine, creating a systematic pipeline for text-to-Cypher task generation, and designing new evaluation metrics.
Advanced Knowledge Transfer: Refined Feature Distillation for Zero-Shot Quantization in Edge Computing
Authors: Inpyo Hong, Youngwan Jo, Hyojeong Lee, Sunghyun Ahn, Sanghyun Park
Date: 2024-12-26 00:00:00
Tasks: Edge-computing, Quantization, Transfer Learning

We introduce AKT (Advanced Knowledge Transfer), a novel method to enhance the training ability of low-bit quantized (Q) models in the field of zero-shot quantization (ZSQ). Existing research in ZSQ has focused on generating high-quality data from full-precision (FP) models. However, these approaches struggle with reduced learning ability in low-bit quantization due to its limited information capacity. To overcome this limitation, we propose effective training strategy compared to data generation. Particularly, we analyzed that refining feature maps in the feature distillation process is an effective way to transfer knowledge to the Q model. Based on this analysis, AKT efficiently transfer core information from the FP model to the Q model. AKT is the first approach to utilize both spatial and channel attention information in feature distillation in ZSQ. Our method addresses the fundamental gradient exploding problem in low-bit Q models. Experiments on CIFAR-10 and CIFAR-100 datasets demonstrated the effectiveness of the AKT. Our method led to significant performance enhancement in existing generative models. Notably, AKT achieved significant accuracy improvements in low-bit Q models, achieving state-of-the-art in the 3,5bit scenarios on CIFAR-10. The code is available at https://github.com/Inpyo-Hong/AKT-Advanced-knowledge-Transfer.
Unifying Tree-Reweighted Belief Propagation and Mean Field for Tracking Extended Targets
Authors: Weizhen Ma, Zhongliang Jing, Peng Dong, Henry Leung
Date: 2024-12-26 00:00:00

This paper proposes a unified tree-reweighted belief propagation (BP) and mean field (MF) approach for scalable detection and tracking of extended targets within the framework of factor graph. The factor graph is partitioned into a BP region and an MF region so that the messages in each region are updated according to the corresponding region rules. The BP region exploits the tree-reweighted BP, which offers improved convergence than the standard BP for graphs with massive cycles, to resolve data association. The MF region approximates the posterior densities of the measurement rate, kinematic state and extent. For linear Gaussian target models and gamma Gaussian inverse Wishart distributed state density, the unified approach provides a closed-form recursion for the state density. Hence, the proposed algorithm is more efficient than particle-based BP algorithms for extended target tracking. This method also avoids measurement clustering and gating since it solves the data association problem in a probabilistic fashion. We compare the proposed approach with algorithms such as the Poisson multi-Bernoulli mixture filter and the BP-based Poisson multi-Bernoulli filter. Simulation results demonstrate that the proposed algorithm achieves enhanced tracking performance.