Harnessing and Evaluating the Intrinsic Extrapolation Ability of Large Language Models for Vehicle Trajectory Prediction (2025)

Abstract: Emergent abilities of large language models (LLMs) have significantly advanced their application in autonomous vehicle (AV) research. Safe integration of LLMs into vehicles, however, necessitates their thorough understanding of dynamic traffic environments. Towards this end, this study introduces a framework leveraging LLMs’ built-in extrapolation capabilities for vehicle trajectory prediction, thereby evaluating their comprehension of the evolution of traffic agents’ behaviors and interactions over time. The framework employs a traffic encoder to extract spatial-level scene features from agents’ observed trajectories to facilitate efficient scene representation. To focus on LLM’s innate capabilities, scene features are then converted into LLM-compatible tokens through a reprogramming adapter and finally decoded into predicted trajectories with a linear decoder. Experimental results quantitatively demonstrate the framework’s efficacy in enabling off-the-shelf, frozen LLMs to achieve competitive trajectory prediction performance, with qualitative analyses revealing their enhanced understanding of complex, multi-agent traffic scenarios.

## introduction recent studies suggest that when the model parameter number and the training data volume surpass critical thresholds, llms can exhibit sophisticated, high-level "emergent abilities". these include incontext learning (brown, 2020), instruction following @xcite and zero-shot extrapolation @xcite . the advanced extrapolation ability enables llms to create and predict coherent content beyond their initial input, underscoring their potential for creative problem-solving and indicating preliminary * corresponding author manifestations of artificial general intelligence (agi) @xcite . simultaneously, llms are drawing increasing interest in the av domain, with some researchers anticipating a "gpt moment" that could revolutionize this field @xcite . off-the-shelf and general-purpose llms such as gpt are being leveraged to process textual descriptions of dynamic and complex traffic scenes, aiming for highlevel scene understanding and transparent decisionmaking @xcite . nevertheless, the rapid expansion of llm applications in safety-critical and dynamic av contexts introduces several critical concerns: q1: should time series data, such as traffic agents' trajectories, be encoded and processed in a textual format by llms? q2: can llms accurately model and comprehend the temporal evolution of traffic agents' behaviors and interactions? to answer the above questions, this study explores trajectory encoding strategies to enhance llms' comprehension of dynamic traffic scenes and examine the potential to harness llms' builtin extrapolation capabilities for vehicle trajectory prediction. in advanced av systems, prediction models analyze the observed trajectories of interested vehicles and their surrounding agents to forecast future waypoints, thus preventing collisions and enabling safe, efficient driving decisions. in this work, assessing llms' applicability to trajectory prediction offers an intuitive and quantitative measure of their understanding of sophisticated, dynamic traffic scenarios. however, applying llm for trajectory forecasting entails the following challenges: to address the identified challenges, this work introduces a framework that exploits built-in extrapolation capabilities of general-purpose, off-the-shelf llms for vehicle trajectory prediction. initially, instead of directly inputting text-formatted trajectories, the framework begins with a traffic scene encoder, which extracts spatial-level features from observed trajectories, capturing traffic agents' spatial layout and social interactions at each moment. secondly, these scene features are mapped into the text embedding space via a reprogramming adapter @xcite , where each feature is reprogrammed as a combination of multiple text tokens from llm vocabulary. this practice enables llms to comprehend the encoded features without re-training while mitigating catastrophic forgetting risk of their pre-existing capabilities. furthermore, the scene tokens-reprogrammed scene features-serve as a compact representation of multi-agent trajectories (form t idk tokens to t tokens), enhancing local semantic richness by encoding environmental context. finally, the llm processes the scene tokens alongside task-descriptive text tokens, with its output decoded into predicted trajectories using a simple linear decoder. the main contributions in this work can be summarized as follows: notably, this study is limited to evaluating the efficacy of llms in analyzing trajectory time series data, with predictions based on high-definition (hd) maps being reserved for future research. recent research investigates the mechanisms underpinning the versatile and adaptive intelligence of llms. studies by @xcite propose that llms function essentially as advanced pattern recognition systems, with the self-attention mechanism resembling principal component analysis. research by @xcite suggests that llms can leverage task-relevant language cues for time series forecasting. another prominent hypothesis is that extensive and diverse datasets compel llms to cultivate generalized, functional "neural circuits" @xcite . their vast parameter space ensures adequate redundancy and variability, enabling these circuits to specialize and fine-tune for specific tasks @xcite . time series forecasting: time series forecasting is pivotal in application such as trajectory prediction @xcite , energy usage management and financial investments. historically, forecasting methods have progressed from traditional statistical models @xcite to advanced deep learning-based methods. presently, transformer-based architectures, exemplified by informer @xcite , which utilize self-attention mechanisms for efficient parallel processing, represent the forefront of sequence modeling advancements. however, recent research @xcite , challenges the efficacy of transformers in time series forecasting, demonstrating that fully connected networks (fcn) can significantly outperform previous transformer models. subsequent research, termed patchtst @xcite , reinforces this by demonstrating that transformers exhibit limitations in time series analysis due to insufficient semantic granularity of individual time series data points. to mitigate this, patchtst segments time series into overlapping subseries-level patches before encoding, thereby actively enriching the local semantic of time series embeddings. meanwhile, recent researches underscore the potential of leveraging llms for time series forecasting. a seminal work in this direction is gpt4ts @xcite , which integrates an llm into the patchtst framework while fine-tuning its normalization and output layers. expanding on this breakthrough, time-llm @xcite introduces a learnable reprogramming adapter to map patch embeddings into text embedding space while keeping the llm parameters frozen. additionally, time-llm offers an innovative perspective, indicating that time series forecasting can be reframed as a nlp task solvable by off-the-shelf, text-only llms. trajectories input text a traffic scene encoder extracts spatial-level features from observed trajectories of traffic agents, capturing their spatial layouts and interactions at each timestamp. next, a reprogramming adapter maps these scene features into the text embedding space, translating them into combinations of llm vocabulary tokens. these reprogrammed scene features, referred to as scene tokens, are concatenated with text tokens that describe the prediction task and then fed into a pre-trained, forzen llm backbone. finally, a linear decoder, which performs a matrix transformation, converts the llm's output into predicted trajectories. traffic scene encoder: effective and efficient traffic scene encoding is fundamental to trajectory forecasting task, as it endows models with a comprehensive understanding of both social interactions (agent-to-agent interactions) and temporal dependencies (evolution of agent movements over time) @xcite . for instance, tra-jectron++ @xcite ) models traffic scenes as directed graphs, separately encoding social interactions and temporal correlations. hivt @xcite , on the other hand, employs a hierarchical framework, initially capturing social interactions and then using a temporal model for trajectory prediction. in contrast, agentformer @xcite argues that separate or sequential modeling of social and temporal correlations can lead to information loss, advocating for joint modeling to improve accuracy. while advanced joint encoding schemes can potentially improve prediction performance, the proposed framework prioritizes the utilization of an encoder that only captures spatial layouts and social interactions among traffic agents. this design ensures that temporal correlations are managed by llms, thereby allowing a focused evaluation of their extrapolation capabilities for trajectory prediction. specifically, the framework first vectorizes the spatial states of the target vehicle 0 and its neighboring agents @xmath0, 2, ..., i} for each timestamp @xmath1, 2, ..., t } as follows: next, a cross-attention layer is used to encode the interactions between target vehicle and neighboring agents into the vector h ′ @xmath2: here, the query vector q 0 @xmath3, the key vector k i @xmath4: finally, a learnable fusion scheme is utilized to integrate the neighboring agents interactions and the target vehicle's spatial state according to: reprogramming adapter: recent researches indicates the emergent capabilities of llms position them as general-purpose artificial assistants @xcite . consequently, there is growing interest in extending llms to additional modalities, such as images, to broaden their functional scope and applicability. specifically, blip-2 @xcite utilizes image features extracted from a resnet backbone as key and value vectors, which are then integrated with learnable query vectors and processed through a cross-attention layer to generate visual tokens comprehensible to llms. similarly, llava @xcite derives visual tokens by directly mapping the image features, produced by the clip encoder @xcite , into the text embedding space with a trainable fcn. in contrast, research on adapting llms for trajectory data is limited due to several factors: 1) data interpretability: unlike image pixel values, structured trajectory data is inherently more interpretable due to its organized and sequential nature. towards this end, the proposed framework adopts the trainable reprogramming adapter @xcite , reprogram : r @xmath5, to projects the extracted scene feature h t from equation (4) into the llms' text embedding space, formulated as: in essence, the scene feature h t is transformed into a combination of multiple llm vocabulary tokens, b scene @xmath6, via the learned mapping function reprogram. this transformation yields a compact textual representation of h t , enhancing input tokens' local semantic richness and facilitating llms' efficient comprehending of scene information. compared with blip-2 and llava, this adaption scheme allows for a more targeted assessment of llms' extrapolation ability using the representations they have learned from natural language data. refer to appendix a for further details of this section. linear trajectory decoder: architectures such as rnn and lstm are frequently employed as trajectory decoders due to their proficiency in capturing temporal dependencies. these autoregressive models function similarly to vehicle motion models @xcite , generating future waypoints iteratively based on preceding predictions to maintain both temporal and spatial coherence in the decoding process. nonetheless, utilizing autoregressive architectures to decode llm outputs can inadvertently transfer the prediction challenge to the sophisticated decoder, potentially obscuring the evaluation of llms' innate capabilities. for example, if the decoder mirrors a constant velocity motion model, an llm with limited extrapolation ability might still appear to perform well by simply outputting constant speeds for constant-velocity trajectories. this decoupling can mask the llm's true predictive capabilities, as it relies on the decoder model to handle extrapolation tasks. to avoid distributing the prediction task to sophisticated decoders, the proposed framework adopts a simple linear model to decode the llm's output through: overall, despite employing the cascading architecture depicted in figure 1 , the proposed framework ensures that temporal dependencies are managed solely by llms through a meticulously designed sequence of encoder, adapter and decoder modules, thereby facilitating a targeted assessment of llms' extrapolation capabilities in trajectory prediction. dataset: the proposed method is evaluated using the nuscenes dataset @xcite , a comprehensive dataset for autonomous driving featuring 1000 scenes from boston and singapore. each scene, lasting 20 seconds (s) and annotated at a frequency of 2 hz, includes up to 23 semantic object classes. comparison methods: a total of four comparison methods is considered in this work: 1) zs-llm @xcite , which utilizes llm in a zero-shot paradigm by embedding text-formatted trajectories into the template fine-tuned for chat interactions. 2) ic-llm (brown, 2020) incorporates task-specific exemplars into input text and extends zc-llm by utilizing llms' in-context learning capabilities. 3) cot-llm @xcite , extending zc-llm, utilizes llms' chain-of-thought capabilities to infer future trajectories. 4) time-llm @xcite represents a seminal approach that leverages off-the-shelf llms for time series forecasting, thus is determined as the baseline method in this work. evaluation metric: trajectory prediction performance is mainly evaluated through the metrics of average displacement error (ade) and final displacement error (fde) @xcite . ade measures the average euclidean distance between the predicted waypoints and the ground-truth waypoints in a specified prediction horizon. fde quantifies the euclidean distance between the fi-nal predicted waypoint and the final ground-truth waypoint. additionally, the missing rate (mr) metric is introduced to quantify the proportion of prediction failures, where a prediction failure occurs when the number of waypoints generated by llms deviates from the expected count of predicted timestamps. implementation detail: the observation window in this work comprises 4 timestamps in 2 seconds (2s), while the prediction window encompasses 12 timestamps in 6s. the framework's training objective is to minimize the l2 loss between the predicted and ground-truth trajectories, using the adam optimizer with a cosine annealing learning rate scheduler initialized at 1 × 10 -4 . to ensure reproducibility, implementation details and the trained model checkpoints are available at here. see appendix b for more details on the comparison methods, evaluation metrics and considered llms. ## conclusion to assess the proficiency of off-the-shelf, text-only llms in understanding temporal dynamics of traffic agents, this study presents a framework leveraging the built-in extrapolation capabilities of llms for vehicle trajectory forecasting. key findings from our comprehensive evaluation include: 1) given text-formatted observed trajectories, llms can, to some extent, predict simple linear trajectory of target vehicle by leveraging in-context, taskspecific exemplars. 2) however, as the density of neighboring traffic agents increases, the predictive performance of llms deteriorates significantly, underscoring their limitation in handling the excessive tokens required for multi-agent trajectory representation. 3) to mitigate this limitation, this work propose to treat trajectory data as a distinct modality from text. specifically, the framework represents multiple trajectories as semantically enriched, spatial-level scene tokens. this practice significantly enhances the prediction performance and enables vehicle turning maneuver predictions without hd maps.