Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting
Abstract: Deep motion forecasting models have achieved great success when trained on a massive amount of data.
Yet, they often perform poorly when training data is limited.
To address this challenge, we propose a transfer learning approach for efﬁciently adapting pre-trained forecasting models to new domains, such as unseen agent types and scene contexts.
Unlike the conventional ﬁne-tuning approach that updates the whole encoder, our main idea is to reduce the amount of tunable parameters that can precisely account for the target domain-speciﬁc motion style.
To this end, we introduce two components that exploit our prior knowledge of motion style shifts: (i) a low-rank motion style adapter that projects and adjusts the style features at a low-dimensional bottleneck; and (ii) a modular adapter strategy that disentangles the features of scene context and motion history to facilitate a ﬁne-grained choice of adaptation layers.
Through extensive experimentation, we show that our proposed adapter design, coined MoSA, outperforms prior methods on several forecasting benchmarks. Code available at https://github.com/vita-epfl/motion-style-transfer
Introduction
Motion forecasting is an essential pillar for the successful deployment of autonomous systems in environments comprising various heterogeneous agents.
It presents the challenges of modeling (i) universal etiquette ( e.g. , goal-directed behaviors, avoiding collisions) that govern general motion dynamics of all agents; and (ii) social norms ( e g. , the minimum separation distance, preferred speed) that inﬂuence the navigation styles of different agents across different locations.
Owing to the success of deep neural networks on large-scale datasets, learning prediction models in a data-driven manner has become a de-facto approach for motion forecasting and has shown impressive results [1, 2, 3, 4].
However, existing deep forecasting models suffer from inferior performance when they encounter novel scenarios [5, 6, 7, 8].
For instance, a network trained with large-scale data for pedestrian forecasting struggles to directly generalize to cyclists.
Some recent methods propose to incorporate strong priors robust to the underlying distribution shifts [9, 10, 11].
Yet, these priors often make strong assumptions on the distribution shifts, which may not hold in practice. This shortcoming motivates the following transfer learning paradigm: adapting a forecasting model pretrained on one domain with sufﬁcient data to new domains such as unseen agent types and scene contexts as efﬁciently as possible .
One common transfer learning approach is ﬁne-tuning a pretrained model on the data collected from target domain.
However, directly updating the model is often sample inefﬁcient, as it fails to exploit the inherent structure of the distributional shifts in the motion context.
In the forecasting setup, the physical laws behind motion dynamics are generally invariant across geographical locations and agent types: all agents move towards their goal and avoid collisions. As a result, the distribution shift can be largely attributed to the changes in the motion style, deﬁned as the way an agent interacts with its surroundings. Given this decoupling of motion dynamics, it can be efﬁcient for an adaptation algorithm to only account for the updates in the target motion style.
In this work, we efﬁciently adapt a deep forecasting model from one motion style to another.
We refer to this task as motion style transfer .
We retain the domain-invariant dynamics by freezing the pre-trained network weights. To learn the underlying shifts in style during adaptation, we introduce motion style adapters (MoSA), which are new modules inserted in parallel to the encoder layers.
The style shift learned by MoSA is injected into the frozen pre-trained model.
We hypothesize that the style shifts across forecasting domains often reside in a low-dimensional space.
To formulate this intuition, we design MoSA as a low-dimensional bottleneck, inspired by recent works in language [12, 13].
Speciﬁcally, MoSA comprises two trainable matrices with a low rank. The ﬁrst matrix is responsible for extracting the style factors to be updated, while the second enforces the updates.
MoSA learns the style updates by adding and updating less than 2% of the parameters in each layer .
In low-resource settings, it can be difﬁcult for MoSA to distinguish the relevant encoder layers updates from the irrelevant ones, resulting in sub-optimal performance. To facilitate an informed choice of adaptation layers, we propose a modularized adaptation strategy.
Speciﬁcally, we consider forecasting architectures that disentangle the ﬁne-grained scene context and past agent motion using two independent low-level encoders.
This design allows the ﬂexible injection of MoSA to one encoder while leaving the other unchanged.
Given the style transfer setup, our modular adaptation strategy yields substantial performance gains in the low-data regime.
We empirically demonstrate the efﬁciency of MoSA on the state-of-the-art model Y-Net [2] on the heterogenous SDD [14] and inD [15] datasets in various style transfer setups.
Next, we highlight the potential of our modularized adaptation strategy on two setups: Agent Motion Style Transfer and
Scene Style Transfer.
Finally, to showcase the generalizability of MoSA in self-driving applications, we adapt a large-scale model trained on one part of the city to an unseen part, on the Level 5 Dataset [16].
Through extensive experimentation, we quantitatively and qualitatively show that given just 10-30 samples in the new domain, MoSA improves the generalization error by 25% on SDD and inD.
Moreover, our design outperforms standard ﬁne-tuning techniques by 20% on the Level 5 dataset.
Related Work
Motion forecasting.
Classical models described the interactions between various agents based on domain knowledge but often failed to model complex social interactions in crowds [17, 18, 19].
Following the success of Social LSTM [1], various data-driven forecasting models have been proposed to capture social interactions directly from observed data [3, 20, 21, 4, 22, 23].
These methods heavily rely on a large and diverse set of training data, which may not be readily available for novel agents and locations.
In this work, we efﬁciently adapt a pretrained model to unseen target domains.
Distribution shifts.
The primary challenge in adapting to new domains lies in tackling the underlying distributional shifts. In the motion context, negative data augmentation techniques have been applied in a limited scope to reduce collisions [10] and off-road predictions [24] on new domains. Closely related to our work, Liu et al.
[25] proposed to reuse the majority of pre-trained parameters for efﬁcient adaptation. However, there exist key differences in the methodology: (1) we does not require access to multiple training domains with varying styles in order to perform adaptation (2) we introduce low-rank adapters instead of ﬁnetuning existing parameters, to model domain shifts.
Domain adaptation is another paradigm that allows a learning algorithm to observe unlabelled test samples. While this approach is effective for supervised visual tasks [26, 27, 28], it is not ideal for motion forecasting where the crucial challenge is sample efﬁciency as labels (future trajectories) are fairly easy to acquire. Therefore, we propose to perform transfer learning using limited data.
Transfer learning. The standard approach of ﬁne-tuning the entire or part of the network [29, 30] has been shown to outperform feature-based transfer strategy [31, 32]. In the motion context, transfer learning given limited data often requires special architecture designs like external memory [33] and meta-learning objectives [34, 35] that require access to multiple training environments. Wang et al.
[36] performed online adaptation across different scenarios for vehicle prediction domains. Recently, there has been a growing interest in developing parameter-efﬁcient ﬁne-tuning (PET) methods in both language and vision, as they yield a compact model [37, 12, 13] and show promising results in outperforming ﬁne-tuning in low-resource settings [13, 38]. Similar in spirit to PET methods, we introduce additional parameters in our network that account for the updates in target style.
Motion Style.
the popular work of Robicquet et al.
[14] deﬁned navigation style as the way different agents interact with their surroundings.
It introduced social sensitivity as two handcrafted descriptions of agent style and provided them as input to the social force model [17].
In this work, we model style ℎ !"
[Figure]
as a latent variable that is learned in a data-driven manner.
Furthermore, we decouple motion style into scene-style components and agent-style components to favour efﬁcient adaptation.
Method
Deep neural networks have shown remarkable performance in motion forecasting thanks to the availability of large-scale datasets.
However, these models often struggle to generalize when unseen scenarios are encountered in the real world due to underlying differences in motion style.
In this section, we ﬁrst formally introduce the problem setting of motion style transfer.
Subsequently, we introduce the design of our style adapter modules that help to effectively tackle motion style transfer.
Finally, we describe the application of style adapters to a modularized architecture in order to perform style transfer in a more efﬁcient manner.
3.1 Motion Style Transfer
Motion style.
Modelling agent motion behavior involves learning the social norms ( e.g. , minimum separation distance to others, preferred speed, valid areas of traversal) that dictate the motion of the agent in its surroundings.
These norms differ across agents as well as locations.
For instance, the preferred speed of pedestrians differs from that of cyclists; the separation distance between pedestrians in parks differs from that in train stations.
To describe these agent-speciﬁc (or scenespeciﬁc) elements that govern underlying motion behavior, we deﬁne the notion of “motion style”.
Motion style is the collective umbrella that models the social norms of an agent given its surroundings.
Problem statement.
Consider the inductive transfer learning setting for deep motion forecasting across different motion styles.
Speciﬁcally, we are provided a forecasting model trained on large quantities of data comprising a particular set of style(s) and our goal is to adapt the model to the idiosyncrasies of a target style as efﬁciently as possible.
We denote the model input and ground-truth future trajectory of an agent i using x i and y i respectively.
The input x i comprises the past trajectory of the agent, surrounding neighbors, and the surrounding context map.
The context can take various forms like RGB images or rasterized maps.
We assume that the data corresponding to an agent type is generated by an underlying distribution P X,Y ( · ; s ) parameterized by s , the style of the agent.
As mentioned earlier, the style is dictated by both the agent type and its surroundings.
Training.
The forecasting model has an encoder-decoder architecture (see Fig. 1) with weights W enc and W dec respectively.
The training dataset, D S of size N is given by ∪ s ∈ S D s =
( x i , y i ) i ∈{ 1 ,...,N } , where S is a collection of motion styles observed within the dataset.
The model is trained to minimize a loss objective L , such as the negative log likelihood (NLL) [1, 3] loss or variety loss [39]:
[Equation]
Adaptation.
When a novel scenario with style s (cid:48) ( s (cid:48) / ∈ S ) is encountered, it leads to a distribution shift and the learned model often struggles to directly generalize to the corresponding dataset
D s (cid:48) =
( x (cid:48) i , y (cid:48) i ) i ∈{ 1 ,...,N target } of size N target .
The common approach to tackling such shifts is to ﬁne-tune the entire or part of the pretrained model.
Fine-tuning optimizes an objective similar to training, but on the new dataset:
[Equation]
In this work, we aim to develop an adaptation strategy for efﬁcient motion style transfer, i.e. , cases where N target is small ( N target (cid:28) N ).
Often, motion behaviors do not change drastically across domains.
Instead, most of the behavioral dynamics are governed by universal physical laws ( e.g. , inﬂuence of inertia, collision-avoidance).
We therefore propose to freeze the weights of the pretrained forecasting model and introduce motion style adapters, termed MoSA , to capture the target motion style.
As shown Fig.
1, we adapt a pre-trained forecasting model by ﬁne-tuning W MoSA with the following objective:
[Equation]
3.2 Motion Style Adapters
Our main intuition is that the style shifts across forecasting domains are usually localized – they are often due to the changes in only a few variables of the underlying motion generation process.
Therefore, during style transfer, we only need to adapt the distribution of this small portion of latent factors, while keeping the rest of the factors constant.
These updates would correspond to the changes in motion style ( s → s (cid:48) ) in the target domain, as the general principles of motion dynamics ( e.g. , avoid collisions, move towards goal) remain the same across domains for all agents.
We design motion style adapters, referred to as MoSA, to carry out these updates.
Our proposed MoSA design comprises a small number of extra parameters added to the model during adaptation (see Fig. 1).
Each module comprises two trainable weight matrices of low rank, denoted by A and B .
The ﬁrst matrix A is responsible for inferring the style factors that are required to be updated to match the target style, while the second matrix B performs the desired update.
The low rank r realizes our intuition that style updates reside in a low-dimensional space, by restricting the number of style factors that get updated ( r (cid:28) d where d is the dimension size of an encoder layer).
Therefore, during adaptation, the weight updates of the encoder are constrained with our low-rank decomposition W MoSA = BA .
The pretrained model is frozen and only A and B are trained.
For brevity, let us consider the adaptation of encoder layer l with input h l and output h l
+
As shown in Fig.
1, W lenc and W lMoSA are multiplied with the same input h l , and their respective output vectors are summed coordinate-wise as shown below:
[Equation]
It has been shown that initialization plays a crucial role in parameter-efﬁcient transfer learning [37, 12].
Therefore, following common practices, matrices A and B are initialized with a near-zero function [12], so that the original network is unaffected when training starts.
Furthermore, the initialization provides ﬂexibility to these modules to ignore certain layers during motion style updates.
Despite this ﬂexibility, the total number of extra parameters is signiﬁcant and can be inconducive to efﬁcient style transfer.
Therefore, to further boost sample efﬁciency, we present a modular adaptation strategy which we describe next.
3.3 Decoupling Motion Style Adapters
Motion style can be decoupled into scene-speciﬁc style and agent-speciﬁc style.
Scene-speciﬁc style dictates the changes in motion due to physical scene structures.
For instance, cyclists prefer to stay on the lanes while pedestrians move along sidewalks.
The agent-speciﬁc style captures the underlying [Figure] navigation preferences of different agents like distance to others and preferred speed.
Modularizing the encoder into two parts that account for the physical scene and agent’s past motion independently can help to decouple the functionality of our motion style adapters and further improve performance.
Consider the modularized motion encoder designs shown in Fig.
The modularized encoder models the input scene and agent’s past history independently.
The fusion encoder then fuses the two representations together.
This design has the advantage to decouple the task of the style adapters into scene-speciﬁc updates and agent-speciﬁc updates.
Given the modularized setup, the nature of the underlying distribution shifts can help guide which modules within the model are required to be updated to the target style.
As demonstrated in Section.
4.4, given different categories of style transfer setups, decoupling style adapters can improve the adaptation performance while signiﬁcantly reducing the number of updated parameters.
Experiments
We evaluate our method on a total of three datasets to study the performance of motion style adapters:
Stanford Drone Dataset (SDD) [14], the Intersection Drone Dataset (InD) [15], and Level 5 Dataset (L5)
We evaluate each method over ﬁve experiments with different random seeds.
More implementation details and ablations are summarized in the Appendix.
Baselines.
We use the state-of-the-art Y-Net model [40] on SDD and inD, and the Vision Transformer (ViT) [41] on L5 across all methods.
We compare the following: • Full Model Finetuning (FT) [29]: we update the weights of the entire model. • Partial Model Finetuning (ET) [25]: we update the weights of the Y-Net encoder for SDD and inD , and the last two layers of ViT for Level 5. • Parallel Adapters (PA) [42]: we insert a convolutional layer with a ﬁxed ﬁlter size in parallel to each encoder layer and update the weights of these added layers.
This baseline does not incorporate the low-rank constraint.
• Adaptive Layer Normalization [43, 44]: we update the weights and biases of the layer normalization. • Motion Style Adapters (MoSA)
[Ours]: we insert our motion style adapters in parallel to each encoder layer in SDD and inD, and in parallel to query and value matrices of multi-headed attention in Level 5.
During modularized adaptation, we add our modules only across the speciﬁed encoders.
Metrics.
We use the established Average Displacement Error (ADE) and Final Displacement Error (FDE) metrics for measuring the performance of model predictions.
ADE is calculated as the l 2 error between the predicted future and the ground truth averaged over the entire trajectory while FDE is the l 2 error between the predicted future and ground truth for the ﬁnal predicted point [1].
For multiple predictions, the ﬁnal error is reported as the min error over all predictions [39].
Additionally, we deﬁne the generalization error as the error of the pretrained model on the target domain.
The more the dissimilarity between the source domain and target domain, the higher the generalization error. [Table] [Figure] [Table]
[Figure]
Motion Style Transfer across Agents on SDD ( 🚶  🚴 )
[Table] 4.1 Motion Style Transfer across Agents on Stanford Drone Dataset
We perform short-term prediction, where the trajectory is predicted for the next 4.8 seconds, given 3. seconds of observation.
The Y-Net model is trained on pedestrian data across all scenes and adapted to the cyclists’ data in deathCircle 0 , as there exists a clear distinction between the motion style of pedestrians and cyclists (see Fig. 3).
We adapt the model using N target = { 10 , 20 , 30 } samples.
Tab. 1 quantiﬁes the performance of various adaptation techniques.
The model trained on pedestrians does not generalize to cyclists as evidenced by the high generalization error of 58 pixels.
Our MoSA design reduces this error by ∼ 30% using only 30 samples.
Moreover, MoSA outperforms the baselines while keeping the pretrained model frozen and updating only 2% additional parameters.
Fig.
4 illustrates the updates in the Y-Net goal decoder output (red means increase in focus and blue means decrease in focus) on model adaptation using MoSA.
Adapted Y-Net successfully learns the style differences between the behavior of pedestrians and cyclists: 1) it correctly infers valid areas of traversal, 2) effectively captures the multimodality of cyclists, and 3) updates the motion style parameters as the new cyclist goal positions (red) are farther from the end of the observation position, compared to the un-adapted goal positions (blue).
4.2 Motion Style Transfer across Scenes on Intersection Drone Dataset
We perform long-term prediction, where trajectory in the next 30 seconds is predicted, givenseconds of observation.
The Y-Net model is trained on pedestrians in { scene 2 , scene 3 , scene 4 } and tested on unseen scene scene 1 .
We adapt the model using N target = { 10 , 20 , 30 } samples.
Despite the long-term prediction setup, the generalization error is 33 pixels which is lower compared to SDD, as the target domain is more similar to the source domain.
Tab. 1 quantiﬁes the performance of scene style transfer across all methods.
Using just 30 samples, MoSA improves the generalization error by ∼ 40% and outperforms its counterparts.
The superior performance in comparison to PA justiﬁes the importance of the low-rank constraint.
[Figure]
[Figure]
Error in meters for 5 seeds.
4.3 Motion Style Transfer across Scenes on Lyft Level 5 Dataset
We divide the L5 dataset into two splits based on the data collection locations thereby, constructing a scene style shift scenario.
We train the ViT-Tiny model on the majority route and adapt it to the smaller route not seen during training.
To simulate low-resource settings, we provide the frames, sampled at different rates, that cover the unseen route only once.
Fig.
6 quantitatively evaluates the performance of various adaptation strategies.
MoSA performs superior in comparison to different baselines while adding and updating only 5% of the full model parameters.
It is apparent that our low-rank design is a smarter way of adapting models in the motion context as compared to ﬁne-tuning the model.
Fig.
7 empirically validates that a bigger pre-training dataset size results in better adaptation performance.
Finally, Fig.
5 qualitatively illustrates the improvement of the attention heatmaps of the last layer of ViT post model adaptation using MoSA.
MoSA helps to better focus on the different possible future routes and the vehicles in front.
Additional visualizations are provided in the Appendix.
S
A
S+F
A+F
S+A+F 9.
[Figure]
S
A
S+F
A+F
S+A+F
F D E
[Figure]
4.4 Modular Motion Style Adapters
Now, we demonstrate the effectiveness of applying our adapters on top of a modularized architecture on two setups: Scene generalization and Agent Motion generalization.
As shown in Fig.
2, we modularize the Y-Net architecture.
We treat scene and agent motion independently for the ﬁrst two layers of the encoder before fusing the learned representations.
We refer to this design as Y-Net-Mod.
Given Y-Net-Mod, we consider ﬁve cases based on modules on which MoSA is applied: (1) scene only [S], (2) agent motion encoder only
[A], (3) scene and fusion encoder [S+F], (4) agent motion and fusion encoder [A+F], and (5) scene, agent motion and fusion encoders together [S+A+F].
Agent motion generalization: In scene 1 of inD, cars and trucks share the same scene context differing only in their velocity distribution.
Fig.
8 represents the performance of style transfer from cars to trucks on 20 samples under ﬁve different adaptation cases.
It is interesting to note that adapting the agent motion encoder alone
[A] performs the best while including the scene encoder for adaptation deteriorates performance ([S] worse than [A], [S+A+F] worse than [A+F]).
Scene generalization: We train the Y-Net-Mod model on pedestrian data on scene ids = { 2 , 3 , 4 } and adapt it on scene 1 of inD. Fig.
9 represents the performance of scene style transfer on 20 samples in the ﬁve cases.
Contrary to the previous setup, adapting the scene encoder
[S] is clearly more important than the agent motion encoder [A].
Further, adapting the agent encoder deteriorates performance ([S+A+F] worse than [S+F]).
It is clear that modularization helps to boost the performance of MoSA.
Limitations
We demonstrated the effectiveness of decoupling our proposed motion style adapters using Y-NetMod.
However, during training, we do not enforce any constraints on the learning objective to favor effective modularization that can result in more efﬁcient adaptation.
Developing training strategies that allow quick adaptation using MoSA is a potential line of future work.
Another limitation is the requirement of human intervention in determining the modules to be adapted given the nature of the style shift.
A future direction is automating the selection of layers to be adapted for target domains.
Conclusion
We tackle the task of efﬁcient motion style transfer wherein we adapt a pre-trained forecasting model using limited samples from an unseen target domain.
We hypothesize that the underlying shift across domains often resides in a low-dimensional space.
We formulated this intuition into our motion style adapter (MoSA) design, which is trained to infer and update the style factors of variation in the target domain while keeping the pre-trained parameters frozen.
Additionally, we present a modularized adaptation strategy that updates only a subset of the model given the style transfer setup.
Extensive experimentation on three real-world datasets demonstrates the effectiveness of our approach. Acknowledgments
This work was supported by Honda R&D Co. Ltd, the Swiss National Science Foundation under the
Grant 2OOO21-L and EPFL.
We also thank VITA members and reviewers for their valuable comments.