SimLingo β Predictive Fusion (fdrop03)
InternVL2-1B (InternViT-300M + Qwen2-0.5B, LoRA r=32 Ξ±=64) camera-VLA driving policy for CARLA, with a two-stage cross-attention fusion where InternViT vision tokens are the query:
- Stage 1 β KV = 360Β° LiDAR β BEV occupancy histogram (current scene).
- Stage 2 β KV = DeltaTok-forecast future embedding β a frozen DINOv3 ViT-B/16 tokenizer plus a trained Predictor that forecasts the next delta token without seeing the future frame.
Enriched tokens replace <IMG_CONTEXT> in place. Training code, config, and the data-collection
pipeline: github.com/nacef112/kaut-project. Built on
carla_garage and
SimLingo.
Recipe
route_as |
target_point_command β GPS target point and the language command |
predict_route_as_wps |
True |
speed_wps_mode |
2d |
fusion_dropout_prob |
0.3 β withholds BEV and forecast TOGETHER, per sample, train-only. Forces the target-point/command path to carry gradient instead of only the fusion shortcut. |
| forecast | single horizon (not multi-horizon) |
num_context_frames / delta_frame_gap |
4 / 1 |
| DeltaTok tokenizer | included in this repo (deltatok_best_b2dheldout.ckpt) |
Files
| file | size | notes |
|---|---|---|
fdrop03_step050000_weights.ckpt |
3.2 GB | epoch=2, step=50000 β the warm-start every downstream checkpoint in this project line continued from. Use this one unless you specifically want the later epoch. |
fdrop03_final_epoch003_weights.ckpt |
3.2 GB | epoch=3, step=82840 β further trained, same recipe |
deltatok_best_b2dheldout.ckpt |
2.0 GB | DeltaTok forecast tokenizer (single-horizon) that Stage-2 was trained against |
SHA256SUMS.txt |
β | integrity check |
Both weight files load state_dict only (optimizer/scheduler state stripped), verified to load
strict=True into a freshly constructed model β 0 missing, 0 unexpected keys, 1317.6M params.
Load
import torch
sd = torch.load("fdrop03_step050000_weights.ckpt", map_location="cpu", weights_only=False)
state_dict = sd.get("state_dict", sd)
# model = hydra.utils.instantiate(cfg.model, ...) -- see the training repo for cfg construction
model.load_state_dict(state_dict, strict=True)
The model class (DrivingModel) and its Hydra config live in the training repo linked above β
this is weights only, not a standalone inference package. DINOV3_BACKBONE env var overrides
the (gated) facebook/dinov3-vitb16-pretrain-lvd1689m backbone path if you have a local copy;
see the training repo's README for requesting access.
What this is not
Not a multi-horizon, trimodal-fusion, LoRA-only-SFT, or CoT/adaptive-CoT checkpoint β those are separate lines built on top of this same architecture, not included here.