SensorMAE

Pretrained encoders of the paper "Do Pretrained Backbones Transfer to Physical Sensors? A Systematic Study of Multimodal Initialization Across the Sensor Stack".

SensorMAE is a multimodal masked autoencoder pretrained on 16 public driving and robotics datasets (~826k aligned samples) with RGB, thermal LWIR/MWIR, reflective NIR/SWIR, event cameras, stereo depth and semantic segmentation. Code, data preparation and the downstream benchmarks: github.com/DTU-PAS/sensormae.

Files

File	Encoder	Parameters	Pretraining initialized from
sensormae-base-mm	ViT-B/16	101.7M	MultiMAE ViT-B
sensormae-base-vit	ViT-B/16	101.7M	ViT-B (ImageNet-21k)
sensormae-small-vit	ViT-S/16	29.6M	ViT-S (ImageNet-21k)
sensormae-tiny-vit	ViT-Ti/16	9.5M	ViT-Ti (ImageNet-21k)

Each model is provided as .safetensors and as .pth (the file the benchmarking code reads); both hold the same tensors. Only the encoder is included:

• patch_embed.<modality>.proj.{weight,bias}: patch embedding (16x16) per modality
• pos_embed.<modality>: positional embedding per modality (14x14 grid, 224x224 input)
• modality_embed.<modality>: modality embedding
• cls_token and the 12 transformer blocks blocks.<i>.* (timm ViT layout)

with modalities rgb (3 channels), lmwir (LWIR/MWIR, 1), snir (NIR/SWIR, 1), depth (1) and event (10: 5 time bins x 2 polarities over 50 ms). The segm patch embedding expects 64-channel embedded Cityscapes labels; the label embedding itself is not part of the release.

Usage

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

path = hf_hub_download("dtu-pcas/sensormae", "sensormae-base-mm.safetensors")
state_dict = load_file(path)

To fine-tune on segmentation or detection with any subset of sensors, download the .pth files into benchmarking/weights/ of the code repository and use the provided configs:

hf download dtu-pcas/sensormae --include "*.pth" --local-dir benchmarking/weights

License

MIT.