projectsidewalk/sidewalk-tagger-ai-curbramp

Project Sidewalk Tagger: Curb Ramp

A multi-label classifier that predicts which accessibility tags apply to a Curb Ramp label in a crop of a Google Street View image. Project Sidewalk serves it through sidewalk-ai-api to suggest tags for crowdsourced labels.

This is the DINOv2 model trained on the cleaned dataset in Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset (ASSETS '24). Training and evaluation code is in sidewalk-tagger-ai.

Provenance

validated-dino-cls-b-curbramp-tags-best.pth is a byte-for-byte copy of the file of the same name in projectsidewalk/sidewalk-tagger-ai-models at commit 65959dbc80b87e4f39385204c4b639cbcf58e1a8, which holds every model from the paper and remains available unchanged.

sha256: 4d00193aed73fc199049f31cebade51f236bfca92ad9f76adf08a9d08a272833

Model

• Backbone: DINOv2 ViT-B/14 with 4 register tokens (dinov2_vitb14_reg4), fine-tuned.
• Head: Linear(768, 256) → ReLU → Linear(256, 10), with a sigmoid per class.
• Input: RGB crop resized to 256 px and padded to a multiple of 14, normalized with ImageNet mean and std.
• Format: a PyTorch state dict; it loads with torch.load(..., weights_only=True).

The architecture is custom rather than a transformers class. See DinoVisionTransformerClassifier and ImageTagger in sidewalk_ai_api/tagger.py for loading and preprocessing.

Classes

The output has 10 classes, in this order. NULL entries are unused placeholders that preserve the class indices from training.

Index	Tag
0	missing tactile warning
1	narrow
2	not enough landing space
3	not level with street
4	NULL
5	points into traffic
6	debris / pooled water
7	steep
8	surface problem
9	NULL

Thresholds

Sigmoid scores aren't calibrated probabilities. The per-tag thresholds Project Sidewalk uses to suggest that a tag is present or absent are in tag_thresholds.json in sidewalk-ai-api.

Evaluation

See the paper for per-tag results on the test split of sidewalk-tagger-ai-validated. REPRODUCE_RESULTS.md in sidewalk-tagger-ai reproduces them.

Citation

@inproceedings{10.1145/3663548.3688531,
  author = {Liu, Xinlei and Wu, Kevin and Kulkarni, Minchu and Saugstad, Michael and Rapo, Peyton Anton and Freiburger, Jeremy and Hosseini, Maryam and Li, Chu and Froehlich, Jon E.},
  title = {Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset},
  year = {2024},
  publisher = {Association for Computing Machinery},
  doi = {10.1145/3663548.3688531},
  booktitle = {Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility},
  articleno = {103},
  numpages = {12}
}