SmolVLM 500M Instruct for ExecuTorch WebGPU

This repository contains an ExecuTorch conversion of HuggingFaceTB/SmolVLM-500M-Instruct for the ExecuTorch WebGPU backend, so the model runs entirely in a web browser on the visitor's GPU. It powers the corresponding demo in the ExecuTorch WebGPU Playground (launching soon on executorch.ai).

Base model: HuggingFaceTB/SmolVLM-500M-Instruct
Format: ExecuTorch .pte programs delegated to the WebGPU backend
Quantization: int8 vision encoder and token embeddings; decoder as exported
Total size: 893.9 MB

About the original model

A compact vision-language model that answers questions about images, describes them and transcribes text in them. It pairs a 93M SigLIP vision encoder with the SmolLM2-360M language model.

Developed by: Hugging Face
Paper: SmolVLM: Redefining small and efficient multimodal models
507M parameters
English
Based on the Idefics3 architecture

For training data, evaluation results and full details, see the original model card: HuggingFaceTB/SmolVLM-500M-Instruct.

Intended use and limitations

Image description, visual question answering and reading text in images.

Not for high-stakes decisions about people (for example identity, employment or credit). It can describe images inaccurately or invent details; see the misuse and out-of-scope section of the upstream card.

What changed in this conversion

Vision encoder and token embeddings quantized to int8
Vision encoder, prefill and decoder exported as separate programs
Context capped at 512 tokens

Inputs and outputs

vlm_vision_q8.pte: one image to visual tokens
vlm_prefill.pte and vlm_decoder.pte: prompt prefill, then token-by-token decoding; embeddings come from embed_tokens.i8.bin with embed_tokens.scale.f32.bin
Tokenize with the files in tokenizer/; vlm_meta.json lists the special tokens and sizes

Files

File	Size	SHA-256
`embed_tokens.i8.bin`	45.1 MB	`d3ea47247be0b070…`
`embed_tokens.scale.f32.bin`	192.5 KB	`adefdadaae81d572…`
`tokenizer/special_tokens_map.json`	1.0 KB	`aa0ff906077086df…`
`tokenizer/tokenizer.json`	3.4 MB	`5ece781dc8d2b2f3…`
`tokenizer/tokenizer_config.json`	27.6 KB	`36c6fd44d07d10fd…`
`vlm_decoder.pte`	370.7 MB	`d92697a8c32eeaa1…`
`vlm_meta.json`	201 B	`b36fcf7ef7da8c2a…`
`vlm_prefill.pte`	370.7 MB	`1e8671eb77ae2f60…`
`vlm_vision_q8.pte`	103.8 MB	`5f09d6028d9bb945…`

Full checksums are in manifest.json. The playground verifies them after download.

Running it

These programs were exported with the ExecuTorch WebGPU backend (backends/webgpu) and validated with the WebAssembly runtime build used by the ExecuTorch WebGPU Playground (launching soon on executorch.ai). That runtime build is not published yet; until it is, the playground is the supported way to run them. It downloads these files once, caches them in the browser, and runs everything locally.

Requirements: a browser with WebGPU enabled.

Provenance

Converted from the upstream checkpoint by the ExecuTorch team with the WebGPU export path.
A reproducible export recipe will be published alongside the open-source ExecuTorch WebGPU runtime.

Citation

If you use this model, please cite the original work:

@article{marafioti2025smolvlm,
  title={SmolVLM: Redefining small and efficient multimodal models},
  author={Andrés Marafioti and Orr Zohar and Miquel Farré and Merve Noyan and Elie Bakouch and Pedro Cuenca and Cyril Zakka and Loubna Ben Allal and Anton Lozhkov and Nouamane Tazi and Vaibhav Srivastav and Joshua Lochner and Hugo Larcher and Mathieu Morlon and Lewis Tunstall and Leandro von Werra and Thomas Wolf},
  journal={arXiv preprint arXiv:2504.05299},
  year={2025}
}

License

This is a derivative of the upstream model and is distributed under the same terms (apache-2.0); see LICENSE. Modifications (conversion to the ExecuTorch format and quantization) by Meta Platforms, Inc.