Hugging Face's logo
Activity Feed

AI & ML interests

None defined yet.

Recent Activity
wscholl  updated a Space 25 days ago
squishai/README
wscholl  updated a model 25 days ago
squishai/gemma-3-4b-it-bf16-squished
wscholl  updated a model 25 days ago
squishai/gemma-3-1b-it-bf16-squished
View all activity

Organization Card
Community About org cards
Squish

Squish: pre-squished models for Apple Silicon

Fast local LLMs on Apple Silicon. Sub-second model loads. Beats Ollama on throughput, tail latency, and full-response time. One OpenAI and Ollama-compatible daemon. No cloud, no API keys, fully offline.

License: BUSL-1.1 PyPI Python Homebrew Platform CI Konjo Gate Coverage Ruff Docs 🤗 Models OpenAI API compatible Ollama compatible Stars

  ███████╗██╗  ██╗██╗  ██╗         █████╗     █████╗ ██╗  ██╗        ██████╗ ██████╗ ██╗ ██╗
  ██╔════╝██║  ██║╚██╗██╔╝        ██╔══██╗   ██╔══██╗╚██╗██╔╝        ╚════██╗╚════██╗╚═╝██╔╝
  ███████╗███████║ ╚███╔╝         ╚██████║   ╚█████╔╝ ╚███╔╝          █████╔╝ █████╔╝  ██╔╝
  ╚════██║╚════██║ ██╔██╗          ╚═══██║   ██╔══██╗ ██╔██╗          ╚═══██╗██╔═══╝  ██╔╝
  ███████║     ██║██╔╝ ██╗         █████╔╝██╗╚█████╔╝██╔╝ ██╗        ██████╔╝███████╗██╔╝██╗
  ╚══════╝     ╚═╝╚═╝  ╚═╝         ╚════╝ ╚═╝ ╚════╝ ╚═╝  ╚═╝        ╚═════╝ ╚══════╝╚═╝ ╚═╝
     faster cold start                faster long-prompts                   less RAM

   ██████╗    ███████╗███████╗        ██████╗ ██╗  ██╗        ██╗███╗   ██╗████████╗██████╗
  ██╔═████╗   ██╔════╝██╔════╝        ╚════██╗██║  ██║        ██║████╗  ██║╚══██╔══╝╚════██╗
  ██║██╔██║   ███████╗███████╗         █████╔╝███████║        ██║██╔██╗ ██║   ██║    █████╔╝
  ████╔╝██║   ╚════██║╚════██║        ██╔═══╝ ╚════██║        ██║██║╚██╗██║   ██║    ╚═══██╗
  ╚██████╔╝██╗███████║███████║        ███████╗     ██║        ██║██║ ╚████║   ██║   ██████╔╝
   ╚═════╝ ╚═╝╚══════╝╚══════╝        ╚══════╝     ╚═╝        ╚═╝╚═╝  ╚═══╝   ╚═╝   ╚═════╝
     cold load · 0.33–0.53s         tok/s · beats Ollama              quant default

   ██╗ ██╗███╗   ███╗███████╗        ██████╗    ███████╗ ██████╗          ██╗ ██████╗  ██████╗
  ███║███║████╗ ████║██╔════╝        ╚════██╗   ██╔════╝██╔════╝         ███║██╔═████╗██╔═████╗
  ╚██║╚██║██╔████╔██║███████╗         █████╔╝   ███████╗██║  ███╗        ╚██║██║██╔██║██║██╔██║
   ██║ ██║██║╚██╔╝██║╚════██║        ██╔═══╝    ╚════██║██║   ██║         ██║████╔╝██║████╔╝██║
   ██║ ██║██║ ╚═╝ ██║███████║        ███████╗██╗███████║╚██████╔╝         ██║╚██████╔╝╚██████╔╝
   ╚═╝ ╚═╝╚═╝     ╚═╝╚══════╝        ╚══════╝╚═╝╚══════╝ ╚═════╝          ╚═╝ ╚═════╝  ╚═════╝
     repeat TTFT · KV hit                GB · smaller on disk              inference modules

Run a model in one command

brew tap konjoai/squish
brew install squish
squish run qwen2.5:7b      # pulls a pre-squished model + starts a local server

The daemon serves an OpenAI (/v1/*) and Ollama (/api/*) API on port 11435. Point any existing client at it and go. Prefer pipx? pipx install squish-ai.

Models in this org

Every model below is INT4-quantized (4-bit, group size 64, affine) into MLX format and ready for squish run. Sizes are the actual on-disk download from this org; raw is the source bf16 checkpoint.

Model Run it Raw (bf16) Squished Saved Context
Qwen2.5-1.5B-Instruct squish run qwen2.5:1.5b 3.1 GB 0.9 GB 72% 32,768
Qwen2.5-7B-Instruct squish run qwen2.5:7b 15.2 GB 4.3 GB 72% 32,768
Qwen3-0.6B squish run qwen3:0.6b 1.2 GB 0.35 GB 71% 40,960
Qwen3-4B squish run qwen3:4b 8.0 GB 2.3 GB 72% 40,960
Qwen3-8B squish run qwen3:8b 16.4 GB 4.6 GB 72% 40,960
Llama-3.2-1B-Instruct squish run llama3.2:1b 2.5 GB 0.7 GB 72% 131,072
Llama-3.2-3B-Instruct squish run llama3.2:3b 6.4 GB 1.8 GB 72% 131,072
Gemma-3-1B-Instruct squish run gemma3:1b 2.6 GB 0.8 GB 72% 32,768
Gemma-3-4B-Instruct squish run gemma3:4b 9.9 GB 2.6 GB 74% 131,072

squish run <id> downloads these exact weights and loads them in under a second: no compression wait, no Python environment, no cloud.

Why Squish

Squish separates how a model's weights are stored from how they run: stored compressed and Metal-native, then mmap-ed straight into unified memory, with no dtype-conversion pass. It runs as a persistent daemon whose two-tier KV cache reuses prefill across requests instead of re-running it.

Measured on an Apple M3 (16 GB), Qwen2.5-7B vs Ollama, thermally controlled (Squish INT4/INT3 vs Ollama Q4_K_M):

Metric Ollama Squish
Full response @ 4,000-token prompt 37.5 s 3.8 s (up to 9.8× faster)
Cold start (load + first token, 1.5B) 20–30 s ≈ 0.5 s (54× load)
Decode throughput @ 75 tokens 20.3 tok/s 24.0 tok/s (INT3)
Repeat-prompt TTFT (KV cache hit) ~160 ms 4–11 ms
Peak RAM during inference 5.14 GB 3.50 GB
Disk (7B INT4 / INT3) 4.36 GB 4.00 / 3.56 GB
Cold short-prompt TTFT 167 ms 192 ms (honest loss)

The one place Ollama wins is single-token latency on a cold, novel prompt, stated plainly. Full methodology and ablations: BENCHMARKS.md.

What it doesn't do

If any of these matter, Ollama or LM Studio is the right call:

  • No GPU outside Apple Silicon. It's MLX-based; CUDA users want vLLM or llama.cpp.
  • No multi-user serving. One developer, one machine, not a production API.
  • No multimodal. Text only.
  • Slower first token on a cold, short prompt than Ollama (192 ms vs 167 ms).

Links

All models here are pre-squished by Squish. Run any of them in one command on Apple Silicon.

datasets 0

None public yet