qualitymaterial/Flint-NVFP4-FreeToken

Flint NVFP4 for FreeToken — experimental setup

Description

A reproducible starting point for running Recursive Flint's official NVFP4 GGUF through FreeToken with CPU/GPU expert offloading. It pairs a pinned experimental Flint integration with launch scripts and recorded compatibility checks, so users can reproduce the tested environment without building a new quantization.

Flint is a text-only mixture-of-experts model developed by Recursive for coding-agent workflows, including code investigation, code generation and multi-turn tool use. The upstream model has approximately 39.8 billion total parameters and 2.5 billion active parameters per token. This package adds setup and validation material; it does not train, fine-tune, quantize or modify the model weights.

Release contents: Linux/WSL installer, FreeToken API launcher, Windows WSL launcher, pinned upstream references, license and validation report. Intended users: developers experimenting with local coding assistants and OpenAI-compatible clients. Status: experimental, initially distributed privately; local Ubuntu WSL serving has passed the short acceptance checks below.

This repository contains setup scripts and measured compatibility results, not model weights or a new quantization. Downloaded weights come directly from Recursive's official NVFP4 GGUF.

The runtime uses FreeToken PR #797, pinned to 711a86ef11c67bbe6c69ff59370594f64c6ef1b8. It is an independent experimental package, not an official FreeToken or Recursive release. The integration code belongs to its upstream contributors.

Why I put this setup together

I wanted to use Flint for local coding and agent work on the PC I already own: an RTX 5070 Ti with 16 GB VRAM and 64 GB system RAM. My earlier FreeToken setups with Ornith and Qwen3.8 Distill followed the same idea: use system RAM for a model larger than the GPU's memory, keep useful cache capacity, and make it practical to use from FreeToken Desktop.

Flint's official NVFP4 GGUF is about 22.86 GB, so the complete file cannot fit in my 16 GB GPU. FreeToken's CPU/GPU expert offloading made this a workable local setup. NVFP4 was a practical choice because an official checkpoint already existed and it passed our compatibility checks; I did not need to create another quantization. Q4 was also explored, but those tests do not establish that either format is better for every workload.

The goal is a useful daily setup, with responsive generation and room for reusable context, without buying a larger GPU or keeping a rented GPU running. Local inference avoids a hosted inference API's per-token charges; the PC, electricity, storage and any external tools still have their own costs. Runpod was used temporarily for compatibility testing. Daily use runs on my own machine.

In my follow-up use on October 23, 2026, Flint worked well in FreeToken with a 323K token cache, and generation felt very fast. That is my experience with this setup, not a controlled speed comparison. A cache-pool allocation is not the same as a per-conversation context limit or a completed long-context test. The exact allocation and a matching launch recipe for that later configuration have not been captured in this package; its included launcher still defaults to the earlier tested 8K profile.

Choice	Reason on my PC
Official Flint NVFP4	Use the existing official quantization for local coding and agent experiments
RTX 5070 Ti 16 GB + 64 GB RAM	Work with the hardware I already own
CPU/GPU expert offloading	Use system RAM for weights that exceed VRAM capacity
FreeToken Desktop	Keep model selection and chat in the interface I already use
Large reusable token cache	Preserve capacity for repeated context while balancing weights and runtime memory
Pinned setup and recorded checks	Give others a concrete starting point and show what was actually tested

Requirements and setup

Linux x86-64 or Ubuntu WSL2, NVIDIA CUDA 13 compatible driver, Python 3.12 managed by uv, git, and substantial free disk space (allow 70 GB). The GGUF is 22.86 GB and needs additional system RAM, GPU cache and workspace. The local 64 GB host / 16 GB RTX 5070 Ti passed the 8K short checks described below. Other hardware and larger contexts need separate validation. Do not run another large GPU model at the same time.

In Linux or Ubuntu WSL, install uv using its official instructions, download this repository, then run:

bash install.sh
bash serve.sh

Windows users run installation inside Ubuntu WSL first. Start-Flint.ps1 then starts that same isolated environment. This package does not replace the FreeToken Desktop engine or register a model in its model picker. It serves FreeToken's local API at http://127.0.0.1:1919/v1; connect an OpenAI-compatible client there. Stop with Ctrl+C. The engine must finish loading before inference succeeds; the HTTP health route alone does not establish model readiness.

The maintainer's separate local Desktop integration was subsequently verified using a WSL daemon and an imported model entry. Its machine-specific Desktop adapter and shortcut are not included in this portable package; installing these scripts alone does not perform that integration.

The installer downloads pinned third-party FreeToken wheels and source. Nightly asset URLs can disappear when upstream replaces the release; if an asset is missing, installation stops. Do not silently substitute another build.

The launcher starts one request at 8K context with hybrid expert offload and a 2 GiB GPU expert cache. FLINT_CONTEXT=32768 bash serve.sh selects 32K. Its --moe-cpu-layers auto is a WSL pinning adjustment; this WSL profile passed the local short acceptance checks. The cloud run used the same settings except that adjustment.

Verified results, 2026-10-23

On a disposable Linux Runpod RTX 5090 32 GB using the pinned integration and matching base wheels:

• Official NVFP4 passed visible text, generated Python against five functional cases, a tool-call/result round trip and streaming.
• NVFP4 recovered three keys from a server-reported 29,528-token prompt under a 32K profile. The original helper exit was 1 because of a tokenizer return-object counting bug; the saved server usage and answer establish successful retrieval. This is one synthetic retrieval, not a broad context benchmark.
• 217 existing reference/config/tokenizer/parser checks passed for NVFP4; 217 also passed for Unsloth Q4.
• Q4 passed three serving checks, but its coding text failed Python parsing. That response was not retained, so the cause is unresolved; Q4 long-context was not tested.

Local follow-up on 2026-10-23: Ubuntu WSL on an RTX 5070 Ti 16 GB passed all four short serving checks, including generated interval-merging code against five cases, tool roundtrip and streaming. Profile: 8192 tokens, 2 GiB expert cache, hybrid execution, eight CPU workers and automatic pinning. Runtime occupied approximately 5.4 GB GPU memory including other host use. Python's base runtime needed a Linux path without spaces for FlashInfer compilation; the installer now handles a spaced base path by copying that runtime into the isolated installation. This new automatic copy path has syntax validation; the local repair used the equivalent manual copy.

Desktop follow-up on the same PC: FreeToken Desktop successfully imported Flint, started the WSL engine, and returned two requested exact replies in Chat. The first took about 60 seconds including initial CUDA sampling-kernel compilation; the second took 975 ms for 20 tokens (Desktop-reported 20.5 tokens/s). These are short integration checks, not a general throughput benchmark. Reopening Desktop through the local Flint shortcut retained the running model.

Native Windows engine support, local 32K retrieval, full 262K context, concurrent-agent workload and general coding benchmarks remain unverified by these recorded checks. The later 323K cache observation above is a maintainer report, not a validated 323K conversation window. See RESULTS.md for the earlier cloud evidence. Preserve returned reasoning_content and tool calls in conversation history. Flint uses the glm47 tool parser and optional flint reasoning parser.

Licenses and attribution

These original setup scripts are Apache-2.0. FreeToken and the official model have their own Apache-2.0 licenses/notices, available in their linked repositories; their files are fetched directly, not redistributed here. No FreeToken engine source, weights, credentials or private user data are bundled.