FreeSolo-341M

Paper · GitHub · Models · Usage

RoPE-free long-range attention, trained at 4k with zero-shot length extrapolation.

FS scores all completed chunk mean keys and reads four selected chunks per query and head. This 341M research base model was trained on FineWeb-Edu at a 4,096-token context.

Try it

hf download zekun-shi/FreeSolo-341M freesolo-source.zip --local-dir ./FreeSolo-341M
python -m pip install ./FreeSolo-341M/freesolo-source.zip
freesolo-generate --model zekun-shi/FreeSolo-341M --output generation.json

The short retrieval example asks for the hidden number 5437923. Start with CUDA-enabled PyTorch; Usage gives the tested installation and speed-replication command.

The idea

FreeSolo learns simple chunk mean keys that support search, norm pruning, and hierarchical summarization. RoPE stays in the local window; long-range attention reads selected original tokens without rotating their queries and keys.

Results

Context	2k	4k	8k	16k
RULER mean	30.70	23.26	16.45	11.84

12 tasks at 2k/4k; 13 at 8k/16k, where QA-2 is available. 100 examples per task, seed 42. These are checkpoint evaluation results; the figures show the paper's model-family comparison.

This is a base model without instruction tuning. Retrieval is task dependent, especially when several facts or values must be combined. Training details, artifact checksums, and citation are in Usage.