arxiv:2609.02881

Graph Machine: Towards Better Pretraining via Edges

Published on Sep 2

· Submitted by

Authors:

Abstract

A Graph Machine architecture uses sparse dynamic routing and differentiable pointer chasing to maintain linear state complexity, enabling efficient replacement of dense Transformer layers with minimal loss degradation.

Generated by thinkingmachines/Inkling-Small

We introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves O(n) complexity in its sparse layers without restricting the potentially accessible state size to O(1). Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.

View arXiv page View PDF Project page GitHub 1 Add to collection

Community

lintaihou

Paper submitter about 3 hours ago

Author here! How many tokens does an attention head need to retrieve? In these experiments, just 2–4 out of 4,096 can be enough.

Graph Machine replaces 75% of Qwen3-0.6B's dense attention layers with sparse layers using dynamic pointers. Attention follows pointers to retrieve content; referral passes addresses through a form of pointer chasing with gradients. The sparse layers have linear compute.

I pretrained from scratch for 15.7B FineWeb-Edu tokens using a standard LLM recipe. Compared with the dense baseline, the best model retrieving 2 tokens per KV head raises validation loss by just 0.014; at 4 tokens, the best model improves it by 0.003.

Code · Checkpoints · Training curves

Happy to discuss the architecture and results!