harshpreet931/cut-laya-onnx

Laya for the browser (8-bit ONNX)

Laya by Convai Innovations, converted to run in a web page with ONNX Runtime Web. It powers cut., an editor that can't write, only cut.

File	Size	What changed
encoder.onnx	493 MB	8-bit weight-only matmuls (MatMulNBits, block 32, symmetric); token embeddings in fp16
head.onnx	37 MB	8-bit weight-only matmuls
tokenizer.json, rl_agent_config.json	3.6 MB	unchanged

The original checkpoint is 1.7 GB as ONNX in fp32.

How it was made

1. Exported with laya-ts/scripts/export_onnx.py from Laya's repo (the script checks the ONNX outputs against PyTorch within 1e-4).
2. Shrunk with scripts/shrink_browser_model.py. 4-bit weights were tried and dropped: they changed the top label on 11 of 57 test lines.

How close is it?

On cut's 7 sample posts (57 lines, one five-way choice question per line), run in Chrome on WebGPU, it gives the same top label as the full PyTorch model on all 57 lines, with probabilities within 0.021. That's a small, hand-made set, not a benchmark.

Using it

With laya-ts in a browser:

import { Agent } from "laya-ts";
const agent = await Agent.load("harshpreet931/cut-laya-onnx");

cut uses a vendored copy of laya-ts that also runs the head on WebGPU and batches lines into one pass: about 0.8 s for a 9-line post on an Apple M4 Pro.

License

Apache-2.0, like the original. Laya is by Convai Innovations; this repo only changes the format and precision of its weights.