det-train

Bitwise-deterministic training arithmetic at any parallelism. The contract: outputs are fixed functions of the inputs, independent of tiling, K-sharding, world size, and repetition, so a loss curve is torch.equal-identical whether the same global computation runs on one GPU or sixty-four.

Mechanism (Kulisch long accumulator): each GEMM output owns a fixed array of signed int64 digits in radix 2^32 spanning the full fp32 product exponent range. Every product is formed exactly in fp64 (bf16 and f32 significands make every product exact) and its base-2^32 limbs are added into the digits. Accumulation is therefore exact integer addition into fixed bins, which is associative by construction, so any fold tree over any K-decomposition is bit-identical. Sharded partials compose by adding digit arrays; distributed reduction composes by a single int64 all-reduce of digit arrays. There is no data-dependent window and no floor loss, so the result is not merely run-to-run repeatable but invariant to how the computation is split across tiles, shards, and ranks. Results carry the exact fp64-product sum rounded once to fp32.

Usage

import torch
from kernels import get_kernel

dt = get_kernel("phanerozoic/det-train", version=1, trust_remote_code=True)

layer = dt.DetLinear(4096, 4096, device="cuda")
y = layer(x); y.square().mean().backward()      # deterministic fwd and bwd

# tensor parallelism over K: shards compose bit-identically
a1 = dt.det_gemm_partial(x[:, :2048], w[:, :2048])
a2 = dt.det_gemm_partial(x[:, 2048:], w[:, 2048:])
y_sharded = dt.det_finalize(dt.det_combine([a1, a2]))
assert torch.equal(y_sharded, dt.det_gemm(x, w))

# data/tensor parallel: sum long-accumulator digits across ranks, exactly
acc = dt.det_all_reduce(dt.det_gemm_partial(x_shard, w_shard))
y = dt.det_finalize(acc)                        # identical at any world size

version selects the release branch; trust_remote_code is required by kernels for publishers without the trusted-publisher mark. K-shard boundaries are arbitrary; any partition composes exactly.

API

Verified behavior

• Sharding and grouping invariance: det_gemm over full K equals the det_combine/det_finalize composition of 2-, 4-, 8-, and 16-way K-shards, and any fold grouping of the partials, bitwise, for bf16 and f32 (L4 sm_89 and RTX PRO 6000 sm_120).
• Real multi-GPU: a K-sharded GEMM reduced across 4 GPUs via one NCCL int64 all-reduce of the long-accumulator digits is torch.equal to the single-process result, and every rank at world size 1 and 4 emits the identical result digest. This is the loss-curve world-size invariance property, demonstrated at the kernel.
• Accuracy: results carry the exact fp64-product sum rounded once to fp32; maximum relative error against an fp64 matmul reference is at the fp32 rounding floor.
• Determinism: repeated runs, and a training run whose forward is computed by sharded composition, are bitwise identical.

Scope and cost

The kernel accumulates on fp64 scalar cores, not tensor cores; throughput is measured and reported below, and the intended regimes are debugging runs, bisection across cluster topologies, evaluation training, silent-data- corruption detection, and audits, where the property is worth the throughput. K <= 2^19 per GEMM call; bf16 and f32 inputs.

License

Apache-2.0.