PIN: folded models, and what you can add to them
0.9080 on CIFAR-10 from 270,277 stored values, standing for 2,415,919,104 connections: a fold of 8,939 times. Code for the PIN papers, archived on Zenodo with a DOI.
Code for the PIN papers. Every script here is one file that runs on its own, downloads or verifies its own data, checks itself before doing any real work, prints a table, and writes results to disk as it goes. There is no package to install and no configuration to write: paste one into a hosted notebook and it runs.
That shape is deliberate. The work is done from a phone as often as from a desk, so a script that needs four files and a setup step is a script that does not get run.
What a fold is, in one paragraph
An ordinary weight matrix stores one number per connection. A folded one stores a much smaller set of VALUES and a PARTITION saying which connections share which value:
W[i, j] = v[ idx(i, j) ]
A convolution is one choice of idx. So are a patch layer, a butterfly, a
block-circulant matrix, and an arbitrary tying. The framework separates the
partition from the values, which makes the architecture a data structure
rather than a code path, and lets a model be reshaped by swapping the index
while keeping what it learned.
Start here
| you want to | run |
|---|---|
| see a folded model reach 0.9080 on CIFAR-10 | resnet.py |
| see the fast path checked against the general one | sliceconv.py |
| build the saved bases everything else reads | testbed_all.py |
| see a member added to a frozen base | injection.py |
| see a model fail to notice its own ignorance | confidence_member.py |
resnet.py takes about half an hour on a modest GPU and prints its own
timing estimate after ten measured steps, so you can stop it early if that
is not what you want.
Every figure and table in v5, and what produced it
| section | claim | script |
|---|---|---|
| §1 | 0.9080 on CIFAR-10 from 270,277 values | resnet.py |
| §2.1 | pooling, and the depth curve inverting | pooling.py |
| §2.2 | depth has a low optimum; reach does not predict it | grow_depth.py, depth_law.py |
| §2.3 | the stacks are chaotic and self-regulating | criticality.py, homeostat.py |
| §2.4 | blindness, visible in the gradients | strain.py |
| §3 | four curricula, four nulls | grown.py, curriculum.py, scramble.py |
| §4.1 | injection at exactly zero disturbance | injection.py |
| §4.2 | head capacity, and containment withdrawn | head_capacity.py, provisioning.py, bodies.py |
| §4.3 | a member from five examples, in 0.6 ms | sample_efficiency.py |
| §4.4 | the weight as a dial, and negative weights | member_weight.py |
| §4.5 | correction is bounded; extension is not | gate.py, ignorance.py |
| §4.6 | ignorance is invisible at the output | confidence_member.py |
| §4.7 | qualifying a member: concede or override | qualify.py |
| §4.9 | borrowing across bases | borrow.py, calibration.py |
| §4.10 | what a buyer is actually paying for | marketplace.py |
| §5 | the cascade, and confidence signals | cascade.py, routing_signals.py |
| §6 | determinism, and the 0.002 noise floor | determinism.py |
Supporting: testbed_all.py trains the saved bases; sliceconv.py holds the
fast convolution and its check; switch.py measures what reshaping costs;
make_v5_figures.py draws the six figures from the measured values.
Reproducibility
Folded training was not reproducible until recently and now is. GPU atomic
scatter completes in hardware order and floating-point addition is not
associative, so the same seed produced models differing by about 0.002 in
final accuracy. A fixed-order reduction is exact, and it is cheap here
because the partition never changes during training, so the sort is paid
once for the whole run. determinism.py measures both.
Two protocol notes, learned the hard way and worth inheriting:
- Compare paired within a seed. An unpaired spread misled this project three times, once hiding a result at 2.6 deviations.
- Read best epoch, not final, whenever an arm overfits. A control with more epochs to overfit with once produced an apparent +0.0051 that was really +0.0001.
What is not here
The papers withdraw things, and the code keeps the scripts that produced the withdrawn results as well as the corrections. That is deliberate. A project that never retracted anything is one nobody checked.
Superseded versions of a script are not included; where a fault mattered it is described in the file that replaced it, in a comment at the place it happened. Several of those comments are the most useful documentation in the repository: an index sized on uniform test data that asked for 149 GB against a real one, a member applied inside a suppression dead zone, a storage figure that omitted the head it was compared against.
Requirements
Python 3, NumPy, and CuPy if a GPU is present. Every script falls back to
NumPy on the CPU, more slowly. Datasets are fetched through Keras and
cached; resnet.py caches to a directory you choose so a restarted session
does not download CIFAR again.
Nothing else. There is no framework dependency, no custom kernel, and no compiled extension. A folded convolution is expressed in operations any array library already has, which is part of the claim.
Citing
The papers carry the results and the reasoning; this repository carries what produced them. Cite the paper for a finding and this for the method that produced it.
Licence
Apache 2.0. It permits commercial use and includes an explicit patent grant, which matters more here than the licence text: the framework describes mechanisms someone may want to build on, and a permissive licence without a patent clause leaves that ambiguous.