YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Archify Free-Pipeline Research
Exploratory research on whether redundant full-repository scouting or specialized scouting produces better source-grounded architecture representations when a stronger model composes the final map.
Research question
Does redundant full-repository scouting or specialized scouting produce better source-grounded architecture representations when composed by a stronger model?
Status: Exploratory. A newer larger-repo run is encouraging for fan-out scouting and supports the variance/low-overlap hypothesis; no full orchestration strategy has been shown superior yet.
Why this exists
This repository preserves a research trail around Archify as a cognitive checkpoint after agentic coding work. The goal is not to prove that one orchestration strategy wins. The current value is narrower and more honest:
- keep source-grounded claims instead of model prose only;
- compare what different models notice or omit;
- separate technical pipeline success from human-comprehensible architecture output;
- document failure modes before declaring a harness reliable.
Current evidence snapshot
| Dataset / target repo | Source-grounded concepts | Model diagrams measured | Visual-check pass | Visual-check fail | Not run |
|---|---|---|---|---|---|
| HAI-MCP | 37 | 8 | 6 | 1 | 1 |
| HAI-TIKTOK | 54 | 7 | 3 | 2 | 2 |
| codex-archify-experiment | 21 | 4 | 3 | 0 | 1 |
| Total | 112 | 19 | 12 | 3 | 4 |
Visual-check means automated browser evidence from Archify, not human perceptual approval. A pass says the artifact fit/readability checks passed in the measured browser viewports; it does not prove semantic superiority.
New variance signal
A later larger run produced a stronger signal for the fan-out hypothesis:
110 evidence items from 6 maps -> 53 verified concepts
| Consensus level | Verified | Unverified |
|---|---|---|
| >=3 models | 12 | 0 |
| <=2 models | 41 (77%) | 9 |
| Model | Clusters |
|---|---|
| muse-spark-1.3 | 22 |
| nemotron-3-ultra | 22 |
| ling-3.0-flash-fin | 18 |
| mimo-v2.5 | 17 |
| muse-spark-1.2 | 11 |
| nemotron-3.5-lightning | 0 |
This is encouraging for fan-out scouting: a filter that requires at least three agreeing models would discard about 77% of the verified clusters in this run. Incomplete candidates can still be useful when their source references survive into the merge, which is exactly the failure-tolerant behavior the harness is meant to test.
The stronger claim remains limited. The run supports, but does not confirm, the idea that larger projects produce lower overlap between model views. Two projects differ in structure and assignment content, and 77% of the found clusters is not automatically 77% of all important project information. Also, nemotron-3.5-lightning needs clarification: if its candidate was formally usable but contributed 0 clusters, the report must distinguish candidate usability from merge-usable source evidence.
Current interpretation: low-consensus findings matter and must be preserved for the stronger aggregator. Whether that becomes a better owner-facing architecture representation depends on the composer producing a map that is actually more understandable, not only more comprehensive.
Representative artifacts
To see how model choice changed the output, open the dedicated comparison page:
docs/model-choice-impact.md— per-model table with HTML maps, screenshots, structure counts, visual-check status, and the latest six-model merge contribution.docs/external-online-artifacts.md— external online artifact links that are part of the research trail but not imported into this repo.
HAI-MCP, Opus map: visual-check pass
- 37 merged source-grounded concepts in the claim set.
- 12 components, 12 connections, 3 guided views.
- Automated visual-check: pass, 0 diagnostics, 0 overflow viewports.
HAI-MCP, composed map: more evidence is not automatically better
- Same 37-concept claim base, composed into 13 components and 13 connections.
- Automated visual-check: fail, 3 diagnostics, 2 overflowing measured viewports.
- Lesson: aggregation can preserve more findings while producing a worse first-screen artifact.
HAI-TIKTOK, Opus map: larger repo case
- 54 merged source-grounded concepts.
- 12 components, 11 connections, 3 guided views.
- Automated visual-check: pass.
codex-archify-experiment, Muse map: failure-lineage as data
- 21 merged source-grounded concepts.
- 10 components, 9 connections, 3 guided views.
- Automated visual-check: pass.
Experimental design
The controlled comparison still needs to be run cleanly:
same repository
same commit
same model pool
same budget class
│
├── A: redundant full-repository scouting
│
└── B: specialized scouting by analysis area
│
same stronger aggregator
same architecture-map assignment
↓
source correctness · important findings · omissions
comprehensibility · time · cost · failure behavior
What is measured
The project tracks more than whether HTML was produced:
- source correctness: paths, lines, functions, and responsibilities;
- important findings: central architecture facts, including minority discoveries;
- omissions: important components absent from a candidate map;
- visual artifact health: viewport containment, readability, diagnostics;
- human usefulness: whether the owner can explain the system top-down from the artifact;
- robustness: timeouts, invalid JSON, missing scouts, unavailable model slugs;
- cost/time: model cost, wall time, and wasted runs.
Data
data/architecture-runs.csv— compact run table.data/architecture-runs.json— machine-readable summary and run metrics.data/latest-specialized-scouting-result.md/.json— newer 110-evidence / 53-concept variance signal reported by Samuel.data/published-generated-artifacts.json— index of the published Archify website artifacts.data/model-choice-impact.json— derived model-choice comparison table for online reading.data/external-online-artifacts.json— machine-readable index of external online artifacts.data/hai-mcp.claims.md— 37 verified HAI-MCP concepts.data/hai-tiktok.claims.md— 54 verified HAI-TIKTOK concepts.data/codex-archify-experiment.claims.md— 21 verified codex-archify-experiment concepts.Wissehscaflichterversuchsaufbau.md— German research framing and caveats.
Published Archify website artifacts
The generated Archify results are committed under generated/ so they can be opened directly from GitHub:
- Archify HTML maps:
generated/*.architecture*.html - Archify JSON IR:
generated/*.architecture*.json - automated visual-check reports:
generated/*.visual-check.jsonandgenerated/*.visual-check.html - visual evidence screenshots:
generated/*.visual-check.*.png - merged claim summaries:
generated/*.claims.md
Raw .logs/ and nested run logs are intentionally not published; they are local execution traces, not owner-facing evidence.
Current conclusion
The low-overlap/fan-out hypothesis now has supporting evidence from a larger run, but no complete orchestration strategy has been shown superior yet.
The scientific value so far is exploratory: the repository contains hypotheses, failure modes, source-grounded claim sets, screenshots, and measurement criteria that make the next comparison auditable instead of anecdotal.