docs/DEPLOYMENT.md

Deployment

Status tags: IMPLEMENTED · VERIFIED · MEASURED · OPEN · DEFERRED · BLOCKED · BY DESIGN.

The live SatQuery AI system runs across three private repositories plus one public umbrella repository, serving a static frontend on Cloudflare Pages, a thin orchestrator / gateway on Render, and a CPU inference service in a GitHub Codespace reached over an outbound tunnel. The monorepo working copy is not the deployed source.

This document is the exhaustive deployment reference: the four tiers, every live revision, every environment variable (with measured live values), the deploy mechanics per tier, cold-start behaviour, the five historical backend blockers, the platform traps, and the superseded design that the active one replaced. It is written to be readable without the source tree, but every non-obvious claim carries the file it came from.

The single most important trap in this document. deploy/ inside the monorepo is stale and untracked. It is not the deployed source. The deployed backend is SatQuery-Backend/main.py; the monorepo's deploy/render/main.py is an earlier, tunnel-less revision. Edits must go to the three real repositories, never to the local deploy/ tree.

Hostnames are deliberately not published. The orchestrator's public hostname appears throughout this release as <backend-host>. The deployment is documented for reproducibility — the topology, the environment-variable names, the timeout chain and the failure modes are all exact — without advertising the live endpoint. The three deployment repositories are private and are not part of this release.

Companions. architecture/02-deployment-topology.md (the long-form topology treatment), OPERATIONS.md (running the live system), SECURITY.md (the trust boundary), architecture/10-observability-and-ops.md (health, traces and the operator surface), LIMITATIONS.md §4 (operational limitations), RESEARCH_NOTES.md §6 (the auto-mode fallthrough), and TESTING.md (how the deploy-time properties are tested).

---

1. How to read this document

Convention	Meaning
VERIFIED	Read from a live endpoint, a git ref, or a file on disk during the release reconnaissance
MEASURED	A value with a recorded number and its source
IMPLEMENTED	Code exists; whether it ran is stated separately
NOT RUN	The work has not been executed
OPEN	A known defect or gap that is not closed
BY DESIGN	Deliberately absent, and the reason is recorded
UNKNOWN	UNKNOWN — not established from the available evidence

Every revision, environment variable and finding below was read from a file or a live endpoint. Where a value could not be established, the text says so rather than guessing.

1.1 The three deploy sources versus the working copy

Artifact	Location	Role
Frontend source	Anish-lab-blip/SatQuery-Frontend (private)	staged from the monorepo's frontend/ contents
Backend source	Anish-lab-blip/SatQuery-Backend (private)	Render orchestrator (main.py, tunnel client)
Inference source	Anish-lab-blip/SatQuery-Inference (private)	Codespace FastAPI + deploy/codespace/tunnel_agent.py
Public umbrella	Anish-lab-blip/SatQuery-AI (public)	the intended release home
Working copy	C:/Users/anish/satquery-ai	local only, no git remote

The working copy's deploy/ is untracked (git ls-files deploy/ returns empty) and stale. Its deploy/render/main.py is a tunnel-less revision (532 lines by the project's own record); the deployed SatQuery-Backend/main.py is 768 lines and carries the tunnel client (docs/FINAL_DELIVERY_TODO.md §1.1). The tunnel agent (deploy/codespace/tunnel_agent.py) is not present in the monorepo working copy at all — it lives in the SatQuery-Inference repository, and the monorepo's deploy/codespace/launch.sh refers to it as a path that only exists in the deployed checkout.

---

2. Live revisions (VERIFIED)

Read from the GitHub API during the release reconnaissance (release/CURRENT_RELEASE_STATE.md §1).

Component	Repository	Visibility	Branch	Revision	Host
Frontend	Anish-lab-blip/SatQuery-Frontend	private	main	2d7ae53b482d	Cloudflare Pages → satquery.pages.dev
Backend / orchestrator	Anish-lab-blip/SatQuery-Backend	private	main	89d80eaddec5	Render → <backend-host>
Inference	Anish-lab-blip/SatQuery-Inference	private	main	5a0936ace491	Codespace potential-space-trout-r4ppw969w45j2pvvw, port 8000
Public umbrella	Anish-lab-blip/SatQuery-AI	public	main	3dcabd32da41 ("Initial commit")	this release home
Monorepo (working copy)	C:/Users/anish/satquery-ai	local only	master	9d57aed	no git remote; 334 dirty entries
Hugging Face	thundercode/SatQuery	public	main	lastModified 2026-09-25T16:26:53Z	2 files only: .gitattributes, 25-byte README.md

Notes that must not be smoothed over:

• The public umbrella SatQuery-AI contains only README.md (13 bytes: # SatQuery-AI). At the time of the reconnaissance it was effectively empty; it is the intended home for this release.
• The dirty-entry count is a snapshot. release/CURRENT_RELEASE_STATE.md §1 records 334 dirty entries at release reconnaissance; docs/FINAL_DELIVERY_TODO.md §1.1 records 323 (294 untracked, 20 modified, 8 deleted) at an earlier capture. The working tree changed between the two captures.
• The three deployed repositories are private. Their GitHub links return 404 for an outside audience. This is BY DESIGN (see §11.4).
• The Hugging Face repository thundercode/SatQuery carried only two files at reconnaissance (.gitattributes and a 25-byte README.md). It is not the runtime inference host; the Codespace resolves pinned backbones from the Hub at run time.

2.1 A note on the "HEAD re-read" verification

Nine deployed frontend files were re-read from the GitHub API and found sha256 byte-identical to the local copies, with the deployed HEAD re-read independently (verify_deployed_head.py, release/CURRENT_RELEASE_STATE.md §5). Three live-validation passes ran against successive HEADs: pass 1 against ff46eba42b18 + d413d3672311, passes 2 and 3 against the final HEAD 2d7ae53b482d. No run id is shared between passes.

---

3. The four tiers

Browser
  │  HTTPS
  ▼
Cloudflare Pages  —  satquery.pages.dev                (static frontend, 11 pages)
  │  HTTPS / JSON  →  /api/*
  ▼
Render            —  <backend-host> (orchestrator / gateway)
  │  outbound long-poll  POST /tunnel/agent
  ▼
GitHub Codespace  —  FastAPI inference, CPU, port 8000
  │  build_space_app()
  ▼
specialists: SmolVLM · RemoteCLIP · MiniLM · CROMA · STANet
  │
  ▼
ResultEnvelope  →  tunnel  →  Render  →  browser

flowchart LR
  U[Browser] -->|HTTPS| CF["Cloudflare Pages<br/>static frontend"]
  CF -->|"HTTPS JSON<br/>/api/health · /api/capabilities · /api/infer · /api/assets"| R["Render<br/>orchestrator / gateway"]
  R -->|"outbound long-poll<br/>POST /tunnel/agent"| C["GitHub Codespace<br/>FastAPI inference :8000"]
  C --> S[(SmolVLM · RemoteCLIP<br/>MiniLM · CROMA · STANet)]
  C -->|ResultEnvelope| R
  R -->|"envelope + error translation"| CF

The important inversion: the middle arrow is outbound from the inference host, not inbound to it. That is the whole reason the design works for a private repository (§4).

3.1 Cloudflare Pages — the static tier

Serves the frontend. No backend, no secrets, and no API calls of its own on the static pages. The one exception is the Analyze console (mission.html), which calls the gateway.

• Staged by: scripts/stage_pages.mjs (builds a Pages bundle).
• Deployed with: npx wrangler pages deploy.
• Deploy result (measured, docs/DEPLOYMENT_DECISION.md §7): 60 files staged, 39,173,936 B (37.36 MiB) total, largest file assets/video/satquery-launch-50s.mp4 at 22,710,313 B (21.66 MiB), with 3,504,087 B of headroom under the 25 MiB per-file limit; 0 missing references; 0 external network dependencies (HERMETIC); exit 0.
• Env vars: none (static). The Pages project name / domain is still open (§12).

The hermetic claim is scoped. docs/DEPLOYMENT_DECISION.md §3 audited frontend/ (excluding .tools/) and found zero occurrences of fetch(, XMLHttpRequest, axios, EventSource, WebSocket, /v1/, import.meta.env or process.env. docs/DEPLOYMENT_TOPOLOGY.md §3.1 narrows this: the "no API calls of any kind" statement holds for every static page except mission.html, which calls the orchestrator. The monorepo README.md's older claim that the frontend is "hermetic — no backend calls" is materially stale.

3.2 Render — the gateway

A deliberately thin, stateless orchestrator. It holds no model, no state, no database, and performs no auth (deploy/render/main.py module docstring; plan §73/§74). Its responsibilities, from docs/DEPLOYMENT_ARCHITECTURE.md §2:

Responsibility	Detail
Schema validation	reject malformed requests before they cost inference
Size limits	whole-request body cap, shared with the Codespace
Rate limiting	per-IP count + window — fairness, NOT a security control (docs/DEPLOYMENT_ARCHITECTURE.md §5.2)
CORS allowlist	the Pages origin; never *
Request ids	correlate a request across tiers
Timeouts	sit inside the task budget (§7.5)
Secret custody	holds credentials that must never reach the browser
Error translation	upstream failures → the documented error envelope (§7.6)

It is not a model host. It has no database, no auth, and no queue.

The gateway's proxied routes (from deploy/render/main.py):

Gateway route	Upstream	Notes
GET /api/health	answered locally	reports the orchestrator's own config; never answers for the Codespace
POST /api/infer	POST {codespace}/v1/analyze	wake-then-proxy; sets `X-SatQuery-State: waking
GET /api/capabilities	GET {codespace}/v1/capabilities	no second copy of the capability table
POST /api/assets	POST {codespace}/v1/assets	raw/multipart body relayed verbatim

Hard rule. The gateway must not retry POST /api/infer on its own — a retry would consume inference a second time. The client decides on retry. The reason is recorded in code (deploy/render/main.py docstring; docs/STEP7_BACKEND_CHAIN_REPORT.md §11).

No second capability table. The gateway proxies /v1/capabilities and nothing else decides "what can this deployment do?". The authoritative sources are core.planner.CAPABILITY_ASSETS and SpecialistSpec.requires_assets (docs/DEPLOYMENT_ARCHITECTURE.md §2.2).

3.2.1 The CORS allowlist is assembled, not just read

deploy/render/main.py::_allowed_origins assembles the allowlist in a fixed order:

1. SATQUERY_ALLOWED_ORIGINS — the operator's comma-separated list (authoritative for extra origins).
2. _PRODUCTION_ORIGINS — https://satquery.pages.dev, always present, so a missing env var cannot take the live site down.
3. _DEV_ORIGINS — 10 explicit host:port pairs (localhost and 127.0.0.1 × ports 3000/5500/5173/8000/8080), added unless SATQUERY_ALLOW_DEV_ORIGINS is one of 0/false/no/"".

A wildcard * raises ValueError — checked both in _allowed_origins and in GatewayConfig.__post_init__, because CORSMiddleware does not run that validator (deploy/render/main.py). The list is deliberately explicit, never a regex or suffix match, so allowing localhost for development cannot admit an arbitrary remote site. The health payload reports the effective list, so a production deployment can prove from outside that the dev origins were turned off.

3.2.2 The wake flow

deploy/render/main.py::ensure_codespace_up() returns (base_url, woke):

1. GET the Codespace via the GitHub API (deploy/render/codespaces.py::get_codespace).
2. If state != "available", POST .../start (start_codespace; GitHub returns 202, and 204 is also seen in practice).
3. Poll GET {base}/v1/health until 200 or until SATQUERY_WAKE_TIMEOUT_S elapses.

Polling knobs: _WAKE_POLL_INTERVAL_S = 2.0, _WAKE_HEALTH_TIMEOUT_S = 10.0 (deploy/render/main.py). The public base URL is derived by forwarded_url(), which prefers the Codespace JSON's web_url and rewrites its trailing port segment, falling back to https://{name}-{port}.app.github.dev. That host pattern is an isolated assumption: the module's own docstring records that it "was not verifiable from the build environment (no live Codespace to inspect)".

3.3 GitHub Codespace — the inference tier

Runs the real inference service: build_space_app() from app/space_app.py, served by deploy/codespace/serve.py on $PORT, in CPU mode. It honours the four-endpoint contract (/v1/health, /v1/capabilities, /v1/analyze, /v1/assets), imports cheaply without torch, reuses app/serving.py as the composition root, and degrades rather than crashes on absent artifacts (docs/DEPLOYMENT_ARCHITECTURE.md §3.3).

The serve entrypoint is deliberately tiny (deploy/codespace/serve.py):

from app.space_app import build_space_app
import uvicorn

app = build_space_app()

if __name__ == "__main__":
    port = int(os.environ.get("PORT", "8000"))
    uvicorn.run(app, host="0.0.0.0", port=port)

The composition root (app/serving.py::build_serving_controller) resolves device from SATQUERY_DEVICE. It wires three capabilities through the registry's builders= seam without editing configs/base.yaml:

Capability	Wired artifact	Why the seam
change	artifacts/change/levir_change_v001/head.pt	change.checkpoint_path is unset; adding it to config would move Config.hash
change_vqa	artifacts/change_vqa/run/head.pt + the same STANet	closes the F2 train/serve skew (training and serving must share one detector)
optical_sar	CROMA (resolved from the pinned identity) + artifacts/optical_sar/fusion_head_production_v001/head.pt	croma.checkpoint_path is unset, so the encoder was unreachable by default

The seam is a call-site argument (core/registry.py's builders= override), not config, so Config.hash stays 78f1e3700da15aa1 (app/serving.py docstring). Degrade, do not crash: absent artifacts yield available: false with a reason; corrupt artifacts raise ModelLoadError. The two are deliberately not conflated.

3.3.1 The Codespace launcher and its survivability design

.devcontainer/devcontainer.json sets postStartCommand: bash deploy/codespace/launch.sh, so the inference server and the tunnel agent start on every Codespace start. launch.sh is more defensive than it looks, and the reasons are recorded in the script:

• Preflight (refuse to start half-configured). It checks import yaml, pydantic, fastapi, uvicorn, httpx and import app.space_app, exiting non-zero with a diagnostic if either fails. httpx is checked explicitly because tunnel_agent.py imports it directly and it was previously absent from requirements.txt, so the agent "died instantly and the supervised restart loop hid the error in a log file".
• Stale-serve detection. A stamp file (/tmp/satquery-serve.stamp) records rev=<HEAD> asset_enabled=<…> asset_dir=<…>. If the running server's stamp disagrees with the current checkout and environment, the serve process is restarted, because "a stale serve process is worse than no process: it answers /v1/health and /v1/capabilities from OLD code".
• The tunnel agent is supervised and immortal. setsid alone is not enough in Codespaces — the lifecycle shell that runs postStartCommand can still reap the process group, which "showed up in production as 'the agent announced once, then vanished'". The launcher therefore uses setsid + nohup + </dev/null around a supervising while true wrapper that re-launches the agent if it exits, so the agent is "effectively immortal for the life of the Codespace".
• Post-launch verification. After a 4-second wait it checks the agent process is alive and that the log contains a successful announce (announced to hub), because "backgrounding with all output discarded means a crashing agent is completely invisible".

launch.sh refers to bash deploy/codespace/doctor.sh in two diagnostics. doctor.sh is not present in the monorepo working copy; it lives in the deployed SatQuery-Inference checkout. UNKNOWN — not established from the available evidence whether it is present in that repository, as the private repository was not readable for this documentation pass.

3.3.2 The warm_cache.py pre-warm

deploy/codespace/post_create.sh (postCreateCommand) installs the lean CPU requirements and runs python deploy/codespace/warm_cache.py, which pre-downloads the pinned HF models into the HF cache so the first /v1/analyze is fast. It is idempotent, reports per-model OK/SKIPPED/FAILED status, and never aborts on a single miss (deploy/codespace/README.md).

3.4 Hugging Face — the model tier

Holds the six trained artifacts and the model card. It is not the runtime inference host; the Codespace resolves the pinned backbones from the Hub at run time. At reconnaissance the public repository thundercode/SatQuery contained two files only (.gitattributes and a 25-byte README.md) — the model card / weights publication is a separate workstream from this deployment.

---

4. Why the transport is an outbound tunnel

The inference host is a Codespace in a private repository. A forwarded port for a private repo returns 302, so an inbound-forwarding design cannot work. Instead:

• the Codespace runs deploy/codespace/tunnel_agent.py (from SatQuery-Inference);
• the agent dials out to POST /tunnel/agent and long-polls;
• work is executed against http://127.0.0.1:8000 locally.

This inverts the usual direction: the inference host needs no inbound firewall hole, and GitHub's port-forwarding relay, port visibility and the repository's visibility are all irrelevant. It also means the transport is only alive while the agent is polling.

Measured: GET /api/health reported tunnel.agent_connected: true with a non-zero completed counter, and POST /api/infer {} returned 422 invalid_request with the response header x-satquery-transport: tunnel (release/CURRENT_RELEASE_STATE.md §1; docs/FINAL_DELIVERY_TODO.md §6 E-03).

When the Codespace is stopped, the agent stops polling → GET /api/health reports tunnel.agent_connected: false and POST /api/infer parks until SATQUERY_TUNNEL_TIMEOUT_S (150 s), then returns tunnel_offline (503, recoverable: true) (docs/DEPLOYMENT_TOPOLOGY.md §2).

The forwarded-port path is dead, not merely unused: it returns 302 for the private repo. The GitHub-API wake path (POST /user/codespaces/{name}/start) still exists in deploy/render/codespaces.py, but the tunnel design relies on the agent reconnecting on Codespace start via the devcontainer postStartCommand.

---

5. The full live health payload (VERIFIED, probed)

{
  "status": "ok",
  "service": "satquery-orchestrator",
  "tunnel": {
    "agent_connected": true,
    "agent_id": "codespaces-fd1038",
    "pending": 0,
    "completed": 97
  },
  "config": {
    "codespace_name": "potential-space-trout-r4ppw969w45j2pvvw\n",
    "codespace_port": 8000,
    "transport_mode": "auto",
    "tunnel_timeout_s": 150.0,
    "wake_timeout_s": 120.0,
    "upstream_timeout_s": 90.0,
    "device": "cpu",
    "has_github_token": true
  }
}

Source: release/CURRENT_RELEASE_STATE.md §1. The completed counter is a live, monotonically increasing value — later captures recorded completed: 314 (docs/FINAL_DELIVERY_TODO.md §1.4) and completed: 338 (docs/FINAL_DELIVERY_REPORT.md §3). The count is a runtime fact, not a fixed constant; do not quote it as a stable figure.

Two things in this payload are load-bearing:

1. codespace_name still carries a trailing \n. This is B-02, cosmetic and OPEN; the wake path strips it (_codespace_name() calls .strip()), so only the /api/health reporting payload shows the raw value (§8.2).
2. transport_mode is auto. This is the root shape of B-07 (§8.1).

5.1 The live capability contract (VERIFIED, probed)

GET /api/capabilities → schema_version 1.0, six entries, all available: true (release/CURRENT_RELEASE_STATE.md §1):

task	requires_pair	max_assets	notes
vqa	false	1	SmolVLM weights fetched from the HF Hub on first use
caption	false	1	SmolVLM weights fetched from the HF Hub on first use
grounding	false	1	RemoteCLIP encoder fetched from the HF Hub on first use
change	true	2	—
change_vqa	true	2	—
optical_sar	true	2	modalities: ["optical","sar"]

The capability table is served by the single adapter app/deployment.py, derived from the registry's spec table plus filesystem presence. The adapter emits only contract vocabulary (loaded/absent/unavailable/not_requested/evicted) and — precisely because it must not load a model to answer a metadata request — it never emits loaded or evicted (docs/DEPLOYMENT_ARCHITECTURE.md §3.3.1). available: false always carries a non-null reason.

---

6. Environment variables

6.1 Render (gateway) — measured live values

Variable	Value (live)	Purpose
CODESPACE_NAME	potential-space-trout-r4ppw969w45j2pvvw	which Codespace to wake
CODESPACE_PORT	8000	the inference port
SATQUERY_ALLOWED_ORIGINS	https://satquery.pages.dev	CORS allowlist (never *)
SATQUERY_DEVICE	cpu	device preference
SATQUERY_TRANSPORT	auto	tunnel first, then forward (§8.1)
SATQUERY_TUNNEL_TIMEOUT_S	150	how long to wait on the tunnel
SATQUERY_WAKE_TIMEOUT_S	120	how long to wait for a cold start
SATQUERY_UPSTREAM_TIMEOUT_S	90	gateway → upstream budget
GITHUB_TOKEN	present	Codespace control (existence only; never recorded here)
PORT	platform-supplied	Render's own listen port

Source: docs/DEPLOYMENT_TOPOLOGY.md (measured 2026-09-25 live note) and release/CURRENT_RELEASE_STATE.md §1.

render.yaml in the monorepo declares the blueprint's env vars: PORT, SATQUERY_ALLOWED_ORIGINS, GITHUB_TOKEN, CODESPACE_NAME (sync: false — set in the dashboard), plus CODESPACE_PORT: "8000", SATQUERY_DEVICE: "cpu", SATQUERY_WAKE_TIMEOUT_S: "120", SATQUERY_UPSTREAM_TIMEOUT_S: "90" (render.yaml). The blueprint does not declare SATQUERY_TRANSPORT or SATQUERY_TUNNEL_TIMEOUT_S — those are set in the live dashboard and are part of the deployed SatQuery-Backend revision, not the monorepo's stale blueprint.

Measured absence. There is no SATQUERY_UPSTREAM_URL and no HF_TOKEN in the live config. The transport is the outbound tunnel, not a forwarded port. This contradicts the older docs/DEPLOYMENT_TOPOLOGY.md §3.2 table and docs/DEPLOYMENT_ARCHITECTURE.md §4, which predate the tunnel design (release/CURRENT_RELEASE_STATE.md §1 note; docs/FINAL_DELIVERY_TODO.md §1.7 item 4).

6.2 Codespace (inference)

Variable	Purpose
PORT	platform-assigned; must be read (historical blocker #2, §9)
SATQUERY_DEVICE	cpu | cuda | mps | null; read without importing torch
SATQUERY_MAX_FILE_BYTES	per-file cap, shared with Render so the two layers cannot disagree
SATQUERY_ASSET_ENABLED / SATQUERY_ASSET_DIR	both required for /v1/assets; fails closed (503) otherwise
SATQUERY_ASSET_MAX_FILES / SATQUERY_ASSET_TTL_S	optional handle capacity / lifetime
SATQUERY_HUB_URL	the Render orchestrator the tunnel agent dials out to; default https://<backend-host>

Sources: docs/DEPLOYMENT_TOPOLOGY.md §3.3; deploy/codespace/launch.sh.

.devcontainer/devcontainer.json sets containerEnv: SATQUERY_DEVICE=cpu, PORT=8000, SATQUERY_ASSET_ENABLED=1, SATQUERY_ASSET_DIR=/tmp/satquery-assets. The launcher re-exports the asset variables on every start because containerEnv is only applied when the container is created — "setting it there alone would leave an already-running Codespace unconfigured until a rebuild. This script runs on every start and is therefore the effective source of truth" (deploy/codespace/launch.sh).

Asset-store defaults, from docs/DEPLOYMENT_ARCHITECTURE.md §4: handle capacity 32, TTL 900 s. A malformed or non-positive value falls back to the default rather than becoming a zero TTL. The store refuses rather than evicts a live handle, so a full store answers 503 (ambiguous with an unconfigured store — see docs/DEPLOYMENT_ARCHITECTURE.md §5.1).

6.3 Config-loader environment overrides

Two registry values can be overridden from the environment without editing the YAML (core/config.py):

Variable	Effect
SATQUERY_PRECISION	overrides training.precision
SATQUERY_TORCH_COMPILE	overrides deployment.torch_compile ("true" → True)

Both are still validated by the loader. Setting SATQUERY_TORCH_COMPILE=true fails startup, because finding C-8 forbids torch.compile on the (historical) ZeroGPU target — and the loader hard-fails on deployment.torch_compile is True (core/config.py; configs/deploy.yaml header; docs/STEP8_FINAL_CONFORMANCE_AUDIT.md §6.1). This is an example of the loader refusing an incoherent configuration rather than silently accepting it.

6.4 The environment-variable vocabulary, and where it moved

The active design kept the env-var vocabulary and moved only the host names. The superseded design used SATQUERY_SPACE_URL; the tunnel design uses the Codespace name/port pair plus SATQUERY_HUB_URL on the inference side (docs/DEPLOYMENT_TOPOLOGY.md §5). The older SATQUERY_UPSTREAM_URL name is not set live.

---

7. Deploy mechanics per tier

Tier	Mechanism
Frontend → Cloudflare Pages	scripts/stage_pages.mjs builds a Pages bundle; npx wrangler pages deploy
Backend → Render	render.yaml blueprint; main.py exposes the ASGI object app
Inference → Codespace	deploy/codespace/serve.py binds build_space_app() to $PORT; .devcontainer/ forwards 8000 and starts the tunnel agent via postStartCommand
Repository writes	the GitHub Git Data API — blob → tree → commit → PATCH ref

7.1 Frontend deploy (measured)

cd C:/Users/anish/satquery-ai

node scripts/stage_pages.mjs \
  --out=.deploy/dist-final \
  --include=_headers \
  --include=robots.txt \
  --include=assets/img/eo/provenance.json \
  --include=assets/img/eo/CREDITS.md

npx wrangler pages deploy "C:/Users/anish/satquery-ai/.deploy/dist-final" --project-name <name>

(docs/DEPLOYMENT_DECISION.md §7.) _headers and robots.txt must be force-included because no page references them; provenance.json and CREDITS.md likewise. The measured staging result is quoted in §3.1.

_headers cannot un-cache an asset — it concatenates. See §10 for the Cloudflare trap and the cache-busting consequence (the EO pair was renamed to new -720 URLs rather than given a new rule).

7.2 Backend deploy

render.yaml is the blueprint: runtime: python, plan: free, buildCommand: pip install -r deploy/render/requirements.txt, startCommand: uvicorn deploy.render.main:app --host 0.0.0.0 --port $PORT, healthCheckPath: /api/health. The deployed SatQuery-Backend repository is the source of truth; the monorepo's render.yaml is a snapshot of the tunnel-less revision.

7.3 Inference deploy

deploy/codespace/serve.py binds build_space_app() to $PORT. .devcontainer/devcontainer.json forwards 8000 as public and runs launch.sh on every start. The launcher's stale-serve detection (§3.3.1) means a code or environment change causes the running server to be restarted rather than left answering from old code.

7.4 Repository writes: the GitHub Git Data API

Every deployed file is uploaded as a blob whose sha256 is computed locally and verified against the uploaded blob, then assembled into a tree, committed, and the branch ref patched (blob → tree → commit → PATCH ref). This means:

• each file is content-verified rather than trusted;
• deletions are expressed explicitly as sha: null tree entries;
• the deploy is idempotent — re-running it with identical content produces no change.

Measured: 9 deployed files were re-read from the API and found sha256 byte-identical to the local copies, with the deployed HEAD re-read independently (verify_deployed_head.py; release/CURRENT_RELEASE_STATE.md §5).

7.5 The timeout relationship (do not invert)

gateway upstream timeout  <  agent.timeout_seconds  ≤  the inference host's own request budget

Both bounds are derived from the frozen config, not chosen (docs/BACKEND_DEPLOYMENT_RUNBOOK.md §4.2):

Quantity	Value	Source
agent.timeout_seconds	120 s	configs/base.yaml
gpu_duration_vqa	20 s	configs/deploy.yaml
gpu_duration_grounding	45 s	configs/deploy.yaml
gpu_duration_change	30 s	configs/deploy.yaml
gpu_duration_optical_sar	45 s	configs/deploy.yaml
Largest single gpu_duration_*	45 s	derived

So the upstream timeout belongs above 45 s (the longest a single call may run) and below 120 s (the host's own request budget). GatewayConfig.__post_init__ refuses a timeout ≤ 45 s and ≥ 120 s (docs/PHASE19_FINAL_HARDENING.md §3.2). The live value is SATQUERY_UPSTREAM_TIMEOUT_S = 90.

7.6 The error contract

Upstream failures are wrapped in the v1 envelope {"error": {"code", "message", "detail", "recoverable"}} (deploy/render/main.py docstring; docs/DEPLOYMENT_ARCHITECTURE.md §2.3):

Condition	Status	recoverable	Code
Connection error to the Codespace	502	true	upstream_unreachable
Wake times out	504	true	wake_timeout
Non-JSON upstream body	502	true	schema_validation_error
Missing GITHUB_TOKEN / CODESPACE_NAME	500	false	orchestrator_config_error
Malformed request JSON	400	false	invalid_request

The code is passed through unchanged — the gateway must not remap the taxonomy in core/errors.py, because a gateway that remapped codes would make the frontend's error handling unpredictable (docs/DEPLOYMENT_ARCHITECTURE.md §2.3). A non-JSON upstream error is never relayed verbatim (defect G-4, docs/STEP7_BACKEND_CHAIN_REPORT.md §13).

---

8. Cold start (documented, not hidden)

Render's free tier sleeps when idle, and the Codespace may be stopped. Before a request can be served, Render must start the Codespace (if stopped) and wait for the tunnel agent to reconnect. The frontend shows "Waking inference engine…" during this.

Property	Value
Cold start	tens of seconds
Tunnel wait before falling through	SATQUERY_TUNNEL_TIMEOUT_S = 150 s
Wake wait	SATQUERY_WAKE_TIMEOUT_S = 120 s
Upstream budget	SATQUERY_UPSTREAM_TIMEOUT_S = 90 s
Codespace idle timeout	30 min (GitHub REST: idle_timeout_minutes=30)

Sources: docs/DEPLOYMENT_TOPOLOGY.md §2; docs/FINAL_DELIVERY_TODO.md §6 E-04.

Cold start is documented rather than papered over: an honest "this will take a while the first time" is better than a silent hang.

8.1 The transport_mode: auto fallthrough — B-07 (OPEN)

SATQUERY_TRANSPORT=auto means: try the tunnel; on timeout, fall through to the forward path.

The forward path to a private repo returns 302 quickly — but the wake step still consumes SATQUERY_WAKE_TIMEOUT_S (120 s) first. So a worst-case failed request takes roughly

150 s (tunnel timeout)  +  120 s (wake timeout on a 302)  ≈  249 s

This is the root shape of the observed transient tunnel gap, and it is why a request can appear to hang and then fail (release/CURRENT_RELEASE_STATE.md §6; docs/FINAL_DELIVERY_TODO.md §5 B-07: "in auto transport mode a tunnel timeout falls through to the forward path (SatQuery-Backend/main.py:546), which then burns wake_timeout_s=120 on a 302 → the observed 504").

A patch (fix-b07-forward-unavailable.patch) was authored and verified (git apply --check clean, py_compile clean, applies to the deployed 89d80eaddec5). It adds:

• forward_unavailable (503, terminal 302/401/403 on the forward path), and
• upstream_timeout (504, tunnel healthy but slow), and
• the codespace_name .strip() fix.

Status: B-07 is OPEN. The patch is prepared but NOT deployed. The deployed health payload still shows the trailing \n and the fallthrough remains live.

8.2 B-02 — the trailing newline (OPEN, cosmetic)

The /api/health payload reports codespace_name with a trailing \n. This is B-02, confirmed still live during the reconnaissance. It is cosmetic: the wake path is safe because _codespace_name() calls .strip() (SatQuery-Backend/main.py:123-124) and the wake path uses it (main.py:357); only the health-reporting payload (main.py:619) reads the raw env var (docs/FINAL_DELIVERY_TODO.md §4 P2-T03). Fix = change line 619 to _codespace_name(), then Render redeploys. Deferred because a live-backend redeploy before the demonstration was not judged worth the risk. Status: OPEN (cosmetic).

---

9. The five historical backend blockers

Before any backend could boot, five verified blockers had to be closed. Each was re-verified as a real blocker (docs/DEPLOYMENT_DECISION.md §8), and the current design closes them:

#	Blocker (verified)	How it is closed
1	requirements.txt declared no fastapi / uvicorn / httpx / starlette	the runtime installs the ASGI stack so build_space_app() and the gateway can import
2	no code read $PORT — a platform port would be ignored	deploy/codespace/serve.py binds to $PORT; Render reads its own
3	hand-rolled CORS raised 405 on OPTIONS, so browser preflight failed	the gateway registers OPTIONS explicitly / uses Starlette's CORSMiddleware
4	module-level app = create_app() swallowed config errors into app = None	construction errors now propagate (fail-fast) instead of leaving a dead app
5	adapter integrity unverified on load (_adapter_sha256 computed but never compared)	the load path compares the digest against an expected value, or fails startup

docs/DEPLOYMENT_TOPOLOGY.md §4 records these honestly as "closed by construction / to be verified on the first live run" at the time it was written. The live system subsequently ran and served all six tasks (docs/FINAL_DELIVERY_REPORT.md §4), which is the evidence that the blockers are closed in production.

9.1 The defects the first real run found

docs/STEP7_BACKEND_CHAIN_REPORT.md §13 records that the ASGI layer had never executed, and that running it surfaced four live defects immediately:

ID	Defect	Severity	Status
G-1	request: Request never resolved (an in-function import left Request out of __globals__), so every POST body was misread as a missing query parameter and no handler ever ran	Critical	FIXED
G-2	an unsupported force_task enum value was forwarded upstream instead of refused locally	High	FIXED
G-3	an empty HF_TOKEN produced Authorization: Bearer , which httpx rejects → a crash reported as an upstream failure	High	FIXED
G-4	a non-JSON upstream error body was relayed verbatim, breaking the error contract and leaking internal text	High	FIXED

The lesson recorded there is worth carrying: "the first hour of actually running the gateway found a critical defect that had been invisible for as long as nobody could run it". The gateway is the validation boundary; a gateway that misreads every body while the tests stay green is exactly the failure a documented blocker hides.

---

10. Platform traps (recorded so they are not rediscovered)

Trap	Detail
Cloudflare _headers CONCATENATE	Two matching rules are merged, not overridden. A specific rule nested under a broad /assets/img/* rule yields max-age=604800, …, max-age=0, must-revalidate — and Chromium takes the FIRST max-age. The file's own "later rules override" comment is false. Measured live 2026-09-25 (docs/FINAL_DELIVERY_TODO.md §1.7 item 9).
Cloudflare 308 redirect	X.html → /X. Reference the extensionless path.
Forwarded port returns 302	for a private repo — this is why the tunnel exists (§4).
Tunnel agent must start on boot	via the devcontainer postStartCommand, or a restarted Codespace comes up with agent_connected: false.
Never retry /api/infer at the gateway	a retry consumes inference twice (§3.2).
deploy/ is stale and untracked	not the deployed source (§1.1).
Edge-cache serves deleted files	The old EO pair URLs still answer 200 from Cloudflare's edge cache (CF-Cache-Status: HIT, Age: 1076) although the files are deleted; a cache-busted request returns 404. Nothing references them (LIVE_VALIDATION_POSTFIX.md, "Known residuals").
containerEnv applies only at container creation	hence launch.sh re-exports the asset variables on every start (§6.2).
setsid alone does not survive postStartCommand	the lifecycle shell can reap the process group; the launcher uses setsid + nohup + </dev/null plus a supervising wrapper (§3.3.1).

---

11. The superseded design, and what did NOT change

The earlier design ran inference on an HF Space with ZeroGPU (5 GPU-min/day, @spaces.GPU decoration) behind a Railway gateway (docs/DEPLOYMENT_ARCHITECTURE.md §1; docs/DEPLOYMENT_TOPOLOGY.md §5). The active design changes three things:

1. CPU-first instead of ZeroGPU. No code change was required — device_preference honours SATQUERY_DEVICE and defaults to CPU, every specialist defaults to device="cpu", and all placement is .to(device) (never .cuda()). ZeroGPU's GPU-minute quota and @spaces.GPU decoration are no longer on the critical path (docs/DEPLOYMENT_DECISION.md §5).
2. A real, always-buildable inference environment. A Codespace gives a reproducible container without a GPU quota or a Space's cold-start constraint. The wake flow replaces ZeroGPU lazy loading as the cold-start story.
3. No GPU quota to protect at the gateway. Rate/size limits remain, but as fairness controls rather than quota protection.

What did NOT change:

Unchanged	Detail
the four-endpoint contract	/v1/health, /v1/capabilities, /v1/analyze, /v1/assets
the gateway responsibility table	§3.2 above
the env-var vocabulary	only host names moved (SATQUERY_SPACE_URL → the Codespace name/port pair)
the config freeze	78f1e3700da15aa1

configs/deploy.yaml still describes the old HF-Space/ZeroGPU target (platform: huggingface-spaces, sdk: gradio, zerogpu: true, the gpu_duration_* values). It is frozen paperwork: no Gradio runtime exists in code, and editing it would move Config.hash. It is left undisturbed (docs/DEPLOYMENT_DECISION.md §4; configs/deploy.yaml header).

11.1 The ZeroGPU/Gradio target, in full, and why it is inert

configs/deploy.yaml carries registry: false, which makes its non-membership in the config registry machine-readable; core/config.py reads exactly one file (configs/base.yaml) through a single yaml.safe_load and never globs configs/*.yaml. scripts/validate_deploy_config.py asserts the manifest is inert and that its deployment: block is byte-for-byte equal to configs/base.yaml's. There is no Gradio runtime: no import gradio, no gr.Blocks, no gr.Interface, and the one ZeroGPU code path — spaces.GPU(duration=…) inside decorate_gpu() — "is never applied to any route"; routes use plain @api.get/@api.post (docs/DEPLOYMENT_DECISION.md §4). The real entrypoint is FastAPI: build_space_app().

The spaces package is not installed, so the @spaces.GPU(duration=…) path has never executed; decorate_gpu() returns an identity decorator when spaces is absent, which is the correct CPU behaviour (docs/PHASE19_FINAL_HARDENING.md §5.2). Status: REJECTED (superseded; frozen paperwork only).

11.2 The stale hf/ docs

hf/SETUP.md and hf/README.md assert that the project "does not own any model weights … ships no weights, no binaries, and no large artifacts" and that "this environment has no Hugging Face credentials". Both were false at release time — six trained artifacts exist (release/CURRENT_RELEASE_STATE.md §6). This is a documentation defect, not a deployment defect; it is recorded in LIMITATIONS.md §6.

11.3 The stale monorepo README.md

The monorepo README.md calls the frontend "hermetic — no backend calls" (it calls /api/* on Render), puts Render/Codespace as "in progress" (both deployed), describes a 4-endpoint /v1/* contract (the live gateway contract is /api/*), omits the tunnel, and points at the stale untracked deploy/ as the deployment source (release/CURRENT_RELEASE_STATE.md §6).

11.4 Why the three deploy repos are private

SatQuery-Frontend, SatQuery-Backend and SatQuery-Inference are private; their links 404 for an outside audience. This is BY DESIGN — the public umbrella SatQuery-AI is the audience-facing home. The only public code repository at reconnaissance was Anish-lab-blip/SatQuery-AI (docs/FINAL_DELIVERY_TODO.md §4 P9-T01).

---

12. Deployment preconditions (open items)

1. Cloudflare Pages project name / domain — needed for the deploy command and the Sitemap: line in robots.txt. sitemap.xml was deliberately not created because it needs a canonical production domain (docs/DEPLOYMENT_DECISION.md §6).
2. Artifacts present, or capabilities honestly ship available: false — the service degrades rather than crashes on absent artifacts.
3. HF_TOKEN on Render if the HF proxy path is used — not used in the live config (measured absence, §6.1).
4. Codespace .devcontainer/ forwarding :8000 and starting the tunnel agent — both present in the live deployment.
5. The five blockers in §9 closed and verified on the first live run — closed; the live system served all six tasks (docs/FINAL_DELIVERY_REPORT.md §4).
6. The B-07 patch deployed — NOT done; B-07 remains OPEN (§8.1).

---

13. What is NOT RUN / OPEN / BLOCKED for this topic

Item	State
B-07 tunnel fallthrough (auto mode ≈ 249 s worst case)	OPEN — patch prepared, NOT deployed
B-02 codespace_name trailing \n	OPEN (cosmetic)
A deployed system-level load test	NOT RUN
Multi-region / HA deployment	NOT RUN
A production database / persistence	BY DESIGN (absent)
Auth / user accounts / queue	BY DESIGN (absent; plan §73/§74)
The ZeroGPU/Gradio deployment target	REJECTED (superseded; frozen paperwork only)
doctor.sh present in the deployed Inference repo	UNKNOWN — not established from the available evidence
A latency benchmark of the deployed path	NOT RUN (see PERFORMANCE.md, LIMITATIONS.md §3)

---

14. Where the evidence lives

Evidence	Location
Live health + capabilities probe	release/CURRENT_RELEASE_STATE.md §1
Active topology contract	docs/DEPLOYMENT_TOPOLOGY.md
Backend contract (gateway responsibilities, env vocabulary, failure modes)	docs/DEPLOYMENT_ARCHITECTURE.md §1.1, §2, §3.3, §4, §5
Deployment decision + the frozen HF manifest	docs/DEPLOYMENT_DECISION.md
Operator runbook (local verification, timeout window, rollback)	docs/BACKEND_DEPLOYMENT_RUNBOOK.md
Deployment packaging / open items	docs/PHASE18_DEPLOYMENT_PACKAGING.md
Final hardening + ASGI defects	docs/PHASE19_FINAL_HARDENING.md, docs/STEP7_BACKEND_CHAIN_REPORT.md
The undeployed B-07 patch	session scratch: fix-b07-forward-unavailable.patch
Deployed-artifact integrity check	verify_deployed_head.py (session scratch)
Live validation (3 passes, 24 runs)	.workbuddy-ai/scratch/live_validation/
Delivery status + blocker register	docs/FINAL_DELIVERY_TODO.md, docs/FINAL_DELIVERY_REPORT.md