mattxchou/webpage-markdown-api

Webpage to Markdown API (v1)

A single-endpoint micro-API for AI app developers: send a URL, get back clean article markdown. Sold on RapidAPI.

POST /extract { "url": "https://..." } → { "url", "title", "markdown", "word_count" }

Extraction uses trafilatura (best-in-class readability extraction, no headless browser in v1).

---

Run locally

cd ~/workspace/money/rapidapi/markdown-api
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# ADMIN_KEY protects the /admin/* key-management endpoints. Set any strong value.
export ADMIN_KEY="$(openssl rand -hex 32)"
uvicorn app:app --host 0.0.0.0 --port 8000

Health check: curl http://localhost:8000/health

Generate / revoke API keys

All admin calls need the X-Admin-Key header (the ADMIN_KEY env value).

# Create a key on a tier (free | pro | ultra). The full key is shown ONCE.
curl -s -X POST http://localhost:8000/admin/keys \
  -H "Content-Type: application/json" \
  -H "X-Admin-Key: $ADMIN_KEY" \
  -d '{"tier":"pro"}'

# List keys with this month's usage vs quota
curl -s http://localhost:8000/admin/keys -H "X-Admin-Key: $ADMIN_KEY"

# Revoke by key prefix (first 12 chars, shown in the list)
curl -s -X DELETE http://localhost:8000/admin/keys/md_ab12cd34ef \
  -H "X-Admin-Key: $ADMIN_KEY"

Keys are stored as SHA-256 hashes in keys.json (the raw key is never persisted).

Call the API

curl -s -X POST http://localhost:8000/extract \
  -H "Content-Type: application/json" \
  -H "X-API-Key: md_YOUR_KEY" \
  -d '{"url":"https://docs.python.org/3/library/json.html"}'

Metering

• Usage is counted per key, per calendar month (UTC) in usage.json.
• Only successful extractions (HTTP 200) count toward quota. Failed fetches, 422s, and 4xx auth errors are free.
• When a key hits its monthly cap, /extract returns 429 {"detail": {"error": "monthly_quota_exceeded", ...}}. Quota resets on the 1st.
• Per-minute rate limits (in-memory sliding window, per key): free 10/min · pro 60/min · ultra 300/min → 429 rate_limit_exceeded.
• On RapidAPI, map each RapidAPI plan to a key tier: when a user subscribes, generate a key on the matching tier and send it to them; when they cancel, revoke the key.

Pricing tiers (set these on the RapidAPI listing)

RapidAPI plan	Price	Key tier	Quota	Rate limit
Basic (Free)	$0	free	100 req/mo	10/min
Pro	$9/mo	pro	10,000 req/mo	60/min
Ultra	$29/mo	ultra	100,000 req/mo	300/min

Suggested RapidAPI listing fields: category Data / Text Analysis (or "Artificial Intelligence"), endpoint POST /extract, auth via X-API-Key header — import openapi.yaml when creating the API on RapidAPI.

Configuration (env vars)

Var	Default	Purpose
ADMIN_KEY	(required for /admin/)*	Protects key-management endpoints. Generate with openssl rand -hex 32.
SSRF_GUARD	1 (on)	Rejects URLs resolving to private/internal IPs. Turn off (SSRF_GUARD=0) only in proxied dev environments where DNS is intercepted (all hosts resolve to the proxy range). Leave on in production (Render).
PORT	8000	Listen port (Render injects its own).

Option A — Render (free tier, recommended for the listing)

1. Push this directory to a Git repo.
2. Render dashboard → New → Blueprint → select the repo (render.yaml is picked up automatically).
3. Render generates ADMIN_KEY for you — copy it from the dashboard's Environment tab.
4. Your API is live at https://markdown-extract-api.onrender.com (HTTPS ✓, works for RapidAPI).
5. Point the servers: URL in openapi.yaml at the Render URL before importing the spec into RapidAPI.

Notes: Render free web services spin down after inactivity (first request after idle takes ~30–60s). If that hurts conversions, the $7/mo Starter plan keeps it always-on.

Option B — cloudflared quick tunnel (testing only)

Gives you a public HTTPS URL in seconds, no account needed. Good for testing the RapidAPI spec import before committing to Render:

# install once: https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/downloads/
uvicorn app:app --host 127.0.0.1 --port 8000 &
cloudflared tunnel --url http://localhost:8000
# → https://<random>.trycloudflare.com  (HTTPS ✓)

Caveat: the trycloudflare.com URL changes on every restart and the tunnel dies when your machine sleeps — do not use this for the live RapidAPI listing, only for testing.

Error reference

Code	Meaning
400 invalid_url	URL isn't a fully-qualified http(s) URL
400 unresolvable_host / blocked_host	DNS failed, or URL resolves to a private/internal IP (SSRF guard)
401 missing_api_key / invalid_api_key	Bad or missing X-API-Key
422 extraction_failed	Graceful failure: no readable content — page is JS-rendered, paywalled, or login-walled. v1 has no headless browser
429 rate_limit_exceeded	Per-minute limit hit
429 monthly_quota_exceeded	Monthly plan cap hit
502 fetch_failed	Target site unreachable / blocked the request
504 fetch_timeout	Page took longer than 30s to fetch+extract

v2 ideas (stubbed, not built)

• Headless-browser fallback (Playwright) for JS-heavy pages instead of the 422.
• Redis-backed rate limiting + usage (multi-worker safe; current in-memory limiter is single-process).
• Webhook on quota thresholds (80%/100%) per key.
• Per-key custom quotas overriding tier defaults.
• Response caching for repeated URLs (hash of URL → markdown, short TTL).