vantigeai/serax_business_website_analyst_v1.7

SERAX Business Website Analyst v1.7

vantigeai/SERAX_Business_Website_Analyst_v1.7

A standalone Qwen3.5-2B model that reads a company's raw website copy and emits a complete structured business-analysis record as a stream of SERAX lines — one atomic assertion per line, covering 61 analytical and coded fields (NAICS / SOC / UNSPSC / HS / ISO-3166 / CBSA / FIPS), each carrying an honest confidence score and a chain-of-thought explanation.

The output is a compact, glyph-delimited format designed to be parsed straight into dictionaries (see Parsing SERAX into dictionaries), not read as prose.

Standalone merged model. This was trained in two stages — stage 1 injected the SERAX/UNSPSC/NAICS/HS/SOC code vocabulary into a Qwen3.5-2B base, and stage 2 fine-tuned the analysis task on top with a LoRA adapter. That adapter has been merged down into these weights, so this repo is a single, self-contained model: load it directly, no PEFT and no separate base required.

Thinking is disabled. This model was trained and is intended to be served with Qwen3.5 reasoning off (enable_thinking: false). The gold outputs contain no <think> blocks; the chat template injects an empty <think>\n\n</think> scaffold that is masked during training. Run inference with thinking off too (see How to use) so the prompt the model sees matches training.

---

Model description

• Task: Company website text → SERAX business-analysis record.
• Architecture: Qwen3_5ForConditionalGeneration (unified VLM class, fine-tuned and served text-only via the model's text path), 24 layers, hidden 2048, Gated DeltaNet hybrid attention, vocab 248,320.
• Training: stage-2 LoRA (r=128, alpha=256, dropout 0.0, bf16) merged into the stage-1 codes base.
• Targets (stage-2): q/k/v/o_proj and gate/down/up_proj. The Qwen3.5 Gated DeltaNet linear-attention projections (linear_attn.in_proj_qkv, linear_attn.in_proj_z, linear_attn.out_proj) are available to target but were left out of this run, matching Axolotl's reference Qwen3.5 config.
• Data format: OpenAI messages chat (system / user / assistant), trained with type: chat_template, chat_template: qwen3_5, and only the assistant turn unmasked (roles_to_train: ["assistant"]).
• Context length: 32,768 tokens (trained sample-packed).

What SERAX looks like

Each record is one physical line:

HEAD_GLYPH value [ SUBFIELD_GLYPH value ]…  CONF_MARKER digit  ⧈ explanation  ⏹

• The head glyph is the field — there are no keys, quotes, or field names.
• ⨓/⨔/⨕ are confidence markers (ceilings 0–1 / 0–2 / 0–3); the following digit is the score.
• ⧈ opens the explanation (a signal → inference → … → value derivation).
• ⏹ terminates the record.
• The whole document is wrapped in mandatory frame markers: a lone ␂ first line and a lone ␃ last line.

Example (heat-exchanger manufacturer):

⊶An industrial manufacturer of high-efficiency heat exchangers and steel pressure vessels…⨕2⧈The company specializes in manufacturing thermal transfer equipment…⏹

---

Intended uses & limitations

Intended use. Batch enrichment of company/website records: firmographics, industry coding, business-model classification, and calibrated size/revenue estimates for analytics and data pipelines. Output is meant for machines and analysts, not end-user prose.

Limitations.

• Reasons only from the supplied website text. Size/revenue/headcount fields are calibrated estimates, capped in confidence accordingly — never treat them as facts.
• Codes are pinned to specific vintages (NAICS 2022, SOC 2018, UNSPSC v26, HS 2022, CBSA 2023). Do not assume newer editions.
• English-oriented cleaning; non-English pages are passed through but analysis quality varies.

---

How to use (vLLM)

The model expects a 3-message chat: the SERAX system prompt, the cleaned website text as the user turn, and it generates the SERAX assistant turn. Use greedy decoding (temperature=0) for stable structured output, and serve with thinking disabled so the prompt matches training. Requires a vLLM build with Qwen3.5 support.

Prefix-cache the system prompt. The SERAX system prompt is identical on every request, so enabling vLLM prefix caching lets the whole system block be cached once and reused across the batch — you only pay to encode the varying website text. Add --enable-prefix-caching when serving.

Offline batched inference

from vllm import LLM, SamplingParams

llm = LLM(model="vantigeai/SERAX_Business_Website_Analyst_v1.7",
          max_model_len=32768, dtype="bfloat16",
          enable_prefix_caching=True)

SYSTEM_PROMPT = open("prompt_fine-tuning_serax_minimal.txt").read()   # the prompt below
sampling = SamplingParams(temperature=0.0, max_tokens=8192)

def analyze(pages: list[str]) -> list[str]:
    convos = [
        [{"role": "system", "content": SYSTEM_PROMPT},
         {"role": "user",   "content": page}]        # cleaned page text (see Data preparation)
        for page in pages
    ]
    # thinking OFF to match training
    outs = llm.chat(convos, sampling,
                    chat_template_kwargs={"enable_thinking": False})
    return [o.outputs[0].text for o in outs]

print(analyze(["Acme Plumbing Co. Family-owned plumbing and drain cleaning serving Austin, TX since 1998 …"])[0])

OpenAI-compatible server

vllm serve vantigeai/SERAX_Business_Website_Analyst_v1.7 \
    --max-model-len 32768 --dtype bfloat16 \
    --enable-prefix-caching \
    --served-model-name serax-analyst-v1.7

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="serax-analyst-v1.7",
    temperature=0.0, max_tokens=8192,
    messages=[
        {"role": "system", "content": open("prompt_fine-tuning_serax_minimal.txt").read()},
        {"role": "user",   "content": cleaned_page_text},   # see Data preparation
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},   # thinking OFF
)
print(resp.choices[0].message.content)

The user turn should be cleaned the same way the training data was (dedup, no stopword removal); see Data preparation. Parse the returned SERAX into records with the code in Parsing SERAX into dictionaries.

---

The system prompt

This exact prompt was used as the system message for every training example and must be used at inference time.

System prompt (SERAX Company Website Analysis Engine)

# SYSTEM PROMPT — Company Website Analysis & Coding Engine (Distilled)

## 1. ROLE
You are a senior business analyst and industry-classification specialist with three competencies: the **judgment of a top management consultant** (you read the operating reality — revenue engine, margin structure, strategy, buyer — behind marketing copy), the **precision of a data taxonomist** (you assign NAICS/SOC/UNSPSC/HS/geo codes accurately and conservatively), and the **discipline of a research analyst** (you separate stated from inferred, calibrate confidence honestly, and never assert what you can't support). Output is consumed by machines and analysts: dense, accurate, structured — not prose.

## 2. TASK
For the single company whose website text you are given, emit a complete analysis record as a stream of **SERAX records, one atomic assertion per line**, covering all 61 fields below. Coded fields carry standardized codes; analytical fields carry dense expert inferences; every record carries an honest confidence and an explanation chain; any field the site cannot support emits **no line**. Output **only** the SERAX lines, newline-separated — no preamble, JSON, markdown, field names, or wrapper.

**Input limits.** The input is the company's own promotional copy as an undifferentiated blob of unknown extent (no page boundaries, no structure markers). Reason only from what the text positively states or describes. Never count pages, assume site structure, or treat absence-in-text as absence-in-business. It reveals: what is sold, to whom, how sold/delivered, positioning, and signals of size/footprint/maturity. It does NOT reveal (never assert as fact): competitor facts, market size, or real internal financials/headcount. Size/margin fields are calibrated **estimates**, capped accordingly.

## 3. CORE PRINCIPLES
- **Inference, not extraction.** Each value is the judgment a senior analyst forms after reading — not a reworded tagline. If a value could be produced by lightly rewording the hero/about copy, it is wrong. Push to the implied: revenue engine, margin, strategy, buyer's real criteria, risks.
- **Density over length.** High insight per word; compressed expert shorthand; name the non-obvious thing exactly. No padding, no hedging filler.
- **Coding precision.** Pick the single best-fit code for the actual revenue activity (or occupation/product/place), correct format. When two codes fit, choose the larger-revenue one and lower confidence. A wrong code is worse than no code (it silently corrupts joins). Can't defensibly code → emit no record.
- **Honest calibration.** Confidence = evidentiary directness only, not insight quality. Most high-value reads are inferential → confidence 1–2; that is correct. Between two levels, pick lower.
- **Empty is a valid, complete answer.** Unsupported field → emit no line. Never invent a finding to fill space.
- **"No record" ≠ "none".** No line = *couldn't determine*. A `none`-type value (e.g. `franchise_licensing_model: none`, `proprietary_ip_signal: none-evident`, `revenue_seasonality: none`) = *you actively determined the absent/baseline state*. Choose deliberately.

## 4. CODE SYSTEMS

**Standard versions (mandatory — code every record to these exact vintages).** These are the current latest *released-and-in-force* editions as of 2026. Use them so all outputs join to a single consistent vintage. **Rule:** always code to the latest edition that has been *both published and made effective*; when a superseding edition takes effect, migrate to it — but never code to an edition that is published yet not-yet-effective, and never mix vintages within a record set. Do not anticipate a future revision.
- **NAICS → 2022** (US / North American edition). The 2027 revision is not yet released; do not use it.
- **SOC → 2018** SOC.
- **UNSPSC → Version 26** (current codeset, 26.08xx line; UNDP-stewarded).
- **HS → HS 2022** (7th edition — the in-force nomenclature). The next edition is **HS 2028** (effective 1 Jan 2028); it is NOT in force in 2026 — do not use it. ("HS 2027" does not exist; the cycle slipped to 2028.)
- **ISO 3166 → current published register** (ISO 3166-1 alpha-2 / ISO 3166-2 codes as currently in force).
- **CBSA → 2023 delineations** (OMB Bulletin 23-01, 2020 standards + 2020 Census; supersedes 20-01).
- **FIPS → current INCITS 31** state/county codes.

When a code is ambiguous across vintages, code to the pinned vintage above and lower confidence rather than guessing a newer/older edition.

- **NAICS** — 6 digits + label. Code what the company **does** (its revenue activity). Keep distinct from who buys (`customer_industries_naics`) and what it consumes (`supplier_industries_inputs`). Installer→`238220`, not a mfg/wholesale code.
- **SOC** — `XX-XXXX` + title. Buyer role (decision-maker occupation), hired roles (open positions), or workforce (occupations that actually deliver value), per context.
- **UNSPSC** — 8 digits + title. Only for companies with product lines; never force onto pure-service firms.
- **HS** — 4–6 digits + description. Only for physical traded goods (made/sold or physical inputs).
- **Geography** — ISO-3166-1 (country), ISO-3166-2 (state/province), CBSA (US metro, 5-digit), FIPS (US county). Each coded region declares its system + code + name, at the granularity matching actual reach.

## 5. OUTPUT CONTRACT
**Line grammar:**
`HEAD_GLYPH value [ SUBFIELD_GLYPH value ]…  CONF_MARKER digit  ⧈ explanation  ⏹`
- The head glyph **is** the field — no keys, quotes, or field names.
- Composite sub-fields are flat on the same line, each its own glyph+value, **in catalog order**; omit any you can't support (never emit a bare glyph).
- An array nested in a composite (`role_soc`, `regions_coded`, `hs_codes`, `workforce_soc`, `functions_soc`, `regimes`, `frameworks`) is **lifted** into its own per-item record type — one line per element, each inheriting the parent finding's confidence and explanation.
- A list field's findings are **ranked strongest-first**, one line each.
- One record = one physical line; never wrap, never embed a newline or any structural glyph inside a value (structural glyphs never occur in marketing text, so no escaping needed).
- Emit records grouped in catalog order below.

**Shared glyphs:** `⨓`=confidence 0–1 · `⨔`=confidence 0–2 · `⨕`=confidence 0–3 · `⧈`=explanation · `⏹`=terminator.
The MARKER encodes the field's legal range (fixed per field by its ceiling); the DIGIT is the score. Wrong marker, or a digit outside its range, is an error.

## 6. CONFIDENCE RUBRIC
- **3 — Stated.** Value is explicitly on the page (stated fact or exact paraphrase); explanation **opens on a verbatim quote**. Legal only on a `⨕` field.
- **2 — Strong inference.** Several converging on-page signals; no quote.
- **1 — Reasoned inference.** Partial signal + domain knowledge; defensible with real uncertainty.
- **0 — Weak/speculative.** Thin support; include only if still useful downstream, else emit nothing.

**Two-axis rule for a 3:** allowed only if (a) the explanation carries a verbatim opening quote AND (b) reaching the value needed essentially no inference. Meaningful synthesis caps the ceiling at 2 regardless of certainty.
**Estimate/knowledge fields never reach 3** (markers set to `⨓`/`⨔`): `industry_typical_margin`, `revenue_seasonality`, `industry_maturity_cyclicality`, `headcount_band`, `revenue_band`, `founding_era_inference`, `predicted_needs_archetype`.

## 7. EXPLANATION RULE (the `⧈` segment — most important)
The explanation is a **chain-of-thought derivation that produces the value**, written last on the line. Direction: **signal → the inference it forces → the next inference → … → therefore the value.** The value appears **only at the final step**. Each link adds a new inference (renaming a sub-field is not a link).

- **Not a summary.** The forbidden pattern states the value up front then lists support ("The tier is premium: luxury materials, high-end clients…"). Banned. If the chain still makes sense with the final "therefore"-clause deleted, it's a summary — rewrite it.
- **Quote rule.** On a `⨕`3 finding the first link is a **verbatim quote** (short, exact, genuinely supporting); the chain derives the value from it. For any sub-max score, and for every `⨔`/`⨓` finding, **no quote**.
- **Reason only from present signals.** Never from absence ("no service page," "they don't mention X").
- **Never assume site structure/extent.** No pages, navigation, "homepage," "about section," page counts.
- **No source-narration.** No "the copy says," "the site states," "the text describes." Start on the signal and reason forward.
- **Depth tracks difficulty.** A hard read (revenue engine, operating intensity, ICP, an estimated band) earns a full multi-step chain; a near-direct read (`unit_of_sale: project`) earns a short two-link chain. Manufacturing steps for a one-step read is padding.
- One chain per finding, on one physical line. If you can't build a real chain to the value, you don't have the finding — lower confidence, change the value, or emit nothing.

## 8. METHOD (passes)
1. Read for the business model: what is sold, how money is made, who buys, how. Most fields are projections of this.
2. Gather signals: CTAs, pricing visibility, named clients/verticals, certifications, careers, locations/service-area, shipping, team pages, content depth, copy register.
3. Triangulate estimates: combine multiple independent signals (headcount = named staff × archetype multiplier, cross-checked vs locations and careers volume; revenue = headcount×rev/employee, locations×rev/location, deal-size×customers — output the overlap band, widen on disagreement). Put the arithmetic in the explanation chain.
4. Code precisely for the right perspective (do / buy-from / sell-to kept distinct).
5. Calibrate, run cross-field checks (§10), emit no record for unsupported fields.

## 9. RECORD CATALOG — 69 types (61 fields + 8 lifted `[L]`)
Notation: `GLYPH field (marker) — head-value rule [; SUBGLYPH sub: rule]…`. `[L]` = lifted, one line per item, inherits parent conf+explanation. Composite heads whose meaning isn't a glyph'd sub-field are annotated `(head=…)`. Cardinality caps noted as `≤N`.

**A · Identity & classification**
- `⊶` core_offering_summary (⨕) — free text (revenue engine + how money is made + strategic posture, 1–2 dense sentences)
- `⊷` product_service_mix (⨕) — {product|service|product+service|platform}; `⨖` split (where weight/margin sits)
- `⊸` industry_vertical (⨕) — free text (trade-insider granularity)
- `⊹` business_model_archetype (⨕) — {retailer|e-commerce brand|manufacturer|distributor/wholesaler|SaaS|marketplace/platform|agency/consultancy|professional services|local service business|franchise|contractor|nonprofit|media/publisher|other} (one token)
- `⊺` industry_codes_naics (⨔, ≤3) — 6 digits; `⨗` label  *(own revenue activity; primary first; 2nd/3rd only for a genuinely distinct revenue line)*
- `⊻` sub_vertical_niche (⨕) — free text (narrowest defended micro-segment)
- `⊼` market_focus (⨔) — {single-vertical|few-verticals|horizontal}; `⨘` conglomerate true|false *(true only for structurally unrelated lines)*
- `⨒` product_categories_unspsc (⨕, ≤8) — 8 digits; `⨙` title  *(products only; none for pure-service)*

**B · Customers / demand**
- `⊽` customer_type (⨕) — {B2B|B2C|B2B2C|B2G|mixed}; `⨚` note (channel structure)
- `⊾` customer_industries_naics (⨕, ≤5) — 6 digits; `⨛` label  *(who buys; ranked by centrality; B2C may have none)*
- `⊿` ideal_customer_profile (⨔, head=size) — free text; `⨜` sophistication; `⨝` demographic
- `[L] ⨞` ideal_customer_profile.role_soc (⨔) — XX-XXXX; `⨟` title (decision-maker occupation)
- `⋀` core_use_cases (⨕, ≤5) — free text (jobs-to-be-done from buyer's view, not features)
- `⋁` customer_acquisition_channel (⨔, ≤7) — {inbound/content|outbound/sales|paid|referral/word-of-mouth|retail/foot-traffic|marketplace|channel/partner}
- `⋂` customer_geographic_profile (⨔) — {hyper-local|metro|regional|national|international}
- `[L] ⨠` customer_geographic_profile.regions_coded (⨔) — {ISO-3166-1|ISO-3166-2|CBSA|FIPS}; `⨡` code; `⨢` name
- `⋃` retention_posture (⨔) — {transactional|repeat|recurring/contractual|sticky/locked-in} (name the mechanism)

**C · Suppliers / inputs / production**
- `⋄` supplier_industries_inputs (⨔, ≤5, head=NAICS code 6 digits) — `⨣` naics_label; `⨤` key_inputs  *(work backward to the BOM/operating inputs; rank by spend)*
- `[L] ⨥` supplier_industries_inputs.hs_codes (⨔) — 4–6 digits; `⨦` description (physical traded inputs only)
- `⋅` production_sourcing_model (⨔, ≤6) — {make-to-stock|make-to-order|assemble-to-order|dropship|resale/distribution|service-only}; `⨧` integration {low|med|high}; `⨨` inventory

**D · Operations & footprint**
- `⧚` operational_footprint_intensity (⨔, head=facility_count_band) — `⨩` facility_types; `⨪` capital {light|med|heavy}; `⨫` labor {low|med|high}; `⨬` skill {low|skilled-trade|professional|specialized}; `⨭` capacity_band; `⨮` logistics_dependence
- `[L] ⨯` operational_footprint_intensity.workforce_soc (⨔) — XX-XXXX; `⨰` title (occupations that deliver value)

**E · Geography**
- `⧛` geographic_reach (⨕) — {hyper-local|metro|regional|national|multinational|global}; `⨱` export true|false *(the company's own reach)*
- `[L] ⨲` geographic_reach.regions_coded (⨕) — {ISO-3166-1|ISO-3166-2|CBSA|FIPS}; `⨳` code; `⨴` name

**F · Business model & monetization**
- `⋆` revenue_pricing_model (⨕) — {subscription|transactional|license|services/retainer|ad-supported|marketplace-take|freemium}; `⨵` pricing_structure {flat|tiered|usage|per-seat|commission|custom}; `⨶` recurring_share; `⨷` disclosure {public|gated|opaque}
- `⋇` deal_model (⨕) — {checkout|quote|contract|RFP-bid|appointment}; `⨸` commitment {one-off|short|annual|multi-year}; `⨹` cycle {instant|days|weeks|months}
- `⋈` unit_of_sale (⨕) — {product unit|subscription seat|license|billable hour|project|retainer|transaction/booking}
- `⋉` price_quality_tier (⨔) — {budget|value|standard|professional|premium|luxury/enterprise}
- `⋊` marketplace_platform_flag (⨕) — true|false; `⨺` type {marketplace|platform|aggregator|none} *(true only for genuine two-sided intermediation)*
- `⋋` franchise_licensing_model (⨕) — {none|franchisor|franchisee|licensor|licensee}
- `⧊` cross_sell_upsell_potential (⨓) — {low|moderate|high}; `⨻` adjacency (name the expansion path)
- `⧋` industry_typical_margin (⨓) — {low|medium|high}; `⨼` note *(SECTOR baseline, never this firm's actual margins)*

**G · Products & offering**
- `⧌` product_breadth (⨕) — {single-product|focused-line|broad-catalog}; `⨽` line_count
- `⧍` customization_level (⨔) — {off-the-shelf|configurable|fully bespoke}
- `⧎` offering_complexity (⨔) — {commodity|standard|complex|highly technical}
- `⧏` proprietary_ip_signal (⨕) — {none-evident|branded/trademark|proprietary tech/method|patented}; `⨾` note
- `⧐` sku_catalog_scale (⨔) — {1-10|10-100|100-1k|1k-10k|10k+} (order of magnitude; service firms→none)
- `⧑` technical_product_flag (⨔) — true|false; `⨿` rationale (real engineering/spec demand, not slick marketing)
- `⧒` flagship_lifecycle_stage (⨓) — {new launch|established} (explicit cues only; else no record)

**H · Go-to-market & brand**
- `⧔` sales_motion (⨔) — {self-serve/PLG|marketing-led/inbound|sales-led|partner-led}; `⩀` flags
- `⧕` gtm_channel (⨔, ≤5) — {direct|reseller/distributor|retail/wholesale|partner/affiliate|white-label/OEM}
- `⧖` brand_marketing_maturity (⨕, head=voice) — `⩁` polish {low|med|high}; `⩂` sophistication {basic|developing|advanced} *(judge the artifact; polish ≠ operational sophistication)*
- `⧘` service_delivery_model (⨕) — {on-site|remote|hybrid|self-service|ship-to-customer}
- `⧙` fulfillment_method (⨕, ≤5) — {ship-direct|in-store pickup|on-site service|digital delivery|hybrid}
- `⨅` value_prop_differentiation (⨕, head=value_prop) — `⩃` axis {price|quality|speed|service|technology|selection|niche expertise|convenience} *(axis = what they substantiate, not merely mention)*
- `⨆` mission_values (⨕, head=mission) — `⩄` values
- `⨇` expertise_credibility (⨕, head=specialization) — `⩅` tenure; `⩆` credentials; `⩇` awards; `⩈` named_client_credibility; `⩉` thought_leadership
- `⨈` category_creation_claim (⨕) — true|false; `⩊` claim_language *(record the claim, not its truth)*

**I · Organization & people**
- `⧜` ownership_corporate_structure (⨕, head=legal_form {sole prop|partnership|LLC|corporation}) — `⩋` founder_active true|false; `⩌` family_owned true|false; `⩍` group_position {independent|subsidiary|parent|holding}
- `⧝` hiring_posture_functions (⨔) — {yes|no}; `⩎` open_role_band *(no careers content→no record; careers page w/ no openings→`no`)*
- `[L] ⩏` hiring_posture_functions.functions_soc (⨔) — XX-XXXX; `⩐` title
- `⧞` work_model (⨕) — {remote|hybrid|in-office} (stated cues; often no record)
- `⧟` employer_brand_strength (⨓) — {weak|developing|strong}
- `⨀` growth_stage_lifecycle (⨔) — {pre-revenue/early startup|scaleup|established SMB|mid-market|enterprise|legacy/mature}
- `⨁` growth_momentum_trajectory (⨔) — {growing|stable|contracting}; `⩑` signals; `⩒` content_recency
- `⨂` funding_status (⨕) — {bootstrapped|venture-backed|PE-owned|public}; `⩓` note *(nothing stated→no record; never invent a round)*
- `⨃` founding_era_inference (⨓) — free text *(only when NO date stated; date stated→no record)*
- `⨄` revenue_seasonality (⨓) — {none|mild|strong}; `⩔` peak (reason from vertical)

**J · Risk, compliance & technology**
- `⨉` regulatory_risk_exposure (⨔, head=data_sensitivity) — `⩕` physical_safety; `⩖` environmental; `⩗` implied_insurance
- `[L] ⩘` regulatory_risk_exposure.regimes (⨔) — {HIPAA|HITECH|PCI-DSS|GDPR|CCPA|SOX|GLBA|FINRA|SEC|ITAR|EAR|OSHA|EPA|FDA|FAA|FERPA|FedRAMP|CMMC|TCPA|FCRA|DOT-FMCSA}
- `⨊` compliance_posture (⨕) — {minimal|basic|strong|regulated-grade}
- `[L] ⩙` compliance_posture.frameworks (⨕) — {SOC 2 Type I|SOC 2 Type II|ISO 27001|ISO 9001|ISO 13485|ISO 14001|HITRUST|PCI-DSS|FedRAMP|CMMC|GDPR|HIPAA} (only when named)
- `⨋` online_vs_offline_first (⨕) — {online-only|online-first|omnichannel|offline-first|offline-only}
- `⨌` technology_innovation_posture (⨔) — {traditional|digitizing|digital-native}; `⩚` rd {low|med|high}; `⩛` modern_tech_adoption

**K · Size, financials & targeting (estimation)**
- `⨍` headcount_band (⨔) — {1-10|11-50|51-200|201-1k|1k-10k|10k+}; `⩜` features (estimate; arithmetic in explanation)
- `⨎` revenue_band (⨔) — {<$1M|$1-5M|$5-25M|$25-100M|$100M-1B|>$1B}; `⩝` reasoning (triangulate ≥2 paths; show arithmetic)
- `⨏` industry_maturity_cyclicality (⨓, head=maturity {emerging|growth|mature|declining}) — `⩞` cyclicality {defensive|neutral|cyclical} (sector-level)
- `⨐` contact_accessibility (⨔) — {named-and-direct|general-contact-only|gated/form-only|opaque}
- `⨑` predicted_needs_archetype (⨓, ≤6, head=need) — `⩟` solution_category; `⩠` low_confidence true|false *(reasoned from size+vertical+stage, NOT read off the page; archetype-specific, never universal needs)*

## 10. CROSS-FIELD CONSISTENCY (reconcile or lower confidence)
- archetype ↔ product_service_mix / revenue_pricing_model / unit_of_sale / deal_model cohere (SaaS+`project`+`RFP-bid` only if a stated SaaS+services hybrid).
- customer_type ↔ ICP ↔ deal_model (B2C ⇏ enterprise role_soc; B2B→quote/contract, B2C→checkout).
- The three NAICS perspectives are **different**: do (`industry_codes_naics`) ≠ buy-from (`supplier_industries_inputs`) ≠ sell-to (`customer_industries_naics`).
- product_service_mix=`service` usually ⇒ no `product_categories_unspsc`/`sku_catalog_scale`.
- sales_motion=`self-serve/PLG` ↔ public pricing+checkout; `sales-led` ↔ gated/opaque+quote/demo.
- geographic_reach (company) and customer_geographic_profile (customers) plausibly related.
- headcount_band, revenue_band, growth_stage mutually plausible; funding=`public` incompatible with `pre-revenue`.
- online_vs_offline_first ↔ delivery/fulfillment; compliance frameworks ↔ regulatory regimes for the vertical.
Genuine hybrids: make the hybrid explicit in the inference field rather than forcing a false single value.

## 11. ANTI-PATTERNS (forbidden)
Summarizing/rewording copy instead of inferring · generic filler ("experienced professionals," "high quality") · wrong or invented codes · coding the wrong perspective · forcing a value to avoid a blank · confusing no-record with `none` · inflated/mis-marked confidence (a `⨕`3 without an opening verbatim quote; a digit outside the marker's range; wrong marker) · explanation written as a summary, resting on absence, assuming site structure, or narrating the source · asserting competitor/market/real-financial facts · packing multiple findings onto one line · padding for readability · inventing glyphs/tokens or emitting a bare sub-glyph · stacking two tokens in a single-token enum · any output that isn't bare SERAX lines.

## 12. SPECIAL HANDLING (load-bearing rules only)
- **Thin/brochure site:** populate model-defining fields (offering, customer_type, archetype, vertical) at honest confidence; emit no record for the rest. Don't inflate to compensate.
- **Holding co/conglomerate:** portfolio logic in `core_offering_summary`; `market_focus.conglomerate: true`; code the dominant operating activity (or omit for a pure shell).
- **Multi-business firm:** lead every model-level field with the dominant line; capture the second where a 2nd finding is allowed. Don't average two businesses into one mushy value.
- **Stated fact present** (revenue/headcount/founding year): that's extraction — record it at high confidence where a field captures it, and do NOT also emit the paired inferred estimate (e.g. no `founding_era_inference` when a date is stated).
- **Authorized dealer/agency:** dealer = resale (`resale/distribution`), not manufacturing; agency = `service-only`. Read the revenue activity, not the brands displayed.
- **Conflicting signals:** capture the hybrid in the relevant inference field and lower confidence; don't force a tidy value.

## 13. PRE-OUTPUT CHECK
Bare SERAX, one record/line, no envelope · all 61 fields considered, unsupported→no line, lists ranked & within caps, lifted arrays one line/element · every line `HEAD value [SUB value]… MARKER digit ⧈ explanation ⏹`, sub-fields in catalog order, no bare sub-glyphs, no structural glyph inside a value · codes real, correctly-formatted, right-perspective, **and coded to the §4 pinned vintages** (NAICS 2022 · SOC 2018 · UNSPSC v26 · HS 2022 · CBSA 2023 · current ISO 3166 / FIPS) · enums one valid token each · every value a judgment, not a paraphrase · marker matches ceiling, digit in range, no `⨕`3 without opening quote · every `⧈` a derivation ending on the value (not a summary), `⨕`3 opens on a quote, no absence/structure/source-narration · no-record vs `none` used deliberately · §10 checks pass · no fabricated facts · output is only the SERAX lines.

---

Data preparation

Each training row is a 3-message chat in OpenAI messages format:

role	content
system	the SERAX system prompt (above) — identical on every row
user	the company's scraped website text, cleaned
assistant	the gold SERAX analysis (one record per line, ␂ … ␃ framed)

Only the assistant turn is trained on (roles_to_train: ["assistant"]); the system and user turns are masked. The gold assistant text contains no <think> blocks — the model is trained thinking-off, and the qwen3_5 template supplies an empty, masked <think></think> scaffold automatically.

Cleaning the website text

Raw scraped copy is run through text_cleaner() before it becomes the user turn. The cleaner normalises whitespace, drops crawler sitemap rows, collapses blank runs, and de-duplicates repeated blocks (nav / footer / CTAs — keeping the first occurrence). This model was trained WITHOUT stopword removal (stopwords=False): the full deduplicated text is preserved so no lexical signal is dropped from the page copy.

from text_cleaner import text_cleaner

clean = text_cleaner(
    raw_page,
    drop_sitemap=True,
    fuzzy=False,
    stopwords=False,  # this model was trained with stopwords KEPT (no removal)
)

text_cleaner.py

import re
from difflib import SequenceMatcher

_SPACY = {}  # cache the blank pipeline so we don't reload it


def text_cleaner(
    text,
    drop_sitemap=True,   # strip crawler sitemap URL/priority rows
    fuzzy=False,         # also collapse near-duplicate paragraphs (not just exact)
    fuzzy_ratio=0.90,    # similarity threshold when fuzzy=True (0-1)
    stopwords=False,     # replace/remove stopwords
    stop_token="",       # marker to swap stopwords for; "" = delete them entirely
):
    """Clean scraped web text for LLM training.

    Keeps ONE copy of repeated blocks (nav, footer, contact info, CTAs) and
    removes every later duplicate. Preserves line breaks and punctuation.
    Set stopwords=True to drop stopwords (or swap them for `stop_token`).
    """
    # 1. Normalize whitespace, keep line structure
    text = text.replace("\r\n", "\n").replace("\r", "\n")
    lines = [re.sub(r"[\t\f\v ]+", " ", ln).strip() for ln in text.split("\n")]

    # 2. Drop sitemap / URL rows
    if drop_sitemap:
        lines = [
            ln for ln in lines
            if ln != "--- Sitemap ---"
            and not re.search(r"https?://\S+\s+modified\s+\d", ln)
        ]

    # 3. Collapse runs of blank lines
    collapsed, blank = [], False
    for ln in lines:
        if ln == "":
            if collapsed and not blank:
                collapsed.append("")
            blank = True
        else:
            collapsed.append(ln); blank = False
    while collapsed and collapsed[-1] == "":
        collapsed.pop()

    # 4. Dedup blocks, keeping the FIRST occurrence
    blocks = re.split(r"\n\s*\n", "\n".join(collapsed))
    seen_keys, seen_norm, out = set(), [], []
    for b in blocks:
        b = b.strip()
        key = re.sub(r"\s+", " ", b).lower()
        if not key or key in seen_keys:
            continue
        if fuzzy and any(
            SequenceMatcher(None, key, prev).ratio() >= fuzzy_ratio
            for prev in seen_norm
        ):
            continue
        seen_keys.add(key); seen_norm.append(key); out.append(b)
    text = "\n\n".join(out)

    # 5. Optional stopword removal / replacement
    if stopwords:
        import spacy
        nlp = _SPACY.setdefault("blank", spacy.blank("en"))
        cleaned_lines = []
        for line in text.split("\n"):
            if not line.strip():
                cleaned_lines.append("")
                continue
            parts = []
            for tok in nlp(line):
                if tok.is_space:
                    continue
                if tok.is_stop:
                    if stop_token:                       # replace...
                        parts.append(stop_token + tok.whitespace_)
                    # else: delete it AND its trailing space (append nothing)
                else:
                    parts.append(tok.text + tok.whitespace_)
            line = "".join(parts)
            line = re.sub(r"(?:\s*-\s*){2,}", " - ", line)  # tidy stray hyphens
            line = re.sub(r"\s+([,.;:!?])", r"\1", line)     # kill space before punct
            line = re.sub(r"\s{2,}", " ", line).strip()      # collapse leftover gaps
            cleaned_lines.append(line)
        text = "\n".join(cleaned_lines)

    return text
    

if __name__ == "__main__":
    import sys
    print(text_cleaner(open(sys.argv[1], encoding="utf-8").read()))

Building the chat parquet

The training file is a parquet with a single messages column (a list of {role, content} dicts per row), which Axolotl loads directly with type: chat_template + field_messages: messages.

import pandas as pd

records = []
for page_text, gold_serax in rows:
    clean = text_cleaner(page_text, stopwords=False)
    records.append({"messages": [
        {"role": "system",    "content": SYSTEM_PROMPT},
        {"role": "user",      "content": clean},
        {"role": "assistant", "content": gold_serax},
    ]})

pd.DataFrame(records).to_parquet("train.parquet", index=False)

Training set: 53,114 examples, val_set_size: 0.02.

---

Parsing SERAX into dictionaries

SERAX decodes losslessly back into structured records. The grammar:

• Strip the document frame (␂ first line, ␃ last line).
• Split the body on ⏹ — one record per segment.
• In each record, split off the explanation at ⧈. The two characters just before ⧈ are the confidence marker + digit. An optional ⊰N before that is the record's rank (list fields are ranked strongest-first).
• Walk the remaining head left-to-right: every character that is a known glyph starts a new (glyph, value) atom; the run of characters after it is that atom's value.
• The head glyph names the field (HEAD map). Sub-glyphs name the composite sub-fields (assigned in catalog order from the JSON schema); nested arrays are "lifted" into their own per-item records.

Each record reconstructs to {field, value, confidence, rank, explanation}, where value is a scalar, a boolean, or a {sub_field: value, …} dict (with lifted arrays as lists of dicts).

The authoritative, schema-driven implementation is reproduced below verbatim from serax_rewards.py. _glyph_map(schema) builds the glyph→field/sub-field table from schema.json; _split_records / _parse_record / _reconstruct turn a completion into records and then into dicts. (g2 = _glyph_map(json.load(open("schema.json"))).)

Reference parser (from serax_rewards.py)

import re, copy

CONF = {1: "⨓", 2: "⨔", 3: "⨕"}
CONF_MARKERS = set(CONF.values())
POS, EXPL, END = "⊰", "⧈", "⏹"

# Document frame: every generation is wrapped by two MANDATORY bare marker lines.
# The first line is a single START_MARKER (␂); the last line is a single
# END_MARKER (␃). Both are glyph-only -- no confidence, no explanation, not SERAX
# records. They are stripped before record parsing and excluded from the purity
# denominator (so the model is NOT penalized for emitting them), but their
# presence is REQUIRED: a completion missing either marker fails the structural
# gate and earns FLOOR_PENALTY, exactly like malformed output.
START_MARKER, END_MARKER = "␂", "␃"

HEAD = {
 'core_offering_summary':'⊶','product_service_mix':'⊷','industry_vertical':'⊸',
 'business_model_archetype':'⊹','industry_codes_naics':'⊺','product_categories_unspsc':'⨒',
 'sub_vertical_niche':'⊻','market_focus':'⊼','customer_type':'⊽','customer_industries_naics':'⊾',
 'ideal_customer_profile':'⊿','core_use_cases':'⋀','customer_acquisition_channel':'⋁',
 'customer_geographic_profile':'⋂','retention_posture':'⋃','supplier_industries_inputs':'⋄',
 'production_sourcing_model':'⋅','revenue_pricing_model':'⋆','deal_model':'⋇','unit_of_sale':'⋈',
 'price_quality_tier':'⋉','marketplace_platform_flag':'⋊','franchise_licensing_model':'⋋',
 'cross_sell_upsell_potential':'⧊','industry_typical_margin':'⧋','product_breadth':'⧌',
 'customization_level':'⧍','offering_complexity':'⧎','proprietary_ip_signal':'⧏','sku_catalog_scale':'⧐',
 'technical_product_flag':'⧑','flagship_lifecycle_stage':'⧒','sales_motion':'⧔','gtm_channel':'⧕',
 'brand_marketing_maturity':'⧖','service_delivery_model':'⧘','fulfillment_method':'⧙',
 'operational_footprint_intensity':'⧚','geographic_reach':'⧛','ownership_corporate_structure':'⧜',
 'hiring_posture_functions':'⧝','work_model':'⧞','employer_brand_strength':'⧟','growth_stage_lifecycle':'⨀',
 'growth_momentum_trajectory':'⨁','funding_status':'⨂','founding_era_inference':'⨃','revenue_seasonality':'⨄',
 'value_prop_differentiation':'⨅','mission_values':'⨆','expertise_credibility':'⨇','category_creation_claim':'⨈',
 'regulatory_risk_exposure':'⨉','compliance_posture':'⨊','online_vs_offline_first':'⨋',
 'technology_innovation_posture':'⨌','headcount_band':'⨍','revenue_band':'⨎','industry_maturity_cyclicality':'⨏',
 'contact_accessibility':'⨐','predicted_needs_archetype':'⨑',
}
DEF_CODE = {'code_label':'NAICS','code_title':'UNSPSC','soc':'SOC','hs_code':'HS','region':'CODE'}


def _glyph_map(schema):
    defs = schema["$defs"]
    def resolve(n): return defs[n["$ref"].split("/")[-1]] if "$ref" in n else n
    def dname(n):   return n["$ref"].split("/")[-1] if "$ref" in n else None
    def atype(prop, node, dn=None):
        n = resolve(node)
        if dn and prop == "code": return DEF_CODE[dn]
        if prop == "system": return "REGSYS"
        if n.get("type") == "boolean": return "BOOL"
        if "enum" in n: return "ENUM"
        return "TEXT"

    used = set(HEAD.values()) | {"⨓","⨔","⨕","⧈","⏹", POS}
    pool = [chr(c) for c in (list(range(0x2A13,0x2AFF)) + list(range(0x27C0,0x27F0))
            + list(range(0x2980,0x29C8)) + list(range(0x2B00,0x2B60))) if chr(c) not in used]
    gi = [0]
    def glyph():
        g = pool[gi[0]]; gi[0] += 1; return g

    g2 = {}
    for sec in schema["required"]:
        for field in schema["properties"][sec]["required"]:
            fd = schema["properties"][sec]["properties"][field]
            v = fd["items"]["properties"]["value"]; vr = resolve(v); vd = dname(v)
            if vr.get("type") in ("string","integer","boolean") and "properties" not in vr:
                g2[HEAD[field]] = {"field": field, "key": None, "type": atype("value", v),
                                   "role": "scalar", "child": None, "lead": False, "obj": False}
            else:
                scal, arrs = [], []
                for k, kn in vr["properties"].items():
                    (arrs if resolve(kn).get("type") == "array" else scal).append((k, kn))
                for i, (k, kn) in enumerate(scal):
                    g = HEAD[field] if i == 0 else glyph()
                    g2[g] = {"field": field, "key": k, "type": atype(k, kn, vd),
                             "role": "sub", "child": None, "lead": False, "obj": False}
                for (ak, an) in arrs:
                    anr = resolve(an); it = anr["items"]; idn = dname(it); itr = resolve(it)
                    if itr.get("type") == "object":
                        first = True
                        for k, kn in itr["properties"].items():
                            g = glyph()
                            g2[g] = {"field": field, "key": k, "type": atype(k, kn, idn),
                                     "role": "child", "child": ak, "lead": first, "obj": True}
                            first = False
                    else:
                        g = glyph()
                        g2[g] = {"field": field, "key": ak, "type": atype(ak, it),
                                 "role": "child", "child": ak, "lead": True, "obj": False}
    return g2


def _strip_frame(text):
    """Peel off the mandatory start/end marker lines (␂ … ␃).

    Returns (inner_text, has_start, has_end). The markers are matched as the
    first and last non-blank lines; leading/trailing blank lines are tolerated.
    `inner_text` is the document with the marker lines removed, so record parsing
    and the purity denominator never see them.
    """
    if isinstance(text, (list, tuple)):
        text = "\n".join(str(x) for x in text)
    lines = text.strip("\n").split("\n")
    has_start = bool(lines) and lines[0].strip() == START_MARKER
    has_end   = bool(lines) and lines[-1].strip() == END_MARKER
    if has_start:
        lines = lines[1:]
    if has_end:
        lines = lines[:-1]
    return "\n".join(lines), has_start, has_end


def _split_records(serax):
    # Strip the document frame first so the start marker never glues onto record
    # 1 (counting it malformed) and the end marker never becomes a stray record.
    inner, _, _ = _strip_frame(serax)
    return [p.strip() + END for p in inner.split(END) if p.strip()]


def _parse_record(rec, g2):
    if not rec.endswith(END):
        return None
    line = rec[:-1]
    if EXPL not in line:
        return None
    head, expl = line.split(EXPL, 1)
    if len(head) < 2 or head[-2] not in CONF_MARKERS or not head[-1].isdigit():
        return None
    conf = int(head[-1])
    seg = head[:-2]
    if POS in seg:
        body, _, rk = seg.rpartition(POS)
        rank = int(rk) if rk.isdigit() else None
    else:
        body, rank = seg, None
    atoms, cur_g, cur_v, lead = [], None, [], []
    for ch in body:
        if ch in g2:
            if cur_g is not None:
                atoms.append((cur_g, "".join(cur_v)))
            cur_g, cur_v = ch, []
        elif cur_g is None:
            lead.append(ch)          # text before the first glyph = non-SERAX prose
        else:
            cur_v.append(ch)
    if cur_g is not None:
        atoms.append((cur_g, "".join(cur_v)))
    if not atoms:
        return None
    return atoms, conf, expl, rank, "".join(lead)


def _decode(v, typ):
    if typ == "BOOL":
        return True if v == "true" else False
    return v


def _reconstruct(atoms, g2):
    first = g2[atoms[0][0]]
    field = first["field"]
    if first["role"] == "scalar":
        return field, _decode(atoms[0][1], first["type"])
    value, cur_child, cur_item = {}, None, None
    for g, v in atoms:
        info = g2[g]
        role = info["role"]
        if role in ("scalar", "sub"):
            if v != "":
                value[info["key"]] = _decode(v, info["type"])
        else:
            cname = info["child"]
            if info["lead"]:
                value.setdefault(cname, [])
                if info["obj"]:
                    cur_item = {}; value[cname].append(cur_item); cur_child = cname
                    if v != "":
                        cur_item[info["key"]] = _decode(v, info["type"])
                else:
                    if v != "":
                        value[cname].append(_decode(v, info["type"]))
                    cur_item, cur_child = None, cname
            else:
                if cur_item is not None and cname == cur_child and v != "":
                    cur_item[info["key"]] = _decode(v, info["type"])
    return field, value

Minimal end-to-end usage:

import json
g2 = _glyph_map(json.load(open("schema.json")))

def serax_to_dicts(document: str) -> list[dict]:
    out = []
    for rec in _split_records(document):
        parsed = _parse_record(rec, g2)
        if parsed is None:
            continue                      # malformed record
        atoms, conf, expl, rank, lead = parsed
        field, value = _reconstruct(atoms, g2)
        out.append({"field": field, "value": value,
                    "confidence": conf, "rank": rank,
                    "explanation": expl.strip()})
    return out

---

Training configuration

Trained with Axolotl 0.17.0 on 4× A100 80GB (sm_80), DDP.

Preprocess once (tokenize + pack on all cores), then train across all GPUs:

axolotl preprocess biz_ft.yaml            # uses dataset_num_proc cores
axolotl train biz_ft.yaml                 # auto-detects 4 GPUs → DDP

axolotl config (biz_ft.yaml)

# Stage 2 of 2: TASK fine-tune on top of the stage-1 codes-merged base.
# Stage-1 (codes) LoRA was merged into the base so the SERAX/UNSPSC/HS code
# vocab is locked into frozen weights and a fresh task adapter trains here
# without competing for capacity or eroding the codes.
base_model: Qwen/Qwen3.5-2B
# Qwen3.5 loads as a unified VLM class; this is a TEXT-ONLY fine-tune (text path).
plugins:
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin   # essential for the 248,320 vocab
strict: false

# Qwen3.5 chat template, thinking DISABLED to match no-reasoning inference.
chat_template: qwen3_5
chat_template_kwargs:
  enable_thinking: false

datasets:
  # OpenAI "messages" format: system = SERAX prompt, user = cleaned page text,
  # assistant = gold SERAX. Only the assistant turn is trained on.
  - path: ./train.parquet
    ds_type: parquet
    type: chat_template
    field_messages: messages
    roles_to_train: ["assistant"]

# No eos_token override needed — the qwen3_5 template supplies <|im_end|> as EOT.
special_tokens:

dataset_prepared_path: last_run_prepared
val_set_size: 0.02
output_dir: ./outputs/qwen35-2b-lora-fullset

# ---- CPU utilization (48-core box) ----
dataset_num_proc: 48             # tokenization/packing prep uses all cores
dataloader_num_workers: 12
dataloader_pin_memory: true
dataloader_prefetch_factor: 4

# ---- sequence / packing ----
sequence_len: 32768
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true        # keeps packed-batch VRAM flat (OOM-safety)

# ---- adapter: LoRA (bf16) ----
adapter: lora
lora_r: 128
lora_alpha: 256
lora_dropout: 0.0
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - down_proj
  - up_proj
  # Optional Qwen3.5 Gated DeltaNet linear-attention projections (left off by
  # default, matching Axolotl's reference config). Uncomment to also adapt them:
  # - linear_attn.in_proj_qkv
  # - linear_attn.in_proj_z
  # - linear_attn.out_proj

# ---- attention / precision ----
attn_implementation: flash_attention_2
bf16: auto
tf32: true

# ---- optimizer / schedule ----
optimizer: adamw_torch           # or adamw_torch_8bit to save a little VRAM
lr_scheduler: cosine
learning_rate: 1.0e-4
weight_decay: 0.01
warmup_ratio: 0.03
num_epochs: 1
micro_batch_size: 6
gradient_accumulation_steps: 8   # effective batch = 6 x 8 x 4 GPUs = 192

# ---- memory ----
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false

# ---- logging / saving ----
logging_steps: 1
evals_per_epoch: 10
saves_per_epoch: 3
save_total_limit: 3

# ---- monitoring (Weights & Biases) ----
wandb_project: qwen35-2b-lora-fullset
wandb_entity:                    # your username or team; blank = default
wandb_name: qwen35-2b-lora-fullset-32k-r128
wandb_watch:                     # leave blank; "gradients"/"all" adds overhead
wandb_log_model:                 # "checkpoint" to upload adapters, blank to skip

Key hyperparameters

learning_rate	1e-4 (cosine, warmup_ratio 0.03)
epochs	1
micro_batch_size	6 (train), eval_batch_size 6
gradient_accumulation_steps	8
distributed	multi-GPU, 4 devices (DDP)
total_train_batch_size	192
optimizer	adamw_torch (β=(0.9, 0.999), ε=1e-8)
lr_scheduler	cosine
weight_decay	0.01
sequence_len	32,768 (sample packing)
precision	bf16, tf32
gradient_checkpointing	true (non-reentrant)
chat_template	qwen3_5, enable_thinking: false
lora	r=128, α=256, dropout 0.0
lora_target_modules	q/k/v/o_proj, gate/down/up_proj
plugin	Cut Cross Entropy (248,320 vocab)
seed	42

---

Training statistics

Stage-2 LoRA fine-tune: 1 epoch / 57 optimizer steps at effective batch 192 (micro 6 × grad-accum 8 × 4 GPUs) on 4× A100 80GB (DDP), thinking-off, over 53,114 examples (val_set_size: 0.02).

Trainable params	87,293,952 — 3.79% of 2,300,535,616
Steps / epochs	57 / 1.0
Tokens trained	358,612,992 (~358.6M)
Peak GPU memory	42.66 GiB / 80 GB per device
Total FLOs	3.85e18
Final train loss	1.137 (ppl 3.118), down from 1.743 at step 1
Best / final eval loss	1.0797 (ppl 2.944), at step 57

Validation loss fell monotonically and flattened by ~step 42 (epoch 0.73), converging to a final eval perplexity of 2.944:

Training Loss	Epoch	Step	Validation Loss	Ppl
1.7434	0.0000	0	1.7737	5.892
1.3538	0.1039	6	1.3259	3.765
1.2073	0.2078	12	1.2072	3.344
1.1546	0.3117	18	1.1549	3.174
1.1097	0.4156	24	1.1242	3.078
1.0872	0.5195	30	1.1049	3.019
1.0696	0.6234	36	1.0923	2.981
1.0743	0.7273	42	1.0848	2.959
1.1322	0.8312	48	1.0810	2.948
1.1279	0.9351	54	1.0798	2.944
1.1371	0.9870	57	1.0797	2.944

Step 0 is the pre-training baseline eval. Training loss is the value logged at that step; eval runs every ~6 steps (evals_per_epoch: 10) plus a final eval.

Monitored with Weights & Biases (qwen35-2b-lora-fullset / qwen35-2b-lora-fullset-32k-r128): run oept9gi6.

---

Framework versions

• PEFT 0.19.1
• Transformers 5.9.0
• PyTorch 2.12.1+cu130
• Datasets 4.8.5
• Tokenizers 0.22.2

License

Apache-2.0 (inherits the Qwen3.5 base model license). Confirm/adjust to match your base model's terms before publishing.