runpodai09/poker_platform

QwenPoker Platform V2

A contract-first training and serving platform for poker AI agents backed by Qwen 3.5-4B. The platform shares a single observation, action, prompt and artifact contract across Phase 0 (OCR data preparation), Phase 1 (supervised fine-tuning), Phase 2 (RL self-play) and production serving.

Project Structure

poker_platform/
├── apps/                  # Deployable applications
│   └── new_poker_game/    # Production poker game (backend + frontend + Docker)
├── configs/               # Locked training and environment contracts (JSON)
├── contracts/             # Pinned tokenizer and chat-template snapshots
│   └── qwen35_tokenizer/
├── data/                  # Training data
│   └── raw/               # Raw OCR source files for Phase 0 replay
├── docs/                  # Design documents and progress tracking
├── runtime/               # External runtime dependencies
│   └── ocr_parser_v2/     # OCR parser and config required by Phase 0
├── scripts/               # Production entry-point scripts for every phase
├── src/                   # Core library
│   └── qwenpoker_platform/  # Shared platform package (62 modules)
├── tests/                 # Unit and integration tests (238 tests)
├── model_registry.json    # Lane catalog and promotion history
├── pyproject.toml         # Build system and dependency specification
├── README.md              # This file
└── RELEASE_GATES.md       # Binding release gate conditions

src/qwenpoker_platform/ — Core Library Modules

Module	Purpose
models.py	PokerObservationV2, PokerDecisionV2 and shared Pydantic contracts
enums.py	PolicyAction, Street, ConcreteAction, PokerPosition enumerations
engine.py	Poker engine interface, action resolution and legal-action canonicalization
prompt.py	Prompt rendering (poker-prompt-v2) for SFT, RL and serving
prompt_cache.py	Tokenized prompt and candidate sequence caching layer
sampler.py	Trie-based structured action sampler (constrains LLM to legal JSON)
torch_sampler.py	GPU-accelerated tensor version of the structured sampler
actor_critic.py	StructuredActorCritic — policy + value head with batched inference
inference.py	StructuredPolicyInference — serving-optimized single-observation inference
codec.py	JSON schema generation, canonical encoding and action parsing
ocr_replay.py	Full hand replay engine for OCR source validation
production_adapter.py	Adapts live game engine state to PokerObservationV2
dataset.py	TrainingExampleV2, fingerprinting, provenance and split definitions
split_planner.py	Group-stratified train/validation/test/challenge split planning
lineage.py	Data lineage manifest and reconciliation equations
manifest.py	Deployment manifest, SHA256 verification and artifact digests
handoff.py	Phase 0 → Phase 1 handoff with quarantine tracking
analytics.py	Phase 0 distribution analytics (street, action, source, quality)
credit.py	GAE, session-balanced weighting and credit assignment config
phase2_rollout.py	LightweightTransition, StructuredRolloutBuffer, PPO loss
phase2_trainer.py	PPO optimizer step with optional adapter-based KL reference
phase2_orchestrator.py	Full Phase 2 training loop (rollout → update → checkpoint → promote)
phase2_session_runner.py	Single self-play session rollout collection
phase2_protocol.py	Phase 2 hyperparameters, session planning and OCR replay scheduling
self_play.py	Opponent league, seat assignment and focal learner rotation
session.py	Multi-hand session state machine (bankroll, wipeout, horizon)
bankroll.py	Bankroll prior sampling from observed OCR stack distributions
curriculum.py	Gradual player introduction (2→3→4→5 player curriculum)
ema.py	Float32 exponential moving average with content-addressable hashing
lora_audit.py	Qwen 3.5 LoRA target inventory and trainable parameter audit
qwen35_training.py	Model loading with LoRA injection and 4-bit quantization
semantic_sft.py	Semantic categorical SFT objective (replaces token-level JSON loss)
sft_trainer.py	HuggingFace Trainer subclass with semantic policy loss
serving.py	Serving response contract with deterministic legal fallback
environment.py	Training environment contract and runtime audit
source_integrity.py	Archive-directory equivalence and tar traversal protection
policy_objective.py	Weighted action NLL and reference KL divergence
rl_objective.py	RL-specific loss utilities
rl_state.py	Curriculum, budget and RL state tracking contracts
evaluation.py	Phase 1 evaluation metrics (accuracy, macro-F1, calibration)
metrics.py	Action rate, terminal PNL, rank and session metric contracts
phase2_checkpoint.py	Resumable checkpoint with full RNG and optimizer state
phase2_gate.py	Paired seat-swapped promotion gate
phase2_promotion.py	EMA promotion with statistical confidence requirements

scripts/ — Entry Points

Script	Purpose
replay_phase0_ocr.py	Production and smoke-mode OCR replay pipeline
preflight_phase1.py	Phase 1 preflight (environment, data, contract verification)
train_phase1.py	Phase 1 SFT training with semantic objective
evaluate_phase1.py	Phase 1 evaluation on validation / release-test / challenge sets
audit_phase1_adapter.py	Adapter save/load and merge/unmerge parity audit
build_phase2_protocol.py	Phase 2 protocol generation from measured baseline
preflight_phase2.py	Phase 2 preflight (lineage, engine parity, league, adapter chain)
train_phase2.py	Phase 2 RL self-play training

configs/ — Locked Configurations

File	Purpose
phase1_environment_a40_v1.json	A40 environment contract (Python, CUDA, packages, VRAM)
phase1_qwen35_4b.json	Phase 1 LoRA hyperparameters and training config
phase1_selection_v1.json	Composite checkpoint selection criteria
phase2_environment_a100_v1.json	A100 environment contract
phase2_qwen35_4b.json	Phase 2 RL hyperparameters

Model Identity

Base model: Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. The 248 LoRA targets across the Qwen 3.5 language backbone (120 linear-attention, 32 full-attention, 96 MLP projections) are audited before training begins and after the adapter is attached. Vision, convolution, embedding, LM head and MTP modules are excluded.

Installation

python -m pip install -e ".[train]"

For test dependencies:

python -m pip install -e ".[test]"

Quick Start

Phase 0 — OCR Data Preparation

Smoke test (integration check only, not valid Phase 1 input):

python scripts/replay_phase0_ocr.py \
  --mode smoke \
  --max-files-per-source 10 \
  --pseudonym-key-file /workspace/secrets/phase0_hmac.key \
  --output-directory artifacts/phase0_smoke

Production run (processes all legacy and new OCR sources):

python scripts/replay_phase0_ocr.py \
  --mode production \
  --pseudonym-key-file /workspace/secrets/phase0_hmac.key \
  --code-revision 0123456789abcdef0123456789abcdef01234567 \
  --container-digest sha256:0123456789abcdef... \
  --output-directory artifacts/phase0_production

Phase 1 — Supervised Fine-Tuning

Preflight verification:

python scripts/preflight_phase1.py \
  --data-directory artifacts/phase0_production \
  --report artifacts/phase1_preflight.json

Semantic benchmark (20 optimizer steps):

python scripts/train_phase1.py \
  --data-directory artifacts/phase0_production \
  --output-directory outputs/phase1_benchmark \
  --benchmark-steps 20

Full Phase 1 training:

python scripts/train_phase1.py \
  --data-directory artifacts/phase0_production \
  --output-directory outputs/phase1

Adapter lifecycle audit:

python scripts/audit_phase1_adapter.py \
  --adapter outputs/phase1/final \
  --data-directory artifacts/phase0_production \
  --report outputs/phase1/adapter_lifecycle.json

Validation evaluation:

python scripts/evaluate_phase1.py \
  --adapter outputs/phase1/final \
  --data-directory artifacts/phase0_production \
  --output-directory outputs/phase1/evaluation

Release-test and challenge evaluation (final candidate only):

python scripts/evaluate_phase1.py \
  --adapter outputs/phase1/final \
  --data-directory artifacts/phase0_production \
  --output-directory outputs/phase1/release_evaluation \
  --include-release

Phase 2 — RL Self-Play

Protocol generation (requires measured legacy focal-learner decision count):

python scripts/build_phase2_protocol.py \
  --phase0-directory artifacts/phase0_production \
  --baseline-learner-decisions <measured_legacy_focal_learner_decisions> \
  --planned-sessions <planned_session_count> \
  --output artifacts/phase2/protocol.json

Phase 2 preflight:

python scripts/preflight_phase2.py \
  --phase0-directory artifacts/phase0_production \
  --sft-adapter outputs/phase1/final \
  --adapter-lifecycle-report outputs/phase1/adapter_lifecycle.json \
  --phase1-selection-gate outputs/phase1/evaluation/phase1_selection_gate.json \
  --engine-parity-suite artifacts/phase2/engine_parity_suite.json \
  --protocol artifacts/phase2/protocol.json \
  --opponent-league artifacts/phase2/opponent_league.json \
  --session-horizon-prior artifacts/phase0_production/session_horizon_prior_train.json \
  --report artifacts/phase2/preflight.json

Tests

python -m pytest tests/ -q

238 tests, zero failures, zero skips. All tests run on CPU without a GPU.

Release Gates

See RELEASE_GATES.md for binding gate conditions across all phases.