vmishra20/orion-9m-decision-model

ScratchLM Decision Model — Pre-Medical Checkpoint

Overview

This repository contains a custom PyTorch decoder/decision model trained from scratch and adapted on a multi-dataset decision task.

Model summary

Property	Value
Total parameters	9.365M (9,365,249)
Trainable parameters in the currently loaded model	9.365M (9,365,249)
Framework	PyTorch
Model family	Custom decoder-only / decision model
Pretraining corpus	HuggingFaceFW/fineweb, sample-10BT configuration, streamed subset
Pretraining tokens	179,150 tokens.

Architecture

• Model family: custom decoder-only language/decision model implemented in PyTorch.
• Pretraining: initialized and pretrained from scratch; no pretrained backbone is implied by this model card.
• Attention: MLA-style latent/compressed query and key-value projections.
• Position encoding: RoPE applied to the positional query/key channels.
• Depth: recurrent/looped Transformer blocks, reusing shared blocks across multiple passes.
• Feed-forward routing: sparse Mixture-of-Experts (MoE), with top-k expert routing.
• Embedding/output: tied token embedding and output weights, if enabled in the exported architecture.
• Task adaptation: supervised fine-tuning on a multi-dataset decision/classification corpus.

Configuration

Setting	Value
Hidden size	256
Attention heads	4
Maximum sequence length	512
Query latent rank	128
KV latent rank	64
Non-RoPE Q/K head dimension	32
RoPE Q/K head dimension	16
Value head dimension	32
Unique/shared Transformer blocks	4
Recurrent passes per block stack	2
Number of MoE experts	4
Experts selected per token	2
MoE hidden dimension	384
Vocabulary size	16000

Implementation note: The architecture is custom PyTorch code. A weights file plus this README does not automatically make the model compatible with Hugging Face AutoModel or pipeline. The exported model code, configuration, and tokenizer must be tested together in a clean environment before claiming standard Transformers compatibility.

Pretraining

• Dataset: HuggingFaceFW/fineweb
• Configuration: sample-10BT
• Method: streaming ingestion of a bounded subset; the entire FineWeb corpus was not downloaded.
• Actual token count: 179,150 tokens.

Supervised fine-tuning (SFT)

Orion’s first version was fine-tuned on a combined dataset of nine general science, commonsense, and reasoning benchmarks, comprising 80,658 normalized multiple-choice examples: OpenBookQA (5,457), ARC-Easy (2,821), ARC-Challenge (1,418), SciQ (12,000), CommonsenseQA (10,962), PIQA (12,000), Social-IQA (12,000), WinoGrande (12,000), and HellaSwag (12,000). The data was split into 72,592 training examples and 8,066 validation examples (90/10), with a maximum sequence length of 160 tokens. The subsequent Reinforcement Learning from Comparative Demonstrations (RLCD) stage used preference pairs derived from these same questions, with the correct answer as the preferred response and a sampled incorrect answer as the rejected response. The generated preference dataset contained 72,592 training pairs and 8,066 validation pairs.

Intended use

Research and experimentation with compact language/decision models, custom attention mechanisms, recurrent Transformer blocks, and sparse MoE.

Limitations

• A small model may have limited factual knowledge, instruction following, calibration, and multi-step reasoning.
• Next-token pretraining and decision-task SFT do not establish clinical validity or medical safety.
• Benchmark scores must be reported with the dataset split, exact evaluation script, number of examples, and checkpoint hash.
• This is not a medical device and must not be used as a substitute for professional medical advice.
• Dataset licenses and attribution requirements should be checked before redistributing model weights or training artifacts.

Files

Check the repository contents for the model weights, configuration, tokenizer artifacts, inference code, and dependency list. Validate that a fresh environment can load the exported weights and reproduce a sample forward pass before uploading this repository publicly.