sizzlebop/onca-3.0-12b-GGUF

ONCA 3.0 (12B) GGUF

This repository provides GGUF quantizations for Joesh1/onca-3.0-12b.

ONCA 3.0 is a specialized 11.95B-parameter oncology research model derived from google/gemma-4-12B. It uses the text-only decoder within Gemma 4's unified architecture and is fine-tuned to generate compact, structured JSON outputs across six oncology tasks: pathological TNM classification (T, N, M staging), clinical trial triage, trial criterion reasoning, and CIViC molecular-evidence classification.

All GGUF files were converted from the original sharded safetensors weights using llama.cpp at native F16 precision, followed by quantization into standard k-quant variants.

ONCA 3.0 is a research model, not a medical device, clinical decision system, or staging authority. It is intended strictly for research evaluation and structured information extraction workflows.

---

Available Files and Quantizations

File	Quant Type	Size	Description / Recommendation
onca-3.0-12b-F16.gguf	F16	22.20 GB	Full precision base conversion. Highest fidelity reference weights.
onca-3.0-12b-Q8_0.gguf	Q8_0	11.80 GB	Near-lossless 8-bit quantization. Recommended for rigorous research evaluation.
onca-3.0-12b-Q6_K.gguf	Q6_K	9.11 GB	Excellent retention of structured generation accuracy with reduced VRAM demands.
onca-3.0-12b-Q5_K_M.gguf	Q5_K_M	7.96 GB	Strong balance of memory efficiency and schema adherence.
onca-3.0-12b-Q4_K_M.gguf	Q4_K_M	6.87 GB	Fast 4-bit quant. Recommended default for local workstation inference and desktop GPUs.
onca-3.0-12b-Q3_K_M.gguf	Q3_K_M	5.67 GB	Lower memory footprint when running on constrained devices.
onca-3.0-12b-Q2_K.gguf	Q2_K	4.50 GB	Maximum compression for testing on memory-limited hardware.

---

Technical Specifications

• Foundation Architecture: Gemma 4 Unified (Gemma4UnifiedForConditionalGeneration)
• Modality: Text only (multimodal vision/audio heads are not active in this checkpoint)
• Parameter Count: 11.95B parameters
• Context Window: Up to 262,144 tokens (evaluation sequences capped at 4,096 tokens)
• Base Precision: float16
• License: Apache-2.0

---

Supported Oncology Tasks and Schemas

Every query should specify its task in the prompt. The model outputs compact JSON without conversational filler or hidden chain-of-thought tokens:

1. Pathological T Staging (tnm_t):

   {"task":"tnm_t","label":"T2"}
   

2. Pathological N Staging (tnm_n):

   {"task":"tnm_n","label":"N1"}
   

3. Pathological M Staging (tnm_m):

   {"task":"tnm_m","label":"M0"}
   

4. Trial-Space Triage (trial_space):

   {"task":"trial_space","decision":"CONSIDER"}
   

5. Trial-Criterion Reasoning (trial_criterion):

   {"task":"trial_criterion","criterion_type":"inclusion","native_decision":"included","canonical_decision":"SATISFIED","evidence_sentence_ids":[2,4]}
   

6. CIViC Molecular Evidence (civic_evidence):

   {"task":"civic_evidence","evidence_type":"PREDICTIVE","evidence_level":"A","evidence_direction":"SUPPORTS","clinical_significance":"SENSITIVITY_RESPONSE"}
   

---

Prompt Format

This model uses Gemma 4 conversational turn markers:

<bos><start_of_turn>user
You are an oncology research assistant that outputs only valid JSON matching the requested schema.

Task: tnm_t
Cancer type: Pancreatic ductal adenocarcinoma
Pathology excerpt: Tumor measures 3.2 cm in greatest dimension, limited to the pancreas.<end_of_turn>
<start_of_turn>model
{"task":"tnm_t","label":"T2"}<end_of_turn>

---

Quickstart Guide

Option 1: llama.cpp CLI

Run local inference directly with llama-cli:

llama-cli -m onca-3.0-12b-Q4_K_M.gguf \
  -p "<bos><start_of_turn>user\nYou are an oncology research model. Return only JSON.\n\nTask: tnm_t\nCancer type: Lung adenocarcinoma\nReport: Invasive adenocarcinoma measuring 2.4 cm.<end_of_turn>\n<start_of_turn>model\n" \
  -n 128 \
  --temp 0.0 \
  -ngl 99

Option 2: llama.cpp Server

Launch an OpenAI-compatible HTTP server:

llama-server -m onca-3.0-12b-Q4_K_M.gguf \
  --port 8080 \
  -c 8192 \
  -ngl 99

Query the endpoint:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "onca-3.0-12b",
    "messages": [
      {"role": "user", "content": "Task: trial_space\nPatient: 62yo male with metastatic CRC.\nTrial space: First-line metastatic colorectal cancer immunotherapy."}
    ],
    "temperature": 0.0
  }'

Option 3: Ollama

Create a custom Ollama model using a Modelfile:

FROM ./onca-3.0-12b-Q4_K_M.gguf

TEMPLATE """<bos>{{- range .Messages }}
<start_of_turn>{{ .Role }}
{{ .Content }}<end_of_turn>
{{- end }}
<start_of_turn>model
"""

PARAMETER stop "<end_of_turn>"
PARAMETER temperature 0.0

Build and run:

ollama create onca-12b -f Modelfile
ollama run onca-12b "Task: tnm_m\nCancer type: Breast carcinoma\nReport: No evidence of distant metastasis."

Option 4: LM Studio

1. Place onca-3.0-12b-Q4_K_M.gguf inside your local LM Studio directory.
2. Select the model from the model dropdown.
3. Configure the chat preset for Gemma syntax (<start_of_turn>user / <start_of_turn>model).
4. Set temperature to 0.0 for deterministic structured JSON output.

---

Original Model and Citation

• Original Hugging Face Repository: Joesh1/onca-3.0-12b
• Base Foundation Model: google/gemma-4-12B
• Primary Research Focus: Structured clinical NLP, pathological staging extraction, and precision oncology trial screening.