QEV2-12B-INT8
Inference-only release Β· merged LoRA Β· 8bit LLM.int8
License: QueryPie AI Research License v1.0
Personal non-commercial research and experimentation, and non-commercial academic research and evaluation are permitted. Commercial use requires prior written permission from QueryPie or a separate commercial license agreement. For commercial use, contact QueryPie. Business operations, product development and commercially motivated proofs of concept are not covered by the free research permission. Third-party foundation components retain their original licenses.
English Β· νκ΅μ΄ Β· ζ₯ζ¬θͺ
Overview
QEV2-12B-INT8 is a text-only typed-decision model built on Gemma 4, with merged rank-16 LoRA updates and a pointer head. It supports choice, noul and ordinal score questions in one forward pass and returns option probabilities instead of generated text.
Base: google/gemma-4-12B-it Β· revision: 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7 Β· format: INT8 (LLM.int8)
Training data
The project training corpus was constructed from publicly available datasets and internal corporate DLP data using data augmentation. Raw training, calibration, validation and test data are not distributed. This checkpoint includes DLP-specific adaptation using internal DLP data in addition to the public-data general stage.
Self-contained package
This inference-only release includes a complete LoRA-merged backbone, FP32 pointer head, tokenizer, calibration temperature and custom Transformers inference code. It needs no separate adapter or external base download. Training scripts, adapter files, optimizer state, conversion tools and datasets are not distributed. Publishing full weights does not technically prevent further fine-tuning; applicable license terms still govern use.
βInference-onlyβ describes the supplied package, not a technical training lock. LICENSE permits modification and fine-tuning for the permitted personal non-commercial or non-commercial academic research purposes. Commercial or business use, including evaluation and fine-tuning for such purposes, requires prior written permission or a separate commercial agreement.
INT8 means 8-bit bitsandbytes LLM.int8, not 4-bit or integer-only execution. The merged backbone Linear weights use threshold 6.0; pointer head, embeddings, normalization and outlier paths retain floating-point precision. CUDA is required by the validated INT8 runtime. Memory savings do not guarantee faster responses.
Quick start
cd QEV2-12B-INT8
pip install -r requirements.txt
python example.py --device cuda:0
from transformers import AutoModel
model = AutoModel.from_pretrained(".", trust_remote_code=True,
device="cuda:0", local_files_only=True)
result = model.decide("This is a public product announcement.", {
"dlp": {"type": "noul", "instructions": "Does this contain personal contact information?",
"criteria": {"false": "No", "true": "Yes"}}
})
print(result)
Run these commands after downloading the complete repository snapshot. To load a published Hub release directly, replace "." with its repository ID and omit local_files_only=True. Use AutoModel and decide(); standard Transformers pipelines, Hub widgets and hosted Inference Providers are not validated for this custom interface.
Historical source evaluation
The benchmark table below describes the historical UNMERGED BF16 source checkpoint, not new measurements of this merged or INT8 release. No full benchmark was rerun. Merge and quantization can change probabilities or selections; validation.json reports only a 9-question EN/KO/JA synthetic check and serialized reload parity.
| Metric | Value |
|---|---|
| MMLU-ProX-Lite Β· 16,384 questions | 43.69% |
| Belebele Β· 24,622 questions | 68.89% |
| Typed-Decision Β· 540 questions / 270 states | 91.85% |
| Equal-panel mean accuracy | 68.14% |
| DLP accuracy Β· 3,167 documents | 98.29% |
| DLP recall Β· threshold 0.50 | 99.69% |
| DLP F2 Β· threshold 0.50 | 98.98% |
DLP FP/FN: 50/4. Stored checkpoint temperature: 1.627309. Historical/reused benchmarks, not a new independent confirmation. Public benchmark pretraining contamination cannot be ruled out. KO/JA translations were not native-speaker reviewed. Probabilities are not correctness guarantees.
Evaluation data sources and QEV-specific preparation
These are internal comparative results on source datasets converted into a common typed-decision input format for QEV/KEV/JEV, not official scores on the complete unmodified source benchmarks. Evaluation provenance is separate from the training-corpus description. Neither source nor processed datasets are included.
- MMLU-ProX-Lite: li-lab/MMLU-ProX-Lite, MIT. The question becomes state and options become choice criteria. cot_content is excluded; blank/N/A options are removed, with original question/answer and option order retained and validated. The fixed successful length-compatible intersection contains 16,384 questions from 17,052 source questions across 29 common languages.
- Belebele: facebook/belebele, CC BY-SA 4.0. flores_passage becomes state, the question becomes instructions, four answers become choice criteria, and correct_answer_num maps to the label. The same fixed comparison procedure retains 24,622 questions from 26,100 source questions across 29 common languages. Processed data is for internal evaluation; neither source nor transformed passages are redistributed.
- Private Typed-Decision: QueryPie project private-general-v1. Policy/rules, numbers/dates, routing, missing information, ordering/grades and everyday reasoning are generated from templates with code/rule-solver labels. English templates are localized to KO/JA with labels, option keys and numbers preserved; automatic integrity checks are used without native-speaker review. Entire states with identical q1/q2 type/instructions/criteria are removed, leaving 270 states / 540 questions, 180 per language, from 450 states / 900 questions.
- DLP: QueryPie internal multilingual held-out test corpus, 3,167 documents (EN 1,065 / JA 1,038 / KO 1,064). Original text, IDs and labels are preserved; text becomes state and a noul question asks for sensitive/non-sensitive classification. Any ground-truth finding makes a document positive; finding spans/counts/types are not evaluated. There are 1,281 positives and 1,886 negatives, with threshold 0.50. Document text and labels remain private. MMLU/Belebele use a fixed item-ID intersection admitted under state 384 / per-question row 1,024 / packed 2,048, without truncation. The general composite is the equal-weight mean of question-level accuracies for the three general panels. Recorded upstream revisions and processed-file SHA-256 values are provided in EVALUATION_DATA_SOURCES.json. Source dataset licenses are not replaced by the QueryPie model license.
Limits and intended use
Default state validation is len(encoded_state) + 1 β€ 2,048, so the user state itself may contain up to 2,047 tokenizer tokens. Each complete question row, including state, instructions, options and structural/template tokens, must fit within 2,688 tokens. The sum of encoded row lengths must not exceed 10,752 tokens; repeated state tokens count once per row. Up to 4 named questions and 2β10 options per question are accepted. Oversized inputs are rejected without truncation. Token counts depend on the model tokenizer. Historical quality was measured under state 384 / row 1,024 / packed 2,048; this does not establish quality at the higher serving limits.
The interface supports general typed decisions and document-level DLP classification. For DLP, it predicts whether sensitive content is present; it does not extract finding spans, counts or types. choice uses named options; noul requires false/true criteria and returns a true probability; score uses an ordered list and returns a zero-based selected index and probabilities over indices, not a continuous scalar score. confidence is the maximum option probability, not a measured correctness rate. It uses no text generation. Reliability across arbitrary domains is not established.
The validated hardware is NVIDIA H200/CUDA. CPU and other runtimes are not performance-validated. Custom code requires trust_remote_code=True.
The provided INT8 loader requires a CUDA device and BF16 floating-point paths.
License and attribution
The QueryPie-trained modifications (including merged LoRA), pointer head and QueryPie-specific code use QueryPie AI Research License v1.0. Original Qwen/Gemma foundation weights and tokenizer assets retain Apache-2.0; independently granted upstream rights are not restricted. See THIRD_PARTY_LICENSES.md, LICENSES/Apache-2.0.txt, and NOTICE for scope and attribution. Training data is not relicensed or distributed. Personal non-commercial research and experimentation, and non-commercial academic research and evaluation are permitted. Commercial use requires prior written permission from QueryPie or a separate commercial license agreement. For commercial use, contact QueryPie. Business operations, product development and commercially motivated proofs of concept are not covered by the free research permission. Third-party foundation components retain their original licenses. Generic save_pretrained() is disabled to prevent dropping internal assets. Copy or download the complete repository. validation.json and MANIFEST.sha256 describe package checks.
Release validation
| Comparison | Max probability difference | Changed selections / 9 |
|---|---|---|
| Original BF16 β merged BF16 | 0.00600743 | 0 |
| Merged BF16 β INT8 | 0.00457034 | 0 |
| Original BF16 β INT8 | 0.00832382 | 0 |
| Saved model β reloaded model | 0.00000000 | 0 |
- Downloads last month
- 33