sajibsrs/raging1.0-0.5B-MLX-4bit

sajibsrs/raging1.0-0.5-MLX-4bit

A lightweight 494M parameter model fine-tuned specifically for RAG ingestion pipelines. Optimized for Apple Silicon via MLX 4-bit quantization.

Purpose

raging1.0-0.5-MLX-4bit is a formatting specialist designed to sit at the front door of your RAG pipeline. It handles three core ingestion tasks at ~122 tok/s on Apple M4:

1. Knowledge Graph Extraction - Entity-relation triples as [subject, relation, object] JSON arrays
2. Structured JSON Formatting - Strict schema adherence with correct data types
3. Concise Summarization - Factual compression in 1-3 sentences

Benchmarks vs Base (Qwen2.5-0.5B-Instruct)

Metric	Base	raging-1.0	Delta
KG Triple Precision	0.33	0.67	+103%
Throughput (tok/s)	64	122	+90%
TTFT (ms)	680	338	-50%
Output Length Stability	22.1	17.1	-23%
JSON Schema Accuracy	100%	100%	-
Summarization Compression	0.69	0.68	-

Benchmarks run on Apple M4, 10-sample subset, averaged over 3 independent runs.

Quick Start

from mlx_lm import load, generate

model, tokenizer = load("sajibsrs/raging1.0-0.5-MLX-4bit")

messages = [
    {"role": "system", "content": "Extract entity-relation triples as JSON array of [subject, relation, object]. Keep elements 1-4 words. Output ONLY JSON array."},
    {"role": "user", "content": "Microsoft was founded by Bill Gates and Paul Allen in Albuquerque."}
]

prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
response = generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=False)
print(response)

Training Details

• Base Model: Qwen/Qwen2.5-0.5B-Instruct
• Method: QLoRA (4-bit quantized base, LoRA rank 16, all 24 layers)
• Trainable Parameters: 2.93M (0.59%)
• Dataset: ~19.5K curated examples (summarization + structured JSON + KG extraction)
• Iterations: 1,000 (batch size 2, LR 1e-5)
• Hardware: Apple M4 Mac Mini (16GB unified memory)
• Framework: mlx-lm

Known Limitations

• KG Boundary Detection: May produce compound objects (e.g., "Bill Gates and Paul Allen in Albuquerque" instead of separate triples). Recommend post-processing with regex splitting.
• Pronoun Resolution: May output pronouns ("he", "the company") as subjects instead of resolved entity names.
• Not a Chat Model: Fine-tuned for single-turn structured extraction. Multi-turn conversational quality is degraded compared to base.
• JSON Complexity: Handles flat and simple nested schemas reliably. Deeply nested or conditional schemas may require fallback to a larger model.

Recommended Pipeline Integration

Use raging1.0-0.5-MLX-4bit as a fast first-pass processor. Validate outputs with Pydantic/jsonschema and fall back to a 7B+ model on validation failure:

import json
from pydantic import BaseModel, ValidationError

def safe_extract(raw_output, schema):
    try:
        return schema.model_validate(json.loads(raw_output))
    except (json.JSONDecodeError, ValidationError):
        return None  # Fallback to larger model

License

Apache 2.0 (inherited from Qwen2.5-0.5B-Instruct)

Author

sajibsrs