LocalLLaMA/typed-decisions
Benchmark • Updated • 3.2k • 42.2k • 261
How to use shubhdotai/d3-edge-finetuned with Transformers:
# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline("zero-shot-classification", model="shubhdotai/d3-edge-finetuned") # pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("shubhdotai/d3-edge-finetuned", device_map="auto")Two fine-tunes of vllm-sr/d3-edge (0.6B decision model), trained with Unsloth + LoRA on d3's own readout and merged into the BF16 weights. Each folder is a complete d3 package that works with system_one.
| Folder | Trained on |
|---|---|
typed-decisions/ |
LocalLLaMA/typed-decisions train (1,200 cases × 5 questions, soft gold labels) |
banking77/ |
Banking77 (3,000 training messages, 77 intents) |
from huggingface_hub import snapshot_download
from transformers import AutoModel
path = snapshot_download("shubhdotai/d3-edge-finetuned", allow_patterns = "banking77/*") # or "typed-decisions/*"
model = AutoModel.from_pretrained(f"{path}/banking77", trust_remote_code = True)
print(model.system_one(state = "...", questions = {...}))
For banking77/, ask one choice question whose options are the 77 Banking77 intent names, sorted alphabetically (option codes are assigned by position, as in training).
Typed decisions test (400 cases, 2,000 questions). KL is KL(gold ‖ model), lower is better:
| workflow | d3-edge accuracy | d3-edge kl | d3-edge fine-tuned accuracy | d3-edge fine-tuned kl | d3 (API) accuracy | d3 (API) kl |
|---|---|---|---|---|---|---|
| agent_trace_observability | 0.35 | 1.412 | 0.738 | 0.081 | 0.654 | 0.484 |
| customer_service | 0.396 | 0.9 | 0.808 | 0.102 | 0.792 | 0.251 |
| invoice_processing | 0.504 | 1.557 | 0.84 | 0.074 | 0.782 | 0.529 |
| security_incidents | 0.436 | 0.837 | 0.782 | 0.066 | 0.746 | 0.373 |
| overall | 0.422 | 1.176 | 0.792 | 0.081 | 0.744 | 0.41 |
Banking77 test (3,076 messages):
| model | Banking77 accuracy |
|---|---|
| d3-edge | 0.895 |
| d3-edge fine-tuned | 0.917 |
| d3 (API) | 0.922 |
d3 (API) is the hosted large vllm-sr/d3 model, zero-shot.