litillabs/litil-contract-playbook-3b

LiTiL Contract Playbook 3B

What this model does

LiTiL Contract Playbook applies the playbook rule you provide to a contract clause and recommends the next review action. It returns one of seven actions—including accept, redline, seek approval, escalate, or reject—together with a reason and usable fallback language in a predictable JSON object.

Use it as the policy layer in a contract-intelligence stack. Clause classifiers and extractors can first identify the provision and key facts; the workflow then supplies the matching playbook rule to this model. Its structured result can populate a review screen, route an approval, start a redline, or send the clause to a specialist while preserving the rule that drove the decision.

• Useful for: consistent first-pass clause review, redline routing, and escalation under your rules
• Give it: a clause, the relevant playbook rule, and the available context
• It returns: one of seven review actions with a reason and fallback language in JSON

Item	Value
Hugging Face repository	litillabs/litil-contract-playbook-3b
Tested adapter revision	ee221045c94b15166e5ed513f3c189c8ce2b3665
Base model	Qwen/Qwen2.5-3B-Instruct
Format	PEFT LoRA adapter, 119,801,528 bytes
Training dataset	Programmatically generated M9 Playbook v1 examples
Playbook	M9 Contract Action Policy, Playbook v1
Input context	Up to 1,728 rendered prompt tokens in the retained run
Output	One JSON object, up to 320 generated tokens

The playbook

This adapter was built from the locally authored M9 Contract Action Policy — Playbook v1 for B2B SaaS, cybersecurity services, and data-processing agreements. It assumes a US commercial setting and directs the model to apply the supplied policy rather than invent statutory rules.

The versioned playbook contains 70 rules across 13 clause families: limitation of liability, indemnity, confidentiality, AI use of customer data, DPA and security, assignment and change of control, auto-renewal, termination, IP ownership, publicity, warranty disclaimer, subprocessors, and audit rights.

Prompt and output contract

Use the exact system prompt above and render the user message in this order:

contract_type: {contract_type}
company_side: {company_side}
deal_context: {JSON object}
clause_type: {clause_type}
playbook_excerpt: {playbook_excerpt}
clause_text: {clause_text}

The output must be one JSON object with exactly these fields:

Field	Type and meaning
action	accept, redline, fallback_1, fallback_2, business_approval, legal_escalation, or reject
risk_level	low, medium, high, or critical
issue_tags	Array of tags from the supplied playbook vocabulary
risky_text	Array of excerpts from the supplied clause
playbook_basis	The rule basis from the supplied playbook excerpt
recommended_fallback	Fallback text or direction from the playbook
missing_facts	Required deal-context facts that were not supplied
explanation	Concise explanation citing the playbook rule basis

The retained evaluation used greedy decoding, prompt truncation at 1,728 tokens, and max_new_tokens=320. Parse with a strict JSON decoder and validate the keys, enumerated values, and issue-tag vocabulary before using a decision downstream.

Measured results

Evaluation	Cases	Qwen base	LiTiL Contract Playbook
Development action accuracy	383	11.49%	85.12%
Evaluation action accuracy	925	25.41%	84.97%
Evaluation binary decision accuracy	925	72.54%	87.89%
Valid JSON	925	99.78%	100.00%

On the same training rows, a TF-IDF lookup baseline reached 67.68% exact action accuracy and 75.35% binary decision accuracy. The adapter's 925-case exact-action Wilson interval is 82.53–87.13%.

These examples apply rule IDs represented in training, and their reference decisions come from the same versioned playbook resolver. The scores measure playbook execution within that interface.

Training and data

Training used 5,560 programmatically generated examples, with 383 development and 925 evaluation examples. A row-level lineage check matched all 5,560 training rows to the deterministic generators: 4,780 came from the original literal-template pool and 780 came from the coverage generator. Those generators use fixed synthetic templates and the generic test playbook; they do not load client document sources. No client files or client contract text appear in this post-training dataset.

The adapter was trained for four epochs with maximum sequence length 2048, per-device batch size 1, gradient accumulation 8, BF16, learning rate 2e-4, cosine scheduling, warmup ratio 0.03, and no packing. LoRA uses rank 16, alpha 32, and dropout 0.05. The retained training specification targets a RunPod NVIDIA L4 24 GB GPU through SkyPilot.

Runtime sizing

The 3B base contains about 6 GB of BF16 weights and the adapter adds about 114 MB. Allow roughly 8–10 GB of accelerator memory for the tested 2K-token envelope in BF16. Four-bit base weights contain about 1.5 GB of weight data; a practical short-prompt setup commonly needs 4–6 GB after quantization metadata, activations, and the KV cache. The recorded training configuration fits on a 24 GB L4.

Intended use

Use this adapter to automate a first-pass playbook action when the relevant policy excerpt and deal facts are supplied. Validate the returned JSON, retain the playbook rule beside the result, and let the action code determine whether the next step is acceptance, a redline, an approval request, or specialist review.

Use the model

Install the runtime:

python -m pip install -U torch transformers peft accelerate safetensors huggingface_hub

This example uses the LiTiL Labs release repository.

import json

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE_ID = "Qwen/Qwen2.5-3B-Instruct"
ADAPTER_ID = "litillabs/litil-contract-playbook-3b"

SYSTEM_PROMPT = """You are a contract-review assistant. Apply the provided playbook to the clause and return a JSON object only.

Output schema (return ONLY valid JSON, no prose, no markdown fences):
{
  "action": "accept|redline|fallback_1|fallback_2|business_approval|legal_escalation|reject",
  "risk_level": "low|medium|high|critical",
  "issue_tags": [],
  "risky_text": [],
  "playbook_basis": "",
  "recommended_fallback": "",
  "missing_facts": [],
  "explanation": ""
}

Rules:
- Decide ONLY based on the playbook excerpt and deal context provided. Do NOT invent legal authority.
- If a required fact is missing from deal_context, set action to business_approval and list the missing fact.
- issue_tags should use the playbook's tag vocabulary.
- explanation should cite the playbook rule basis.
"""


def render_user(data: dict) -> str:
    return (
        f"contract_type: {data['contract_type']}\n"
        f"company_side: {data['company_side']}\n"
        f"deal_context: {json.dumps(data['deal_context'])}\n"
        f"clause_type: {data['clause_type']}\n"
        f"playbook_excerpt: {data['playbook_excerpt']}\n"
        f"clause_text: {data['clause_text']}\n"
    )


example = {
    "contract_type": "MSA",
    "company_side": "customer",
    "deal_context": {
        "ARR": None,
        "counterparty_distressed_or_competitor": False,
        "counterparty_identity": "globex_inc",
        "customer_type": "startup",
        "data_sensitivity": "medium",
        "urgency": "fast_track",
        "vendor_type": "incumbent",
    },
    "clause_type": "confidentiality",
    "playbook_excerpt": (
        "Customer-side confidentiality: must be mutual; ≥3yr term + perpetual "
        "for trade secrets; broad permitted recipients (affiliates, advisors)."
    ),
    "clause_text": (
        "Each party shall maintain the other party's Confidential Information "
        "in confidence for a period of 4 years from the date of disclosure. "
        "Permitted recipients are limited to a single named employee of the "
        "receiving party."
    ),
}

dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
tokenizer = AutoTokenizer.from_pretrained(BASE_ID)
base = AutoModelForCausalLM.from_pretrained(
    BASE_ID,
    dtype=dtype,
    device_map="auto",
)
model = PeftModel.from_pretrained(
    base,
    ADAPTER_ID,
).eval()

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": render_user(example)},
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    truncation=True,
    max_length=1728,
).to(next(model.parameters()).device)

with torch.inference_mode():
    generated = model.generate(
        **inputs,
        max_new_tokens=320,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

text = tokenizer.decode(
    generated[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
).strip()
decision = json.loads(text)

required_keys = {
    "action",
    "risk_level",
    "issue_tags",
    "risky_text",
    "playbook_basis",
    "recommended_fallback",
    "missing_facts",
    "explanation",
}
if set(decision) != required_keys:
    raise ValueError(f"Unexpected output keys: {sorted(decision)}")
print(json.dumps(decision, indent=2))

Saved adapter output for that exact example:

{
  "action": "redline",
  "risk_level": "medium",
  "issue_tags": ["permitted_recipients_too_narrow"],
  "risky_text": ["limited to a single named employee"],
  "playbook_basis": "Permitted recipients must include affiliates and advisors.",
  "recommended_fallback": "Expand permitted recipients to affiliates, contractors under NDA, professional advisors.",
  "missing_facts": [],
  "explanation": "Per playbook rule CONF-C-03 (clause: confidentiality, side: customer): action=redline, risk=medium. Permitted recipients must include affiliates and advisors. Issues flagged: permitted_recipients_too_narrow. Recommended fallback: Expand permitted recipients to affiliates, contractors under NDA, professional advisors."
}

Developed by LiTiL Labs.