Counsel-27B-FP8

Counsel-27B is a general-purpose language model with specialist legal capabilities, built by JudicialMind. It handles everyday chat, reasoning, writing and coding like a modern general model, and on legal work it reads statutes, regulations, contracts and case law closely, answers from the provisions and authorities that support the answer, and runs legal research and document workflows that call tools.

This repository holds Counsel-27B in FP8 for fast serving: 30 GB of weights instead of 55 GB, with the same drafter, runtime and modes. The BF16 weights are at judicialmind/Counsel-27B.

Highlights

• 310 tokens/s for one user on one H100 80 GB. With the Counsel drafter, FP8 is 42% faster than BF16 (218) and 6.1× BF16 without the drafter (51).
• Document editing at up to 3,627 tokens/s. Copy mode with Counsel's GDN replay kernel returns a revised 5,000-token contract in under 2 seconds.
• Base-model quality on Harvey LAB at 43% lower cost (BF16 weights). On the Harvey LAB benchmark Counsel-27B passed 88.2% of criteria (Qwen3.8-27B base: 87.2%) with 40% fewer input and 57% fewer output tokens: $0.98 per task instead of $1.71.
• 128K context on one 80 GB GPU in every mode, served through counsel-runtime.

Speed in action

Real-time replays of measured runs on one NVIDIA H100: every token appears when the server produced it.

Counsel-27B-FP8 with the Counsel drafter writes a complete non-disclosure agreement at 244 tokens/s, 4.8× standard decoding.

Copy mode with the GDN replay kernel revises ten contracts (42,331 tokens) in 14.5 seconds at 4,028 tokens/s, 80× standard decoding, with every clause placed correctly.

_{Music: "Inspired" (first video) and "Aitech" (second video) by Kevin MacLeod (incompetech.com), licensed under CC BY 4.0.}

Capabilities

General

• Instruction following and multi-turn conversation.
• Step-by-step reasoning in thinking mode, with the reasoning returned separately from the final answer.
• Function calling with JSON tool schemas.
• Everyday writing, summarisation and coding.

Legal

• Regulatory and statutory questions: identifies the obligations a provision imposes, who they bind and under what conditions, and answers from the text rather than from general impressions.
• Cross-references: follows "subject to", "as defined in" and "notwithstanding" links between provisions and brings the connected passages into the answer.
• Case-law research: synthesises an answer from retrieved judgments and search results, citing the authorities it relies on and separating holdings from commentary.
• Contracts: reviews clauses, explains risk and allocation (conditions, warranties, indemnities, limitation, termination), compares drafts and proposes redlines.
• Legal agent workflows: plans multi-step tasks, calls document-search and matter-management tools, and reports tool results faithfully, including failures and missing information, instead of claiming success.

Model details

Model	Counsel-27B-FP8
Type	Decoder-only causal language model, hybrid attention
Parameters	26.9 B
Layers	64 (48 gated linear-attention layers, 16 gated full-attention layers, one in every four)
Context	128K tokens (131,072) as served by counsel-runtime; 262,144 maximum positions
Vocabulary	248,320 tokens
Weights	FP8 (e4m3) with one scale per weight matrix and dynamic activation scaling for the attention, feed-forward and linear-attention projections; embeddings, output head, norms and linear-attention gates in BF16. Sharded model-*.safetensors (30 GB, including the built-in MTP head); drafter in drafter/ (BF16, 3.8 GB)
Language	English

Files

File	Contents
model-*.safetensors, model.safetensors.index.json	Model weights (FP8), including the built-in multi-token-prediction (MTP) head
config.json, generation_config.json	Model, quantization and default generation settings
configuration_counsel.py, modeling_counsel.py	Model classes for transformers and the chat() interface
tokenizer.json, tokenizer_config.json	Tokenizer
chat_template.jinja	Chat format, including thinking mode and tool calls
drafter/	Counsel drafter (DFlash2, 1.9 B parameters, block 16) for --mode drafter
runtime/	counsel-runtime: OpenAI-compatible server for vLLM and SGLang, with the drafter, MTP, plain and copy modes and Counsel's GDN replay kernel
LICENSE	CC BY-NC 4.0: non-commercial research use
LICENSE-APACHE	Apache License 2.0 of the Qwen3.8-27B base model and of vLLM, which the GDN replay kernel modifies

Serving

runtime/ holds counsel-runtime, an OpenAI-compatible server for vLLM and SGLang with Counsel's recommended settings: thinking on at reasoning effort xhigh, reasoning and tool calls in the standard OpenAI fields, and a 128K-token (131,072) context window. It serves the model as counsel-27b on port 8000. For use with transformers, take the BF16 weights from judicialmind/Counsel-27B.

huggingface-cli download judicialmind/Counsel-27B-FP8 --local-dir Counsel-27B-FP8

# vLLM 0.32.0
pip install "./Counsel-27B-FP8/runtime[vllm]"
counsel-install-kernels                             # once: Counsel's GDN replay kernel, used by --mode copy
counsel-serve-vllm Counsel-27B-FP8                  # drafter mode (default)
counsel-serve-vllm Counsel-27B-FP8 --mode copy      # document editing

# SGLang 0.5.22, in its own Python environment (needs the CUDA 13 toolkit: SGLang compiles kernels on first start)
pip install "./Counsel-27B-FP8/runtime[sglang]"
counsel-serve-sglang Counsel-27B-FP8

--mode	How the next tokens are drafted	Best for
drafter (default)	the Counsel drafter proposes 15 tokens per step	questions, research and drafting for one to a few users at a time
mtp	the built-in multi-token-prediction head proposes 7 tokens per step	the same, without loading the 3.8 GB drafter
plain	no drafting: one token per step	busy servers: the most total throughput with many users at once
copy	copied from the document in the prompt: up to 256 tokens per step on vLLM (with the GDN replay kernel); SGLang has no prompt-lookup drafting, so it runs the Counsel drafter; thinking off by default	editing documents: amendments, redlines, returning a revised contract

Every mode serves the same weights: the drafter, the MTP head and copy drafts only propose tokens, and Counsel checks each one. To switch mode, restart the server with another --mode. --tensor-parallel-size 2 (vLLM) or --tp-size 2 (SGLang) splits one replica over two GPUs; --gpus N runs one replica per GPU; any other server flag passes straight through.

Recommended settings

	Thinking	Direct
temperature	0.6 – 0.8	0.3 – 0.7 (0 for deterministic output)
top_p	0.95	0.95

Prompt and output can use the full 128K-token context.

Results

Speed

Decode speed for one user in tokens per second on one H100 80 GB (SXM) with counsel-runtime (vLLM 0.32.0, SGLang 0.5.22), 16 held-out legal prompts (thinking on, greedy, up to 1,024 output tokens):

Mode	FP8, vLLM	FP8, SGLang	BF16, vLLM
drafter	310	323	218
mtp	201	149	139
plain	80	86	51

Document editing: 2,400 to 3,600 tokens/s

When Counsel edits a document, almost every token of its answer is already in the prompt. Copy mode drafts up to 256 tokens at a time straight from the document, and Counsel's GDN replay kernel makes those long drafts fit and verify fast on one GPU, so the revised document comes back about 50 to 70 times faster than plain BF16 decoding.

Use it for amendments and conforming changes, redlines (party renames, defined terms, dates and amounts), inserting or deleting clauses, and returning the full revised agreement after changes are agreed.

counsel-install-kernels                            # once per vLLM environment
counsel-serve-vllm Counsel-27B-FP8 --mode copy

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
reply = client.chat.completions.create(
    model="counsel-27b", temperature=0,
    messages=[{"role": "user", "content": (
        'Add the sentence "This obligation survives termination of this Agreement for three (3) years." at the end of '
        "paragraph [p-12]. Return the complete revised agreement with everything else unchanged.\n\n" + agreement_text)}],
)
print(reply.choices[0].message.content)

Measured on one H100 80 GB (thinking off, greedy, one user); every answer was checked against the expected revised document.

Use case	What Counsel returns	copy, FP8, vLLM	copy, BF16, vLLM	copy, FP8, SGLang (Counsel drafter)	Edits correct (FP8)
Targeted amendment: add one clause to a 1,600-token contract excerpt	the full revised excerpt, 1,573 tokens	2,432	2,608	827	8 of 8
Long-document amendment: the same edit in a 5,000-token excerpt	the full revised document, 5,082 tokens	3,627	4,060	274	4 of 4

Harvey LAB: quality and cost (BF16 weights)

Harvey LAB benchmark, both models served with vLLM at temperature 0.8 and reasoning effort xhigh, graded by the same judge.

	Counsel-27B	Qwen3.8-27B (base)
Criteria passed	88.2%	87.2%
Tasks fully passed	8	15
Input / output tokens per task	1.96M / 31K	3.29M / 72K
Cost per task ($0.45 per million input, $3.20 per million output tokens)	$0.98	$1.71

How the speed-ups work

• FP8 weights. Half the bytes of BF16 to read for every token, and FP8 tensor-core matrix multiplies.
• Counsel drafter. A 1.9 B-parameter drafter proposes 15 tokens in one forward pass; Counsel verifies them in one step and keeps the longest correct run, 5.6 tokens per step on average.
• Built-in MTP head. The multi-token-prediction layer kept in the checkpoint drafts 7 tokens, 4.3 accepted per step.
• Copy drafting. When editing, nearly every token of the answer is already in the prompt: an n-gram lookup on the GPU drafts up to 256 tokens and Counsel accepts 100 to 160 of them per step.
• GDN replay kernel. Counsel's 48 gated linear-attention layers carry a recurrent state. Stock vLLM keeps 1 + 256 copies of it per request to undo rejected drafts, which does not fit next to a 27 B model on one 80 GB GPU. The replay kernel keeps one copy and replays the accepted tokens from cached projections.
• Chunked verify. The 257-token verify window runs through the chunked linear-attention kernel instead of token by token.

Use

Counsel-27B is for general and legal work. On legal matters its output is not legal advice: check citations and conclusions against the primary sources before relying on them.

License

Counsel-27B-FP8 (weights, drafter and runtime) is released under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0), for non-commercial research use. For commercial use, contact surya@judicialmind.ai.

It is built on Qwen3.8-27B by the Qwen team, released under the Apache License 2.0; the weights in this repository are modified from the language model of that release. The GDN replay kernel in runtime/counsel_runtime/kernels/ modifies vLLM and keeps vLLM's Apache License 2.0 notices. The Apache License 2.0 text for both is in LICENSE-APACHE.