userrepgen-4b-8bit (MLX)

8-bit MLX conversion of microsoft/UserRepGen-4B for Apple Silicon. Text-only (vision weights not included). A smaller 6-bit version performed just as well in testing and is the recommended choice for most users.

UserRepGen-4B reads a multi-turn user–assistant conversation and outputs a JSON description of the user: their task and their user_behavior. It is not a chatbot.

Usage

The model must be prompted with the included representation_prompt.txt template, with thinking disabled.

import json
from mlx_lm import load, generate
from huggingface_hub import hf_hub_download

repo = "xquantize/userrepgen-4b-8bit"
model, tokenizer = load(repo)
template = open(hf_hub_download(repo, "representation_prompt.txt")).read().strip()

conversation = [
    {"role": "user", "content": "why does my python script say KeyError: 'name'"},
    {"role": "assistant", "content": "The key 'name' isn't in the dictionary. Can you share the code?"},
    {"role": "user", "content": "oh it's nested under data['user']['name']. thanks!!"},
]
history = "\n".join(f"[{m['role']}]: {m['content']}" for m in conversation)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": template.format(conversation_history=history)}],
    add_generation_prompt=True, enable_thinking=False, tokenize=False,
)
out = generate(model, tokenizer, prompt=prompt, max_tokens=1024)
print(json.loads(out))

Evaluation

Tested against the original bf16 weights (also run in MLX) on 5 varied conversations (Excel help, Python debugging, travel planning, writing edits, health question), using greedy decoding as in Microsoft's reference script.

Version	Bits/weight	Valid JSON	Outputs with repeated sentences
bf16 (original)	16	5/5	0/5
8-bit (this repo)	8.5	5/5	1/5
6-bit	6.5	5/5	0/5
4-bit, group size 32	5.0	4/5	1/5
4-bit, mixed 4/6	4.7	3/5	2/5
4-bit (plain)	4.5	3/5	1/5

4-bit is not recommended for this model. All three 4-bit variants tested broke on at least one conversation, falling into long repetition loops that produced invalid JSON. 6-bit and 8-bit both matched bf16; the 8-bit's one repeated sentence (on the Excel example) still produced valid JSON, and the difference between 6-bit and 8-bit is within noise for a test this small.

Details

• Size: ~4.2GB download; runs comfortably on Macs with 16 GB+ unified memory
• Speed: ~17 s for a full representation of an 8-turn conversation on an M3 Pro (18 GB)
• Max input: the original model supports prompts up to 8,192 tokens
• Converted with: mlx-lm 0.32.0 mlx_lm.convert --hf-path microsoft/UserRepGen-4B -q --q-bits 8

Limitations

The evaluation above is a small sanity check (5 conversations, greedy decoding), not a full benchmark. For the model's intended use, training details and known limitations, see the original model card.

Credits

Original model by Microsoft, released under the MIT license. Converted with mlx-lm by xquantize.