xybytz/SilentReasoner-7B

NOTE: THIS IS EXPERIMENTAL SOFTWARE. PLEASE ENJOY, BUT DO NOT USE FOR PRODUCTION CRITICAL STUFF! SilentReasoner-7B (SoftCOT)

This model is an experimental fine-tune of Qwen2.5-7B-Instruct designed to compress explicit Chain-of-Thought (CoT) reasoning into 8 continuous latent tokens.

Instead of generating 1,000+ tokens of Let's think step by step... text before answering, this model processes the logic invisibly using continuous embeddings, then jumps straight to the final answer.

It acts like a compiled executable: it skips conversational filler, writes dense/Pythonic code, and retrieves optimal algorithms instantly. 📊 Benchmarks (vs. Baseline Qwen2.5-7B-Instruct) Metric Baseline Qwen-7B SilentReasoner-7B HumanEval (Pass@1) 80.0% 85.0% ✅ AIME 2024 (Pass@1) 6.7% 10.0% ✅ AIME Total Tokens Generated 23,910 16,673 AIME Total KV Cache Used 1549 MB 1100 MB ⚡ Efficiency Gains on Reasoning Tasks

Token Reduction: 30.3% fewer tokens generated.
VRAM Reduction: 29.0% smaller KV cache footprint.
Zero Conversational Filler: Skips the "Sure, here is the code!" text entirely.

💻 How to run it

⚠️ CRITICAL: Because this model uses inputs_embeds to inject the 8 continuous latent vectors, standard model.generate(input_ids=...) or HF pipeline() will output blank text.

You must inject the latent embeddings. Please use the inference.py script provided in this repository.

1. Installation

pip install torch transformers peft bitsandbytes

2. Run the Chatbot

Download the repository, open a terminal in the folder, and run: bash

python inference.py

3. Code Implementation

If you want to build on top of this model, here is the core logic for injecting the latents: python

import torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig from peft import PeftModel import os

model_path = "your_username/SilentReasoner-7B" base_model_name = "Qwen/Qwen2.5-7B-Instruct"

Load 4-bit Base Model

bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, ) base_model = AutoModelForCausalLM.from_pretrained(base_model_name, quantization_config=bnb_config, device_map="auto") model = PeftModel.from_pretrained(base_model, model_path) model.eval()

Load the 8 Latent Embeddings

extras = torch.load(os.path.join(model_path, "softcot_extras.pt"), map_location="cuda") latent_embeddings = torch.stack([extras["latent_embeddings"][str(i)] for i in range(8)]).unsqueeze(0).to(torch.bfloat16)

Format prompt as raw text (NOT chat template)

prompt = "What is 2 + 2?" text = f"Problem: {prompt}\nSolution: " input_ids = tokenizer(text, return_tensors="pt").input_ids.to("cuda")

with torch.no_grad(): # Inject latents prob_emb = model.get_input_embeddings()(input_ids) full_emb = torch.cat([prob_emb, latent_embeddings], dim=1) mask = torch.ones(1, full_emb.size(1), device="cuda", dtype=torch.long)

# Generate
outputs = model.generate(
    inputs_embeds=full_emb, 
    attention_mask=mask, 
    max_new_tokens=512, 
    do_sample=False
)

print(tokenizer.decode(outputs[0, 1:], skip_special_tokens=True))

🧠 How it works (SoftCOT Architecture)

During training, an auxiliary GRU decoder was attached to the 8 latent tokens. The model was forced to route its explicit reasoning through these 8 continuous vectors to reconstruct the Chain-of-Thought text. At inference, the auxiliary decoder is discarded. The base model has learned to use these 8 latents as a compressed "mental scratchpad", allowing it to bypass generating hundreds of visible thought tokens.