Sludge-Base-9B

Sludge-Base-9B is trentmkelly's experimental continued-pretraining LoRA adapter for Qwen3.5-9B-Base, trained on 15 million tokens of forum text. This repository contains the final PEFT adapter and tokenizer; the original base weights are required separately. It is not a merged standalone model.

The model learns raw post continuation. It has not undergone a separate chat or instruction-tuning stage. The included chat template is inherited from the tokenizer, not evidence of chat training. Outputs can reproduce offensive forum language, formatting artifacts, unsupported claims, repetition, or invented dialogue turns.

Training

• Base revision: 68c46c4b3498877f3ef123c856ecfde50c39f404.
• Exactly 3,000,000 training tokens from each of the five datasets listed above, for 15,000,000 tokens total, including EOS and excluding padding.
• 150,000 held-out tokens total; whole threads split before post sampling.
• Original plaintext with no content filtering or deduplication. Empty/null records contribute no text. Final selected records may be truncated to meet token budgets.
• Packed sequence length 2,048; one epoch, 7,325 optimizer steps.
• BF16 frozen base; FP32 trainable LoRA parameters with BF16 autocast.
• LoRA rank 32, alpha 64, dropout 0.05, all eligible linear modules; 86,556,672 trainable parameters.
• AdamW, learning rate 1e-4, cosine schedule, 3% warmup, batch size 1, gradient accumulation 1, no gradient checkpointing, one RTX A6000 48 GB.

The exact source revisions, token budgets, and preparation details are recorded in data_manifest.json; selected training settings are in training_config.json.

Evaluation

On the same fixed sample of 16 held-out packed blocks, loss decreased from 3.10996 before training to 2.63632 at the final step. Perplexity decreased from 22.4201 to 13.9617. These are in-domain next-token prediction metrics on a small validation sample, not a general capability or chat-quality benchmark.

eval_results.json contains the final metrics. Generated completion probes used temperature 0.8, top-p 0.95, top-k 50, and repetition penalty 1.0.

Loading

The training environment used PyTorch 2.6.0, Transformers 5.17.0, PEFT 0.20.0, Flash Linear Attention 0.5.2, and Liger Kernel 0.8.2. Authenticate with an account that can access this private repository before loading the adapter.

import torch
from peft import PeftModel
from transformers import AutoTokenizer, Qwen3_5ForCausalLM, Qwen3_5TextConfig

repo = "trentmkelly/Sludge-Base-9B"
base_id = "Qwen/Qwen3.5-9B-Base"
revision = "68c46c4b3498877f3ef123c856ecfde50c39f404"
tokenizer = AutoTokenizer.from_pretrained(repo)
tokenizer.pad_token = tokenizer.eos_token
config = Qwen3_5TextConfig.from_pretrained(base_id, revision=revision)
config.pad_token_id = tokenizer.pad_token_id
base = Qwen3_5ForCausalLM.from_pretrained(
    base_id,
    revision=revision,
    config=config,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
    device_map={"": 0},
)
model = PeftModel.from_pretrained(base, repo).eval()
inputs = tokenizer("The internet was a mistake because", return_tensors="pt").to("cuda")
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        top_k=50,
        repetition_penalty=1.0,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
        logits_to_keep=1,
    )
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))