mrhimanshu/Owen-ai-text-detection

Qwen3-0.6B three-class AI-text detector

A fine-tune of Qwen/Qwen3-0.6B that sorts text into three classes instead of the usual two:

Label	Class	Meaning
0	HUMAN	Written by a person, no model involvement
1	AI_ASSISTED	A person wrote it; a model rephrased or copyedited it
2	AI	Generated by a model from scratch

The middle class is the one a binary detector cannot express, and it is the most common real-world case. rasbt/human-vs-ai-50k ships only classes 0 and 2, so class 1 was built for this model by asking Qwen/Qwen3-1.7B to edit human texts at two intensities — light copyedit, moderate rewrite — accepting an edit only inside fixed similarity and length bands, which keeps class 1 a real middle ground rather than a blurred copy of either neighbour. 6,105 edited rows were kept. Each one inherits its seed's split and group_id, so a human text and its edited variant can never land on opposite sides of a split boundary.

Test-set results

Held-out split, scored only after checkpoint selection and temperature scaling.

Metric	Value
Accuracy (3-class)	95.67%
Macro F1 (3-class)	91.35%
Brier score	0.0676
Log loss	0.1260
AI-involvement accuracy	95.81%
Human false-positive rate	4.04%

The last row is the number that matters when this runs on real people: the share of genuinely human texts flagged as having AI involvement.

Two things you have to replicate

1. The readout token. Logits are read from a <|im_end|> token appended immediately after the text, at a position that varies with input length. Encode with add_special_tokens=False, truncate to 1023 tokens, append the readout token, then right-pad. Padding first puts the readout in the wrong place.

2. The temperature. Raw logits are uncalibrated. Divide them by 2.3634 (also in detector-config.json) before the softmax, or the reported confidence will be overstated.

AI involvement = P(AI_ASSISTED) + P(AI), thresholded at 0.5, reproduces the familiar binary answer without collapsing the model back to two classes.

Usage

import json

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer

REPO_ID = "mrhimanshu/Owen-ai-text-detection"

tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
tokenizer.padding_side = "right"
model = AutoModelForSequenceClassification.from_pretrained(REPO_ID)
model.eval()

with open(hf_hub_download(REPO_ID, "detector-config.json")) as handle:
    config = json.load(handle)


def classify(texts):
    encoded = tokenizer(
        texts,
        add_special_tokens=False,
        truncation=True,
        max_length=config["max_text_length"],
    )
    # The readout token goes after the text, before any padding.
    encoded = {
        "input_ids": [ids + [tokenizer.eos_token_id] for ids in encoded["input_ids"]],
        "attention_mask": [mask + [1] for mask in encoded["attention_mask"]],
    }
    padded = tokenizer.pad(encoded, return_tensors="pt")

    with torch.inference_mode():
        logits = model(**padded).logits.float()

    probabilities = torch.softmax(logits / config["temperature"], dim=-1)
    for text, row in zip(texts, probabilities):
        scores = {model.config.id2label[i]: float(p) for i, p in enumerate(row)}
        scores["AI_INVOLVEMENT"] = scores["AI_ASSISTED"] + scores["AI"]
        print(f"{text[:60]!r} {scores}")


classify(["Some text to score."])

Limitations

• Trained on a 1,024-token window. Longer inputs are truncated, so the verdict describes the opening of a long document only.
• Class 1 is synthetic: one editing model, two intensities. Human editing habits, and other editing models, are not represented.
• English only.
• A detector score is evidence, not proof. It should not be the sole basis for an accusation or a penalty against an individual.