WebScorer-8K

WebScorer-8K is a Qwen3-Embedding-0.6B-based model that assigns a continuous quality score from 0 to 10 to a web document.

This release is the selected low-leakage variant from the ScoreWeb ablation study. It prioritizes excluding low-quality documents while retaining useful high-quality content.

Model architecture

Qwen3-Embedding-0.6B
  + all-layer LoRA in training
  + merged LoRA weights for release
  + final hidden sequence
  + last non-padding token
  + LayerNorm
  + Linear(1024 -> 1)
  + 10 * sigmoid

There is no multi-layer feature fusion and no additional bidirectional Transformer encoder. The final 0.6B model is fully merged, so inference does not require PEFT or an adapter.

Training used all 28 Qwen layers with LoRA rank 8, alpha 16, dropout 0.05, covering q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj.

Evaluation

Evaluation cohort: 8,792 validation documents and 9,234 held-out test documents, split by hostname/group.

Split	MAE	Spearman	Within 1	>=7 recall	>=8 recall	<5 leak to >=5
Validation	0.69973	0.82836	76.13%	48.08%	24.63%	8.83%
Test	0.70428	0.82822	75.76%	44.67%	21.37%	8.60%

The alternative last-token MLP reached test MAE 0.70111 and higher high-score recall, but leaked 9.55% of documents with teacher score <5 past the >=5 cutoff. The linear head was selected because low-quality removal is the primary objective.

See experiments/REPORT.md for all architecture comparisons and caveats.

Installation

python -m pip install -r requirements.txt

The custom model code was tested with Transformers 5.17.0. Qwen3 requires Transformers 4.51 or newer.

Python usage

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "s0duku/webscorer-8k"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).cuda().eval()

text = "A tutorial with definitions, examples, and complete code."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=8192)
inputs = {key: value.cuda() for key, value in inputs.items()}

with torch.inference_mode():
    score = model(**inputs).logits.item()

print(score)  # 0..10

trust_remote_code=True is required because the scalar regression head is defined in modeling_webscorer.py.

Command line

Run against a local checkout:

python score_text.py --text "Document text"
python score_text.py --input examples/input.txt

Run against the Hugging Face repository:

python score_text.py \
  --model s0duku/webscorer-8k \
  --input examples/input.txt

Each output line is JSON:

{"index": 0, "score": 7.314421, "preview": "Document text..."}

Inputs above 8,192 Qwen tokens are rejected by default because training excluded documents above this limit. Pass --truncate only when truncation is acceptable.

Training data

The scorer was trained by distilling a Qwen3.8-27B-FP8 teacher rubric into 175,061 labeled documents:

• 156,866 train
• 8,792 validation
• 9,234 test
• Maximum training length: 8,192 Qwen tokens

Splits are grouped by hostname, with document ID as fallback. The dataset is not a human-adjudicated gold standard.

The companion label dataset is available at s0duku/scoreweb-qwen3.8-labels.

Intended use

• Rank or filter Chinese/English web documents for pretraining data selection.
• Pre-screen content before a more expensive evaluator.
• Compare relative document quality after calibrating thresholds on a target domain.

Limitations

• The score reproduces a specific teacher rubric, not universal document quality.
• The model was trained on documents up to 8,192 Qwen tokens.
• Long documents, unseen domains, paywalled pages, and JavaScript-heavy pages may shift the score distribution.
• A fixed threshold of 5 may need recalibration for a new collection.
• Low-score leakage remains concentrated at teacher scores 3 and 4.
• This model should assist, not replace, policy, license, safety, or human review.

Repository layout

.
├── config.json
├── modeling_webscorer.py
├── model.safetensors
├── score_text.py
├── tokenizer.json
├── tokenizer_config.json
└── experiments/
    ├── REPORT.md
    ├── README.md
    ├── final_evaluation.json
    ├── final_run_config.json
    ├── final_data_manifest.json
    ├── results.csv
    ├── original_source/
    └── tests/

Upload to Hugging Face

The directory is already laid out as a model repository:

pip install -U huggingface_hub
export HF_ENDPOINT=https://hf-mirror.com
hf auth login --force
hf repos create s0duku/webscorer-8k --type model --public --exist-ok
hf upload s0duku/webscorer-8k . . --repo-type model \
  --exclude '.git/*' \
  --exclude '**/__pycache__/*' \
  --exclude '**/*.pyc'

License and provenance

This fine-tuned release and repository code are licensed under the MIT License. The upstream base model Qwen/Qwen3-Embedding-0.6B remains subject to its original Apache-2.0 license and attribution. See NOTICE for details.

Selected source checkpoint:

data/experiments/scoreweb-qwen06b-v4-alllora-lastlinear-finalhidden-all175k-5ep-20261023/
checkpoints/checkpoint-17500

The released model.safetensors is the merged final model produced by Transformers Trainer after reloading the best validation-MAE checkpoint.

SHA-256:

723b9c1c5ca895591786e31080c42f0e641b8c3661720dea636bca42b0fed841  model.safetensors