qikp's Educational Scorer (QES)

QES is a model with an identical purpose to HuggingFaceFW/fineweb-edu-classifier, and is trained on a subset of its data.

The biggest difference is that QES is a fine-tune of huawei-noah/TinyBERT_General_4L_312D instead of Snowflake/snowflake-arctic-embed-m.

Mozilla Firefox includes a model fine-tuned on the same base model as QES for form autofill, so the base model's reliability is proven.

Training data

The first parquet shard of HuggingFaceFW/fineweb-edu-llama3-annotations was used. Additionally, a padding data collator was used.

Training details

Training took 20 minutes and 49 seconds on a single T4 GPU from Google.

Model was trained as a FP32/FP16 hybrid as the Turing architecture does not support bfloat16.

The default batch size and learning rate was used.

The model was trained for 2 epochs.

Limitations

The model deviates by up to around three quarters of a point or so during limited internal testing compared to the final FineWeb-Edu dataset. This accuracy is not guaranteed.

As such, it should only be used in constrained circumstances or circumstances involving colossal amounts of data.