genshinssb/hw1-hc3-detector

HC3 Human vs. ChatGPT Answer Classifier

This model classifies English answers from the HC3 dataset as either human-written (label 0) or ChatGPT-generated (label 1). It fine-tunes sentence-transformers/all-MiniLM-L6-v2 with a two-class classification head.

Evaluation results

Model	Held-out test accuracy
Frozen sentence embeddings + logistic regression baseline	0.8449
Fine-tuned classifier	0.9961

The fine-tuned model improved test accuracy by approximately 15.12 percentage points over the baseline in this run. The test set contained 4,668 answers.

Data and training

The notebook used the English HC3 data at revision 4d0ff18143b5a7e1b1e79beb540c04549d1e59d3. It kept the first nonempty human and ChatGPT answer for each eligible question, then split questions into training, validation, and test sets with seed 42. Answers to the same question remained in the same split.

The resulting splits contained 37,334 training answers, 4,666 validation answers, and 4,668 test answers. The baseline used frozen sentence embeddings and logistic regression. The classifier was fine-tuned for five epochs with AdamW, a learning rate of 2e-5, a batch size of 32, and a maximum input length of 256 tokens.

Limitations

These results measure performance on a held-out split of the historical HC3 dataset. They do not establish that the model can reliably identify text from newer AI systems, other writing genres, or individual students. The model may learn dataset-specific differences in style, source, or formatting.

---

license: mit