rocky013/hw1-hc3-detector

HC3 Human vs. ChatGPT Answer Detector

A fine-tuned version of sentence-transformers/all-MiniLM-L6-v2 for binary classification of answers from the HC3 dataset as human-written or ChatGPT-generated. Built for CS546 Homework 1.

Label id	Config name	Meaning
0	LABEL_0	Human-written answer
1	LABEL_1	ChatGPT-generated answer

The model takes only the answer text as input (not the question).

Results

Evaluated on the held-out HC3 test split (4,668 answers, 2,334 per class).

Model	Test accuracy	Macro F1	Misclassified
Baseline: frozen all-MiniLM-L6-v2 embeddings + logistic regression	0.8451	0.8451	723
This model: end-to-end fine-tuned all-MiniLM-L6-v2	0.9871	0.9871	60

Confusion matrix of the fine-tuned model (rows = true label, columns = predicted):

	Pred. human	Pred. ChatGPT
Human	2274	60
ChatGPT	0	2334

All 60 errors are human-written answers wrongly flagged as ChatGPT-generated. No ChatGPT answer was missed.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="rocky013/hw1-hc3-detector")
print(clf("Sure! There are many factors that can contribute to this. Overall, ..."))
# LABEL_1 = ChatGPT-generated, LABEL_0 = human-written

Training details

Data. HC3 English all.jsonl at revision 4d0ff18. For each question, the first nonempty human answer and the first nonempty ChatGPT answer are kept. Empty questions, missing or identical answer pairs, and duplicate questions (case- and whitespace-insensitive) are removed, leaving 23,334 question pairs. Questions are split 80/10/10 with seed 42 before flattening into answers, so both answers to a question always stay in the same split:

Split	Answers
Train	37,334
Validation	4,666
Test	4,668

Procedure. Full fine-tuning of all 22.7M parameters with BertForSequenceClassification, which uses the [CLS] token, the BERT pooler, dropout 0.1 and a linear head.

Hyperparameter	Value
Optimizer	AdamW
Learning rate	2e-5 (constant, no warmup)
Batch size	32
Epochs	5 (5,835 steps)
Max sequence length	256 tokens (truncation, dynamic padding)
Loss	Cross-entropy
Seed	42
Hardware	1 × NVIDIA A100 40GB, about 10 minutes

The training loss fell from 0.69 to below 0.1 within 76 steps, and the mean loss per epoch was 0.081 → 0.013 → 0.007 → 0.004 → 0.004. The validation split was not used for model selection.

Limitations

• In-distribution only. The 98.7% accuracy is measured on the same distribution the model was trained on: 2023-era ChatGPT answers to HC3 questions. It is not expected to transfer to newer LLMs, other domains, prompted "casual" AI text, or human-edited AI text.
• Dataset artifacts. Many human answers in HC3 (especially from reddit_eli5) contain tokenization artifacts such as spaces before punctuation (word ,) and split contractions (do n't), which ChatGPT answers lack. The model may partly rely on these shortcuts rather than genuine authorship signals.
• Asymmetric errors. All test errors are false positives, meaning human text flagged as AI. Wrongly accusing a person of using AI is usually the more harmful error.
• Truncation. Inputs longer than 256 tokens (about 22% of test answers) are truncated.
• Not for high-stakes use. As the HC3 authors and the course note, this historical benchmark is not a reliable detector for current student work or any decision that affects people.

Citation

HC3 dataset:

@article{guo2023hc3,
  title   = {How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection},
  author  = {Guo, Biyang and Zhang, Xin and Wang, Ziyuan and Jiang, Minqi and Nie, Jinran and Ding, Yuxuan and Yue, Jianwei and Wu, Yupeng},
  journal = {arXiv preprint arXiv:2301.07597},
  year    = {2023}
}