HC3 Human vs. ChatGPT Detector
This model is a fine-tuned binary text classifier for distinguishing human-written answers from ChatGPT-generated answers using the Human ChatGPT Comparison Corpus (HC3).
Model Details
- Model type: Transformer sequence classifier
- Base model:
sentence-transformers/all-MiniLM-L6-v2 - Task: Binary text classification
- Language: English
- Labels:
0= human-written1= ChatGPT-generated
Training Data
The model was trained on the English portion of the Human ChatGPT Comparison Corpus (HC3):
- Dataset:
Hello-SimpleAI/HC3 - Source: Guo et al. (2023), How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
- arXiv: 2301.07597
The preprocessing used one nonempty human answer and one nonempty ChatGPT answer per question. Questions were split into train, validation, and test sets before answers were flattened, so answers corresponding to the same question remained in the same split.
Only the answer text was used as model input.
Training Procedure
The model was fine-tuned using:
- Optimizer: AdamW
- Learning rate:
2e-5 - Epochs:
5 - Batch size:
32 - Maximum sequence length:
256 - Number of labels:
2
The full transformer model was fine-tuned together with the binary classification head.
Evaluation Results
Frozen-Embedding Baseline
A Logistic Regression classifier was trained on frozen embeddings from sentence-transformers/all-MiniLM-L6-v2.
- Baseline test accuracy: 0.8449
- Precision:
0.8374 - Recall:
0.8560 - F1:
0.8466
Fine-Tuned Transformer
The fine-tuned transformer achieved:
- Fine-tuned test accuracy: 0.9824
Fine-tuned confusion matrix:
| True label | Predicted Human | Predicted ChatGPT |
|---|---|---|
| Human | 2252 | 82 |
| ChatGPT | 0 | 2334 |
- Downloads last month
- 7