license: apache-2.0
base_model: google/electra-base-discriminator
tags:
- generated_from_trainer
metrics:
- accuracy
- f1
- precision
- recall
model-index:
- name: trueparagraph.ai-ELECTRA
results: []
trueparagraph.ai-ELECTRA
This model is a fine-tuned version of google/electra-base-discriminator on the None dataset. It achieves the following results on the evaluation set:
- Accuracy: 0.9430
- F1: 0.9421
- Precision: 0.9528
- Recall: 0.9316
- Mcc: 0.8862
- Roc Auc: 0.9429
- Pr Auc: 0.9217
- Log Loss: 0.8825
- Loss: 0.2952
Model description
TrueParagraph ELECTRA is a transformer-based model designed for detecting AI-generated text within academic and technical domains, particularly focusing on STEM (Science, Technology, Engineering, and Mathematics) texts. It leverages the ELECTRA architecture, which is known for its efficiency and accuracy in understanding complex text patterns and semantics. ELECTRA uses a novel training approach where it is trained as a discriminator rather than a generator, enhancing its ability to differentiate between real and rephrased text with higher precision. This makes TrueParagraph ELECTRA particularly effective in maintaining academic integrity by identifying potential AI-generated content.
Intended uses & limitations
AI-Generated Text Detection: TrueParagraph ELECTRA is optimized to detect AI-generated paragraphs within academic documents, theses, and research papers. Academic Integrity Enforcement: Useful for educators, researchers, and publishers in verifying the authenticity of written content.
Limitations:
Domain-Specific Performance: While highly effective in STEM-related texts, performance may vary in non-STEM fields due to the specific training dataset used. Potential Bias: The model's predictions might reflect biases present in the training data, particularly in edge cases where AI-generated and human-written text are indistinguishable. False Positives/Negatives: As with any AI model, there may be instances of misclassification, leading to false positives or false negatives, which users should account for when interpreting results.
Training and evaluation data
The model was trained and evaluated on the "pffaundez/16k-trueparagraph-STEM" dataset available on Hugging Face. This dataset comprises 16,000 paragraphs extracted from academic papers and theses across various STEM disciplines. The data includes both human-authored and AI-generated content, providing a balanced and representative sample for training a robust classification model. The dataset is preprocessed to maintain the integrity of technical terminologies, formulas, and citations, ensuring that the model is well-equipped to handle the intricacies of STEM literature.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 5
Training results
| Training Loss | Epoch | Step | Accuracy | F1 | Precision | Recall | Mcc | Roc Auc | Pr Auc | Log Loss | Validation Loss |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.5401 | 0.6297 | 500 | 0.7694 | 0.7044 | 0.9732 | 0.5519 | 0.5963 | 0.7684 | 0.7602 | 3.5789 | 0.6109 |
| 0.3122 | 1.2594 | 1000 | 0.9225 | 0.9231 | 0.9122 | 0.9342 | 0.8452 | 0.9225 | 0.8850 | 1.1485 | 0.2368 |
| 0.2301 | 1.8892 | 1500 | 0.8670 | 0.8811 | 0.7942 | 0.9892 | 0.7573 | 0.8676 | 0.7910 | 1.9476 | 0.3654 |
| 0.1608 | 2.5189 | 2000 | 0.9348 | 0.9364 | 0.9103 | 0.9639 | 0.8711 | 0.9349 | 0.8955 | 1.0090 | 0.2677 |
| 0.1146 | 3.1486 | 2500 | 0.9430 | 0.9421 | 0.9528 | 0.9316 | 0.8862 | 0.9429 | 0.9217 | 0.8825 | 0.2952 |
Framework versions
- Transformers 4.42.4
- Pytorch 2.3.1+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1