|
\n--- |
|
tags: |
|
- Causal Language modeling |
|
- CLM |
|
model_index: |
|
- name: MechDistilGPT2 |
|
results: |
|
- task: |
|
name: Causal Language modeling |
|
type: Causal Language modeling |
|
--- |
|
## MechDistilGPT2 |
|
This model is fine-tuned on text scraped from Mechanical/Automotive pdf books. |
|
|
|
Base model is DistilGPT2(https://huggingface.co/gpt2) (the smallest version of GPT2) |
|
|
|
## Fine-Tuning |
|
* Default Training Args |
|
* Epochs = 3 |
|
* Training set = 200k sentences |
|
* Validation set = 40k sentences |
|
|
|
## Framework versions |
|
* Transformers 4.7.0.dev0 |
|
* Pytorch 1.8.1+cu111 |
|
* Datasets 1.6.2 |
|
* Tokenizers 0.10.2 |
|
|