File size: 3,033 Bytes
6b0315d 5c85d30 6b0315d 5c85d30 c450adf 6b0315d 2f46918 6b0315d 2f46918 6b0315d 99fc095 6b0315d 99fc095 6b0315d 99fc095 8b5e267 99fc095 8b5e267 99fc095 8b5e267 99fc095 8b5e267 6b0315d 2f46918 6b0315d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 |
---
license: apache-2.0
pipeline_tag: text-generation
tags:
- multilingual
- PyTorch
- Transformers
- gpt3
- gpt2
- Deepspeed
- Megatron
datasets:
- mc4
- Wikipedia
widget:
- text: "I know you're tired, but can we go for another walk this evening?\npeter szemraj:\n\n"
example_title: "walk"
- text: "What do you call an alligator who's just had surgery to remove his left arm?\npeter szemraj:\n\n"
example_title: "alligator"
- text: "If you could live anywhere, where would it be?\npeter szemraj:\n\n"
example_title: "dream living place"
- text: "What really makes you angry?\npeter szemraj:\n\n"
example_title: "pet peeve"
- text: "My friend says that she knows every language, but she doesn't speak any of them.. what's wrong with her?\npeter szemraj:\n\n"
example_title: "language"
- text: "What would you change about yourself if you could?\npeter szemraj:\n\n"
example_title: "change"
- text: "My first is in Asia, my second is in Europe, my third is in North America, and my fourth is in South America. What am I?\npeter szemraj:\n\n"
example_title: "continent"
- text: "Can you take me for dinner somewhere nice this time?\npeter szemraj:\n\n"
example_title: "dinner"
- text: "Honey, I have clogged the toilet for the third time this month.. sorry..\npeter szemraj:\n\n"
example_title: "overflow"
- text: "A man pushes his car to a hotel and tells the owner he's bankrupt. Why?\npeter szemraj:\n\n"
example_title: "brain teaser"
inference:
parameters:
min_length: 2
max_length: 64
length_penalty: 0.4
no_repeat_ngram_size: 3
do_sample: True
top_p: 0.95
top_k: 30
temperature: 0.65
repetition_penalty: 3.5
---
# mGPT: fine-tune on message data MWE
This model is a fine-tuned version of [sberbank-ai/mGPT](https://huggingface.co/sberbank-ai/mGPT) on 80k messages. Trained for one epoch, will be updated in a (separate) model repo later.
## Model description
- testing if fine-tuned personality data bleeds over to other languages without being trained in them explicitly
### Usage in python
Install the transformers library if you don't have it:
```
pip install -U transformers
```
load the model into a pipeline object:
```
from transformers import pipeline
import torch
device = 'cuda' if torch.cuda.is_available() else 'cpu'
my_chatbot = pipeline('text-generation',
'pszemraj/mGPT-Peter-mwe',
device=0 if device == 'cuda' else -1,
)
```
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine_with_restarts
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 1
### Framework versions
- Transformers 4.18.0
- Pytorch 1.11.0+cu113
- Datasets 2.1.0
- Tokenizers 0.12.1
|