File size: 3,117 Bytes
6b0315d 5c85d30 c450adf 99544c9 c450adf 99544c9 c450adf 99544c9 c450adf 99544c9 c450adf 99544c9 6b0315d 2f46918 6b0315d 2f46918 6b0315d 99fc095 6b0315d 99fc095 6b0315d 99fc095 8b5e267 99fc095 8b5e267 99fc095 8b5e267 99fc095 8b5e267 6b0315d 2f46918 6b0315d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 |
---
license: apache-2.0
tags:
- multilingual
- PyTorch
- Transformers
- gpt3
- gpt2
- Deepspeed
- Megatron
datasets:
- mc4
- Wikipedia
pipeline_tag: text-generation
widget:
- text: 'I know you''re tired, but can we go for another walk this evening?
peter szemraj:
'
example_title: walk
- text: 'What do you call an alligator who''s just had surgery to remove his left
arm?
peter szemraj:
'
example_title: alligator
- text: 'If you could live anywhere, where would it be?
peter szemraj:
'
example_title: dream living place
- text: 'What really makes you angry?
peter szemraj:
'
example_title: pet peeve
- text: 'My friend says that she knows every language, but she doesn''t speak any
of them.. what''s wrong with her?
peter szemraj:
'
example_title: language
- text: 'What would you change about yourself if you could?
peter szemraj:
'
example_title: change
- text: 'My first is in Asia, my second is in Europe, my third is in North America,
and my fourth is in South America. What am I?
peter szemraj:
'
example_title: continent
- text: 'Can you take me for dinner somewhere nice this time?
peter szemraj:
'
example_title: dinner
- text: 'Honey, I have clogged the toilet for the third time this month.. sorry..
peter szemraj:
'
example_title: overflow
- text: 'A man pushes his car to a hotel and tells the owner he''s bankrupt. Why?
peter szemraj:
'
example_title: brain teaser
inference:
parameters:
min_length: 2
max_length: 64
length_penalty: 0.4
no_repeat_ngram_size: 3
do_sample: true
top_p: 0.95
top_k: 30
temperature: 0.65
repetition_penalty: 3.5
base_model: sberbank-ai/mGPT
---
# mGPT: fine-tune on message data MWE
This model is a fine-tuned version of [sberbank-ai/mGPT](https://huggingface.co/sberbank-ai/mGPT) on 80k messages. Trained for one epoch, will be updated in a (separate) model repo later.
## Model description
- testing if fine-tuned personality data bleeds over to other languages without being trained in them explicitly
### Usage in python
Install the transformers library if you don't have it:
```
pip install -U transformers
```
load the model into a pipeline object:
```
from transformers import pipeline
import torch
device = 'cuda' if torch.cuda.is_available() else 'cpu'
my_chatbot = pipeline('text-generation',
'pszemraj/mGPT-Peter-mwe',
device=0 if device == 'cuda' else -1,
)
```
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine_with_restarts
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 1
### Framework versions
- Transformers 4.18.0
- Pytorch 1.11.0+cu113
- Datasets 2.1.0
- Tokenizers 0.12.1
|