---
license: apache-2.0
tags:
- multilingual
- PyTorch
- Transformers
- gpt3
- gpt2
- Deepspeed
- Megatron
datasets:
- mc4
- Wikipedia
pipeline_tag: text-generation
widget:
- text: 'I know you''re tired, but can we go for another walk this evening?

    peter szemraj:


    '
  example_title: walk
- text: 'What do you call an alligator who''s just had surgery to remove his left
    arm?

    peter szemraj:


    '
  example_title: alligator
- text: 'If you could live anywhere, where would it be?

    peter szemraj:


    '
  example_title: dream living place
- text: 'What really makes you angry?

    peter szemraj:


    '
  example_title: pet peeve
- text: 'My friend says that she knows every language, but she doesn''t speak any
    of them.. what''s wrong with her?

    peter szemraj:


    '
  example_title: language
- text: 'What would you change about yourself if you could?

    peter szemraj:


    '
  example_title: change
- text: 'My first is in Asia, my second is in Europe, my third is in North America,
    and my fourth is in South America. What am I?

    peter szemraj:


    '
  example_title: continent
- text: 'Can you take me for dinner somewhere nice this time?

    peter szemraj:


    '
  example_title: dinner
- text: 'Honey, I have clogged the toilet for the third time this month.. sorry..

    peter szemraj:


    '
  example_title: overflow
- text: 'A man pushes his car to a hotel and tells the owner he''s bankrupt. Why?

    peter szemraj:


    '
  example_title: brain teaser
inference:
  parameters:
    min_length: 2
    max_length: 64
    length_penalty: 0.4
    no_repeat_ngram_size: 3
    do_sample: true
    top_p: 0.95
    top_k: 30
    temperature: 0.65
    repetition_penalty: 3.5
base_model: sberbank-ai/mGPT
---


# mGPT: fine-tune on message data MWE

This model is a fine-tuned version of [sberbank-ai/mGPT](https://huggingface.co/sberbank-ai/mGPT) on 80k messages. Trained for one epoch, will be updated in a (separate) model repo later.

## Model description

- testing if fine-tuned personality data bleeds over to other languages without being trained in them explicitly 

### Usage in python

Install the transformers library if you don't have it:

```
pip install -U transformers
```

load the model into a pipeline object:


```
from transformers import pipeline
import torch
device = 'cuda' if torch.cuda.is_available() else 'cpu'
my_chatbot = pipeline('text-generation', 
                      'pszemraj/mGPT-Peter-mwe',
                      device=0 if device == 'cuda' else -1,
                    )
```




## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine_with_restarts
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 1

### Framework versions

- Transformers 4.18.0
- Pytorch 1.11.0+cu113
- Datasets 2.1.0
- Tokenizers 0.12.1