|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- OpenAssistant/oasst1 |
|
- erfanzar/CC-H2OAI-OASST-1-TRAIN |
|
- erfanzar/CC-OASST-1-TRAIN |
|
language: |
|
- en |
|
- fr |
|
- fa |
|
- nl |
|
metrics: |
|
- bertscore |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
|
|
# OpenSourceTransformers-OST Project |
|
|
|
[OST-OpenSourceTransformers Github](https://github.com/erfanzar/OST-OpenSourceTransformers) |
|
|
|
## Hello community |
|
|
|
this model is only 1B but you can call it somehow an SOTA |
|
|
|
|
|
this model can also run on 4 GB GPU RAM and know dialogs as well |
|
|
|
|
|
### Train Parametes |
|
|
|
- learning-rate : 2e-4 |
|
- sc : cosine lr |
|
- device : T4 GPU * 4 |
|
- batch-size: AutoFind |
|
- train time 12 H |
|
- max sequence length: 1024 |
|
- epochs : 2 |
|
## Usage Code |
|
|
|
```python |
|
|
|
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
from IPython.display import clear_output |
|
import textwrap |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("erfanzar/PGT-1B-2EP") |
|
|
|
model = AutoModelForCausalLM.from_pretrained("erfanzar/PGT-1B-2EP",device_map='auto',load_in_8bit=True) |
|
|
|
|
|
verify_text = lambda txt : '\n'.join([textwrap.fill(txt, width=110) for txt in txt.split('\n')]) |
|
|
|
|
|
def ppp(text:str): |
|
""" |
|
pre processing prompt |
|
""" |
|
return f"<|prompter|> {text} <|endoftext|><|assistant|>" |
|
|
|
def generate(text,max_new_tokens:int=1024,use_ppp:bool=False,b_pair=False): |
|
text = ppp(text) if use_ppp else text |
|
|
|
for i in range(max_new_tokens): |
|
enc = tokenizer(text,return_tensors='pt',add_special_tokens=False) |
|
text_r = text |
|
enc = model.generate(enc.input_ids,max_new_tokens=1,pad_token_id=0) |
|
text = tokenizer.decode(enc[0],skip_special_tokens=False) |
|
text = text[:-4]+tokenizer.eos_token if text[-4:] == '\n\n\n\n' else text |
|
|
|
if text.endswith(tokenizer.eos_token) or text.endswith('\n\n\n\n\n'): |
|
yield text[len(text_r):] if b_pair else text |
|
break |
|
else: |
|
yield text[len(text_r):] if b_pair else text |
|
|
|
|
|
for v in generate('what is a gpu',512,True): |
|
clear_output(wait=True) |
|
print(verify_text(v),end='') |
|
|
|
|
|
``` |
|
|
|
# Pythia-1B |
|
|
|
## Model Details |
|
|
|
### Pretrained Model |
|
|
|
- Developed by: [EleutherAI](http://eleuther.ai) |
|
- Model type: Transformer-based Language Model |
|
- FineTuned Languages: English , Persian , French, And Dutch |
|
- Learn more: [Pythia's GitHub repository](https://github.com/EleutherAI/pythia) for training procedures, config files, and details on how to use. |
|
- Library: [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) |
|
- License: [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) |
|
|
|
|
|
|
|
## NOTE |
|
|
|
The Pythia Suite is **NOT** intended for deployment. It is not in itself |
|
a product and cannot be used for human-facing interactions. For example, |
|
the model may generate harmful or offensive text... |
|
|
|
|
|
and also remember that this model is not good enough for Persian, French, and Dutch at least for this version |
