OpenSourceTransformers-OST Project

Hello community

this model is only 1B but you can call it somehow an SOTA

this model can also run on 4 GB GPU RAM and know dialogs as well

Train Parametes

learning-rate : 2e-4
sc : cosine lr
device : T4 GPU * 4
batch-size: AutoFind
train time 12 H
max sequence length: 1024
epochs : 2

Usage Code



from transformers import AutoTokenizer, AutoModelForCausalLM
from IPython.display import clear_output
import textwrap


tokenizer = AutoTokenizer.from_pretrained("erfanzar/PGT-1B-2EP")

model = AutoModelForCausalLM.from_pretrained("erfanzar/PGT-1B-2EP",device_map='auto',load_in_8bit=True)


verify_text = lambda txt : '\n'.join([textwrap.fill(txt, width=110) for txt in txt.split('\n')])


def ppp(text:str):
  """
  pre processing prompt
  """
  return f"<|prompter|> {text} <|endoftext|><|assistant|>"

def generate(text,max_new_tokens:int=1024,use_ppp:bool=False,b_pair=False):
  text = ppp(text) if use_ppp else text
  
  for i in range(max_new_tokens):
    enc = tokenizer(text,return_tensors='pt',add_special_tokens=False)
    text_r = text
    enc = model.generate(enc.input_ids,max_new_tokens=1,pad_token_id=0)
    text = tokenizer.decode(enc[0],skip_special_tokens=False)
    text = text[:-4]+tokenizer.eos_token if text[-4:] == '\n\n\n\n' else text
  
    if text.endswith(tokenizer.eos_token) or text.endswith('\n\n\n\n\n'):
      yield text[len(text_r):] if b_pair else text
      break
    else:
      yield text[len(text_r):] if b_pair else text


for v in generate('what is a gpu',512,True):
  clear_output(wait=True)
  print(verify_text(v),end='')

Pythia-1B

Model Details

Pretrained Model

Developed by: EleutherAI
Model type: Transformer-based Language Model
FineTuned Languages: English , Persian , French, And Dutch
Learn more: Pythia's GitHub repository for training procedures, config files, and details on how to use.
Library: GPT-NeoX
License: Apache 2.0

NOTE

The Pythia Suite is NOT intended for deployment. It is not in itself a product and cannot be used for human-facing interactions. For example, the model may generate harmful or offensive text...

and also remember that this model is not good enough for Persian, French, and Dutch at least for this version

erfanzar
/

PGT-1B-2EP