mosaicml

/

mpt-7b-chat

like 511

The model doesn't seem to understand the instruction but keeps generating continuing text.

===

import transformers
model = transformers.AutoModelForCausalLM.from_pretrained(
  'mosaicml/mpt-7b-chat',
  trust_remote_code=True
)

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")

import torch 

generate_kwargs = {
    "max_new_tokens": 200,
    "min_new_tokens": 100,
    "temperature": 0.5,
    "do_sample": True, # The three options below used together leads to contrastive search
    #"top_k": 4,
    #"penalty_alpha": 0.6,
    #"no_repeat_ngram_size": no_repeat_ngram_size,
    #**generation_config,
}

prompt = "Write a poem in the tone of Shakespeare on HuggingFace"
with torch.no_grad():
    input_ids = tokenizer(prompt, return_tensors="pt").input_ids
    assert len(input_ids) == 1, len(input_ids)
    if input_ids[0][-1] == 2: # 2 is EOS, hack to remove. If the prompt is ending with EOS, often the generation will stop abruptly.
        input_ids = input_ids[:, :-1]
    input_ids = input_ids.to(0)
    #input_ids = tokenizer(prompt, padding=True, truncation=True, return_tensors="pt").input_ids.to(0)
    generated_ids = model.generate(
        input_ids,
        #stopping_criteria=stopping_criteria,
        **generate_kwargs
    )
    result = tokenizer.batch_decode(generated_ids.cpu(), skip_special_tokens=True)
    print(result[0])

Output:

Write a poem in the tone of Shakespeare on HuggingFace - OpenAI
Write a poem in the tone of Shakespeare on HuggingFace
I am not sure what you are looking for exactly, but here's a poem in the tone of Shakespeare about HuggingFace:
Oh HuggingFace, how sweet thy name doth sound,
Thy AI tools, they make our work so sound,
With pre-trained models, we can find
The answers we seek, without much need
Of coding or of patience, just a little bit
Of fine-tuning, and our models will come alive,
With insights and predictions, they will thrive,
And help us build the future, in a while.
Oh HuggingFace, how sweet thy name doth sound,
Thy AI tools, they make our work so sound,
With pre-trained models, we can find,
The answers we seek, without much need.
Of patience, or of coding, just a little bit
Of fine-