mosaicml

/

mpt-7b-instruct

like 461

I was wondering if there was temperature config or penalty setting to lower the probability of repetition while running from HuggingFace's api? I was trying to generate a dialogue between people and the output looks like:

Person1: I don't know what to say.
Person2: I don't know what to say either.
Person1: I don't know what to say.
Person2: I don't know what to say either.
Person1: I don't know what to say.
Person2: I don't know what to say either.
Person1: I don't know what to say.
Person2: I don't know what to say either.
Person1: I don't know what to say.
Person2: I don't know what to say either.
Person1: I don't know what to say.
Person2: I don't know what to say either.
Person1: I don't know what to say.
Person2: I don't know what to say either.
Person1: I don't know what to say.
Person2: I don't know what to say either.

Here's my current code. I see the options to change temperature and penalty if you were running this from cli and downloaded the entire repo. Using the Huggingface api, would I be changing the penalty in the config section?

torch.cuda.set_per_process_memory_fraction(0.25)
torch.cuda.empty_cache()

model_name = "mosaicml/mpt-7b-instruct

tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")
config = transformers.AutoConfig.from_pretrained(
model_name,
  trust_remote_code=True
)
config.attn_config['attn_impl'] = 'torch'


model = transformers.AutoModelForCausalLM.from_pretrained(
  'mosaicml/mpt-7b-instruct',
  config=config,
  trust_remote_code=True,
  torch_dtype=torch.bfloat16,
)
model.to(device='cuda:3')


INSTRUCTION_KEY = "### Instruction:"
RESPONSE_KEY = "### Response:"
INTRO_BLURB = "Below is an instruction that describes a task. Write a response that appropriately completes the request."
PROMPT_FOR_GENERATION_FORMAT = """{intro}
{instruction_key}
{instruction}
{response_key}
""".format(
    intro=INTRO_BLURB,
    instruction_key=INSTRUCTION_KEY,
    instruction="{instruction}",
    response_key=RESPONSE_KEY,
)
 example = "Write dialogue for two people are a party and meet for the first time. They're both shy and hesitant about starting a conversation. Write 5 lines of dialogue between these two people:"

fmt_ex = PROMPT_FOR_GENERATION_FORMAT.format(instruction=example)


model_inputs = tokenizer(text=fmt_ex, return_tensors="pt").to("cuda:3")

output_ids = model.generate(
    **model_inputs,
    max_new_tokens=1024,
)
output_text = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
print(output_text)