microsoft

/

phi-2

like 3.15k

Tried example from model page :

Instruct: Write a detailed analogy between mathematics and a lighthouse.
Output:

With following code :

# Load model directly
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="microsoft/phi-2")
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")

import time
start_time = time.time()

prompt = """Instruct: Write a detailed analogy between mathematics and a lighthouse.
Output: """


# Function to construct the prompt using the new system prompt template
def get_prompt_with_template(message: str) -> str:
    return SYSTEM_PROMPT_TEMPLATE.format(instruction=message)

response = pipe(prompt)[0]['generated_text']


end_time = time.time()

total_time = end_time - start_time


print(response)

num_tokens = len(tokenizer.encode(response))

print("The script took", total_time, "seconds to run.")
print("Speed : ", num_tokens / total_time, " t/s")

My outputs are :

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
Instruct: Write a detailed analogy between mathematics and a lighthouse.
Output:  HareRay NobelRay
The script took 1.9523024559020996 seconds to run.
Speed :  10.756530032789689  t/s

Tried multiple times on multiple different queries. Why is this so bad?