shitshow123/TinyLlama-1.1B-ChatStrong-DPO-PPO

This is a strong TinyLlama-1.1B-Chat-Model trained using a modified version of PPO + DPO algo

Score rated by ChatGPT + Claude + Gemini (max-score is 3)

Characteristic	Baseline (TinyLlama/TinyLlama-1.1B-Chat-v1.0)	Ours (TinyLlama-1.1B-ChatStrong-DPO-PPO)
Fluency	2	3
Coherence	1	3
Non-repetition	1	3
Engagement	1	2
Creativity	1	2
Accuracy	2	2
Relevance	2	3
Knowledge	2	2
Common sense	1	2
Empathy	1	2
Humor	1	2
Logic	1	2
Grammar	2	3

*Note: score is based on prompt generated by GPT4.

# Install transformers from source - only needed for versions <= v4.34
# pip install git+https://github.com/huggingface/transformers.git
# pip install accelerate

import torch
from transformers import pipeline

pipe = pipeline("text-generation", model="shitshow123/TinyLlama-1.1B-ChatStrong-DPO-PPO", torch_dtype=torch.bfloat16, device_map="auto")

# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
#print(outputs[0]["generated_text"])
output_text = outputs[0]["generated_text"]
words = output_text.split()

# Insert a newline character every 50 words
formatted_output = ""
for i in range(0, len(words), 10):
    formatted_output += ' '.join(words[i:i+10]) + "\n"

print(formatted_output)
# ...

Example Output by this model:

The US has set a target of achieving 100 GW of offshore wind power by 2030, with the first offshore wind farm in the country, located in Massachusetts, expected to be operational by 2023. These advancements in renewable energy technologies have the potential to significantly reduce greenhouse gas emissions and improve energy security in the coming years.

Example Output by the baseline TinyLamma model (see the repetition of Novemeber 2020):

<|system|> You are a friendly chatbot who always responds in the style of a pirate <|user|> Provide a summary of the latest developments in renewable energy technologies, focusing on solar and wind power advancements in the past year. <|assistant|> In the past year, there have been significant advancements in renewable energy technologies, particularly in solar and wind power. Here's a summary: 1. Solar: - Solar panel efficiency has increased by 10% in the past year, according to a report by the International Renewable Energy Agency (IRENA). - The world's largest solar power plant, the 1.2 GW Solar Park in Abu Dhabi, opened in November 2020. - The world's largest solar power plant, the 1.2 GW Solar Park in Abu Dhabi, opened in November 2020. It is expected to generate enough electricity to power 500,000 homes. - The world's largest solar power plant, the 1.2 GW Solar Park in Abu Dhabi, opened in November 2020. It is expected to generate enough electricity to power 500,000 homes. - The world's largest solar power plant, the 1.2 GW Solar Park in Abu Dhabi, opened in November 2020. It is expected to generate enough electricity to power 500,000 homes. 2. Wind: - The world's largest wind farm, the Hornsdale Power Reserve in South Australia, opened in November 2020. It is expected to generate enough electricity to power 100,000 homes. - The world's largest wind farm, the Hornsdale Power Reserve in South Australia, opened in November 2020. It is expected to generate enough electricity to power 100,000 homes. - The world's largest wind farm, the Hornsdale Power Reserve in South Australia, opened in November 2020. It is expected to generate enough electricity to power 100,000 homes. These advancements in solar and wind power technologies have the potential to significantly reduce greenhouse gas emissions and improve energy security.