Quantizations of https://huggingface.co/olliai/Maika-Buddy-3B8-Phi3

From original readme

Model Summary

Maika Buddy Model, fine-tuned from Microsoft Phi3 with our specialized children's conversational dataset, drives BuddyOS to enrich children's communication skills through interactive dialogue. Designed for AI-enhanced toys, Maika encourages children to express themselves, ask questions, and practice active listening in a non-judgmental environment. It not only fosters confidence in communication but also promotes a balanced approach by encouraging activities like outdoor play, hands-on creativity, and sharing experiences with family and friends, ensuring comprehensive developmental support for young users.

Project Website: BuddyOS

Requirements

The code of Maika Buddy has been in the latest Hugging face transformers and we advise you to install transformers>=4.37.0.

Quickstart

Here provides a code snippet to show you how to load the tokenizer and model and how to generate contents.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.set_default_device("cuda")

model_path = "olliai/Maika-Buddy-3B8-Phi3"

model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

prompt = """A chat between a curious kid and an artificial intelligence assistant. The assistant gives helpful, easy-to-understand, detailed, and polite answers to the questions of the kid.
Kid: I have 10 apples. I gave Bob 2 pencil. How many apples do I have now?
AI: """

inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False)

outputs = model.generate(**inputs, max_length=200,
                         do_sample=True,
                        repetition_penalty=1.1,
                        temperature=0.3,
                        top_k=50,
                        top_p=0.95, 
                        eos_token_id=tokenizer.encode("<|endoftext|>"))
text = tokenizer.batch_decode(outputs)[0]
print(text)