oriyonay/vega-v1

Vega v1

Vega v1 is a 1B-parameter GPT model trained from scratch. It was pretrained on 13B tokens using 8 Intel XPUs for 2 weeks, then SFT’d on 2B tokens on 8 XPUs for another 2 days.

It can have basic conversations and recall well known facts. Of course, it hallucinates very often.

Vega v1 is a baseline for my future adventures into LLM training; it is not very useful, but it's fun to play with.

This repository contains the model checkpoint and tokenizer. The included vega_v1_inference.py entry point is the reference inference program.

Vega v1 has:

• Model architecture: vega_v1
• Layers: 14
• Hidden size: 2560
• Vocabulary size: 65536

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('oriyonay/vega-v1', trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained('oriyonay/vega-v1', trust_remote_code=True, device_map='auto').eval()

messages = [
    {'role': 'system', 'content': 'You are Vega v1, a helpful, honest, and concise AI assistant.'},
    {'role': 'user', 'content': 'What is the capital of France?'},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors='pt',
    return_dict=True,
)

inputs = inputs.to(next(model.parameters()).device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    do_sample=True,
)

generated_tokens = outputs[0, inputs['input_ids'].shape[1]:]
response = tokenizer.decode(generated_tokens, skip_special_tokens=True)

print(response)

Because Vega v1 uses custom model code, trust_remote_code=True is required.