tachiwin/tiyat_2_alpha

Tiyat Alpha - Custom LLM

Model Details

• Tokenizer: tachiwin/tokenizer_64k
• Vocabulary Size: 64,000
• Max Sequence Length: 918
• Embedding Dimension: 256
• Number of Layers: 4
• Number of Heads: 4
• Feed-Forward Dimension: 1024
• Total Parameters: ~19.8M

Training Data

• Dataset: tachiwin/tiyat-ground-pretrain-m1024

Training Configuration

• Batch Size: 16
• Learning Rate: 0.0005
• Weight Decay: 0.01
• Gradient Clip: 1.0
• Warmup Ratio: 0.1
• Dropout Rate: 0.1
• Total Steps: 35418
• Final Loss: 4.8666

Usage

from transformers import AutoTokenizer, FlaxAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tachiwin/tiyat_2_alpha")
model = FlaxAutoModelForCausalLM.from_pretrained("tachiwin/tiyat_2_alpha")

# Generate text
inputs = tokenizer("Your prompt here", return_tensors="np")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0]))