garetht/identity-transformer

Identity Transformer

A lightweight transformer implementation designed for testing tensor shapes and experimenting with transformer architecture.

Description

This model implements a minimal transformer that maintains input-output identity. It's useful for:

• Understanding transformer tensor flows
• Testing integration with other components
• Educational purposes to see how transformers work
• Debugging pipelines with a simple model

Model Details

• Vocabulary size: 256 (compatible with the byte-level tokenizer)
• Hidden size: 64
• Number of layers: 1
• Number of attention heads: 2
• Intermediate size: 128
• Activation function: GELU

Usage

from transformers import AutoTokenizer, AutoModel

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("garetht/byte-tokenizer")
model = AutoModel.from_pretrained("garetht/identity-transformer")

# Encode text
text = "Hello, world!"
inputs = tokenizer(text, return_tensors="pt")

# Forward pass
outputs = model(**inputs)

# Process outputs
# outputs.last_hidden_state contains the hidden states

License

This project is open source and available under the MIT License.