summerMC/SummerGPT2

GPT Weight-Derived Reservoir language model.

Architecture

• Reservoir state: 1024
• Reservoir banks: 8
• Bank dimension: 128
• Latent dimension: 256
• Vocabulary basis rank: 192
• Total parameters: 10,564,049

Loading

from transformers import AutoTokenizer, AutoModelForCausalLM

path = "summerMC/SummerGPT2"

tokenizer = AutoTokenizer.from_pretrained(
    path,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    path,
    trust_remote_code=True,
    dtype="auto",
)

This repository contains custom Transformers modeling code.