Instructions to use gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED
- SGLang
How to use gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED with Docker Model Runner:
docker model run hf.co/gpjt/jax-with-mha-bias-fw-fwedu-5050-DEPRECATED
Factorized embeddings and non-embedding parameter allocation for GPT-2
Hi Giles,
In your August Chinchilla writeup, you noted that embeddings and the output head consume roughly 47% of parameters in untied 163M GPT-2 models (and 23% when tied).
At 163,009,536 total parameters (12 layers, 768 hidden width, 50,257 vocab), untied token tables take 50,257 * 768 * 2 = 77.19M parameters. That leaves only 85M parameters for the 12 transformer blocks (7.08M per block). When training over 3.26B tokens, nearly half the parameter budget remains in static lookup tables rather than active layer transformations.
In an open architecture I work on called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture; weights: https://huggingface.co/AndrewThompson1233/maba-101m), we decouple vocabulary size from hidden dimension using a low-rank bottleneck:
Routing untied vocabulary through a rank-128 projection (50k -> 128 -> 768) drops the combined input and output footprint from 77.19M to roughly 13.06M parameters.
This reclaims ~64.1M parameters within the exact same 163M parameter ceiling.
At ~7.08M parameters per block, that saved budget funds 9 additional transformer layers, expanding depth from 12 to 21 layers.
In the context of your eval investigations on why OpenAI GPT-2 outperforms custom small baselines, factorized embeddings allow retaining untied input/output representations without burning 47% of the capacity on lookup tables.
Did you test low-rank factorized projections before choosing between tied and untied weights for the JAX runs?
Best,
Andrew
Hey Andrew,
That's an interesting experiment! My models aren't quite so inventive, I've just been using GPT-2 with minor tweaks (no weight tying, varying datasets, etc) so that I can feel that I have my feet solidly on the ground before building up :-)
I had not really considered trying to reduce the size of the embedding layers because it I felt it would become less of an issue in larger models -- eg. with GPT-2 XL, with 1.64B params, they're only using up 80.41M -- so that's 5% with weight tying, 10% without. But that said, when I was looking at the Qwen 35B MoE (3B active), the problem came right back: it has d_emb of 2,048 and a 248,320 vocab size, so that's about 500M parameters for embeddings (and the same on the output head) -- which means that of their 3B active params per token, only 2B are actually doing our thinking!
What kind of effect on evals (say, test set loss) do you find with your low-rank bottleneck trick?
Cheers,
Giles
Hey Giles,
Spot on observation with sparse MoEs like Qwen. When evaluating active parameter budgets per token, static lookup tables taking up 1B out of 3B active weights creates a massive compute imbalance.
On held-out evaluation and test loss, the effect splits into two regimes:
Iso-architecture (shrinking embedding size without adding layers):
At rank 128 on a 50k vocabulary, the penalty on validation cross-entropy is negligible (typically within +0.02 to +0.04 loss, or <0.5 perplexity delta). Individual tokens without context naturally occupy a low-dimensional intrinsic manifold, so 768 full-rank dimensions for static input representations are mostly redundant.
Iso-parameter (reallocating the recovered weights into depth):
This is where the net gain shows up. Reinvesting that ~64M parameter budget into 6 to 9 additional transformer layers consistently improves validation loss and downstream reasoning benchmarks. 18 to 21 layers of contextual attention and FFN non-linearities store and process far more relational structure than static lookup vectors can.
The main failure mode to watch for during training is early gradient spikes: scaling the bottleneck projection initialization to 1 / sqrt(r) keeps variance uniform and ensures training dynamics match the full-rank baseline from step 1.
Did you notice any gradient variance differences between your tied and untied GPT-2 runs during the first few thousand steps?
Best,
Andrew
Wow, the iso-architecture number really surprises me -- I'd expect a much bigger loss from that. The iso-parameter number surprises me less, but only in the light of the amazing iso-architecture number.
Would you mind if I tried a repro? I'm doing some experiments into dataset curation at the moment (basically trying to repro GPT-2 small quality with essentially the same architecture) but am looking for an experiment to follow up. Happy to collaborate/share results/give you early access to code and so on if you'd like.
From my logs, it looks like I didn't track gradient spikes when I added weight-tying to my model, which looks like an error on my side. I basically just started with no weight tying (following Raschka's "Build an LLM (from Scratch)" and then introduced it for a single training run to see what would happen).
Cheers,
Giles
Hey Giles,
A repro would be fantastic! Absolutely go for it. Very happy to collaborate, compare runs, and look through code or logs whenever you're ready.
If you set up the A/B run against your GPT-2 baseline, two quick implementation details make the reproduction clean and stable:
Bottleneck setup: For untied weights, route input token IDs through Embedding(50257, 128) followed by Linear(128, 768, bias=False). For the LM head, mirror it with Linear(768, 128, bias=False) into Linear(128, 50257, bias=False).
Initialization: Keep standard GPT-2 init on the embedding table, but scale the inner linear projection matrices by std = 1.0 / math.sqrt(128). Without that scaling, the rank collapse during the first 500-1000 warmup steps can artificially inflate early loss.
Whenever you spin up the repo or run logs, feel free to link them or ping me. Excited to see how the numbers look on your dataset curation runs!
Best,
Andrew
Thanks! BTW was thinking this dataset thread was an eccentric place to be discussing this and considering suggesting switching to email, but public > private. Any thoughts on a better forum?
Also: naive intuition is telling me that low rank embeddings (LoRE?) might be less negative on the input side than the output head. Have you tried switching one but not the other? It would be a really interesting ablation study.
Hey Giles,
Your intuition on the output head runs straight into the softmax bottlenek.
On the input side, a rank-128 projection just supplies an initial coordinate before the first block expands and contextualizes it, so the rank cap is largely harmless. The output head is where the constraint bites: logit matrix rank is bounded by min(d, r). Bottlenecking the LM head directly caps the rank of the predicted next-token probability distriution, which hurts entropy on complex predictions.
Testing an asymmetric setup (rank-128 input with a full-rank output head, or r_in = 128 / r_out = 256) is a clean ablation. In untied setups, factorizing just the input still recovers half the embedding tax (~38M params) with zero rank bottleneck on output logits.
To keep this public without clogging your dataset repo, we can move over to the Maba discussions tab:
https://huggingface.co/AndrewThompson1233/maba-v1-architecture/discussions
I can open a thread there on factorized embedding ablations if that works for you.
Best,
Andrew
Yes, good idea, let's move over there -- but in the meantime, I've got the code ready. I'm actually not using the full GPT-2 init setup, just PyTorch defaults, but I think that these changes replicate what you're describing:
https://github.com/gpjt/ddp-base-model-from-scratch/commit/35de747f29c82161f6857a79c296cdbf1f5286a7
The set up is that I now have an optional "lore" config key (low-rank embeddings). That allows input embeddings and the output heads to be switched into a mode where they use your technique. There is an optional "smart_initialise" (later corrected to US English "smart_initialize") that does the initialisation of the projection matrices as you recommend -- I wanted to do a run without that so that I can see stuff go crazy. The rank is also controllable from config.
Does that look solid to you?
Note for anyone else reading: discussion moved here: https://huggingface.co/AndrewThompson1233/maba-v1-architecture/discussions/1