Mitigating chat overfitting on small corpora via block recycling and RoPE
Hi Plimb,
Writing a clean, standalone decoder-only transformer from scratch in PyTorch without relying on external library wrappers is a great engineering exercise.
Looking at your training dynamics and the plateau note in the model card:
Your combined training corpora (Alpaca, Dolly-15k, OASST, and 20k UltraChat) total roughly 15M-25M tokens.
Training a dense 300M model (24 layers, 1024 width) from scratch on ~20M tokens creates extreme parameter-to-data starvation (Chinchilla optimal scaling expects ~6B tokens for 300M weights). The network quickly exhausts unique gradient signals and overfits.
Additionally, learned positional embeddings (nn.Embedding) lock the context strictly to 256 tokens, which fills up after 1-2 conversational turns.
In an open architecture project called Maba (101M reference model: https://huggingface.co/AndrewThompson1233/maba-v1-architecture), we handle parameter efficiency on constrained training sets with two mechanisms:
Deterministic 2-pass block recycling:
Instead of instantiating 24 independent physical layers, train 12 physical blocks and pass representations through them twice with pass conditioning (separate RMSNorm scale vectors for pass 1 and pass 2). This cuts your active weight count roughly in half (~150M params) to directly curb overfitting, while preserving the full 24-layer computational depth needed for multi-step chat responses.RoPE instead of learned positional tables:
Switching from nn.Embedding to Rotary Position Embeddings (RoPE) costs zero parameters and removes the hard 256-token ceiling, allowing the model to extend context dynamically without retraining position tables.
Did you consider reducing physical layer depth or testing weight-tied recurrent passes to match the parameter scale to the size of your instruction corpus?
Best,
Andrew
Hello Andrew,
Thank you so much for your feedback and for all these detailed explanations!
I’d like to clarify that Gladios 0.3B is first and foremost a prototype and an experiment to test different ideas. I’m currently the only one working on this project, and a large part of this prototype was created with the help of AI, particularly for the architecture and the code.
I hadn’t yet considered weight-tied block recycling or RoPE in this way. Your explanations about the relationship between the number of parameters and the amount of data are helping me better understand the model’s current limitations.
So I’m going to look into your suggestions and try testing an architecture with fewer parameters as well as RoPE, then compare the results with the current version.
Thanks again for taking the time to analyze the project and give me such specific advice. This is exactly the kind of feedback that can help me improve the prototype and better understand what I’m doing.
Thanks!
Hi Rusher,
Glad the breakdown was helpful!
Since you are writing clean, standalone PyTorch modules without heavy framework wrappers, implementing RoPE and 2-pass recycling is very straightforward:
A 2-pass loop just wraps your block iteration twice with an outer loop: x = block(x, pass_idx=p).
To stabilize it, you only need two separate RMSNorm scale vectors per block (one for pass 0, one for pass 1) so the network easily distinguishes context ingestion from refinement.
If you run into any questions while implementing RoPE or the pass-conditioning logic in pure PyTorch, feel free to ping here or check out the Maba reference repository.
Looking forward to seeing how the next iteration turns out!
Best,
Andrew
To tell you the truth, I’ve only just started to get a basic sense of how it all works, and I’m still having a hard time understanding everything. It’s all pretty new to me, and right now, AI basically does everything for me. But I’m so fascinated by it that I spend hours researching everything related to AI, how it works, and how it’s actually built.
Like I said, though, it’s really complex, and I’m still in my final years of high school, so I’m basically just learning the ropes XD. And honestly, I’m not really sure where to start.
Hi Rusher,
Honestly, training a 300M parameter model with a custom BPE tokenizer from scratch while still in high school is huge. Most people never touch raw PyTorch tensors or loss curves, so you are already way ahead!
If everything feels overwhelming right now, that is completely normal. ML has a steep learning curve at first. A few practical tips that make learning ten times easier:
Scale down to learn faster:
Training 300M models takes hours and makes debugging hard. Try scaling down to a tiny 10M-15M model on a small, clean dataset (like a slice of TinyStories or Shakespeare). It trains in 15-20 minutes on a free Colab GPU, so you can immediately see the effect when you tweak an attention head or swap an embedding layer.Watch Andrej Karpathy's "Zero to Hero":
If you haven't seen his "Let's build GPT: from scratch, in code" video on YouTube, start there. He builds a clean transformer character-by-character in pure PyTorch and explains every single tensor shape step by step.Inspect minimal architectures:
Don't worry about understanding giant library frameworks like Transformers right away. Look at clean, standalone implementations like nanoGPT or the modeling code in Maba - we intentionally kept Maba's PyTorch files minimal and readable so anyone can follow how the layers, RoPE, and block passes connect.
Keep experimenting and breaking things in code - that is literally how everyone learns. Feel free to ping anytime you run into questions!
Best,
Andrew
Thank you so much for your advice! I’m thinking of working on at least two projects at the same time: one where I can use AI to help me, and another where I’ll try to learn and build things without AI.
Thanks again for the advice, I really appreciate it!
Do you have a Discord account so we can maybe add each other if you'd like?