Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
CodeSoft 
posted an update 1 day ago
Post
61
Over the past week or so, I've been working on some models, those released being CodeSoft/sorbet-25m and CodeSoft/sorbet-v2-25m. In general, I'm a little confused because no matter what hyperparameters I change or datasets I add/remove, the benchmarks never move up.

In a recent project, where I attached a TN-gram block to Sorbet-v2-25M, it still stayed the same on benchmarks despite the TN-gram clearly learning (due to the perplexity being lower with the TN-gram attached). When I changed the corpus to favor higher density text (the first paragraphs of Wikipedia articles and synthetic math), the benchmarks either stayed flat or went down.

Does anyone have ideas on what I can do to improve my models? I'd really appreciate any feedback!
KlondikeDev

You could always try Gated DeltaNet2? Thats part of what I’m trying with Boris-2 — it reaches near parity or surpasses Transformers at a small scale, just not recall.

It might also just be reaching the kinda parameter limit, you might have squeezed the most you can out of those few parameters.

I could be completely wrong, I’m just starting out.

·
CodeSoft

Yeah, my main concern with the benchmarks is that Sorbet-v2 is generally trailing other models in the same parameter range. For example, it scores below Supra2 Medium (25M), BananaMind 2 Mini (25M), and Veyra2 Mango (15M) on most of the benchmarks I’ve tested, although it barely passes them on HellaSwag. On ARC Easy, for example, it’s around 15% behind Supra2 Medium.

That said, you might be right that I’m simply hitting the limit of what I can squeeze out of a ~25M-parameter dense Transformer. That’s actually part of why I’m interested in trying more unusual architectures rather than just making the model bigger.
In this post