TheDrummer/Behemoth-123B-v1.1
Updated
•
1.01k
•
18
Arthur
secretmoon
AI & ML interests
None yet
Recent Activity
liked
a model
15 days ago
TheDrummer/Behemoth-123B-v1.1
liked
a model
about 1 month ago
anthracite-org/magnum-v3-27b-kto
Organizations
secretmoon's activity
liked
a
model
15 days ago
liked
a
model
about 1 month ago
liked
3
models
2 months ago
liked
4
models
3 months ago
updated
a
model
3 months ago
liked
2
models
3 months ago
liked
4
models
4 months ago
liked
a
model
5 months ago
Reacted to
yushun0410's
post with 🚀
5 months ago
Post
4609
Hi Huggingfacers!
Thrilled to introduce Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint. Adam-mini can also achieve 49.5% higher throughput than AdamW on Llama2-7B pre-training.
The design of Adam-mini is inspired by certain Hessian structures we observed on Transformers.
Feel free to try it out! Try switching to Adam-mini with the same hyperparams of AdamW, it would work with only half memory. Hope Adam-mini can help save time, cost, and energy in your tasks!
Paper: "Adam-mini: Use Fewer Learning Rates To Gain More" https://arxiv.org/abs/2406.16793
Code: https://github.com/zyushun/Adam-mini
Thrilled to introduce Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint. Adam-mini can also achieve 49.5% higher throughput than AdamW on Llama2-7B pre-training.
The design of Adam-mini is inspired by certain Hessian structures we observed on Transformers.
Feel free to try it out! Try switching to Adam-mini with the same hyperparams of AdamW, it would work with only half memory. Hope Adam-mini can help save time, cost, and energy in your tasks!
Paper: "Adam-mini: Use Fewer Learning Rates To Gain More" https://arxiv.org/abs/2406.16793
Code: https://github.com/zyushun/Adam-mini
- 1 reply
liked
a
model
5 months ago
liked
a
model
6 months ago