voperl/maseek-mini

Maseek (mini)

Гибридная LLM, собранная с нуля: MLA (DeepSeek-style attention) + SSM (Mamba-style)

• MoE (SwiGLU-эксперты, top-2 из 8). Претрейн на FineWeb-Edu (~300M токенов).

Блоки: 4x attention/SSM-слоя + 1 MoE-слой, d_model=512, 6 слоёв, контекст 1024.

Итог пре-трейна: val_loss 4.1660 ppl 64.5

temperature 1.0: The history of navigation began at the heart of the American Ocean Ocean. The land was first discovered, but it has been known that when it sank many miles of water, or sank, for several months before NASA to be carried by the Canadian Oceanological Union (ISS), but it is believed that the first and most enduring discovery was not entirely possible by the American spaceflight program as it had previously been. The space industry was still in the past that it had come to be known for its own purposes. For the first time the world became a computer-controlled machine that was being built and built on it. The mission was the first to move on the ship to an orbiting spaceflight. It gave it a chance to continue to develop this spaceflight mission. It was also known that a “spaceflight mission” is also known as an air-controlled, multi-disciplinary endeavor to get in and take control of the astronauts with human-inspired and crew of the USS-A-Lincoln Canal Commission

temperature 0.5: The history of navigation began on 1 September 1885. The first of the first of these was the navigation system on the west coast of the island. The first landing of the island was the Cape Cod, the first of its kind to be used by the North Pacific Ocean. The first landing was made possible by the International Space Station (ESA) and the second one. The second landing was made by the South Pacific Ocean and the first Pacific Ocean. It was the first time the first human-space satellite to orbit. The second one was the first one, the second one, the second one, and the second one. The second one was the second one, which was the first time the first one, and the second one. The second one was the second one, and the second one, and the second one, and the second one, which was the second one, and the second one. The second one was the second one, and the third one, and the second one, and

temperature 1.5: The history of navigation began to support naval vessels. But no one would expect such advances until there was an opportunity to explore. A new NASA exploration company plans to construct bridges and build islands and construct maps in space. What started until mid-century, however, the plan never came out. "After the plan was the first land-locked railroad in North America ... a massive island.” (Distant north). That line of work (Dense) in 1843 created on-board by Joseph Tasego. Though Tasego became one of New Orleans merchants, it still had long believed that the first permanent settlement was conducted entirely to meet the plan for ships. The final voyage reached about 8 miles for two thousand dollars per acre, of approximately $800 miles a year. Even by the end of 1863, New Orleans came out of New Orleans: More so than six hundred pounds, two thousand men, had sunk. They arrived to the river; two million men lived under heavy losses. Most of these

Загрузка весов: см. modeling_maseek.py в репозитории (кастомная архитектура).