Hugging Face's logo
Activity Feed

AI & ML interests

None defined yet.
FlameF0X 
posted an update 2 days ago
view post
Post
2899
Hello HuggingFace! (UPDATE)

I tested the current architect of FWKV/Myosotis-1-base (that beinng FWKV) @ different sized and sequence lengths among RWKV and Transformer architecture. I did not include Mamba since that would require a costume kernel.

Note: the evaluation might not be accurate.
CSV avalible @ FlameF0X/evals
  • 8 replies
·
FlameF0X 
posted an update 4 days ago
FlameF0X 
posted an update 8 days ago
view post
Post
2619
Hello HuggingFace!

I would like to share a small preview of a language model that I have been experiencing for a while. In the first attached image there is a small sample of the Myosotis-1, an attempt to make a small 100m parameter flagship model that is built on my bizarre architecture that is somewhat similar to an S4/S5 model with WKV added. I call it FWKV (Feed-Forward WKV). The model is currently still in training because of the nature of RNN-like models. It can also be seen that the model has insane prompt processing and token generation speed (evaluation done on a 2x Titan XP); even for its small size, some similar Transformer models do struggle to get the same results without custom kernels (some Transformer models can achieve this level of throughput on cheap hardware).

In the second image, you can see checkpoint 20k of the model in its next token prediction state (this means it can't chat), ranking in the top 100 on AxiomicLabs/Open_SLM_Leaderboard (the results have not been submitted since the model is not done training).

- Why not just use Transformers?
Have you seen any pure non-Transformers SLMs besides RWKV and Mamba?

- Should you expect this project to become the next LFM or another very fast language model thing on some Raspberry Pi?
No, the model is still an experiment; it's very sensible and prone to collapse (by the time of this post, it can be seen in the 1st image).

- Should you use it?
Maybe not yet; the architecture itself is still very "naive"—that's how I could call it at its current level. If you just want to play with it and see what you could do or how fast the model is on your hardware, then you can do it.

Once the training is finished and I feel satisfied with the model next token prediction (the base model) and "assistants" (the instruction-tuned model) capabilities, I will make open weights at
FWKV with full support of the 🤗 Transformers.
  • 10 replies
·
FlameF0X 
posted an update 2 months ago
view post
Post
327
Hello, people of Hugging Face!

I recently released FlameF0X/TinyMoE-100m-2x8-retrained, a small Mixture of Experts language model trained on the Smollm-Corpus. Built on top of the Mixtral architecture, it’s fully compatible with 🤗 Transformers right out of the box!

The model can produce somewhat coherent text on its own, and for some reason, it generates even more coherent responses when given a ChatLM template.

I’m excited to see what you all come up with, and feel free to fine-tune it if you’d like. In the meantime, I’ll be working on developing the chat-trained version.

Demo: FlameF0X/TinyMoE-Playground
Collection: https://huggingface.co/collections/FlameF0X/tinymoe
FlameF0X 
posted an update 3 months ago
view post
Post
453

My models on the Intel Low-Bit LLM Leaderboard

Figured I'd share where my quantized models landed on Intel/low_bit_open_llm_leaderboard since I hadn't posted about it yet.

FlameF0X/Qwen3-4B-Distilled-Claude-4.6 (NVFP4 and MXFP4) sit at ranks 23 and 24 with 62.68% and 61.18% average, right below the base Qwen3-4B. Not bad considering they were distilled from Claude 4.6 rather than trained from scratch.

FlameF0X/LFM2.5-1.2B-Distilled-Claude-4.6 and FlameF0X/LFM2.5-1.2B-Thinking-CodeX land around rank 47-49, competitive with MiniCPM5-1B and the Qwen3 sub-1B models despite being a larger base architecture.

The funny one is FlameF0X/Qwen2-0.2B-pt and FlameF0X/Qwen2-0.2B-it. They're not properly trained — genuinely undertrained, basically undefined — and they still beat openai/gpt-oss-20b at rank 66. The 20B model. Not sure what that says but it's something.

FlameF0X/LFM2-Research is at the bottom of my lineup but it's a research artifact, not meant to be competitive.

Chart below showing my models vs nearby competitors, with size vs performance on the left.

Chart made by Claude
  • 1 reply
·
FlameF0X 
posted an update 3 months ago
FlameF0X 
submitted a paper to Daily Papers 3 months ago
FlameF0X 
updated a model 4 months ago
FlameF0X 
posted an update 4 months ago
view post
Post
288
I did some testing on the scalability of FWKV. It hits a speed bottleneck at 1B due to the T4’s bandwidth limitations. Theoretically, it should match RWKV’s inference speed if the GPU had more bandwidth. So the 1B size is not accurate.
FlameF0X 
posted an update 4 months ago
view post
Post
278
Greetings Hugging Face!

I started a new project called **FWKV** (Feed-forward Weighted Key Value, or Floored Weighted Key Value), a RWKV-style LM that uses FFNNs (Feed-Forward Neural Networks) instead of RNN and floor(W·K·V). I'm hoping to make it much more efficient and scalable than RWKV.

So far I have:

- https://huggingface.co/FlameF0X/FWKV-29M — this one is undertrained and doesn't have a Space yet. In the attached image you can see its speed on a T4 compared to models with the same configuration.

The only model that's fully working right now is:
- https://huggingface.co/FlameF0X/FWKV-TinyStories — trained on TinyStories for one epoch. The demo Space is FlameF0X/FWKV-demo.
  • 2 replies
·
FlameF0X 
updated a dataset 4 months ago
FlameF0X 
updated a Space 4 months ago
FlameF0X 
published a Space 4 months ago
FlameF0X 
updated a model 4 months ago
FlameF0X 
published a model 4 months ago
FlameF0X 
published a dataset 4 months ago
FlameF0X 
posted an update about 1 year ago
view post
Post
4377
I am very sad to say that the budget in creating of SnowflakeCore-G1 1b and 7b MoE models ran out and I can't pre-train them anymore.
  • 7 replies
·
FlameF0X 
posted an update about 1 year ago
view post
Post
821
the training for SnowflakeCore-G1-1B and 7B would be retaken because now I implemented DeepSpeed and management to use two gpus.
FlameF0X 
posted an update about 1 year ago
view post
Post
278
The development of SnowflakeCore-G1-7B-MoE it getting delay. In the mean time I am working on SnowflakeCore-G1-1B-MoE witch would be a pre-train chatbot.
  • 1 reply
·
FlameF0X 
posted an update about 1 year ago
view post
Post
2959
The development of SnowflakeCore-G1-7B-MoE. I can't say when it would be publish yet because it's big and it requires a lot of computational power.
  • 1 reply
·