vtava
/

TinyCeNN-LM-Distilled

Text Generation

knowledge-distillation

transformer-free

language-modeling

recurrent-neural-network

Model card Files Files and versions

TinyCeNN-LM Distilled

Transformer-free CeNN student distilled from arnir0/Tiny-LLM.

Transformer layers remaining: 0
CeNN recurrent steps: 7
CeNN receptive field: 255 tokens
Training tokens: 10,002,432
Best student CE: 5.247066
Teacher CE: 4.423090
Teacher-gap recovery: 84.92%

Load with the TinyCeNN-LM GitHub package via build_cenn_student().

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for vtava/TinyCeNN-LM-Distilled

Base model

arnir0/Tiny-LLM

Finetuned

(9)

this model