netsrik/frisky-90m

Frisky 90M

Frisky is a 90,128,896-parameter language model trained from scratch with DeepJ on an Apple GPU. It completed base pretraining, mid-training, and response-only instruction fine-tuning.

Model details

This compact language model was created with DeepJ and uses the DeepJ Prism Transformer architecture with 10 layers, a hidden size of 768, 12 attention heads, Q rank 384, KV rank 192 and a 16,384-token vocabulary.

DeepJ Prism uses low-rank Q/KV attention with rotary embeddings, RMSNorm and SwiGLU. It recalculates the full context for every generated token. The model.dj checkpoint and tokenizer.bpe files use DeepJ's versioned binary formats. They are not directly loadable by the Python Transformers library.

Usage

Use DeepJ 0.9.9-alpha or newer:

<dependency>
    <groupId>io.github.kirstenali</groupId>
    <artifactId>deepj</artifactId>
    <version>0.9.9-alpha</version>
</dependency>

Download this repository's model.dj and tokenizer.bpe into downloaded-model. Imports are omitted below.

DeepJChat chat = DeepJChat.load(Path.of("downloaded-model"));
String answer = chat.generate("What is a computer?");

You can also use DeepJ's desktop chat UI. Run it from the DeepJ repository with JDK 20 or newer:

mvn org.openjfx:javafx-maven-plugin:0.0.8:run

Then choose model.dj. Keep tokenizer.bpe in the same directory. The UI applies the checkpoint's saved prompt format automatically and streams the generated text.

Generation settings

Setting	What it changes
Temperature	Lower values favor likely next tokens; higher values give other choices a better chance.
Top-k	Samples from the top k token choices; 0 considers all.
Max tokens	Limits new tokens, not words.
Seed	Controls sampling randomness; changing it can change the response.
Repetition penalty	Makes the model less likely to repeat itself. 1.0 turns it off; try 1.1 if responses loop.

Training data

Base pretraining used FineWeb-Edu. Mid-training and fine-tuning used mixtures derived from SmolTalk, MMLU, GSM8K, and ARC. The MIT license on this repository applies to the model weights, not to the upstream datasets; consult each dataset's terms.

Limitations

Frisky is an experimental small model. It can repeat itself, invent facts, make reasoning errors, and generate incorrect code. Its post-training quality has not yet been established by a rigorous benchmark. Do not use it for consequential decisions.