netsrik/deepj-tinystories-deepseek

DeepJ TinyStories DeepSeek-style

An experimental 1,301,120-parameter causal language model trained with DeepJ. On 100 deterministic TinyStories validation windows (12800 tokens), loss was 2.182273 and perplexity was 8.866.

Model details

This is a compact DeepJ DeepSeek-style model with 4 layers, width 128, 4 attention heads, Q rank 64, KV rank 32, and a 2048-token vocabulary. It is not an exact DeepSeek-V2/V3/R1 implementation and has no incremental KV cache. The model.dj checkpoint and tokenizer.bpe files use DeepJ's versioned binary formats. They are not directly loadable by the Python Transformers library.

Usage

Use DeepJ 0.6.0-alpha from Maven Central, or a later format-compatible release.

<dependency>
    <groupId>io.github.kirstenali</groupId>
    <artifactId>deepj</artifactId>
    <version>0.6.0-alpha</version>
</dependency>

Download this repository's model.dj and tokenizer.bpe into downloaded-model. Imports are omitted below.

Path directory = Path.of("downloaded-model");
BPEModel bpe = BPEModelIO.load(directory.resolve("tokenizer.bpe"));
BPETokenizer tokenizer = new BPETokenizer(bpe);
DeepSeekConfig config = new DeepSeekConfig(
        2048, 128, 128, 4, 4, 384, 64, 32, 0.2f, 1.0f);
DeepSeekModel model = new DeepSeekModel(config, 42L);
model.load(directory.resolve("model.dj"));
String text = TextGenerator.generate(model, tokenizer, config,
        "Once upon a time", 80, 0.8f, 40, 2026L);

Training data

Trained for 10000 steps with batch size 1 and 128-token sequences on TinyStories, a synthetic English dataset licensed CDLA-Sharing-1.0. The BPE vocabulary was trained from a bounded sample of the training split.

Limitations

Small experimental model trained only on synthetic children's stories. It may produce incorrect, repetitive, biased, or unsuitable text. It has not been evaluated for safety or downstream use and is not a general-purpose assistant.