tinygemma-10m

Model Overview

tinygemma-10m is a lightweight causal language model with approximately 13.3 million parameters. It was trained from scratch on the TinyStories dataset, a collection of synthetically generated short narratives using a limited vocabulary that resembles the language comprehension level of a young child.

This is an experimental project aimed at studying the lower bounds of model scale while retaining coherent English generation. It serves as a minimal working example of a transformer-based language model, suitable for educational exploration and lightweight prototyping.

Intended Use

• Educational research on small-scale language models.
• Lightweight text generation for simple prompts, particularly story-like continuations.
• Testing and benchmarking on resource-constrained hardware.

Limitations and Risks

• The model has a vocabulary size of 8192 and a context window of 512 tokens, which restricts its ability to handle complex or technical language.
• Outputs are generated based on statistical patterns and may contain repetitions, factual inaccuracies, or nonsensical content.
• No safety filtering was applied during training. Users should review outputs before using them in any public or sensitive environment.
• Being an experimental model, it is not intended for production use cases or tasks requiring reliable factual knowledge.

Quick Start

Install the required library:

pip install transformers

Load and use the model in Python:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "roshan-soni/tinygemma-10m"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "Once upon a time, there was a small fox named Fin."
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=80)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

· Dataset: roneneldan/TinyStories · Architecture: Decoder-only transformer with 12 layers, embedding dimension 256, hidden dimension 1024, 8 attention heads, and grouped-query attention (1 KV group). Sliding window attention (256 tokens) is used alongside full attention. · Vocabulary: 8192 tokens. · Context length: 512 tokens. · Training precision: float32. · License: MIT

Evaluation

The model was qualitatively evaluated on held-out story samples from the TinyStories dataset. It produces grammatically plausible continuations, though outputs may vary in coherence and often reflect the simple stylistic patterns of the training data. Formal metrics such as perplexity can be computed by users depending on their specific evaluation setups.

Environmental Impact

Training was conducted on modest hardware suitable for a model of this size. Specific details regarding hardware type, duration, and carbon emissions are not currently reported. Users are encouraged to estimate emissions using standard calculators such as the Machine Learning Impact calculator (Lacoste et al., 2019) if reproducing the training.

Citation

If you find this model useful, please cite the TinyStories dataset paper:

@misc{eldan2023tinystories,
  title={TinyStories: How Small Can Language Models Be and Still Speak Coherent English?},
  author={Eldan, Ronen and Li, Yuanzhi},
  year={2023},
  eprint={2305.07759},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

Contact

For questions or suggestions, please open an issue on the Hugging Face repository.

License

This model is released under the MIT license.