Instructions to use kaushik-harsh-99/TinyStories-17M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kaushik-harsh-99/TinyStories-17M with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kaushik-harsh-99/TinyStories-17M", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("kaushik-harsh-99/TinyStories-17M", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kaushik-harsh-99/TinyStories-17M with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kaushik-harsh-99/TinyStories-17M"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kaushik-harsh-99/TinyStories-17M",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/kaushik-harsh-99/TinyStories-17M

SGLang

How to use kaushik-harsh-99/TinyStories-17M with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kaushik-harsh-99/TinyStories-17M" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kaushik-harsh-99/TinyStories-17M",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kaushik-harsh-99/TinyStories-17M" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kaushik-harsh-99/TinyStories-17M",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use kaushik-harsh-99/TinyStories-17M with Docker Model Runner:
```
docker model run hf.co/kaushik-harsh-99/TinyStories-17M
```

TinyStories-17M

TinyStories-17M is a 17.2 million parameter decoder-only Transformer trained entirely from scratch for children's story generation.

The project explores how far a modern Small Language Model (SLM) can be pushed using high-quality synthetic data, an efficient Transformer architecture, and careful training. Despite its compact size, the model is capable of generating coherent, grammatically correct, multi-paragraph children's stories while following a wide variety of prompts.

References

This project is inspired by the TinyStories paper:

TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

Ronen Eldan, Yuanzhi Li (2023)

Paper: https://arxiv.org/abs/2305.07759

Dataset: https://huggingface.co/datasets/roneneldan/TinyStories

Highlights

Trained from scratch
17.2M parameters
Trained on 2.1 million TinyStories-style synthetic stories
Modern LLaMA-style architecture
Rotary Position Embeddings (RoPE)
RMSNorm
SwiGLU Feed Forward Network
PyTorch Scaled Dot Product Attention (Flash Attention compatible)
Tied Input & Output Embeddings
Fully compatible with Hugging Face Transformers

Model Architecture

Property	Value
Architecture	Decoder-only Transformer
Parameters	17,234,304
Layers	8
Hidden Size	384
Attention Heads	6
Feed Forward Size	1024
Context Length	512
Vocabulary Size	8000
Positional Encoding	Rotary Embeddings (RoPE)
Activation	SwiGLU
Normalization	RMSNorm
Weight Tying	Yes

Training

The model was trained from scratch using next-token prediction on a synthetic TinyStories-style dataset.

Training Configuration

Setting	Value
Optimizer	AdamW
Precision	BF16 Mixed Precision
Learning Rate	3e-4
Weight Decay	0.1
Epochs	8
Batch Size	70
Context Length	512
Vocabulary	8000

Training was performed using PyTorch with Automatic Mixed Precision (AMP) and a cosine learning rate schedule.

Dataset

The model was trained on approximately 2.1 million synthetic English children's stories inspired by the TinyStories dataset.

The dataset focuses on:

Simple English
Short coherent narratives
Moral lessons
Everyday situations
Dialogues
Fantasy adventures
Animal stories
Child-friendly vocabulary

Intended Use

TinyStories-17M is intended for:

Story generation
Small Language Model research
Educational projects
Transformer architecture experiments
Learning how decoder-only LLMs work
Efficient inference research

Evaluation

The model was evaluated using a combination of:

Validation loss during training
Manual inspection of generated stories
LLM-assisted qualitative evaluation using ChatGPT-5

Evaluation focused on:

Grammar
Story coherence
Prompt following
Creativity
Character consistency
Repetition
Overall story quality

Qualitative Assessment

Category	Score
Grammar	9.7 / 10
Story Coherence	8.7 / 10
Character Consistency	8.6 / 10
Creativity	8.1 / 10
Repetition Resistance	8.0 / 10
Overall Story Quality	8.62 / 10

Note: These scores are qualitative assessments based on manual inspection and LLM-assisted evaluation. They are intended to provide an approximate indication of the model's strengths and limitations rather than standardized benchmark results.

Generation Examples

The following examples are actual generations produced by TinyStories-17M.

Example 1

Prompt

Once upon a time

Generation

there was a little girl who was very curious. One day she decided to
explore the woods and take a look around. As she walked, the little
girl heard some strange noises coming from behind some trees.She
stopped and looked around but couldn't see anything. Suddenly, a
voice said, "Help me!" The little girl was scared but she asked,
"Who are you?" The voice replied, "I am a giant. I need to tell you
that you must leave these woods and not come back." The little girl
knew she had to do something...

Example 2

Prompt

Ben discovered an old compass hidden inside his grandfather's attic.

Generation

 He decided to explore the attic, but it was very dark and scary. He was
 scared, but he was determined to find out what was behind the boxes. He
 opened one of the boxes and found a heavy chest. He tried to lift it,
 but it was too heavy. He wondered what could be inside. Suddenly, he
 heard a mysterious voice introducing itself as the spirit of the attic,
 encouraging Ben to continue exploring and discover new adventures...

Example 3

Prompt

A lonely dragon met a tiny mouse for the first time.

Generation

The dragon wanted to play with the mouse, but the mouse was scared and ran
away. Hoping to become friends, the dragon searched for the tiny mouse
throughout the forest. Soon a misunderstanding turned into an exciting
chase, and both of them learned that first impressions are not always true...

Note: The examples above are intentionally truncated for brevity. The model continues generating complete children's stories beyond these excerpts.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained(
    "kaushik-harsh-99/TinyStories-17M",
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "kaushik-harsh-99/TinyStories-17M",
    trust_remote_code=True,
)

prompt = "Once upon a time"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.8,
    top_p=0.95,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Repository Structure

model.safetensors
config.json
generation_config.json

configuration_tiny.py
modeling_tiny.py
__init__.py

tokenizer.json
tokenizer.model
tokenizer_config.json
special_tokens_map.json

README.md
LICENSE

Known Limitations

As a 17M parameter language model, TinyStories-17M has several limitations:

May become repetitive during long generations.
Limited world knowledge.
Occasionally produces repetitive dialogue.
Sometimes struggles with uncommon names or complex entities.
Optimized specifically for children's storytelling rather than general-purpose language tasks.

Future Work

The next version of this project focuses on improving both the dataset and training methodology.

Planned improvements include:

Training on approximately 3.7 million curated stories.
Greater diversity of character names and story settings.
Continued pretraining from the current checkpoint.
Larger effective batch size using gradient accumulation.
Longer training schedules.
Improved tokenizer.
Architecture and efficiency experiments.

Acknowledgements

This project was inspired by:

TinyStories

Special thanks to the open-source AI community for making efficient language model research accessible.

Downloads last month: 293

Safetensors

Model size

17.2M params

Tensor type

F32

Dataset used to train kaushik-harsh-99/TinyStories-17M

Paper for kaushik-harsh-99/TinyStories-17M

TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

Paper • 2305.07759 • Published May 12, 2023 • 47