DenseTransformer-152M
Model Description
DenseTransformer-152M is a from-scratch pretrained decoder-only small language model with approximately 152 million parameters.
Its architecture is inspired by GPT-2 but includes several modern improvements:
- 16 Transformer blocks
- d_model = 768
- 12 attention heads
- hidden_dim = 2048
- SwiGLU feed-forward network
- RMSNorm normalization
- Weight tying: input embedding and output head share weights
- Causal self-attention: implemented with PyTorch SDPA, which automatically enables optimized backends
- No explicit positional encoding: relies only on the causal mask
- Vocabulary: GPT-2 tokenizer, 50,257 tokens
The model was trained on a mixture of English Wikipedia and OpenWebText using standard autoregressive language modeling.
Intended Uses
Direct Use
- English text continuation
- Wikipedia / web-style paragraph generation
- Small language model research
- Teaching and experimentation
- As a base model for downstream fine-tuning
Out-of-Scope Use
- Factual question answering: The model is small and has very poor factual accuracy; it will fabricate information.
- Conversational assistant: It has not been instruction-tuned and cannot follow instructions.
- Safety-critical scenarios: Medical, legal, financial, psychological counseling, etc.
- Production environments: It has not been safety-aligned and may generate biased or harmful content.
- Long-context tasks: The effective context length is only 256 tokens; anything beyond that is discarded.
Training Data
The model was trained on the following English data:
| Source | Tokens (approx.) | Notes |
|---|---|---|
| Wikipedia | 25M | English Wikipedia, filtered for obvious harmful content |
| OpenWebText | 300M | Reddit highly-upvoted outbound web pages, basic filtering applied |
| Total | ~325M | Mix ratio: Wikipedia 25% / OpenWebText 75% |
Basic cleaning was applied: deduplication, length filtering, and filtering of pornographic/hateful/child sexualization content.
No comprehensive safety filtering was performed. OpenWebText may still contain ads, garbled text, and biased content.
Training Procedure
- Objective: Autoregressive language modeling, cross-entropy loss
- Optimizer: AdamW (β1=0.9, β2=0.95, weight_decay=0.1)
- Learning rate schedule: Cosine decay with warmup
- Max learning rate: 3e-4
- Warmup: 1000 steps
- Batch size: 24
- Sequence length: 256
- Precision: FP16 mixed precision (AMP)
- Gradient clipping: 1.0
- Training steps: ~28,000 (still training)
- Final validation loss: ~4.42 (weighted Wiki/OWT)
- Hardware: NVIDIA Tesla T4
Evaluation
Validation Loss
| Domain | Validation Loss |
|---|---|
| Wikipedia | 4.43 |
| OpenWebText | 4.42 |
| Weighted Average | 4.42 |
Generation Samples
Given the prompt "Paris is the capital of", the model may generate:
Paris is the capital of the city, and the city is home to the city of St. John's Church.
The city is also home to a number of other religious institutions...
Note: The grammar is mostly correct, but the facts are wrong (Paris is not a U.S. city). This is typical behavior for a small model.
Limitations
- Very poor factual accuracy: The model only does statistical continuation; it has no knowledge graph and will confidently fabricate.
- Weak long-range dependencies: Effective context is 256 tokens; longer generation drifts and repeats.
- No conversational ability: Not instruction-tuned; cannot answer questions.
- No safety alignment: May generate biased, offensive, or inappropriate content.
- Bias: Training data contains historical, cultural, gender, and racial biases; the model amplifies them.
- Hallucination: It will invent people, organizations, events, and numbers.
- Compute: 152M parameters; inference requires ~0.3 GB (FP16) VRAM.
- Language: English only; Chinese ability is very weak.
Bias, Risks, and Ethical Considerations
- The model may generate gender stereotypes, racial bias, and religious bias.
- It may generate misinformation, conspiracy theories, and harmful advice.
- It is unfiltered and may generate pornographic, violent, or hateful content.
- It should not be used to impersonate others, create fake news, or harass.
- Users are responsible for content moderation.
Usage Restrictions
- No commercial use (license: CC BY-NC-SA 4.0).
- Prohibited for illegal, harmful, deceptive, discriminatory, military, or surveillance purposes.
- Prohibited for generating misinformation or impersonating real people.
- Prohibited for medical, legal, financial, or other professional advice.
- Derivative works must use the same license and credit the data sources.
Citation
@misc{densetransformer152m,
title = {DenseTransformer-152M},
author = {littlewildbee},
year = {2026},
howpublished = {\url{https://huggingface.co/littlewildbee/550d}},
note = {Small decoder-only language model pretrained from scratch}
}