---
base_model: roneneldan/TinyStories-1Layer-21M
tags:
- generated_from_trainer
datasets:
- roneneldan/TinyStories
metrics:
- accuracy
model-index:
- name: tinystories_1layer_attn_mlp_C10k_k100
  results:
  - task:
      name: Causal Language Modeling
      type: text-generation
    dataset:
      name: roneneldan/TinyStories
      type: roneneldan/TinyStories
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.5429091526514649
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# tinystories_1layer_attn_mlp_C10k_k100

This model is a fine-tuned version of [roneneldan/TinyStories-1Layer-21M](https://huggingface.co/roneneldan/TinyStories-1Layer-21M) on the roneneldan/TinyStories dataset.
It achieves the following results on the evaluation set:
- Loss: 1.8957
- Accuracy: 0.5429
- Multicode K: 1
- Dead Code Fraction/layer0: 0.0
- Mse/layer0: 611.1572
- Input Norm/layer0: 31.9975
- Output Norm/layer0: 15.0872

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 6
- total_train_batch_size: 48
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.05
- training_steps: 10000

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Accuracy | Multicode K | Dead Code Fraction/layer0 | Mse/layer0 | Input Norm/layer0 | Output Norm/layer0 |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|:-----------:|:-------------------------:|:----------:|:-----------------:|:------------------:|
| 2.5072        | 0.05  | 500   | 2.4764          | 0.4579   | 1           | 0.0                       | 841.1602   | 31.9977           | 4.9114             |
| 2.2285        | 0.1   | 1000  | 2.2265          | 0.4926   | 1           | 0.0                       | 792.3023   | 31.9980           | 7.5524             |
| 2.1472        | 0.16  | 1500  | 2.1584          | 0.5025   | 1           | 0.0                       | 761.8683   | 31.9980           | 8.9239             |
| 2.1144        | 0.21  | 2000  | 2.1128          | 0.5090   | 1           | 0.0                       | 737.1843   | 31.9979           | 9.8992             |
| 2.0847        | 0.26  | 2500  | 2.0791          | 0.5142   | 1           | 0.0                       | 716.9390   | 31.9979           | 10.6577            |
| 2.0439        | 0.31  | 3000  | 2.0482          | 0.5185   | 1           | 0.0                       | 698.7266   | 31.9979           | 11.3599            |
| 2.0263        | 0.37  | 3500  | 2.0253          | 0.5224   | 1           | 0.0                       | 682.2680   | 31.9979           | 12.0105            |
| 1.9906        | 0.42  | 4000  | 2.0066          | 0.5253   | 1           | 0.0                       | 669.1965   | 31.9979           | 12.5568            |
| 1.9852        | 0.47  | 4500  | 1.9898          | 0.5279   | 1           | 0.0                       | 657.5872   | 31.9979           | 13.0526            |
| 1.9687        | 0.52  | 5000  | 1.9757          | 0.5300   | 1           | 0.0                       | 648.2462   | 31.9979           | 13.4496            |
| 1.9672        | 0.57  | 5500  | 1.9620          | 0.5321   | 1           | 0.0                       | 640.0822   | 31.9978           | 13.8078            |
| 1.9441        | 0.63  | 6000  | 1.9513          | 0.5339   | 1           | 0.0                       | 633.8831   | 31.9978           | 14.1018            |
| 1.9408        | 0.68  | 6500  | 1.9397          | 0.5358   | 1           | 0.0                       | 628.0929   | 31.9977           | 14.3550            |
| 1.9256        | 0.73  | 7000  | 1.9302          | 0.5374   | 1           | 0.0                       | 623.2726   | 31.9977           | 14.5534            |
| 1.9204        | 0.78  | 7500  | 1.9225          | 0.5381   | 1           | 0.0                       | 619.4573   | 31.9977           | 14.7258            |
| 1.907         | 0.84  | 8000  | 1.9150          | 0.5393   | 1           | 0.0                       | 616.4379   | 31.9976           | 14.8625            |
| 1.8931        | 0.89  | 8500  | 1.9076          | 0.5408   | 1           | 0.0                       | 613.7874   | 31.9976           | 14.9685            |
| 1.9021        | 0.94  | 9000  | 1.9021          | 0.5417   | 1           | 0.0                       | 612.0126   | 31.9975           | 15.0379            |
| 1.8967        | 0.99  | 9500  | 1.8970          | 0.5426   | 1           | 0.0                       | 610.6121   | 31.9975           | 15.0932            |
| 1.8942        | 1.04  | 10000 | 1.8957          | 0.5429   | 1           | 0.0                       | 611.1572   | 31.9975           | 15.0872            |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.0+cu121
- Datasets 2.16.1
- Tokenizers 0.15.1