| ## Pretrained models for the paper *Scaling up Masked Diffusion Models on Text* | |
| **Scaling law experiments**: We provided all pre-trained models in the *ar_safetensors* and *mdm_safetensors* folders. | |
| For instance, the checkpoint `mdm-1028M-1600e18.safetensors` represents an MDM model with 1,028 million non-embedding | |
| parameters and 1,600e18 training FLOPs. Similarly, the checkpoint `mdm-170M-100e18-rsl-0.01.safetensors` indicates | |
| an MDM model with 170 million non-embedding parameters, 100e18 training FLOPs, and 1% of the dataset subjected | |
| to random sequence lengths during pretraining. | |
| **Math reasoning**: please see the *gsm8k_safetensors* folder. | |
| **Conditional generation**: please see the *sharegpt_safetensors* folder. | |
| **Reverse curse**: please see the *reverse_safetensors* folder | |
| For all models, we provide models in `.pth` and `.safetensors` formats. | |