diffutron
/

DiffutronLM-0.3B-1st-Stage

@@ -11,200 +11,79 @@ language:
 base_model:
 - diffutron/DiffutronLM-0.3B-Base
 pipeline_tag: text-generation
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 base_model:
 - diffutron/DiffutronLM-0.3B-Base
 pipeline_tag: text-generation
+new_version: diffutron/DiffutronLM-0.3B-Instruct
 ---
+# DiffutronLM-0.3B-1st-Stage
+**DiffutronLM-0.3B-1st-Stage** is an intermediate checkpoint of the Diffutron series, a parameter-efficient, Masked Diffusion Language Model (MDLM) designed for the Turkish language.
+This specific model represents the completion of the **first stage of instruction fine-tuning**. It has been trained to grasp the fundamentals of instruction-following in Turkish, serving as a robust foundation before more complex, domain-specific specialization (which is handled in the final `Instruct` model).
+## 📌 Model Details
+* **Model Type:** Masked Diffusion Language Model (MDLM)
+* **Base Architecture:** `jhu-clsp/mmBERT-base` (Multilingual Encoder)
+* **Language:** Turkish
+* **Parameter Count:** 307M (0.3B)
+* **Context Length:** 256 tokens
+* **Training Libraries:** `dllm`, PyTorch
+* **Status:** Intermediate Checkpoint (Stage 1 SFT)
+## 🚀 Training Pipeline for This Checkpoint
+Diffutron replaces traditional next-token autoregressive generation with a discrete diffusion process, generating text by iteratively refining sequences in parallel. To reach this checkpoint, the model underwent two main phases:
+### 1. Continual Pre-training (CPT)
+The multilingual backbone was adapted to Turkish using a high-rank LoRA strategy (r=256, α=256) on ~2 million sequences sourced from Havadis, Temiz-OSCAR, and Turkish Wikipedia. This effectively modeled Turkish morphological nuances without catastrophic forgetting.
+### 2. Stage 1: Foundational Instruction Tuning
+Following CPT, the model underwent full supervised fine-tuning (SFT) to align it with human intent.
+* **Dataset:** `metunlp/LlamaTurk-Instruction-Set`
+* **Objective:** Introduce the model to a broad range of general instructions and establish basic response coherence.
+* **Hyperparameters:** 20 Epochs, Batch Size 16, AdamW optimizer (lr=1e-4), Max Sequence Length 256.
+*(Note: For the most advanced instruction-following capabilities, including complex reasoning, we recommend using the final `DiffutronLM-0.3B-Instruct` model, which includes a second stage of tuning on `InstrucTurca`.)*
+## 📊 Evaluation Results
+Despite being an intermediate checkpoint, the 1st-Stage model demonstrates highly competitive performance against much larger autoregressive baselines on the **CETVEL Benchmark Suite**.
+| Benchmark | **Diffutron-1st (0.3B)-Stage** | Diffutron-2nd-Stage (0.3B) | TURNA (1.1B) | Kumru (2B) | Kanarya (2B) | Llama-3.2 (3B) | Trendyol (7B) | Aya-101 (13B) |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| **Belebele_TR** | **22.22** | 27.00 | 22.56 | 29.00 | 28.11 | 55.78 | 36.22 | 22.89 |
+| **EXAMS_TR** | **25.95** | 27.74 | 23.66 | 30.03 | 30.03 | 26.21 | 28.50 | 22.90 |
+| **IronyTR** | **50.67** | 52.00 | 48.33 | 51.00 | 50.00 | 50.17 | 50.00 | 52.17 |
+| **News_Cat** | **23.20** | 32.40 | 32.80 | 26.40 | 66.80 | 64.00 | 81.20 | 20.00 |
+| **MNLI_TR** | **33.29** | 32.81 | 34.94 | 36.42 | 33.40 | 34.76 | 35.19 | 27.90 |
+| **STS_TR** | **17.77** | 18.78 | 14.21 | 11.75 | 12.91 | 12.91 | 15.52 | 16.97 |
+| **XCOPA_TR** | **53.80** | 52.00 | 55.80 | 54.00 | 64.20 | 54.60 | 61.00 | 59.60 |
+| **Average** | **32.41** | 34.68 | 33.19 | 34.09 | 40.78 | 42.63 | 43.95 | 31.78 |
+## 💻 Usage
+Inference requires generating text via a discrete diffusion process rather than causal next-token prediction. We recommend using the `dllm` library.
+**Recommended Generation Parameters:**
+* **Steps:** 64 to 128
+* **Temperature:** 0.1
+* **Block Length:** 32
+* **Repetition Penalty:** 1.2
+* **Remask Strategy:** `low_conf`
+## ⚠️ Limitations
+* **Intermediate State:** This model has not undergone the final specialization phase and may struggle with highly complex or multi-turn instructions compared to the final Instruct model.
+* **Context Window:** Restricted to a 256-token context window.
+* **Multilingual Backbone:** Inherits representations from a multilingual encoder, not a natively trained Turkish foundation model.
+## 📝 Citation
+```bibtex
+@misc{diffutron2026,
+  author = {Kocabay, Şuayp Talha and Akkuş, Talha Rüzgar},
+  title = {Diffutron: A Masked Diffusion Language Model for Turkish Language},
+  year = {2026},
+  publisher = {Hugging Face},
+  howpublished = {\url{[https://huggingface.co/collections/diffutron/diffutronlm](https://huggingface.co/collections/diffutron/diffutronlm)}}
+}
+```