Update README.md

README.md

@@ -1,3 +1,76 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- allenai/tulu-3-sft-mixture
+- allenai/llama-3.1-tulu-3-8b-preference-mixture
+language:
+- en
+base_model:
+- HuggingFaceTB/SmolLM2-1.7B-Instruct
+---
+# SmolLM2-MedIT-Upscale-2B
+
+**Model Summary**
+
+SmolLM2-MedIT-Upscale-2B is an expanded version of the [SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct) model, increasing its parameter count to 2 billion. This expansion was achieved by doubling the number of heads in the `q_proj`, `k_proj`, `v_proj`, and `o_proj` layers, resulting in vectors of length 4096, compared to 2048 in the original model.
+
+**Purpose of Expansion**
+
+This model was developed to test the hypothesis that self-attention layers do not extend the "memory" of the model. By broadening the attention layers, we aim to observe the impact on the model's performance and memory capabilities.
+
+**Training Status**
+
+The model has undergone preliminary training focused on assessing the effects of the expanded attention layers. It is not fully trained to its maximum potential. We encourage the community to contribute to its further training; pull requests are welcome.
+
+**Analysis of Expanded Layers**
+
+During fine-tuning, we analyzed the changes in the new parameters of the expanded layers:
+
+- **Minimum percentage of new parameters that changed:**
+  - `q_proj`: 62.17%
+  - `k_proj`: 37.85%
+  - `v_proj`: 99.99%
+  - `o_proj`: 99.98%
+
+- **Maximum percentage of new parameters that changed:**
+  - `q_proj`: 98.86%
+  - `k_proj`: 97.99%
+  - `v_proj`: 99.99%
+  - `o_proj`: 99.99%
+
+- **Average change in new parameters after fine-tuning:**
+  - `q_proj`: 1.838e-07
+  - `k_proj`: 2.277e-07
+  - `v_proj`: 6.490e-07
+  - `o_proj`: 3.924e-07
+
+These results are illustrated in the following charts:
+
+![Percentage of Change](percent_of_change.png)
+![Average Parameter Change](mean_difference.png)
+
+**Usage**
+
+To utilize this model, follow the instructions provided for the original SmolLM2-1.7B-Instruct model, adjusting for the increased parameter size.
+
+**Contributing**
+
+We welcome contributions to further train and evaluate this model. Please submit pull requests with your improvements.
+
+**License**
+
+This model is licensed under the Apache 2.0 License.
+
+**Citation**
+
+If you use this model in your research, please cite it as follows:
+
+```
+@misc{SmolLM2-MedIT-Upscale-2B,
+  author = {Mariusz Kurman, MedIT Solutions},
+  title = {SmolLM2-MedIT-Upscale-2B: An Expanded Version of SmolLM2-1.7B-Instruct},
+  year = {2024},
+  publisher = {Hugging Face},
+  url = {https://huggingface.co/YourUsername/SmolLM2-MedIT-Upscale-2B},
+}
+```