Update README.md

README.md

@@ -7,17 +7,13 @@ datasets:
 - prince-canuma/fineweb-CC-MAIN-2024-10-1B-en
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
-
-## Model Details
-
-### Model Description
+# Model Summary
 <img src="llama-3-6B icon.jpeg" width="500" alt="Llama-3-6B"/>
 
+This is world's first Llama-3 base model with 6B params, it is a pretrained version of [prince-canuma/Llama-3-6B-v0](https://huggingface.co/prince-canuma/Llama-3-6B-v0) which was, downcycled from Meta-Llama-3-8B.
+It was continually pretrained on 1B tokens of enlish only text from fineweb and achieves the following results on the evaluation set:
+- Loss: 2.4942
+
 <!-- Provide a longer summary of what this model is. -->
 
 This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
@@ -30,61 +26,72 @@ This is the model card of a 🤗 transformers model that has been pushed on the
 - **License:** [More Information Needed]
 - **Finetuned from model [optional]:** [More Information Needed]
 
+## Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [Prince Canuma](https://huggingface.co/prince-canuma)
+- **Model type:** Transformer
+- **License:** MIT
+- **Finetuned from model:** prince-canuma/Llama-3-6B-v0
+
 ### Model Sources [optional]
 
 <!-- Provide the basic links for the model. -->
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** https://github.com/Blaizzy/Coding-LLMs-from-scratch/tree/main/Llama-3
+- **Video [optional]:** https://youtube.com/playlist?list=PLDn_JsyofyfTH5_5V1MNb8UYKxMl6IMNy&si=5Y4cm-6wrMOD1Abr
 
 ## Uses
 
 <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+You can use this model to create instruct and chat versions for various use cases such as: Coding assistant, RAG, Function Calling and more.
 
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+### Limitations 
 
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
+This model inherits some of the base model's limitations and some additional ones from it's creation process, such as:
+ - Limited scope for coding and math: According to benchmarks, this model needs more pretraining/finetuning on code and math data to excel at reasoning tasks.
+ - Language Limitations: This model was continually pretrained on english only data. If you are planning to use it for multilingual use cases I recommend fine-tuning or continued pretraining.
 
-## Bias, Risks, and Limitations
+## How to Get Started with the Model
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+Use the code below to get started with the model.
 
-[More Information Needed]
+```python
+from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer
 
-### Recommendations
+# Load model, config and tokenizer
+model_name = "prince-canuma/Llama-3-6B-v0.1"
+model = AutoModelForCausalLM.from_pretrained(model_name)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+inputs = tokenizer(
+[
+   "Who created Python?"
+], return_tensors = "pt")
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+from transformers import TextStreamer
+text_streamer = TextStreamer(tokenizer)
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 200)
 
-## How to Get Started with the Model
+```
 
-Use the code below to get started with the model.
+Output:
+```shell
+<|begin_of_text|>Who created Python? What is Python used for? What is the difference between Python 2 and Python 3? What is the difference between Python and Python 3?
+Python is a programming language that was created by Guido van Rossum in 1991. It is a widely used language for web development, data science, and machine learning. Python is also used for creating software applications and games.
+Python is a powerful language that is easy to learn and use. It has a large library of built-in functions and packages that make it easy to write code. Python is also a very popular language for web development, with many popular web frameworks such as Django and Flask being written in Python.
+Python is also used for data science and machine learning. It has a large library of packages for data analysis, machine learning, and artificial intelligence. Python is also used for creating software applications and games.
+Python 2 and Python 3 are two different versions of the Python language. Python 2 was the original version of the
+```
 
-[More Information Needed]
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+For continued pretrained, I extracted 1B tokens from [Huggingface's FineWeb CC-Main-2024-10](https://huggingface.co/datasets/HuggingFaceFW/fineweb#breakdown-by-dumpcrawl) slice.
 
 ### Training Procedure
 
@@ -95,15 +102,45 @@ Use the code below to get started with the model.
 [More Information Needed]
 
 
-#### Training Hyperparameters
+#### Training hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+The following hyperparameters were used during training:
+- learning_rate: 0.0002
+- train_batch_size: 2
+- eval_batch_size: 2
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 4
+- gradient_accumulation_steps: 8
+- total_train_batch_size: 64
+- total_eval_batch_size: 8
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 100
+- num_epochs: 2
 
-#### Speeds, Sizes, Times [optional]
+### Training results
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+| Training Loss | Epoch | Step  | Validation Loss |
+|:-------------:|:-----:|:-----:|:---------------:|
+| 7.1562        | 0.0   | 1     | 7.1806          |
+| 2.7339        | 0.25  | 5867  | 2.6266          |
+| 2.6905        | 0.5   | 11734 | 2.5872          |
+| 2.6134        | 0.75  | 17601 | 2.5549          |
+| 2.532         | 1.0   | 23468 | 2.5235          |
+| 2.5319        | 1.25  | 29335 | 2.5067          |
+| 2.3336        | 1.5   | 35202 | 2.4968          |
+| 2.3486        | 1.75  | 41069 | 2.4942          |
+
+
+### Framework versions
+
+- PEFT 0.10.0
+- Transformers 4.40.0.dev0
+- Pytorch 2.2.0+cu121
+- Datasets 2.15.0
+- Tokenizers 0.15.0
 
-[More Information Needed]
 
 ## Evaluation
 
@@ -136,42 +173,12 @@ Use the code below to get started with the model.
 #### Summary
 
 
-
 ## Model Examination [optional]
 
 <!-- Relevant interpretability work for the model goes here -->
 
 [More Information Needed]
 
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
 
 ## Citation [optional]
 
@@ -185,4 +192,84 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
       author={Prince Canuma},
       year={2024},
 }
-```
+```
+
+[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
+<details><summary>See axolotl config</summary>
+
+axolotl version: `0.4.0`
+```yaml
+base_model: prince-canuma/Llama-3-6B-v0.1
+model_type: AutoModelForCausalLM
+tokenizer_type: AutoTokenizer
+
+load_in_8bit: false
+load_in_4bit: true
+strict: false
+
+datasets:
+  - path: prince-canuma/fineweb-CC-MAIN-2024-10-1B-en
+    type: completion
+    split: train
+dataset_prepared_path: last_run_prepared
+val_set_size: 0.001
+output_dir: ./llama-3-6b
+save_safetensors: true
+adapter: qlora
+lora_model_dir:
+
+sequence_len: 8192
+sample_packing: false
+pad_to_sequence_len: false
+
+lora_r: 128
+lora_alpha: 128
+lora_dropout: 0.05
+lora_target_modules:
+lora_target_linear: true
+lora_fan_in_fan_out:
+
+
+wandb_project: llama-3-6b
+wandb_entity: 
+wandb_watch:
+wandb_name:
+wandb_log_model:
+
+gradient_accumulation_steps: 8
+micro_batch_size: 2
+num_epochs: 2
+optimizer: paged_adamw_32bit
+lr_scheduler: cosine
+learning_rate: 2e-4
+
+train_on_inputs: false
+group_by_length: false
+bf16: auto
+fp16:
+tf32: false
+
+gradient_checkpointing: true
+early_stopping_patience:
+resume_from_checkpoint:
+local_rank:
+logging_steps: 1
+xformers_attention:
+flash_attention: true
+
+warmup_steps: 100
+evals_per_epoch: 4
+eval_table_size:
+save_steps: 4000
+debug:
+deepspeed:
+weight_decay: 0.0
+fsdp:
+fsdp_config:
+special_tokens:
+    pad_token: "<|reserved_special_token_0|>"
+
+
+```
+
+</details><br>