|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- cerebras/SlimPajama-627B |
|
- bigcode/starcoderdata |
|
- HuggingFaceH4/ultrachat_200k |
|
- HuggingFaceH4/ultrafeedback_binarized |
|
- dumb-dev/cpp-10k |
|
- dumb-dev/Encoding-Detection-w-cChardet-DB |
|
- Neloy262/rust_instruction_dataset |
|
- m-a-p/CodeFeedback-Filtered-Instruction |
|
- sahil2801/CodeAlpaca-20k |
|
- vicgalle/alpaca-gpt4 |
|
language: |
|
- en |
|
--- |
|
# I finetuned TinyLlama/TinyLlama-1.1B-Chat-v1.0 on the following datasets: |
|
- dumb-dev/cpp-10k |
|
- dumb-dev/Encoding-Detection-w-cChardet-DB |
|
- Neloy262/rust_instruction_dataset |
|
- m-a-p/CodeFeedback-Filtered-Instruction |
|
- sahil2801/CodeAlpaca-20k |
|
- vicgalle/alpaca-gpt4 |
|
|
|
## Their LORAs can be found [here](https://huggingface.co/dumb-dev/TinyLlama-1.1B-Chat-rust-cpp-encodings/tree/main/LORAs) |
|
In the final model only the 1e-4 LORAs have been used! |
|
Everything was trained a total of 2 epochs. |
|
|
|
### probably the reason why it works this bad: |
|
Following 3 are fp16, the other ones are fp32: |
|
1. [this](https://huggingface.co/dumb-dev/TinyLlama-1.1B-Chat-rust-cpp-encodings/tree/main/LORAs/300mb-DB-CodeFeedback-Tinyllama) |
|
2. [this](https://huggingface.co/dumb-dev/TinyLlama-1.1B-Chat-rust-cpp-encodings/tree/main/LORAs/tinyllama-rust) |
|
3. [this](https://huggingface.co/dumb-dev/TinyLlama-1.1B-Chat-rust-cpp-encodings/tree/main/LORAs/tinyllama-cpp) |
|
|
|
# If someone knows how to improve, please let me know. |
|
Instagram: dev2care |
