metadata
license: apache-2.0
datasets:
- cerebras/SlimPajama-627B
- bigcode/starcoderdata
- HuggingFaceH4/ultrachat_200k
- HuggingFaceH4/ultrafeedback_binarized
- dumb-dev/cpp-10k
- dumb-dev/Encoding-Detection-w-cChardet-DB
- Neloy262/rust_instruction_dataset
- m-a-p/CodeFeedback-Filtered-Instruction
- sahil2801/CodeAlpaca-20k
- vicgalle/alpaca-gpt4
language:
- en
I finetuned TinyLlama/TinyLlama-1.1B-Chat-v1.0 on the following datasets:
- dumb-dev/cpp-10k
- dumb-dev/Encoding-Detection-w-cChardet-DB
- Neloy262/rust_instruction_dataset
- m-a-p/CodeFeedback-Filtered-Instruction
- sahil2801/CodeAlpaca-20k
- vicgalle/alpaca-gpt4
Their LORAs can be found here
In the final model only the 1e-4 LORAs have been used! Everything was trained a total of 2 epochs.
probably the reason why it works this bad:
Following 3 are fp16, the other ones are fp32:
If someone knows how to improve, please let me know.
Instagram: dev2care