What is this model?
I get that this is base llama2 but what dataset was it trained on to become a coding model? What parameters was it trained at? please add it to the model card
Hi, it was trained on the CodeAlpaca dataset. It doesn't have any card yet because it's not ready at all. I don't recommend using it for now.
Gotcha, hey if you are interested i made a really nice coding dataset here, i planned on training the llama2-13b model it it but my 3080 10gb isnt enough to train it. I bet it would make for an amazing model.
https://huggingface.co/datasets/rombodawg/MegaCodeTraining112k
That's really nice! I'm a big believer in evol instruct :)
Heck yea! Well leave this thread open and let me know if you end up using my training data to train llama2 7b or 13b. Another one to consider is llama2-guanaco. The Guanaco series of models has always had an edge over base llama in instruction following and general accuracy. Links bellow:
https://huggingface.co/Mikael110/llama-2-13b-guanaco-fp16
https://huggingface.co/Mikael110/llama-2-7b-guanaco-fp16
Cool! To be honest, I'm more interested in StarCoder and CodeGen2.5 when it comes to code generation but who knows maybe this new version of Llama is better.
Ive tested most of the starcoder based models extensively, and they are very flawed. I personally would try to focus on creating new coding models based on ai that are extremely high scoring in benchmarks overall like llama2, the main issue is starcoder lacks reasoning skills and although it know how to code, it make loads of mistakes. I think the future of coding models lies in really good instruct models fine tuned on coding. Not coding model fine tunes on intruct
That's an interesting take, have you tried WizardCoder too? It feels like the Fill in the Middle training objective is desirable when it comes to code LLM. I wish phi-1 was open-sourced (or even just its dataset).
Yes, wizard coder has the same issue unfortunately. Guanaco 65b has had the best cosing performance based on my benchmarks, before llama 2 came out. And i feel that shows the future of coding models lies in good instruct models first. I have tested other models like airoboros 65b and it sucked at coding so its not necessarily the size that matters. Although my dataset does include instruct and code from wizardlm data
Have you released your benchmarks somewhere? I'm sure a ton of people would be interested in it. Anyway, thanks for the recommendation, I'll keep it in mind!
Not really, my bencharks arent really normal lol i test models based on if they can develop a game of snake in python and in javascript+html. Its suprizingly difficult to get right. But gpt 3.5 did it first try. And guanaco 65b was the closest to get it right. Its a pretty good benchmark for consitansy in code, logic, and overall perplexity
Its just not something that you can easly score
@mlabonne If you are interested, i have made a version 3 of my megacoding dataset, and you are free to use it to train your future coding models
https://huggingface.co/datasets/rombodawg/LosslessMegaCodeTrainingV3_2.2m_Evol
Looks amazing, I want to give it a try!
Ok let me know when you post the model, id love to try it out