Falcon-40b continued pretraining?
#6
by
KnutJaegersberg
- opened
I gotta question by a CS student from Berlin (https://twitter.com/felix_red_panda), who was suggesting that one could continue pretraining falcon-40b, as it has seen 12b German tokens during pretraining.
Are there plans to continue pretraining on German tokens such a model or another model after bloom-6b?
Yes, I am actively working on this. Feel free to reach out to me if you or your student are interested in participating.
It's not my student, just a person I met on Twitter. Let me try to get him here