Is this a complete dataset?
#1
by
ifmain
- opened
Did you use the full set of fineweb 150b or what percentage?
Hey @ifmain .
This is karpathy's build from https://github.com/karpathy/llm.c/discussions/580 converted to HF format to investigate bfloat16 performance - see https://github.com/karpathy/llm.c/pull/571. The run was 150B tokens, 1.5 epochs over the 100B FineWeb sample dataset.
There's active work underway at https://github.com/karpathy/llm.c so I'd suggest following the developments there as well!
ifmain
changed discussion status to
closed