zz / scripts /download

38.2 kB

Ctrl+K

3 contributors

History: 9 commits

lzwjava

Add working directory note to AMD download script docstring

941c05c 8 days ago

download_and_extract.py

2.12 kB
chore: add ruff pre-commit hooks and apply formatting 26 days ago
download_fineweb.py

1.11 kB
chore: add ruff pre-commit hooks and apply formatting 26 days ago
plan_and_download_fineweb.py

9.08 kB
chore: add ruff pre-commit hooks and apply formatting 26 days ago
plan_and_download_fineweb_gpt3.py

10.1 kB
The changes update the download plan script to handle .tar.gz files (for Llama 3 models) in addition to the previously supported .tar files. The code checks if the target filename ends with .tar.gz and adjusts the extraction command accordingly by using tar xfzf format gzip files. This ensures compatibility with Llama 3 models which use the .tar.gz format. 26 days ago
plan_and_download_fineweb_gpt3_amd.py

8.53 kB
Add working directory note to AMD download script docstring 8 days ago
wget_fineweb_1.sh

184 Bytes
refactor: reorganize project structure 2 months ago
wget_fineweb_mirror_1.sh

603 Bytes
refactor: reorganize project structure 2 months ago
wget_fineweb_mirror_11_20.sh

1.47 kB
refactor: reorganize project structure 2 months ago
wget_fineweb_mirror_2_5.sh

1.47 kB
refactor: reorganize project structure 2 months ago
wget_wikimedia_1.sh

732 Bytes
refactor: reorganize project structure 2 months ago
wget_wikimedia_4.sh

861 Bytes
refactor: reorganize project structure 2 months ago
wget_wikimedia_4_accum.sh

863 Bytes
refactor: reorganize project structure 2 months ago
wget_wikimedia_5.sh

994 Bytes
refactor: reorganize project structure 2 months ago