Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Upload files to the Hub

Sharing your files and work is a very important aspect of the Hub. The huggingface_hub uses a Git-based workflow to upload files to the Hub. You can use these functions independently or integrate them into your own library, making it more convenient for your users to interact with the Hub. This guide will show you how to:

  • Push files with a commit context manager.
  • Push files with the push_to_hub() function.
  • Upload very large files with Git LFS.
  • Push files without Git installed with HfApi

Whenever you want to upload files to the Hub, you need to log in to your Hugging Face account:

  1. Log in to your Hugging Face account with the following command:

    huggingface-cli login
  2. Alternatively, if you prefer working from a Jupyter or Colaboratory notebook, login with notebook_login():

    >>> from huggingface_hub import notebook_login
    >>> notebook_login()

    notebook_login() will launch a widget in your notebook from which you can enter your Hugging Face credentials.

commit context manager

The commit context manager handles four of the most common Git commands: pull, add, commit, and push. git-lfs automatically tracks any file larger than 10MB. In the following example, the commit context manager:

  1. Pulls from the text-files repository.
  2. Adds a change made to file.txt.
  3. Commits the change.
  4. Pushes the change to the text-files repository.
>>> from huggingface_hub import Repository
>>> with Repository(local_dir="text-files", clone_from="<user>/text-files").commit(commit_message="My first file :)"):
...     with open("file.txt", "w+") as f:
...         f.write(json.dumps({"hey": 8}))

Here is another example of how to save and upload a file to a repository:

>>> import torch
>>> model = torch.nn.Transformer()
>>> with Repository("torch-model", clone_from="<user>/torch-model", use_auth_token=True).commit(commit_message="My cool model :)"):
...     torch.save(model.state_dict(), "model.pt")

Set blocking=False if you would like to push your commits asynchronously. Non-blocking behavior is helpful when you want to continue running your script while you push your commits.

>>> with repo.commit(commit_message="My cool model :)", blocking=False)

You can check the status of your push with the command_queue method:

>>> last_command = repo.command_queue[-1]
>>> last_command.status

Refer to the table below for the possible statuses:

Status Description
-1 The push is ongoing.
0 The push has completed successfully.
Non-zero An error has occurred.

When blocking=False, commands are tracked, and your script will only exit when all pushes are completed, even if other errors occur in your script. Some additional useful commands for checking the status of a push include:

# Inspect an error.
>>> last_command.stderr

# Check whether a push is completed or ongoing.
>>> last_command.is_done

# Check whether a push command has errored.
>>> last_command.failed

push_to_hub

The Repository class also has a push_to_hub() function to add files, make a commit, and push them to a repository. Unlike the commit context manager, push_to_hub() requires you to pull from a repository first, save the files, and then call push_to_hub().

>>> from huggingface_hub import Repository
>>> repo.git_pull()
>>> repo.push_to_hub(commit_message="Commit my-awesome-file to the Hub")

However, if you aren’t ready to push a file yet, you can still use git_add() and git_commit() to add and commit your file:

>>> repo.git_add("path/to/file")
>>> repo.git_commit(commit_message="add my first model config file :)")

Once you’re ready, you can push your file to your repository with git_push():

>>> repo.git_push()

Upload with Git LFS

For huge files (>5GB), you need to install a custom transfer agent for Git LFS:

huggingface-cli lfs-enable-largefiles

You should install this for each model repository that contains a model file. Once installed, you are now able to push files larger than 5GB.

Managing files in a repo without Git with the create_commit API

huggingface_hub also offers a way to upload files to the Hub without Git installed on your system with the create_commit() method of HfApi. For example, if you want to upload two files and delete another file in a Hub repo:

>>> from huggingface_hub import HfApi, CommitOperationAdd, CommitOperationDelete
>>> api = HfApi()
>>> operations = [
...     CommitOperationAdd(path_in_repo="LICENSE.md", path_or_fileobj="~/repo/LICENSE.md"),
...     CommitOperationAdd(path_in_repo="weights.h5", path_or_fileobj="~/repo/weights-final.h5"),
...     CommitOperationDelete(path_in_repo="old-weights.h5"),
... ]
>>> api.create_commit(
...     repo_id="lysandre/test-model",
...     operations=operations,
... )

create_commit() uses the HTTP protocol to upload files to the Hub. It automatically takes care of uploading large files and binary files with the Git LFS protocol. There are currently two kind of operations supported by the create_commit() method:

  1. CommitOperationAdd to upload a file to the Hub. If the file already exists, its content will be overwritten. It takes two arguments:
    • path_in_repo: the path in the repository where the file should be uploaded
    • path_or_fileobj: either a path to a file on your filesystem, or a file-like object. The content of the file to upload to the Hub.
  2. CommitOperationDelete to remove a file from a repository. It takes path_in_repo as an argument.

Instead of create_commit(), you can also use the following convenience methods:

  • upload_file() to upload a single file to a repo on the Hub
  • upload_folder() to upload a local directory to a repo on the Hub
  • delete_file() to delete a single file from a repo on the Hub
  • metadata_update() to update a repo’s metadata

All these methods use the create_commit API under the hood. For a more detailed description, visit the hf_api() documentation page.