Hub Python Library documentation

Filesystem API

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v0.23.4).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Filesystem API

The HfFileSystem class provides a pythonic file interface to the Hugging Face Hub based on fsspec.

HfFileSystem

HfFileSystem is based on fsspec, so it is compatible with most of the APIs that it offers. For more details, check out our guide and fsspec’s API Reference.

class huggingface_hub.HfFileSystem

< >

( *args **kwargs )

Parameters

Access a remote Hugging Face Hub repository as if were a local file system.

Usage:

>>> from huggingface_hub import HfFileSystem

>>> fs = HfFileSystem()

>>> # List files
>>> fs.glob("my-username/my-model/*.bin")
['my-username/my-model/pytorch_model.bin']
>>> fs.ls("datasets/my-username/my-dataset", detail=False)
['datasets/my-username/my-dataset/.gitattributes', 'datasets/my-username/my-dataset/README.md', 'datasets/my-username/my-dataset/data.json']

>>> # Read/write files
>>> with fs.open("my-username/my-model/pytorch_model.bin") as f:
...     data = f.read()
>>> with fs.open("my-username/my-model/pytorch_model.bin", "wb") as f:
...     f.write(data)

__init__

< >

( *args endpoint: Optional = None token: Union = None **storage_options )

Parameters

  • use_listings_cache, listings_expiry_time, max_paths — passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching. skip_instance_cache — bool If this is a cachable implementation, pass True here to force creating a new instance even if a matching instance exists, and prevent storing this instance. asynchronous — bool loop — asyncio-compatible IOLoop or None

Docstring taken from fsspec documentation.

Create and configure file-system instance

Instances may be cachable, so if similar enough arguments are seen a new instance is not required. The token attribute exists to allow implementations to cache instances if they wish.

A reasonable default should be provided if there are no arguments.

Subclasses should call this method.

exists

< >

( path **kwargs )

Docstring taken from fsspec documentation.

Is there a file at the given path

find

< >

( path: str maxdepth: Optional = None withdirs: bool = False detail: bool = False refresh: bool = False revision: Optional = None **kwargs )

Parameters

  • path — str maxdepth — int or None If not None, the maximum number of levels to descend withdirs — bool Whether to include directory paths in the output. This is True when used by glob, but users usually only want files.
  • kwargs are passed to ls. —

Docstring taken from fsspec documentation.

List all files below path.

Like posix find command without conditions

get_file

< >

( rpath lpath callback = <fsspec.callbacks.NoOpCallback object at 0x7fd9c79ab850> outfile = None **kwargs )

Docstring taken from fsspec documentation.

Copy single remote file to local

glob

< >

( path **kwargs )

Docstring taken from fsspec documentation.

Find files by glob-matching.

If the path ends with ’/’, only folders are returned.

We support "**", "?" and "[..]". We do not support ^ for pattern negation.

The maxdepth option is applied on the first **** found in the path.

kwargs are passed to ls.

info

< >

( path: str refresh: bool = False revision: Optional = None **kwargs ) dict with keys

Returns

dict with keys

name (full path in the FS), size (in bytes), type (file, directory, or something else) and other FS-specific keys.

Docstring taken from fsspec documentation.

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation should calls ls and could be overridden by a shortcut. kwargs are passed on to “ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include `‘size’: None“.

invalidate_cache

< >

( path: Optional = None )

Docstring taken from fsspec documentation.

Discard any cached directory information

isdir

< >

( path )

Docstring taken from fsspec documentation.

Is this entry directory-like?

isfile

< >

( path )

Docstring taken from fsspec documentation.

Is this entry file-like?

ls

< >

( path: str detail: bool = True refresh: bool = False revision: Optional = None **kwargs )

Docstring taken from fsspec documentation.

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

  • full path to the entry (without protocol)
  • size of the entry, in bytes. If the value cannot be determined, will be None.
  • type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

modified

< >

( path: str **kwargs )

Docstring taken from fsspec documentation.

Return the modified timestamp of a file as a datetime.datetime

rm

< >

( path: str recursive: bool = False maxdepth: Optional = None revision: Optional = None **kwargs )

Docstring taken from fsspec documentation.

Delete files.

start_transaction

< >

( )

Docstring taken from fsspec documentation.

Begin write transaction for deferring files, non-context version

url

< >

( path: str )

Get the HTTP URL of the given path

walk

< >

( path *args **kwargs )

Docstring taken from fsspec documentation.

Return all files belows path

List all files, recursing into subdirectories; output is iterator-style, like os.walk(). For a simple list of files, find() is available.

When topdown is True, the caller can modify the dirnames list in-place (perhaps using del or slice assignment), and walk() will only recurse into the subdirectories whose names remain in dirnames; this can be used to prune the search, impose a specific order of visiting, or even to inform walk() about directories the caller creates or renames before it resumes walk() again. Modifying dirnames when topdown is False has no effect. (see os.walk)

Note that the “files” outputted will include anything that is not a directory, such as links.

< > Update on GitHub