Hugging Face's logo

Hub Python Library documentation

Filesystem API

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Filesystem API

The HfFileSystem class provides a pythonic file interface to the Hugging Face Hub based on fsspec.

HfFileSystem

HfFileSystem is based on fsspec, so it is compatible with most of the APIs that it offers. For more details, check out our guide and fsspec’s API Reference.

class huggingface_hub.HfFileSystem

< >

( *args **kwargs )

Parameters

Access a remote Hugging Face Hub repository as if were a local file system.

Usage:

>>> from huggingface_hub import HfFileSystem

>>> fs = HfFileSystem()

>>> # List files
>>> fs.glob("my-username/my-model/*.bin")
['my-username/my-model/pytorch_model.bin']
>>> fs.ls("datasets/my-username/my-dataset", detail=False)
['datasets/my-username/my-dataset/.gitattributes', 'datasets/my-username/my-dataset/README.md', 'datasets/my-username/my-dataset/data.json']

>>> # Read/write files
>>> with fs.open("my-username/my-model/pytorch_model.bin") as f:
...     data = f.read()
>>> with fs.open("my-username/my-model/pytorch_model.bin", "wb") as f:
...     f.write(data)

__init__

< >

( *args endpoint: typing.Optional[str] = None token: typing.Union[bool, str, NoneType] = None **storage_options )

Parameters

  • use_listings_cache, listings_expiry_time, max_paths — passed to DirCache, if the implementation supports directory listing caching. Pass use_listings_cache=False to disable such caching.
  • skip_instance_cache — bool If this is a cachable implementation, pass True here to force creating a new instance even if a matching instance exists, and prevent storing this instance.
  • asynchronous — bool
  • loop — asyncio-compatible IOLoop or None

Docstring taken from fsspec documentation.

Create and configure file-system instance

Instances may be cachable, so if similar enough arguments are seen a new instance is not required. The token attribute exists to allow implementations to cache instances if they wish.

A reasonable default should be provided if there are no arguments.

Subclasses should call this method.

exists

< >

( path **kwargs )

Docstring taken from fsspec documentation.

Is there a file at the given path

find

< >

( path: str maxdepth: typing.Optional[int] = None withdirs: bool = False detail: bool = False refresh: bool = False revision: typing.Optional[str] = None **kwargs )

Parameters

  • path — str
  • maxdepth — int or None If not None, the maximum number of levels to descend
  • withdirs — bool Whether to include directory paths in the output. This is True when used by glob, but users usually only want files.
  • kwargs are passed to ls. —

Docstring taken from fsspec documentation.

List all files below path.

Like posix find command without conditions

get_file

< >

( rpath lpath callback = <fsspec.callbacks.NoOpCallback object at 0x7f0cf89a07c0> outfile = None **kwargs )

Docstring taken from fsspec documentation.

Copy single remote file to local

glob

< >

( path **kwargs )

Docstring taken from fsspec documentation.

Find files by glob-matching.

If the path ends with ’/’, only folders are returned.

We support "**", "?" and "[..]". We do not support ^ for pattern negation.

The maxdepth option is applied on the first **** found in the path.

kwargs are passed to ls.

info

< >

( path: str refresh: bool = False revision: typing.Optional[str] = None **kwargs ) → dict with keys

Returns

dict with keys

name (full path in the FS), size (in bytes), type (file, directory, or something else) and other FS-specific keys.

Docstring taken from fsspec documentation.

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to “ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include `‘size’: None“.

invalidate_cache

< >

( path: typing.Optional[str] = None )

Parameters

  • path — string or None If None, clear all listings cached else listings at or under given path.

Docstring taken from fsspec documentation.

Discard any cached directory information

isdir

< >

( path )

Docstring taken from fsspec documentation.

Is this entry directory-like?

isfile

< >

( path )

Docstring taken from fsspec documentation.

Is this entry file-like?

ls

< >

( path: str detail: bool = True refresh: bool = False revision: typing.Optional[str] = None **kwargs )

Parameters

  • path — str
  • detail — bool if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).
  • kwargs — may have additional backend-specific options, such as version information

Docstring taken from fsspec documentation.

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

  • full path to the entry (without protocol)
  • size of the entry, in bytes. If the value cannot be determined, will be None.
  • type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

modified

< >

( path: str **kwargs )

Docstring taken from fsspec documentation.

Return the modified timestamp of a file as a datetime.datetime

rm

< >

( path: str recursive: bool = False maxdepth: typing.Optional[int] = None revision: typing.Optional[str] = None **kwargs )

Parameters

  • path — str or list of str File(s) to delete.
  • recursive — bool If file(s) are directories, recursively delete contents and then also remove the directory
  • maxdepth — int or None Depth to pass to walk for finding files to delete, if recursive. If None, there will be no limit and infinite recursion may be possible.

Docstring taken from fsspec documentation.

Delete files.

start_transaction

< >

( )

Docstring taken from fsspec documentation.

Begin write transaction for deferring files, non-context version

url

< >

( path: str )

Get the HTTP URL of the given path

walk

< >

( path *args **kwargs )

Parameters

  • path — str Root to recurse into
  • maxdepth — int Maximum recursion depth. None means limitless, but not recommended on link-based file-systems.
  • topdown — bool (True) Whether to walk the directory tree from the top downwards or from the bottom upwards.
  • on_error — “omit”, “raise”, a collable if omit (default), path with exception will simply be empty; If raise, an underlying exception will be raised; if callable, it will be called with a single OSError instance as argument
  • kwargs — passed to ls

Docstring taken from fsspec documentation.

Return all files belows path

List all files, recursing into subdirectories; output is iterator-style, like os.walk(). For a simple list of files, find() is available.

When topdown is True, the caller can modify the dirnames list in-place (perhaps using del or slice assignment), and walk() will only recurse into the subdirectories whose names remain in dirnames; this can be used to prune the search, impose a specific order of visiting, or even to inform walk() about directories the caller creates or renames before it resumes walk() again. Modifying dirnames when topdown is False has no effect. (see os.walk)

Note that the “files” outputted will include anything that is not a directory, such as links.

< > Update on GitHub

←Inference Endpoints Utilities→