Repository: huggingface/datasets
File: src/datasets/fingerprint.py (Python)

Code before:
# A dataset fingerprint is updated after each transform.
# Re-running the same transforms on a dataset in a different session results in the same fingerprint.
# This is possible thanks to a custom hashing function that works with most python objects.

# Fingerprinting is the main mechanism that enables caching.
# The caching mechanism allows to reload an existing cache file if it's already been computed.


#################
# Caching
#################

_CACHING_ENABLED = True
_TEMP_DIR_FOR_TEMP_CACHE_FILES: Optional["_TempCacheDir"] = None
_DATASETS_WITH_TABLE_IN_TEMP_DIR: Optional[weakref.WeakSet] = None


class _TempCacheDir:
    """
    A temporary directory for storing cached Arrow files with a cleanup that frees references to the Arrow files
    before deleting the directory itself to avoid permission errors on Windows.
    """

    def __init__(self):
        # Check if TMPDIR is set and handle the case where it doesn't exist
        tmpdir = os.environ.get("TMPDIR")
        # Normalize the path to handle any path resolution issues
        if tmpdir:
            tmpdir = os.path.normpath(tmpdir)
            if not os.path.exists(tmpdir):
                # Auto-create the directory if it doesn't exist
                # This prevents tempfile from silently falling back to /tmp
                try:
                    os.makedirs(tmpdir, exist_ok=True)
                    logger.info(f"Created TMPDIR directory: {tmpdir}")
                except OSError as e:
                    raise OSError(
                        f"TMPDIR is set to '{tmpdir}' but the directory does not exist and could not be created: {e}. "
                        "Please create it manually or unset TMPDIR to fall back to the default temporary directory."
                    ) from e
            # If tmpdir exists, verify it's actually a directory and writable
            elif not os.path.isdir(tmpdir):
                raise OSError(
                    f"TMPDIR is set to '{tmpdir}' but it is not a directory. "
                    "Please point TMPDIR to a writable directory or unset it to fall back to the default temporary directory."
                )

        # Explicitly pass the directory to mkdtemp to ensure TMPDIR is respected
        # This works even if tempfile.gettempdir() was already called and cached
        # Pass dir=None if tmpdir is None to use default temp directory
        self.name = tempfile.mkdtemp(prefix=config.TEMP_CACHE_DIR_PREFIX, dir=tmpdir)

Review comment (suggestion):
we need to support the same environment variables as mkdtemp

```suggestion
        tmpdir = os.environ.get("TMPDIR") or os.environ.get("TEMP")  or os.environ.get("TMP") 
```

Code after:
# A dataset fingerprint is updated after each transform.
# Re-running the same transforms on a dataset in a different session results in the same fingerprint.
# This is possible thanks to a custom hashing function that works with most python objects.

# Fingerprinting is the main mechanism that enables caching.
# The caching mechanism allows to reload an existing cache file if it's already been computed.


#################
# Caching
#################

_CACHING_ENABLED = True
_TEMP_DIR_FOR_TEMP_CACHE_FILES: Optional["_TempCacheDir"] = None
_DATASETS_WITH_TABLE_IN_TEMP_DIR: Optional[weakref.WeakSet] = None


class _TempCacheDir:
    """
    A temporary directory for storing cached Arrow files with a cleanup that frees references to the Arrow files
    before deleting the directory itself to avoid permission errors on Windows.
    """

    def __init__(self):
        # Check if TMPDIR is set and handle the case where it doesn't exist
        tmpdir = os.environ.get("TMPDIR") or os.environ.get("TEMP") or os.environ.get("TMP")
        # Normalize the path to handle any path resolution issues
        if tmpdir:
            tmpdir = os.path.normpath(tmpdir)
            if not os.path.exists(tmpdir):
                # Auto-create the directory if it doesn't exist
                # This prevents tempfile from silently falling back to /tmp
                try:
                    os.makedirs(tmpdir, exist_ok=True)
                    logger.info(f"Created TMPDIR directory: {tmpdir}")
                except OSError as e:
                    raise OSError(
                        f"TMPDIR is set to '{tmpdir}' but the directory does not exist and could not be created: {e}. "
                        "Please create it manually or unset TMPDIR to fall back to the default temporary directory."
                    ) from e
            # If tmpdir exists, verify it's actually a directory and writable
            elif not os.path.isdir(tmpdir):
                raise OSError(
                    f"TMPDIR is set to '{tmpdir}' but it is not a directory. "
                    "Please point TMPDIR to a writable directory or unset it to fall back to the default temporary directory."
                )

        # Explicitly pass the directory to mkdtemp to ensure TMPDIR is respected
        # This works even if tempfile.gettempdir() was already called and cached
        # Pass dir=None if tmpdir is None to use default temp directory
        self.name = tempfile.mkdtemp(prefix=config.TEMP_CACHE_DIR_PREFIX, dir=tmpdir)