Repository: activeloopai/Hub
File: deeplake/api/dataset.py (Python)

Code before:

        structure.create_missing(ds)

        unstructured.structure(
            ds,
            progressbar,
            num_workers,
        )

        return ds

    @staticmethod
    def ingest_classification(
        src: Union[str, pathlib.Path],
        dest: Union[str, pathlib.Path],
        image_params: Optional[Dict] = None,
        label_params: Optional[Dict] = None,
        dest_creds: Optional[Dict] = None,
        progressbar: bool = True,
        summary: bool = True,
        num_workers: int = 0,
        token: Optional[str] = None,
        connect_kwargs: Optional[Dict] = None,
        **dataset_kwargs,
    ) -> Dataset:
        """Ingests a dataset from a source and stores it as a structured dataset to destination.

        Args:
            src (str, pathlib.Path): Local path to where the unstructured dataset is stored or path to csv file.
            dest (str, pathlib.Path): - The full path to the dataset. Can be:
                - a Deep Lake cloud path of the form ``hub://org_id/datasetname``. To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use 'activeloop login' from command line)
                - an s3 path of the form ``s3://bucketname/path/to/dataset``. Credentials are required in either the environment or passed to the creds argument.
                - a local file system path of the form ``./path/to/dataset`` or ``~/path/to/dataset`` or ``path/to/dataset``.
                - a memory path of the form ``mem://path/to/dataset`` which doesn't save the dataset but keeps it in memory instead. Should be used only for testing as it does not persist.
            image_params (Optional[Dict]): A dictionary containing parameters for the images tensor.
            label_params (Optional[Dict]): A dictionary containing parameters for the labels tensor.
            dest_creds (Optional[Dict]): A dictionary containing credentials used to access the destination path of the dataset.
            progressbar (bool): Enables or disables ingestion progress bar. Defaults to ``True``.
            summary (bool): If ``True``, a summary of skipped files will be printed after completion. Defaults to ``True``.
            num_workers (int): The number of workers to use for ingestion. Set to ``0`` by default.
            token (Optional[str]): The token to use for accessing the dataset.
            connect_kwargs (Optional[Dict]): If specified, the dataset will be connected to Deep Lake, and connect_kwargs will be passed to :meth:`Dataset.connect <deeplake.core.dataset.Dataset.connect>`.
            **dataset_kwargs: Any arguments passed here will be forwarded to the dataset creator function see :func:`deeplake.empty`.

        Returns:
            Dataset: New dataset object with structured dataset.

        Raises:
            InvalidPathException: If the source directory does not exist.
            SamePathException: If the source and destination path are same.
            AutoCompressionError: If the source director is empty or does not contain a valid extension.

Review comment (bug):
@ProgerDav In this PR, we only handle images, correct? If so, that should be clear in the docstring.

Sorry, my comment might be in the wrong place. I'm referring to the first line in the docstring of ingest_classification

Code after:

        structure.create_missing(ds)

        unstructured.structure(
            ds,
            progressbar,
            num_workers,
            shuffle,
        )

        return ds

    @staticmethod
    def ingest_classification(
        src: Union[str, pathlib.Path],
        dest: Union[str, pathlib.Path],
        image_params: Optional[Dict] = None,
        label_params: Optional[Dict] = None,
        dest_creds: Optional[Dict] = None,
        progressbar: bool = True,
        summary: bool = True,
        num_workers: int = 0,
        shuffle: bool = True,
        token: Optional[str] = None,
        connect_kwargs: Optional[Dict] = None,
        **dataset_kwargs,
    ) -> Dataset:
        """Ingests a dataset of images from a source and stores it as a structured dataset to destination.

        Args:
            src (str, pathlib.Path): Local path to where the unstructured dataset of images is stored or path to csv file.
            dest (str, pathlib.Path): - The full path to the dataset. Can be:
                - a Deep Lake cloud path of the form ``hub://org_id/datasetname``. To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use 'activeloop login' from command line)
                - an s3 path of the form ``s3://bucketname/path/to/dataset``. Credentials are required in either the environment or passed to the creds argument.
                - a local file system path of the form ``./path/to/dataset`` or ``~/path/to/dataset`` or ``path/to/dataset``.
                - a memory path of the form ``mem://path/to/dataset`` which doesn't save the dataset but keeps it in memory instead. Should be used only for testing as it does not persist.
            image_params (Optional[Dict]): A dictionary containing parameters for the images tensor.
            label_params (Optional[Dict]): A dictionary containing parameters for the labels tensor.
            dest_creds (Optional[Dict]): A dictionary containing credentials used to access the destination path of the dataset.
            progressbar (bool): Enables or disables ingestion progress bar. Defaults to ``True``.
            summary (bool): If ``True``, a summary of skipped files will be printed after completion. Defaults to ``True``.
            num_workers (int): The number of workers to use for ingestion. Set to ``0`` by default.
            shuffle (bool): Shuffles the input data prior to ingestion. Since data arranged in folders by class is highly non-random, shuffling is important in order to produce optimal results when training. Defaults to ``True``.
            token (Optional[str]): The token to use for accessing the dataset.
            connect_kwargs (Optional[Dict]): If specified, the dataset will be connected to Deep Lake, and connect_kwargs will be passed to :meth:`Dataset.connect <deeplake.core.dataset.Dataset.connect>`.
            **dataset_kwargs: Any arguments passed here will be forwarded to the dataset creator function see :func:`deeplake.empty`.

        Returns:
            Dataset: New dataset object with structured dataset.

        Raises: