Title: Do we need to implement `_prepare_split`?
Author: jmwoloso (NONE)
State: closed
Created: 2022-12-12T01:38:54Z

### Describe the bug

I'm not sure this is a bug or if it's just missing in the documentation, or i'm not doing something correctly, but I'm subclassing `DatasetBuilder` and getting the following error because on the `DatasetBuilder` class the `_prepare_split` method is abstract (as are the others we are required to implement, hence the genesis of my question):

```
Traceback (most recent call last):
  File "/home/jason/source/python/prism_machine_learning/examples/create_hf_datasets.py", line 28, in <module>
    dataset_builder.download_and_prepare()
  File "/home/jason/.virtualenvs/pml/lib/python3.8/site-packages/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/jason/.virtualenvs/pml/lib/python3.8/site-packages/datasets/builder.py", line 793, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/jason/.virtualenvs/pml/lib/python3.8/site-packages/datasets/builder.py", line 1124, in _prepare_split
    raise NotImplementedError()
NotImplementedError
```

### Steps to reproduce the bug

I will share implementation if it turns out that everything should be working (i.e. we only need to implement those 3 methods the docs mention), but I don't want to distract from the original question.




### Expected behavior

I just need to know if there are additional methods we need to implement when subclassing `DatasetBuilder` besides what the documentation specifies -> `_info`, `_split_generators` and `_generate_examples`

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-135-generic-x86_64-with-glibc2.2.5
- Python version: 3.8.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.1

Comments:
Hi! `DatasetBuilder` is a parent class for concrete builders: `GeneratorBasedBuilder`, `ArrowBasedBuilder` and `BeamBasedBuilder`. When writing a builder script, these classes are the ones you should inherit from. And since all of them implement `_prepare_split`, you only have to implement the three methods mentioned above.

Thanks so much @mariosasko for the fast response! I've been referencing [this page in the docs](https://huggingface.co/docs/datasets/v2.4.0/en/about_dataset_load) because it it pretty comprehensive in terms of what we have to do and I figured since we subclass the `BuilderConfig` the same pattern would hold, but I've also seen the page with those sub-classed builders as well, so that fills in a knowledge gap for me.

cc @stevhliu who may have some ideas on how to improve this part of the docs.

one more question for my understanding @mariosasko. the requirement of a loading script has always seemed counterintuitive to me. if i have to provide a script with every dataset, what is the point of using `datasets` if we're doing all the work of loading it, I can just do that in my code and skip the datasets integration (this of course discounts other potential benefits around metadata management, etc., my example is just simplest use case though for the sake of discussion).

so i figured I would implement my own `BuilderConfig` and `DatasetBuilder` to handle that portion of it and not have to make a script. i _thought_ this would result in `datasets` (via `download_and_prepare`) then making me something that I could load using `load_dataset` moving forward.

Concretely, i envisioned this pattern being possible:

 ```
class MyBuilderConfig(BuilderConfig):
    def __init__(self, name="my_named_dataset", ...):
        super().__init__(name, ...)

class MyDatasetBuilder(GeneratorBasedBuilder):
   BUILDER_CONFIG_CLASS = MyBuilderConfig
    ....

my_builder = MyDatasetBuilder(...)

# this doesn't exactly work like I thought; I don't get a dataset back, but NoneType instead
# though I can see it loading the files and it generates the cache, etc.
my_dataset = my_builder.download_and_prepare()

# load the dataset in the future by referencing it by name and loading from the cached arrow version
new_instance_of_my_dataset = load_dataset("my_named_dataset")
```

I've seen references to the `save_to_disk` method which might be the next step I need in order to load it by name, in which case, that makes sense, then i just need to debug why `download_and_prepare` isn't returning me a dataset, but I feel like I still have a larger conceptual knowledge gap on how to use the library correctly.

Thanks again in advance!

>  the requirement of a loading script has always seemed counterintuitive to me

This is a requirement only for datasets not stored in standard formats such as CSV, JSON, SQL, Parquet, ImageFolder, etc. 

>  if i have to provide a script with every dataset, what is the point of using datasets if we're doing all the work of loading it, I can just do that in my code and skip the datasets integration (this of course discounts other potential benefits around metadata management, etc., my example is just simplest use case though for the sake of discussion)

Our README/documentation lists the main features... 

One of the main ones is that our library makes it easy to work with datasets larger than RAM (thanks to Arrow and the caching mechanism), and this is not trivial to implement.

Regarding the step-by-step builder, this is the pattern:
```python
from datasets import load_dataset_builder
builder = load_dataset_builder("path/to/script") # or direct instantiation with MyDatasetBuilder(...)
builder.download_and_prepare()
dset = builder.as_dataset()
```

ok, that makes sense. thank you @mariosasko. I realized i'd never looked on the hub at any of the files associated with any datasets. just did that now and it appears that i'll need to have a script regardless _but_ that will just contain my custom config and builder classes, so without realizing it I was already making my script,  I just need to wrap that in a file that sits alongside my data (I looked at Glue and realized I was already doing what I thought didn't make sense to have to do, lol).

`download_and_prepare` isn't returning me a dataset though, but I'll look into that and open another issue if I can't figure it out.

`download_and_prepare` downloads and prepares the arrow files. You need to call `as_dataset` on the builder to get the dataset.

ok, I think I was assigning the output of `builder.download_and_prepare` but it's an inplace op, so that explains the `NoneType` i was getting back. Now I'm getting:

```
ArrowInvalid                              Traceback (most recent call last)
<ipython-input-7-3ed50fb87c70> in <module>
----> 1 ds = dataset_builder.as_dataset()

/databricks/python/lib/python3.8/site-packages/datasets/builder.py in as_dataset(self, split, run_post_process, ignore_verifications, in_memory)
   1020 
   1021         # Create a dataset for each of the given splits
-> 1022         datasets = map_nested(
   1023             partial(
   1024                 self._build_single_dataset,

/databricks/python/lib/python3.8/site-packages/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, parallel_min_length, types, disable_tqdm, desc)
    442         num_proc = 1
    443     if num_proc <= 1 or len(iterable) < parallel_min_length:
--> 444         mapped = [
    445             _single_map_nested((function, obj, types, None, True, None))
    446             for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)

/databricks/python/lib/python3.8/site-packages/datasets/utils/py_utils.py in <listcomp>(.0)
    443     if num_proc <= 1 or len(iterable) < parallel_min_length:
    444         mapped = [
--> 445             _single_map_nested((function, obj, types, None, True, None))
    446             for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)
    447         ]

/databricks/python/lib/python3.8/site-packages/datasets/utils/py_utils.py in _single_map_nested(args)
    344     # Singleton first to spare some computation
    345     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 346         return function(data_struct)
    347 
    348     # Reduce logging to keep things readable in multiprocessing with tqdm

/databricks/python/lib/python3.8/site-packages/datasets/builder.py in _build_single_dataset(self, split, run_post_process, ignore_verifications, in_memory)
   1051 
   1052         # Build base dataset
-> 1053         ds = self._as_dataset(
   1054             split=split,
   1055             in_memory=in_memory,

/databricks/python/lib/python3.8/site-packages/datasets/builder.py in _as_dataset(self, split, in_memory)
   1120         """
   1121         cache_dir = self._fs._strip_protocol(self._output_dir)
-> 1122         dataset_kwargs = ArrowReader(cache_dir, self.info).read(
   1123             name=self.name,
   1124             instructions=split,

/databricks/python/lib/python3.8/site-packages/datasets/arrow_reader.py in read(self, name, instructions, split_infos, in_memory)
    236             msg = f'Instruction "{instructions}" corresponds to no data!'
    237             raise ValueError(msg)
--> 238         return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
    239 
    240     def read_files(

/databricks/python/lib/python3.8/site-packages/datasets/arrow_reader.py in read_files(self, files, original_instructions, in_memory)
    257         """
    258         # Prepend path to filename
--> 259         pa_table = self._read_files(files, in_memory=in_memory)
    260         # If original_instructions is not None, convert it to a human-readable NamedSplit
    261         if original_instructions is not None:

/databricks/python/lib/python3.8/site-packages/datasets/arrow_reader.py in _read_files(self, files, in_memory)
    192             f["filename"] = os.path.join(self._path, f["filename"])
    193         for f_dict in files:
--> 194             pa_table: Table = self._get_table_from_filename(f_dict, in_memory=in_memory)
    195             pa_tables.append(pa_table)
    196         pa_tables = [t for t in pa_tables if len(t) > 0]

/databricks/python/lib/python3.8/site-packages/datasets/arrow_reader.py in _get_table_from_filename(self, filename_skip_take, in_memory)
    327             filename_skip_take["take"] if "take" in filename_skip_take else None,
    328         )
--> 329         table = ArrowReader.read_table(filename, in_memory=in_memory)
    330         if take == -1:
    331             take = len(table) - skip

/databricks/python/lib/python3.8/site-packages/datasets/arrow_reader.py in read_table(filename, in_memory)
    348         """
    349         table_cls = InMemoryTable if in_memory else MemoryMappedTable
--> 350         return table_cls.from_file(filename)
    351 
    352 

/databricks/python/lib/python3.8/site-packages/datasets/table.py in from_file(cls, filename, replays)
   1034     @classmethod
   1035     def from_file(cls, filename: str, replays=None):
-> 1036         table = _memory_mapped_arrow_table_from_file(filename)
   1037         table = cls._apply_replays(table, replays)
   1038         return cls(table, filename, replays)

/databricks/python/lib/python3.8/site-packages/datasets/table.py in _memory_mapped_arrow_table_from_file(filename)
     48 def _memory_mapped_arrow_table_from_file(filename: str) -> pa.Table:
     49     memory_mapped_stream = pa.memory_map(filename)
---> 50     opened_stream = pa.ipc.open_stream(memory_mapped_stream)
     51     pa_table = opened_stream.read_all()
     52     return pa_table

/databricks/python/lib/python3.8/site-packages/pyarrow/ipc.py in open_stream(source)
    152     reader : RecordBatchStreamReader
    153     """
--> 154     return RecordBatchStreamReader(source)
    155 
    156 

/databricks/python/lib/python3.8/site-packages/pyarrow/ipc.py in __init__(self, source)
     43 
     44     def __init__(self, source):
---> 45         self._open(source)
     46 
     47 

/databricks/python/lib/python3.8/site-packages/pyarrow/ipc.pxi in pyarrow.lib._RecordBatchStreamReader._open()

/databricks/python/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/databricks/python/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Tried reading schema message, was null or length 0
```



looks like my arrow files are all empty @mariosasko 

![image](https://user-images.githubusercontent.com/7530947/208179977-9ae62c9a-866c-472b-9a09-25d1191188fb.png)


i also see the `incomplete_info.lock` file a level up too. seems like the data isn't being persisted to disk when I call `download_and_prepare`. is there something else i need to do before then, perhaps?

quick update @mariosasko. i got it working! i had to downgrade to `datasets==2.4.0`. testing other versions now and will let you know the results.

I've tested with every version of `datasets>2.4.0` and i get the same error with all of them.