Title: Fork dataset
Author: timothyjlaurent (NONE)
State: closed
Created: 2020-06-30T16:42:53Z

We have a multi-task learning model training I'm trying to convert to using the Arrow-based nlp dataset. 

We're currently training a custom TensorFlow model but the nlp paradigm should be a bridge for us to be able to use the wealth of pre-trained models in Transformers.

Our preprocessing flow parses raw text and json with Entity and Relations annotations and creates 2 datasets for training a NER and Relations prediction heads.

Is there some good way to "fork" dataset-

EG

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 -> DatasetREL

or 

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 + DatasetNER -> DatasetREL

Comments:
To be able to generate the Arrow dataset you need to either use our csv or json utilities `load_dataset("json", data_files=my_json_files)` OR write your own custom dataset script (you can find some inspiration from the [squad](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py) script for example). Custom dataset scripts can be called locally with `nlp.load_dataset(path_to_my_script_directory)`.

This should help you get what you call "Dataset1".

Then using some dataset transforms like `.map` for example you can get to "DatasetNER" and "DatasetREL".


Thanks for the helpful advice, @lhoestq  -- I wasn't quite able to get the json recipe working - 

```
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/ipc.py in __init__(self, source)
     60 
     61     def __init__(self, source):
---> 62         self._open(source)
     63 
     64 
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/ipc.pxi in pyarrow.lib._RecordBatchStreamReader._open()
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
ArrowInvalid: Tried reading schema message, was null or length 0
```

But I'm going to give the generator_dataset_builder a try.

1 more quick question -- can .map be used to output different length mappings -- could I skip one, or yield 2, can you map_batch  

You can use `.map(my_func, batched=True)` and return less examples, or more examples if you want

Thanks this answers my question. I think the issue I was having using the json loader were due to using gzipped jsonl files.

The error I get now is :

```

Using custom data configuration test
---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-38-29082a31e5b2> in <module>
      5 print(ner_datafiles)
      6 
----> 7 ds = nlp.load_dataset("json", "test",  data_files=ner_datafiles[0])
      8 

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526 

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    430                 verify_infos = not save_infos and not ignore_verifications
    431                 self._download_and_prepare(
--> 432                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    433                 )
    434                 # Sync info

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    481             try:
    482                 # Prepare split will record examples associated to the split
--> 483                 self._prepare_split(split_generator, **prepare_split_kwargs)
    484             except OSError:
    485                 raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _prepare_split(self, split_generator)
    736                     schema_dict[field.name] = Value(str(field.type))
    737 
--> 738         parse_schema(writer.schema, features)
    739         self.info.features = Features(features)
    740 

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in parse_schema(schema, schema_dict)
    734                     parse_schema(field.type.value_type, schema_dict[field.name])
    735                 else:
--> 736                     schema_dict[field.name] = Value(str(field.type))
    737 
    738         parse_schema(writer.schema, features)

<string> in __init__(self, dtype, id, _type)

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in __post_init__(self)
     55 
     56     def __post_init__(self):
---> 57         self.pa_type = string_to_arrow(self.dtype)
     58 
     59     def __call__(self):

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in string_to_arrow(type_str)
     32         if str(type_str + "_") not in pa.__dict__:
     33             raise ValueError(
---> 34                 f"Neither {type_str} nor {type_str + '_'} seems to be a pyarrow data type. "
     35                 f"Please make sure to use a correct data type, see: "
     36                 f"https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions"

ValueError: Neither list<item: int64> nor list<item: int64>_ seems to be a pyarrow data type. Please make sure to use a correct data type, see: https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions.
```

If I just create a pa- table manually like is done in the jsonloader -- it seems to work fine. Ths JSON I'm trying to load isn't overly complex -  1 integer field, the rest text fields with a nested list of objects with text fields .

I'll close this -- It's still unclear how to go about troubleshooting the json example as I mentioned above. If I decide it's worth the trouble, I'll create another issue, or wait for a better support for using nlp for making custom data-loaders.