timothyjlaurent (NONE) commented on 2020-09-22 04:46:11:
Sorry,  I just noticed this.
I'm running this on MACOS the version of datasets I'm was 1.0.0 but I've also tried it on 1.0.2. `pyarrow==1.0.1`, Python 3.6

Consider this code:
```python

    loader_path = str(Path(__file__).parent / "prodigy_dataset_builder.py")
    ds = load_dataset(
        loader_path, name="prodigy-ds", data_files=list(file_paths), cache_dir=cache_dir
    )["train"]
    valid_relations = set(vocabulary.relation_types.keys())

    ds = ds.filter(filter_good_rows, fn_kwargs=dict(valid_rel_labels=valid_relations))

    ds = ds.map(map_bpe_encodings, batched=True, fn_kwargs=dict(tokenizer=vocabulary.tokenizer), num_proc=10)

    # add all feature data
    ner_ds: Dataset = ds.map(
        add_bio_tags,
        fn_kwargs=dict(ner_label_map=vocabulary.ner_labels, tokenizer=vocabulary.tokenizer),
    )
    rel_ds: Dataset = ner_ds.map(
        relation_ds_factory,
        batched=True,
        writer_batch_size=100,
        fn_kwargs=dict(tokenizer=vocabulary.tokenizer, vocabulary=vocabulary),
    )
```
The loader is essentially a jsonloader with some extra error handling. The data is a jsonlines format with text field and a list of span objects and relation objects. 

In the `ner_ds` a field, `ner_labels` is added, this is used in the downstream `relation_ds_factory`. It all runs fine in a single process but I get a KeyError error if run with num_proc set
:

```
  File "/Users/timothy.laurent/src/inv-text2struct/text2struct/model/dataset.py", line 348, in relation_ds_factory
    ner_labels = example["ner_labels"]
KeyError: 'ner_labels'
``` 

This is just one example of what goes wrong.  I've started just saving the dataset as arrow at the end because it takes a long time to map/filter/shuffle and the caching isn't working (tracked it down to byte differences in the pickled functions). 

^^ Interestingly if I heed the warning from Tokenizers and set the environment variable, `TOKENIZERS_PARALLELISM=true` the map just hangs:

```
[I 200921 21:43:18 filelock:318] Lock 5694118768 released on /Users/timothy.laurent/.cache/huggingface/datasets/_Users_timothy.laurent_.cache_huggingface_datasets_prodigy_dataset_builder_prodigy-ds-5f34378723c4e83f_0.0.0_e67d9b43d5cd82c50b1eae8f2097daf95b601a04dc03ddd504f2b234a5fa247a.lock
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.34ba/s]
#0:   0%|                                                                                                            | 0/1 [00:00<?, ?ba/s]
#1:   0%|                                                                                                            | 0/1 [00:00<?, ?ba/s]
#2:   0%|                                                                                                            | 0/1 [00:00<?, ?ba/s]
#3:   0%|                                                                                                            | 0/1 [00:00<?, ?ba/s]
#4:   0%|                                                                                                            | 0/1 [00:00<?, ?ba/s]
#5:   0%|                                                                                                            | 0/1 [00:00<?, ?ba/s]
#6:   0%|                                                                                                            | 0/1 [00:00<?, ?ba/s]
#7:   0%|                                                                                                            | 0/1 [00:00<?, ?ba/s]
#8:   0%|                                                                                                            | 0/1 [00:00<?, ?ba/s]
```