Multi process map did not load cache file correctly
Opened by enze5088 (closed)

### Describe the bug

When I  was training model on Multiple GPUs by DDP, the dataset is tokenized multiple times after main process. 
![1698820541284](https://github.com/huggingface/datasets/assets/14285786/0b2fe054-54d8-4e00-96e6-6ca5b69e662b)

![1698820501568](https://github.com/huggingface/datasets/assets/14285786/dd62bf6f-a58f-41bf-9848-ea4fb3b62b9b)

Code is modified from  [run_clm.py](https://github.com/huggingface/transformers/blob/7d8ff3629b2725ec43ace99c1a6e87ac1978d433/examples/pytorch/language-modeling/run_clm.py#L484)


### Steps to reproduce the bug

```
block_size = data_args.block_size
IGNORE_INDEX = -100
Ignore_Input = False

def tokenize_function(examples):
    sources = []
    targets = []
    for instruction, inputs, output in zip(examples['instruction'], examples['input'], examples['output']):

        source = instruction + inputs 

        target = f"{output}{tokenizer.eos_token}"

        sources.append(source)
        targets.append(target)

    tokenized_sources = tokenizer(sources, return_attention_mask=False)

    tokenized_targets = tokenizer(targets, return_attention_mask=False,
                                  add_special_tokens=False
                                  )

    all_input_ids = []
    all_labels = []
    for s, t in zip(tokenized_sources['input_ids'], tokenized_targets['input_ids']):
        if len(s) > block_size and Ignore_Input == False:
            # print(s)
            continue
        input_ids = torch.LongTensor(s + t)[:block_size]
        if Ignore_Input:
            labels = torch.LongTensor([IGNORE_INDEX] * len(s) + t)[:block_size]
        else:
            labels = input_ids
        assert len(input_ids) == len(labels)
        all_input_ids.append(input_ids)
        all_labels.append(labels)

    results = {
        'input_ids': all_input_ids,
        'labels': all_labels,

    }
    return results



with training_args.main_process_first(desc="dataset map tokenization ", local=False):
    # print('local_rank',training_args.local_rank)
    if not data_args.streaming:
        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on dataset ",
        )
    else:
        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            remove_columns=column_names,
            desc="Running tokenizer on dataset "
        )
```

### Expected behavior

This code should only tokenize the dataset in the main process, and the other processes load the dataset after waiting

### Environment info

transformers == 4.34.1
datasets == 2.14.5

Comments: The inconsistency may be caused by the usage of "update_fingerprint" and setting "trust_remote_code" to "True."
When the tokenizer employs "trust_remote_code," the behavior of the map function varies with each code execution. Even if the remote code of the tokenizer remains the same, the result of "asher.hexdigest()" is found to be inconsistent each time.
This may result in different processes executing multiple maps
![1698841094290](https://github.com/huggingface/datasets/assets/14285786/21fc3c65-e9fd-4a79-b12e-a1d4b9c6cf32)
![1698841117416](https://github.com/huggingface/datasets/assets/14285786/c3e5a530-54d2-4ae6-b902-ce9f85de373b)



The issue may be related to problems previously discussed in GitHub issues [#3847](https://github.com/huggingface/datasets/issues/3847) and [#6318](https://github.com/huggingface/datasets/pull/6318). 
This arises from the fact that tokenizer.tokens_trie._tokens is an unordered set, leading to varying hash results:
`value = hash_bytes(dumps(tokenizer.tokens_trie._tokens))`
Consequently, this results in different outcomes each time for:
`new_fingerprint = update_fingerprint(datasets._fingerprint, transform, kwargs_for_fingerprint)`

To address this issue, it's essential to make `Trie._tokens` a deterministic set while ensuring a consistent order after the final update of `_tokens`.


We now sort `set` and `dict` items to make their hashes deterministic (install from `main` with `pip install git+https://github.com/huggingface/datasets` to test this). Consequently, this should also make the `tokenizer.tokens_trie`'s hash deterministic. Feel free to re-open the issue if this is not the case.