ghomasHudson (CONTRIBUTOR) commented on 2020-06-26 19:04:44:
Here's a fully working implementation based on the `__iter__` function of @zphang.

- I've generated the task choice list in the constructor as it allows us to index into the MultiDataset just like a normal dataset. I'm changing `task_choice_list` into a list of `(dataset_idx, example_idx)` so each entry references a unique dataset example. The shuffling has to be done before this as we don't want to shuffle within each task (we assume this is done by the user if this is what they intend).
-  I'm slightly concerned this list could become very large if many large datasets were used. Can't see a way round it at the moment though.
- I've used `task.info.builder_name` as the dataset name. Not sure if this is correct.
- I'd love to add some of the other `Dataset` methods (map, slicing by column, etc...). Would be great to implement the whole interface so a single dataset can be simply replaced by this.
- This does everything on the individual example-level. If some application required batches all from a single task in turn we can't really do that.

```python
import nlp
import numpy as np

class MultiDataset:
    def __init__(self,tasks):
        self.tasks = tasks

        # Create random order of tasks
        # Using size-proportional sampling
        task_choice_list = []
        for i, task in enumerate(self.tasks):
            task_choice_list += [i] * len(task)
        task_choice_list = np.array(task_choice_list)
        np.random.shuffle(task_choice_list)

        # Add index into each dataset
        # - We don't want to shuffle within each task
        counters = {}
        self.task_choice_list = []
        for i in range(len(task_choice_list)):
            idx = counters.get(task_choice_list[i],0)
            self.task_choice_list.append((task_choice_list[i],idx))
            counters[task_choice_list[i]] = idx + 1


    def __len__(self):
        return np.sum([len(t) for t in self.tasks])

    def __repr__(self):
        task_str = ", ".join([str(t) for t in self.tasks])
        return f"MultiDataset(tasks: {task_str})"

    def __getitem__(self,key):
        if isinstance(key, int):
            task_idx, example_idx = self.task_choice_list[key]
            task = self.tasks[task_idx]
            example = task[example_idx]
            example["task_name"] = task.info.builder_name
            return example
        elif isinstance(key, slice):
            raise NotImplementedError()

    def __iter__(self):
        for i in range(len(self)):
            yield self[i]


def load_multitask(*datasets):
    '''Create multitask datasets per split'''

    def _get_common_splits(datasets):
        '''Finds the common splits present in all self.datasets'''
        min_set = None
        for dataset in datasets:
            if min_set != None:
                min_set.intersection(set(dataset.keys()))
            else:
                min_set = set(dataset.keys())
        return min_set

    common_splits = _get_common_splits(datasets)
    out = {}
    for split in common_splits:
        out[split] = MultiDataset([d[split] for d in datasets])
    return out


##########################################
# Dataset Flattening

def flatten(dataset,flatten_fn):
    for k in dataset.keys():
        if isinstance(dataset[k],nlp.Dataset):
            dataset[k] = dataset[k].map(flatten_fn,remove_columns=dataset[k].column_names)

# Squad
def flatten_squad(example):
    return {"source": "squad context: " + example['context'] + " question: " + example['question'],
          "target":example["answers"]["text"]}
squad = nlp.load_dataset("squad")
flatten(squad,flatten_squad)

# CNN_DM
def flatten_cnn_dm(example):
    return {"source": "cnn_dm: " + example['article'],"target":[example["highlights"]]}
cnn_dm = nlp.load_dataset("cnn_dailymail", "3.0.0")
flatten(cnn_dm,flatten_cnn_dm)

#############################################

mtds = load_multitask(squad,cnn_dm)

for example in mtds["train"]:
    print(example["task_name"],example["target"])
```
Let me know if you have any thoughts. I've started using this in some of my projects and it seems to work. If people are happy with the general approach for a first version, I can make a pull request.