Repository: EleutherAI/lm-evaluation-harness
Issue: Too slow process for ppq
Root cause: Algorithmic Error

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/790212f7-a72b-48c4-b557-df6b0f0b7dae)
When I evaluated model for ppq, I found the processing of ppq was too slow, it is unbearable for 390, 965 ppq test data.
Are you have an idea?

btw, I added the log for the evaluator.py
```
    for task_name, task in task_dict.items():
        print("task_name:",task_name)
        if type(task) == tuple:
            group, task = task
            if task is None:
                continue
        # TODO: make it possible to use a different metric per filter
        # iterate over different filters used
        for key in task.instances[0].filtered_resps.keys():
            doc_iterator = (
                itertools.islice(
                    enumerate(task.test_docs()), lm.rank, limit, lm.world_size
                )
                if task.has_test_docs()
                else itertools.islice(
                    enumerate(task.validation_docs()), lm.rank, limit, lm.world_size
                )
            )
            for doc_id, doc in tqdm(doc_iterator):
                # subset instances to only this document id ; sort by idx
                # print(1)
                requests = list(filter(lambda x: x.doc_id == doc_id, task.instances))
                requests.sort(key=lambda x: x.idx)
                metrics = task.process_results(
                    doc, [req.filtered_resps[key] for req in requests]
                )
                if log_samples:
                    target = task.doc_to_target(doc)
                    example = {
                        "doc_id": doc_id,
                        "doc": doc,
                        "target": target,
                        "arguments": [req.args for req in requests],
                        "resps": [req.resps for req in requests],
                        "filtered_resps": [req.filtered_resps[key] for req in requests],
                    }
                    example.update(metrics)
                    samples[task_name].append(example)
                for metric, value in metrics.items():
                    vals[(task_name, key, metric)].append(value)
```

On a side note I noticed that many of the `glue` tasks (e.g [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/glue/cola/default.yaml)) seem to be missing test splits in the configs even though the [dataset](https://huggingface.co/datasets/glue) includes them. Not sure if this is deliberate but the sizes do match up with those in the [paper](https://openreview.net/pdf?id=rJ4km2R5t7) (pg. 2).

cc @haileyschoelkopf @lintangsutawika 

This is because the GLUE tasks do not supply public labels for the test sets--all labels are marked as `-1` rather than 0 or 1.

To get a test set score one has to submit to the GLUE leaderboard online for scoring of your predictions. (We should add a `--predict_only` flag though to just save outputs though.)

Ah got it.  Should have looked more closely. `qqp` and `mnli_mismatch` do link to the unlabelled test sets so maybe we should fix that.

On the current most recent main-branch commit, I no longer see QQP taking a long time to preprocess. @shiweijiezero -- do you still observe this on the most recent commit? 

If so, could you share some specs of your machine so I can compare to my setup?

> On the current most recent main-branch commit, I no longer see QQP taking a long time to preprocess. @shiweijiezero -- do you still observe this on the most recent commit?
> 
> If so, could you share some specs of your machine so I can compare to my setup?

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/caa09e94-a31c-4f70-9289-46fcfa36c2eb)
Yes! It seems to work normally, thanks.