Repository: open-compass/opencompass
Issue: [Feature] Cached Dataset load 
Root cause: Unimplemented Feature Gap

### Describe the feature

```
06/18 23:09:38 - OpenCompass - INFO - Task [Deepseek/alignment_bench]
06/18 23:09:38 - OpenCompass - DEBUG - Modules of opencompass's model registry have been automatically imported from opencompass.models
06/18 23:09:38 - OpenCompass - DEBUG - Get class `OpenAI` from "model" registry in "opencompass"
06/18 23:09:38 - OpenCompass - DEBUG - An `OpenAI` instance is built from registry, and its implementation can be found in opencompass.models.openai_api
06/18 23:09:38 - OpenCompass - DEBUG - Modules of opencompass's load_dataset registry have been automatically imported from opencompass.datasets
06/18 23:09:38 - OpenCompass - DEBUG - Get class `AlignmentBenchDataset` from "load_dataset" registry in "opencompass"
load dataset
aaa
06/18 23:09:38 - OpenCompass - DEBUG - An `AlignmentBenchDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.subjective.alignbench
06/18 23:09:38 - OpenCompass - INFO - Start inferencing [Deepseek/alignment_bench]
06/18 23:09:38 - OpenCompass - DEBUG - Modules of opencompass's icl_prompt_templates registry have been automatically imported from opencompass.openicl.icl_prompt_template
06/18 23:09:38 - OpenCompass - DEBUG - Get class `PromptTemplate` from "icl_prompt_templates" registry in "opencompass"
06/18 23:09:38 - OpenCompass - DEBUG - An `PromptTemplate` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_prompt_template
06/18 23:09:38 - OpenCompass - DEBUG - Modules of opencompass's icl_retrievers registry have been automatically imported from opencompass.openicl.icl_retriever
06/18 23:09:38 - OpenCompass - DEBUG - Get class `ZeroRetriever` from "icl_retrievers" registry in "opencompass"
06/18 23:09:38 - OpenCompass - DEBUG - An `ZeroRetriever` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_retriever.icl_zero_retriever
06/18 23:09:38 - OpenCompass - DEBUG - Modules of opencompass's icl_inferencers registry have been automatically imported from opencompass.openicl.icl_inferencer
06/18 23:09:38 - OpenCompass - DEBUG - Get class `GenInferencer` from "icl_inferencers" registry in "opencompass"
06/18 23:09:38 - OpenCompass - DEBUG - An `GenInferencer` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_inferencer.icl_gen_inferencer
[2024-06-18 23:09:38,700] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
  0%|                                                                                                                                                                                                                   | 0/1 [00:00<?, ?it/s]06/18 23:09:41 - OpenCompass - DEBUG - {'id': 'cmpl-b3f5d1f4abff47d1a124a6e27988ac7d', 'object': 'chat.completion', 'created': 1718723378, 'model': 'corax-sast-llm-dev', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': '\n高音单簧管（Soprano Clarinet）和高音萨克斯（Soprano Saxophone）的调性是不同的。\n\n高音单簧管的调性通常是B调。这意味着它被设计成能够演奏B调的音阶，而它的指法会使得它能够自然地演奏出其他调性的音。\n\n高音萨克斯的调性通常是E调。E调萨克斯风是萨克斯风家族中最为常见和广泛使用的类型之一，因为它能够很好地与各种乐队和音乐风格配合。\n\n因此，高音单簧管是B调，而高音萨克斯是E调。'}, 'logprobs': None, 'finish_reason': 'stop', 'stop_reason': 151336}], 'usage': {'prompt_tokens': 40, 'total_tokens': 173, 'completion_tokens': 133}}
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.83s/it]
06/18 23:09:41 - OpenCompass - DEBUG - Get class `SubjectiveSizePartitioner` from "partitioner" registry in "opencompass"
06/18 23:09:41 - OpenCompass - DEBUG - An `SubjectiveSizePartitioner` instance is built from registry, and its implementation can be found in opencompass.partitioners.sub_size
06/18 23:09:41 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored.
06/18 23:09:41 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored.
06/18 23:09:41 - OpenCompass - DEBUG - Key eval.given_pred not found in config, ignored.
06/18 23:09:41 - OpenCompass - DEBUG - Additional config: {'eval': {'runner': {'task': {}}}}
06/18 23:09:41 - OpenCompass - INFO - Partitioned into 1 tasks.
06/18 23:09:41 - OpenCompass - DEBUG - Task 0: [Deepseek/alignment_bench]
06/18 23:09:41 - OpenCompass - DEBUG - Get class `LocalRunner` from "runner" registry in "opencompass"
06/18 23:09:41 - OpenCompass - DEBUG - An `LocalRunner` instance is built from registry, and its implementation can be found in opencompass.runners.local
06/18 23:09:41 - OpenCompass - DEBUG - Get class `SubjectiveEvalTask` from "task" registry in "opencompass"
06/18 23:09:41 - OpenCompass - DEBUG - An `SubjectiveEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.subjective_eval
06/18 23:09:43 - OpenCompass - DEBUG - Get class `AlignmentBenchDataset` from "load_dataset" registry in "opencompass"
load dataset
aaa
06/18 23:09:43 - OpenCompass - DEBUG - An `AlignmentBenchDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.subjective.alignbench
06/18 23:09:43 - OpenCompass - INFO - No postprocessor found.
06/18 23:09:43 - OpenCompass - DEBUG - Modules of opencompass's icl_evaluators registry have been automatically imported from opencompass.openicl.icl_evaluator
06/18 23:09:43 - OpenCompass - DEBUG - Get class `LMEvaluator` from "icl_evaluators" registry in "opencompass"
06/18 23:09:43 - OpenCompass - DEBUG - Get class `PromptTemplate` from "icl_prompt_templates" registry in "opencompass"
06/18 23:09:43 - OpenCompass - DEBUG - An `PromptTemplate` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_prompt_template
06/18 23:09:43 - OpenCompass - DEBUG - Get class `OpenAI` from "model" registry in "opencompass"
06/18 23:09:43 - OpenCompass - DEBUG - An `OpenAI` instance is built from registry, and its implementation can be found in opencompass.models.openai_api
06/18 23:09:43 - OpenCompass - DEBUG - An `LMEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.lm_evaluator
06/18 23:09:43 - OpenCompass - DEBUG - Get class `AlignmentBenchDataset` from "load_dataset" registry in "opencompass"
load dataset
aaa
06/18 23:09:43 - OpenCompass - DEBUG - An `AlignmentBenchDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.subjective.alignbench
[2024-06-18 23:09:43,344] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
```


Dataset will be loaded multi times , it can reduce the performance , i think we should cached the dataset when the program loaded the datasets firstly

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!