Repository: huggingface/evaluate
Issue: Too Long Package Import
Root cause: Architectural Constraint

Package `evaluate` takes too much time to import while it is not so heavy as TensorFlow or Torch.

```shell
$ time python -c 'import torch; print(torch.ones(3))' >> /dev/null
python -c 'import torch; print(torch.ones(3))'  0,84s user 0,06s system 95% cpu 0,942 total
$ time python -c 'import evaluate'                   
python -c 'import evaluate'  2,23s user 0,19s system 98% cpu 2,453 total
```

It takes 2.5 seconds against 0.9 seconds for PyTorch! Moreover CLI takes the same time and fails on startup.

```shell
$ time evaluate-cli   
Traceback (most recent call last):
  File "/usr/bin/evaluate-cli", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/evaluate/commands/evaluate_cli.py", line 72, in main
    if args["module_type"] not in ["metric", "comparison", "measurement"]:
KeyError: 'module_type'
evaluate-cli  2,31s user 0,17s system 99% cpu 2,490 total
```

Indeed, that's pretty slow. Did you investigate what caused the slow loading? 

For the CLI you have to pass some arguments, e.g.:
```bash
evaluate-cli create "Awesome Metric"
```

Sure, I started from creating a template for a new metric and noticed that it takes a lot of time as well as plain import of `evaluate`.

The major contribution to import time is given by importing HuggingFace's packages `transformers.pipelines`, `transformers`, `datasets`. Here is top20 of the longest calls.

```
Fri Jan  6 22:04:17 2023    evaluate-cli.profile

         2797701 function calls (2747192 primitive calls) in 2.834 seconds

   Ordered by: cumulative time
   List reduced from 12326 to 9183 due to restriction <'/usr/lib/python3.10/site-packages'>
   List reduced from 9183 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    2.802    2.802 /usr/lib/python3.10/site-packages/evaluate/__init__.py:1(<module>)
        1    0.000    0.000    2.793    2.793 /usr/lib/python3.10/site-packages/evaluate/evaluation_suite/__init__.py:1(<module>)
        1    0.000    0.000    2.127    2.127 /usr/lib/python3.10/site-packages/evaluate/evaluator/__init__.py:1(<module>)
        1    0.000    0.000    1.715    1.715 /usr/lib/python3.10/site-packages/transformers/pipelines/__init__.py:1(<module>)
        1    0.000    0.000    0.825    0.825 /usr/lib/python3.10/site-packages/transformers/pipelines/document_question_answering.py:1(<module>)
        1    0.000    0.000    0.824    0.824 /usr/lib/python3.10/site-packages/transformers/pipelines/question_answering.py:1(<module>)
        1    0.000    0.000    0.824    0.824 /usr/lib/python3.10/site-packages/transformers/data/__init__.py:1(<module>)
        1    0.000    0.000    0.817    0.817 /usr/lib/python3.10/site-packages/transformers/data/metrics/__init__.py:1(<module>)
        1    0.000    0.000    0.800    0.800 /usr/lib/python3.10/site-packages/transformers/utils/import_utils.py:538(is_sklearn_available)
        1    0.000    0.000    0.800    0.800 /usr/lib/python3.10/site-packages/sklearn/__init__.py:1(<module>)
        1    0.000    0.000    0.798    0.798 /usr/lib/python3.10/site-packages/sklearn/base.py:1(<module>)
        1    0.000    0.000    0.797    0.797 /usr/lib/python3.10/site-packages/sklearn/utils/__init__.py:1(<module>)
        1    0.000    0.000    0.792    0.792 /usr/lib/python3.10/site-packages/transformers/pipelines/audio_classification.py:1(<module>)
        1    0.000    0.000    0.792    0.792 /usr/lib/python3.10/site-packages/transformers/pipelines/base.py:1(<module>)
        1    0.000    0.000    0.790    0.790 /usr/lib/python3.10/site-packages/transformers/modelcard.py:1(<module>)
        1    0.000    0.000    0.788    0.788 /usr/lib/python3.10/site-packages/transformers/training_args.py:1(<module>)
        1    0.000    0.000    0.782    0.782 /usr/lib/python3.10/site-packages/transformers/debug_utils.py:1(<module>)
        1    0.002    0.002    0.781    0.781 /usr/lib/python3.10/site-packages/torch/__init__.py:1(<module>)
        1    0.000    0.000    0.662    0.662 /usr/lib/python3.10/site-packages/datasets/__init__.py:1(<module>)
        1    0.000    0.000    0.546    0.546 /usr/lib/python3.10/site-packages/datasets/arrow_dataset.py:1(<module>)
```