Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval python3.10 error
Root cause: External Dependency Breakage

When trying to import `lm_eval`, with python3.10, I get the following error:

```
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1382, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/share/apps/python-3.10.0-shared/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/user/.local/lib/python3.10/site-packages/transformers/pipelines/__init__.py", line 65, in <module>
    from .document_question_answering import DocumentQuestionAnsweringPipeline
  File "/home/user/.local/lib/python3.10/site-packages/transformers/pipelines/document_question_answering.py", line 29, in <module>
    from .question_answering import select_starts_ends
  File "/home/user/.local/lib/python3.10/site-packages/transformers/pipelines/question_answering.py", line 9, in <module>
    from ..data import SquadExample, SquadFeatures, squad_convert_examples_to_features
  File "/home/user/.local/lib/python3.10/site-packages/transformers/data/__init__.py", line 26, in <module>
    from .metrics import glue_compute_metrics, xnli_compute_metrics
  File "/home/user/.local/lib/python3.10/site-packages/transformers/data/metrics/__init__.py", line 19, in <module>
    from scipy.stats import pearsonr, spearmanr
  File "/share/apps/python-3.10.0-shared/lib/python3.10/site-packages/scipy/stats/__init__.py", line 391, in <module>
    from .stats import *
  File "/share/apps/python-3.10.0-shared/lib/python3.10/site-packages/scipy/stats/stats.py", line 180, in <module>
    from . import distributions
  File "/share/apps/python-3.10.0-shared/lib/python3.10/site-packages/scipy/stats/distributions.py", line 8, in <module>
    from ._distn_infrastructure import (entropy, rv_discrete, rv_continuous,
  File "/share/apps/python-3.10.0-shared/lib/python3.10/site-packages/scipy/stats/_distn_infrastructure.py", line 23, in <module>
    from scipy import optimize
  File "/share/apps/python-3.10.0-shared/lib/python3.10/site-packages/scipy/optimize/__init__.py", line 421, in <module>
    from ._shgo import shgo
  File "/share/apps/python-3.10.0-shared/lib/python3.10/site-packages/scipy/optimize/_shgo.py", line 9, in <module>
    from scipy import spatial
  File "/share/apps/python-3.10.0-shared/lib/python3.10/site-packages/scipy/spatial/__init__.py", line 107, in <module>
    from . import distance, transform
  File "/share/apps/python-3.10.0-shared/lib/python3.10/site-packages/scipy/spatial/transform/__init__.py", line 19, in <module>
    from .rotation import Rotation, Slerp
ImportError: /share/apps/python-3.10.0-shared/lib/python3.10/site-packages/scipy/spatial/transform/rotation.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/AutoGPTQ/harness_test.py", line 12, in <module>
    import lm_eval.evaluator as evaluator
  File "/home/user/lm-evaluation-harness/lm_eval/__init__.py", line 1, in <module>
    from .evaluator import evaluate, simple_evaluate
  File "/home/user/lm-evaluation-harness/lm_eval/evaluator.py", line 10, in <module>
    import lm_eval.tasks
  File "/home/user/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 7, in <module>
    from lm_eval.api.task import TaskConfig, Task, ConfigurableTask
  File "/home/user/lm-evaluation-harness/lm_eval/api/task.py", line 16, in <module>
    from lm_eval.api.metrics import (
  File "/home/user/lm-evaluation-harness/lm_eval/api/metrics.py", line 6, in <module>
    import evaluate
  File "/home/user/.local/lib/python3.10/site-packages/evaluate/__init__.py", line 29, in <module>
    from .evaluation_suite import EvaluationSuite
  File "/home/user/.local/lib/python3.10/site-packages/evaluate/evaluation_suite/__init__.py", line 10, in <module>
    from ..evaluator import evaluator
  File "/home/user/.local/lib/python3.10/site-packages/evaluate/evaluator/__init__.py", line 27, in <module>
    from .audio_classification import AudioClassificationEvaluator
  File "/home/user/.local/lib/python3.10/site-packages/evaluate/evaluator/audio_classification.py", line 23, in <module>
    from .base import EVALUATOR_COMPUTE_RETURN_DOCSTRING, EVALUTOR_COMPUTE_START_DOCSTRING, Evaluator
  File "/home/user/.local/lib/python3.10/site-packages/evaluate/evaluator/base.py", line 34, in <module>
    from transformers import Pipeline, pipeline
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/user/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1372, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/user/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1384, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.pipelines because of the following error (look up to see its traceback):
/share/apps/python-3.10.0-shared/lib/python3.10/site-packages/scipy/spatial/transform/rotation.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send
```
Any idea why? What's the recommended version of python I should use?

You should be able to use any version of Python 3.8 and up (we have unit tests for up to 3.11 ). 


It looks like this is the fix you want (installing scipy from source)? https://github.com/scipy/scipy/issues/14936 I didn't realize Scipy might have issues with higher python versions, so this may merit further investigation.



Solved by installing scipy 1.7.2 instead of scipy 1.6.7 which is automatically installed with python>=3.10