Repository: EleutherAI/lm-evaluation-harness
Issue: Followed the documentation but its not working. Details below - 
Root cause: External Dependency Breakage

When running below snippet - 

python main.py \
    --model hf-causal \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0

Getting this error - 

/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/framework/dtypes.py:513: FutureWarning: In the future `np.object` will be defined as the corresponding NumPy scalar.
  np.object,
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/compat/__init__.py", line 42, in tf
    from tensorboard.compat import notf  # noqa: F401
ImportError: cannot import name 'notf' from 'tensorboard.compat' (/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/compat/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/abhijit/LLM_Eval/lm-evaluation-harness/main.py", line 6, in <module>
    from lm_eval import tasks, evaluator, utils
  File "/home/ubuntu/abhijit/LLM_Eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 5, in <module>
    import lm_eval.base
  File "/home/ubuntu/abhijit/LLM_Eval/lm-evaluation-harness/lm_eval/base.py", line 14, in <module>
    from accelerate import find_executable_batch_size
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/accelerate/__init__.py", line 3, in <module>
    from .accelerator import Accelerator
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/accelerate/accelerator.py", line 39, in <module>
    from .tracking import LOGGER_TYPE_TO_CLASS, GeneralTracker, filter_trackers
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/accelerate/tracking.py", line 42, in <module>
    from torch.utils import tensorboard
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py", line 12, in <module>
    from .writer import FileWriter, SummaryWriter  # noqa: F401
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/utils/tensorboard/writer.py", line 16, in <module>
    from ._embedding import (
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/utils/tensorboard/_embedding.py", line 9, in <module>
    _HAS_GFILE_JOIN = hasattr(tf.io.gfile, "join")
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/lazy.py", line 65, in __getattr__
    return getattr(load_once(self), attr_name)
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/lazy.py", line 97, in wrapper
    cache[arg] = f(arg)
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/lazy.py", line 50, in load_once
    module = load_fn()
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/compat/__init__.py", line 45, in tf
    import tensorflow
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/__init__.py", line 41, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/__init__.py", line 46, in <module>
    from tensorflow.python import data
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/__init__.py", line 25, in <module>
    from tensorflow.python.data import experimental
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/experimental/__init__.py", line 96, in <module>
    from tensorflow.python.data.experimental import service
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/experimental/service/__init__.py", line 140, in <module>
    from tensorflow.python.data.experimental.ops.data_service_ops import distribute
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/experimental/ops/data_service_ops.py", line 25, in <module>
    from tensorflow.python.data.experimental.ops import compression_ops
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/experimental/ops/compression_ops.py", line 20, in <module>
    from tensorflow.python.data.util import structure
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/util/structure.py", line 26, in <module>
    from tensorflow.python.data.util import nest
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/util/nest.py", line 41, in <module>
    from tensorflow.python.framework import sparse_tensor as _sparse_tensor
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/framework/sparse_tensor.py", line 29, in <module>
    from tensorflow.python.framework import constant_op
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/framework/constant_op.py", line 29, in <module>
    from tensorflow.python.eager import execute
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 27, in <module>
    from tensorflow.python.framework import dtypes
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/framework/dtypes.py", line 513, in <module>
    np.object,
  File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/numpy/__init__.py", line 319, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'object'.
`np.object` was a deprecated alias for the builtin `object`. To avoid this error in existing code, use `object` by itself. Doing this will not modify any behavior and is safe. 
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations


Hi, I've not seen this error before but it looks like a problem with tensorboard, which is not a dependency on our end.

This should be fixed by doing `pip install numpy==1.23` I believe.

Could you run `pip freeze` and paste that output here? That should provide some insight.

Closing as I can't replicate this and changing numpy version should fix this, please reopen if this issue persists!