AttributeError: module 'threading' has no attribute '_Condition
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

HuggingFace datasets imported by nemo.collections.llm.gpt.data in /opt/NeMo/nemo/collections/llm/gpt/data/__init__.py:15
`from nemo.collections.llm.gpt.data.dolly import DollyDataModule`

/opt/NeMo/nemo/collections/llm/gpt/data/dolly.py:20
`from datasets import load_dataset`

usr/local/lib/python3.10/dist-packages/datasets/__init__.py:17
from .arrow_dataset import Dataset

...

Typo in Python 3.10? should be threading.Condition instead of threading._Condition ? cf [huggingface/datasets:5613](https://github.com/huggingface/datasets/issues/5613) [apache-beam:24458](https://github.com/apache/beam/issues/24458)
Typo in source code ? should be subclassing threading.Condition instead of threading._Condition ?

~/.local/lib/python3.10/site-packages/multiprocess/dummy/__init__.py:87
class Condition(threading._Condition):
     88     # XXX
     89     if sys.version_info < (3, 0):
     90         notify_all = threading._Condition.notify_all.__func__

AttributeError: module 'threading' has no attribute '_Condition'

**Steps/Code to reproduce bug**

`from nemo.collections import llm`

> ---------------------------------------------------------------------------
> AttributeError                            Traceback (most recent call last)
> Cell In[4], line 1
> ----> 1 from nemo.collections import llm
> 
> File /opt/NeMo/nemo/collections/llm/__init__.py:22
>      20 from nemo.collections.llm import peft
>      21 from nemo.collections.llm.api import export_ckpt, finetune, generate, import_ckpt, pretrain, train, validate
> ---> 22 from nemo.collections.llm.gpt.data import (
>      23     DollyDataModule,
>      24     FineTuningDataModule,
>      25     MockDataModule,
>      26     PreTrainingDataModule,
>      27     SquadDataModule,
>      28 )
>      29 from nemo.collections.llm.gpt.data.api import dolly, mock, squad
>      30 from nemo.collections.llm.gpt.model import (
>      31     Baichuan2Config,
>      32     Baichuan2Config7B,
>    (...)
>     103     gpt_forward_step,
>     104 )
> 
> File /opt/NeMo/nemo/collections/llm/gpt/data/__init__.py:15
>       1 # Copyright (c) 2024, NVIDIA CORPORATION.  All rights reserved.
>       2 #
>       3 # Licensed under the Apache License, Version 2.0 (the "License");
>    (...)
>      12 # See the License for the specific language governing permissions and
>      13 # limitations under the License.
> ---> 15 from nemo.collections.llm.gpt.data.dolly import DollyDataModule
>      16 from nemo.collections.llm.gpt.data.fine_tuning import FineTuningDataModule
>      17 from nemo.collections.llm.gpt.data.mock import MockDataModule
> 
> File /opt/NeMo/nemo/collections/llm/gpt/data/dolly.py:20
>      17 from typing import TYPE_CHECKING, List, Optional
>      19 import numpy as np
> ---> 20 from datasets import load_dataset
>      22 from nemo.collections.llm.gpt.data.core import get_dataset_root
>      23 from nemo.collections.llm.gpt.data.fine_tuning import FineTuningDataModule
> 
> File /usr/local/lib/python3.10/dist-packages/datasets/__init__.py:17
>       1 # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.
>       2 #
>       3 # Licensed under the Apache License, Version 2.0 (the "License");
>    (...)
>      12 # See the License for the specific language governing permissions and
>      13 # limitations under the License.
>      15 __version__ = "3.1.0"
> ---> 17 from .arrow_dataset import Dataset
>      18 from .arrow_reader import ReadInstruction
>      19 from .builder import ArrowBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
> 
> File /usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py:77
>      75 from . import config
>      76 from .arrow_reader import ArrowReader
> ---> 77 from .arrow_writer import ArrowWriter, OptimizedTypedSequence
>      78 from .data_files import sanitize_patterns
>      79 from .download.streaming_download_manager import xgetsize
> 
> File /usr/local/lib/python3.10/dist-packages/datasets/arrow_writer.py:27
>      24 from fsspec.core import url_to_fs
>      26 from . import config
> ---> 27 from .features import Audio, Features, Image, Value, Video
>      28 from .features.features import (
>      29     FeatureType,
>      30     _ArrayXDExtensionType,
>    (...)
>      37     to_pyarrow_listarray,
>      38 )
>      39 from .filesystems import is_remote_filesystem
> 
> File /usr/local/lib/python3.10/dist-packages/datasets/features/__init__.py:17
>       1 __all__ = [
>       2     "Audio",
>       3     "Array2D",
>    (...)
>      15     "Video",
>      16 ]
> ---> 17 from .audio import Audio
>      18 from .features import Array2D, Array3D, Array4D, Array5D, ClassLabel, Features, LargeList, Sequence, Value
>      19 from .image import Image
> 
> File /usr/local/lib/python3.10/dist-packages/datasets/features/audio.py:13
>      11 from ..table import array_cast
>      12 from ..utils.file_utils import xopen, xsplitext
> ---> 13 from ..utils.py_utils import no_op_if_value_is_null, string_to_dict
>      16 if TYPE_CHECKING:
>      17     from .features import FeatureType
> 
> File /usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py:37
>      34 from urllib.parse import urlparse
>      36 import multiprocess
> ---> 37 import multiprocess.pool
>      38 import numpy as np
>      39 from tqdm.auto import tqdm
> 
> File ~/.local/lib/python3.10/site-packages/multiprocess/pool.py:609
>     603             self._cond.release()
>     605 #
>     606 #
>     607 #
> --> 609 class ThreadPool(Pool):
>     611     from .dummy import Process
>     613     def __init__(self, processes=None, initializer=None, initargs=()):
> 
> File ~/.local/lib/python3.10/site-packages/multiprocess/pool.py:611, in ThreadPool()
>     609 class ThreadPool(Pool):
> --> 611     from .dummy import Process
>     613     def __init__(self, processes=None, initializer=None, initargs=()):
>     614         Pool.__init__(self, processes, initializer, initargs)
> 
> File ~/.local/lib/python3.10/site-packages/multiprocess/dummy/__init__.py:87
>      81             return None
>      83 #
>      84 #
>      85 #
> ---> 87 class Condition(threading._Condition):
>      88     # XXX
>      89     if sys.version_info < (3, 0):
>      90         notify_all = threading._Condition.notify_all.__func__
> 
> AttributeError: module 'threading' has no attribute '_Condition'

**Expected behavior**

Import should occur without triggering an error

**Environment overview (please complete the following information)**

 - Environment location: [Docker]
 - Method of NeMo install: Docker, docker pull nvcr.io/nvidia/nemo:24.09 docker run docker run --gpus all -it --rm --shm-size=16g --ulimit memlock=-1 --ulimit stack=67108864 nvcr.io/nvidia/nemo:24.09

**Additional context**

Add any other context about the problem here.

Typo in Python 3.10? should be subclassing threading.Condition instead of threading._Condition ?

Ok docker quietened following error log from pip, this is an apache-beam issue, closing issue.

> ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
> cudf 24.4.0 requires pyarrow<15.0.0a0,>=14.0.1, but you have pyarrow 16.1.0 which is incompatible.
> cugraph 24.4.0 requires dask-cuda==24.4.*, but you have dask-cuda 24.8.2 which is incompatible.
> cugraph 24.4.0 requires rapids-dask-dependency==24.4.*, but you have rapids-dask-dependency 24.8.0 which is incompatible.
> cugraph-service-server 24.4.0 requires dask-cuda==24.4.*, but you have dask-cuda 24.8.2 which is incompatible.
> cugraph-service-server 24.4.0 requires rapids-dask-dependency==24.4.*, but you have rapids-dask-dependency 24.8.0 which is incompatible.
> cuml 24.4.0 requires dask-cuda==24.4.*, but you have dask-cuda 24.8.2 which is incompatible.
> cuml 24.4.0 requires rapids-dask-dependency==24.4.*, but you have rapids-dask-dependency 24.8.0 which is incompatible.
> cuml 24.4.0 requires treelite==4.1.2, but you have treelite 4.3.0 which is incompatible.
> dask-cudf 24.4.0 requires rapids-dask-dependency==24.4.*, but you have rapids-dask-dependency 24.8.0 which is incompatible.
> multiprocess 0.70.16 requires dill>=0.3.8, but you have dill 0.3.1.1 which is incompatible.
> tensorflow-metadata 1.16.1 requires protobuf<4.21,>=3.20.3; python_version < "3.11", but you have protobuf 4.25.5 which is incompatible.
> tensorrt-llm 0.12.0 requires nvidia-modelopt~=0.15.0, but you have nvidia-modelopt 0.0.0 which is incompatible.
> tensorrt-llm 0.12.0 requires pynvml>=11.5.0, but you have pynvml 11.4.1 which is incompatible.
> tensorrt-llm 0.12.0 requires transformers<=4.42.4,>=4.38.2, but you have transformers 4.46.2 which is incompatible.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.