Broken offline mode of NeMo
State: open | Pull request: no
Labels: name: bug

**Describe the bug**

`huggingface_hub` has an offline mode ([`HF_HUB_OFFLINE`](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables#hfhuboffline)). `huggingface_hub` looks only into local cache if this is activated, and if a model is found, everything can be used offline without internet connection.

In `NeMo` there is a function call (https://github.com/NVIDIA/NeMo/blob/main/nemo/core/classes/common.py#L863) that checks if the model exists on HuggingFace hub. So even if the model is already cached, this call needs to establish a connection to HuggingFace hub.

**Steps/Code to reproduce bug**

1) Run a script while having internet connection to download a model, e.g.
  ~~~python
  from nemo.collections.asr.models import EncDecCTCModelBPE

  model = EncDecCTCModelBPE.from_pretrained("nvidia/stt_en_conformer_ctc_large")
  ~~~
2) Set the environment variable (`HF_HUB_OFFLINE=1`), disable internet connection & rerun script:
  ~~~python
  from nemo.collections.asr.models import EncDecCTCModelBPE

  model = EncDecCTCModelBPE.from_pretrained("nvidia/stt_en_conformer_ctc_large")
  ~~~

  Traceback:
  ~~~
  Traceback (most recent call last):
    File "test_debug.py", line 21, in <module>
      model = EncDecMultiTaskModel.from_pretrained("nvidia/canary-1b")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File ".venv/lib/python3.11/site-packages/nemo/core/classes/common.py", line 747, in from_pretrained
      class_, nemo_model_file_in_cache = cls._get_hf_hub_pretrained_model_info(
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File ".venv/lib/python3.11/site-packages/nemo/core/classes/common.py", line 873, in _get_hf_hub_pretrained_model_info
      nemo_file_exists = api.file_exists(repo_id=model_name, filename=resolved_model_filename, repo_type="model")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File ".venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
      return fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
    File ".venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 2899, in file_exists
      get_hf_file_metadata(url, token=token)
    File ".venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
      return fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
    File ".venv/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1294, in get_hf_file_metadata
      r = _request_wrapper(
          ^^^^^^^^^^^^^^^^^
    File ".venv/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 278, in _request_wrapper
      response = _request_wrapper(
                 ^^^^^^^^^^^^^^^^^
    File ".venv/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 301, in _request_wrapper
      response = get_session().request(method=method, url=url, **params)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File ".venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
      resp = self.send(prep, **send_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File ".venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
      r = adapter.send(request, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File ".venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 104, in send
      raise OfflineModeIsEnabled(
  huggingface_hub.errors.OfflineModeIsEnabled: Cannot reach https://huggingface.co/nvidia/canary-1b/resolve/main/canary-1b.nemo: offline mode is enabled. To disable it, please unset the `HF_HUB_OFFLINE` environment variable.
  ~~~

**Expected behavior**

It should take the cached model without relying on an internet connection.