Repository: stanford-crfm/helm
Issue: Update retry logic for fetching HuggingFace tokenizer
Root cause: Validation Gap

Intermittently failing

Adapter.adapt {
      3623 instances, choosing 5/1102 train instances, 2521 eval instances
      Creating huggingface/gpt2_tokenizer_fast with Hugging Face Transformers {
      } [11.95s]
      The request failed. Attempt #2, retrying in 11 seconds...
      Creating huggingface/gpt2_tokenizer_fast with Hugging Face Transformers {
      } [23.101s]
      The request failed. Attempt #3, retrying in 45 seconds...
      Creating huggingface/gpt2_tokenizer_fast with Hugging Face Transformers {
      } [16.397s]
      The request failed. Attempt #4, retrying in 81 seconds...
      Creating huggingface/gpt2_tokenizer_fast with Hugging Face Transformers {
      } [28.512s]
      The request failed. Attempt #5, retrying in 150 seconds...
      Creating huggingface/gpt2_tokenizer_fast with Hugging Face Transformers {
      } [36.721s]
    } [4m26.814s]
  } [7m24.558s]
  Error when running narrative_qa:model=text,data_augmentation=canonical:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 126, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 75, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 89, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 110, in run_one
    scenario_state: ScenarioState = adapter.adapt(instances)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/adapter.py", line 332, in adapt
    prompt: str = self.construct_prompt(train_instances, eval_instance)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/adapter.py", line 485, in construct_prompt
    if self.tokenizer.fits_within_context_window(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/tokenizer/gpt2_tokenizer.py", line 87, in fits_within_context_window
    return self.tokenize_and_count(text) + expected_completion_token_length <= self.max_request_length
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/tokenizer/gpt2_tokenizer.py", line 80, in tokenize_and_count
    return len(self.tokenize(text))
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/tokenizer/gpt2_tokenizer.py", line 75, in tokenize
    response: TokenizationRequestResult = self.service.tokenize(TokenizationRequest(text))
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/tokenizer/tokenizer_service.py", line 25, in tokenize
    return self._service.tokenize(self._auth, request)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/proxy/server_service.py", line 100, in tokenize
    return self.client.tokenize(request)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/proxy/auto_client.py", line 113, in tokenize
    return tokenize_with_retry(client=client, request=request)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/proxy/retry.py", line 54, in <lambda>
    return lambda f: lambda *args, **kwargs: _retrying.call(f, *args, **kwargs)
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/six.py", line 719, in reraise
    raise value
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/proxy/auto_client.py", line 107, in tokenize_with_retry
    return client.tokenize(request)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/proxy/huggingface_client.py", line 178, in tokenize
    tokenizer = HuggingFaceTokenizers.get_tokenizer(request.tokenizer)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/proxy/huggingface_tokenizer.py", line 26, in get_tokenizer
    HuggingFaceTokenizers.tokenizers[tokenizer_name] = GPT2TokenizerFast.from_pretrained("gpt2")
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1692, in from_pretrained
    resolved_vocab_files[file_id] = cached_path(
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/transformers/file_utils.py", line 1625, in cached_path
    output_path = get_from_cache(
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/transformers/file_utils.py", line 1849, in get_from_cache
    raise ValueError(
ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.