Repository: stanford-crfm/helm
Issue: GPU summarization metrics require sentencepiece
Root cause: Environment Incompatibility

Attempting to use GPU summarization metrics without `sentencepiece` installed leads to errors:

```

Traceback (most recent call last):
  File "/path/to/site-packages/transformers/convert_slow_tokenizer.py", line 1633, in convert_slow_tokenizer
    return TikTokenConverter(
  File "/path/to/site-packages/transformers/convert_slow_tokenizer.py", line 1533, in converted
    tokenizer = self.tokenizer()
  File "/path/to/site-packages/transformers/convert_slow_tokenizer.py", line 1526, in tokenizer
    vocab_scores, merges = self.extract_vocab_merges_from_model(self.vocab_file)
  File "/path/to/site-packages/transformers/convert_slow_tokenizer.py", line 1502, in extract_vocab_merges_from_model
    bpe_ranks = load_tiktoken_bpe(tiktoken_url)
  File "/path/to/site-packages/tiktoken/load.py", line 144, in load_tiktoken_bpe
    contents = read_file_cached(tiktoken_bpe_file, expected_hash)
  File "/path/to/site-packages/tiktoken/load.py", line 48, in read_file_cached
    cache_key = hashlib.sha1(blobpath.encode()).hexdigest()
AttributeError: 'NoneType' object has no attribute 'encode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/helm/src/helm/benchmark/runner.py", line 216, in run_all
    self.run_one(run_spec)
  File "/path/to/helm/src/helm/benchmark/runner.py", line 307, in run_one
    metric_result: MetricResult = metric.evaluate(
  File "/path/to/helm/src/helm/benchmark/metrics/summarization_metrics.py", line 133, in evaluate
    return super().evaluate(scenario_state, metric_service, eval_cache_path, parallelism=parallelism)
  File "/path/to/helm/src/helm/benchmark/metrics/metric.py", line 143, in evaluate
    results: List[List[Stat]] = parallel_map(
  File "/path/to/helm/src/helm/common/general.py", line 232, in parallel_map
    results = list(tqdm(map(process, items), total=len(items), disable=None))
  File "/path/to/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/path/to/helm/src/helm/benchmark/metrics/metric.py", line 77, in process
    self.metric.evaluate_generation(
  File "/path/to/helm/src/helm/benchmark/metrics/summarization_metrics.py", line 220, in evaluate_generation
    for name, val in self._compute_faithfulness_scores(inp, pred).items()
  File "/path/to/helm/src/helm/benchmark/metrics/summarization_metrics.py", line 152, in _compute_faithfulness_scores
    return {"summac": self.summac.score_one(inp, pred)["score"]}
  File "/path/to/helm/src/helm/benchmark/metrics/summac/model_summac.py", line 426, in score_one
    image = self.imager.build_image(original, generated)
  File "/path/to/helm/src/helm/benchmark/metrics/summac/model_summac.py", line 147, in build_image
    self.load_nli()
  File "/path/to/helm/src/helm/benchmark/metrics/summac/model_summac.py", line 88, in load_nli
    self.tokenizer = AutoTokenizer.from_pretrained(self.model_card)
  File "/path/to/site-packages/transformers/models/auto/tokenization_auto.py", line 953, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/path/to/site-packages/transformers/tokenization_utils_base.py", line 2036, in from_pretrained
    return cls._from_pretrained(
  File "/path/to/site-packages/transformers/tokenization_utils_base.py", line 2276, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/path/to/site-packages/transformers/models/albert/tokenization_albert_fast.py", line 112, in __init__
    super().__init__(
  File "/path/to/site-packages/transformers/tokenization_utils_fast.py", line 139, in __init__
    fast_tokenizer = convert_slow_tokenizer(self, from_tiktoken=True)
  File "/path/to/site-packages/transformers/convert_slow_tokenizer.py", line 1638, in convert_slow_tokenizer
    raise ValueError(
ValueError: Converting from Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast convertors: ['AlbertTokenizer', 'BartTokenizer', 'BarthezTokenizer', 'BertTokenizer', 'BigBirdTokenizer', 'BlenderbotTokenizer', 'CamembertTokenizer', 'CLIPTokenizer', 'CodeGenTokenizer', 'ConvBertTokenizer', 'DebertaTokenizer', 'DebertaV2Tokenizer', 'DistilBertTokenizer', 'DPRReaderTokenizer', 'DPRQuestionEncoderTokenizer', 'DPRContextEncoderTokenizer', 'ElectraTokenizer', 'FNetTokenizer', 'FunnelTokenizer', 'GPT2Tokenizer', 'HerbertTokenizer', 'LayoutLMTokenizer', 'LayoutLMv2Tokenizer', 'LayoutLMv3Tokenizer', 'LayoutXLMTokenizer', 'LongformerTokenizer', 'LEDTokenizer', 'LxmertTokenizer', 'MarkupLMTokenizer', 'MBartTokenizer', 'MBart50Tokenizer', 'MPNetTokenizer', 'MobileBertTokenizer', 'MvpTokenizer', 'NllbTokenizer', 'OpenAIGPTTokenizer', 'PegasusTokenizer', 'Qwen2Tokenizer', 'RealmTokenizer', 'ReformerTokenizer', 'RemBertTokenizer', 'RetriBertTokenizer', 'RobertaTokenizer', 'RoFormerTokenizer', 'SeamlessM4TTokenizer', 'SqueezeBertTokenizer', 'T5Tokenizer', 'UdopTokenizer', 'WhisperTokenizer', 'XLMRobertaTokenizer', 'XLNetTokenizer', 'SplinterTokenizer', 'XGLMTokenizer', 'LlamaTokenizer', 'CodeLlamaTokenizer', 'GemmaTokenizer', 'Phi3Tokenizer']
```

We should add `sentencepiece` as a (possibly optional) dependency.

Also note that the exception message is unhelpful: https://github.com/huggingface/transformers/issues/36291