Repository: stanford-crfm/helm
Issue: `HuggingFace error: Already borrowed` when running tokenizer in multi-threaded mode
Root cause: Resource Mishandling

When running in multi-threaded mode with a Hugging Face fast tokenizer, there are sometimes `HuggingFace error: Already borrowed` errors.

Upstream issue: https://github.com/huggingface/tokenizers/issues/537

Based on skimming the upstream issue, especially this comment:

> This is happening because you have a single tokenizer, that is used by 2 different threads. While the tokenizer is encoding (on one thread), if the other thread tries to modify it, this error happens because it cannot be modified while being used at the same time.

My assessment is that we should just treat `PreTrainedTokenizerBase` as thread hostile and require that we hold a lock when using it. This does reduce parallelism to 1 - I filed #2017 to eventually support tokenization parallelism.

The changes needed are:

- In `HuggingFaceServer`, add an instance variable lock that protects the instance variable [tokenizer](https://github.com/stanford-crfm/helm/blob/v0.3.0/src/helm/proxy/clients/huggingface_client.py#L58-L60).
- In `HuggingFaceTokenizer`, add a instance variable lock that is used inside `_tokenize_do_it()` and `_decode_do_it()` - hold it when `_tokenizer` is used.

There's also some alternate solution where we wrap `PreTrainedTokenizerBase` in our own class `ThreadSafePreTrainedTokenizer` that does the locking, which is more object-oriented-ish.

Unable to recreate the logic.
I used this config:
```
{description: "mmlu:subject=philosophy,model=openai/gpt2", priority: 1},
{description: "mmlu:subject=philosophy,model_deployment=together/falcon-7b", priority: 1},
{description: "mmlu:subject=college_chemistry,model_deployment=together/mistral-7b-v0.1", priority: 1},
```
which uses 3 different huggingface tokenizers. I deleted the cache before each run and ran on 20 instances each
No error was raised.


@danielz02 also reported running into this bug yesterday

> @danielz02 also reported running into this bug yesterday

Upon digging, the error seems to come from the Python binding of the HF Rust tokenizer. The Rust implementation should be natively thread-safe as long as it compiles (provided no unsafe Rust is used), but the interop mechanism with Python  `PyO3` has runtime checks to make sure the tokenizer instance is not mutably borrowed from two threads, resulting in the error. The HF implementation made the design choice to let the exception be thrown for performance reasons instead of making the whole thing thread-safe.

I'm using `--num-threads 1` to get around this, as suggested by @yifanmai 

Just to check, @danielz02 , what is your version of `transformers` and `tokenizers` ?

And could you provide the command that caused the issue?

```
transformers              4.33.3                   pypi_0    pypi
tokenizers                  0.13.3                    pypi_0    pypi
```

Use the command below in the `DecodingTrust` branch, which is in sync with `main` except for our PR.
```bash
helm-run -r decodingtrust_fairness:task=predict_annual_income,train_base_rate=0.0,test_base_rate=0.0,num_train=0,num_test=200,model=simple/model1 -m 10 --suite decodingtrust
```

(You might get some other errors in metric calculation due to our bugs, but the `tokenizer` error usually comes before that.)

I don't yet have a reliable reproduction for this, but I managed to get a couple of stack traces:

- [Stack trace 1](https://gist.github.com/yifanmai/b14fbd9a3f57bd3852b1eff46b867754)
- [Stack trace 2](https://gist.github.com/yifanmai/ddedc71fb3f97168f6787452500f99c3)

Turns out this affects `AnthropicTokenizer` as well, though with a different stack trace:

```
Traceback (most recent call last):
  File "/u/scr/maiyifan/miniconda3/envs/helm-lite/lib/python3.8/site-packages/helm/proxy/tokenizers/caching_tokenizer.py", line 105, in tokenize
    assert type(tokens[0].value) == str
AssertionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/sailhome/maiyifan/.local/lib/python3.8/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/u/scr/maiyifan/miniconda3/envs/helm-lite/lib/python3.8/site-packages/helm/proxy/clients/auto_client.py", line 103, in make_request_with_retry
    return client.make_request(request)
  File "/u/scr/maiyifan/miniconda3/envs/helm-lite/lib/python3.8/site-packages/helm/proxy/clients/anthropic_client.py", line 166, in make_request
    tokenization_result: TokenizationRequestResult = self.tokenizer.tokenize(
  File "/u/scr/maiyifan/miniconda3/envs/helm-lite/lib/python3.8/site-packages/helm/proxy/tokenizers/caching_tokenizer.py", line 117, in tokenize
    raise ValueError(f"Failed to tokenize text with {self.__class__.__name__} tokenizer: {error}") from error
ValueError: Failed to tokenize text with AnthropicTokenizer tokenizer: 
```

Though when I wrote a unit test to try to reproduce the `AnthropicTokenizer` error, I only got `Already borrowed` and not this other error...