Repository: stanford-crfm/helm
Issue: Synonym data augmentation is not thread safe
Root cause: Resource Mishandling

Errors when running with parallelism

```
  File "/Users/tonyhlee/research/mercury/benchmarking/src/benchmark/augmentations/synonym_perturbation.py", line 100, in perturb
    return self.synonyms_substitute(text)
  File "/Users/tonyhlee/research/mercury/benchmarking/src/benchmark/augmentations/synonym_perturbation.py", line 82, in synonyms_substitute
    synsets = wordnet.synsets(word, pos=wordnet_pos)
  File "/Users/tonyhlee/research/mercury/benchmarking/venv/lib/python3.8/site-packages/nltk/corpus/reader/wordnet.py", line 1700, in synsets
    return [
  File "/Users/tonyhlee/research/mercury/benchmarking/venv/lib/python3.8/site-packages/nltk/corpus/reader/wordnet.py", line 1701, in <listcomp>
    get_synset(p, offset)
  File "/Users/tonyhlee/research/mercury/benchmarking/venv/lib/python3.8/site-packages/nltk/corpus/reader/wordnet.py", line 1505, in synset_from_pos_and_offset
    raise WordNetError(
nltk.corpus.reader.wordnet.WordNetError: No WordNet synset found for pos=v at offset=748616.
```

Made `synonyms` thread-safe in https://github.com/stanford-crfm/benchmarking/pull/790 (I don't get any errors when running with `parallelism > 1` but did not fix the randomness issue https://github.com/stanford-crfm/benchmarking/issues/774 yet, since the fix might depend on how we parallelize exactly).

At the same time, running with parallelism > 1 does not speeds things up. According to this [post](https://superfastpython.com/threadpoolexecutor-slower/), ThreadPoolExecutor should not speed things up for CPU-bound processes (which should be the case here) due to a Global Interpreter Lock. (I briefly tried ProcessPoolExecutor but got some weird deadlocking?!)

Taking a step back, I am not sure augmentations are the bottleneck here. Running `boolq:data_augmentation=canonical -m 100` gives me:
- augmentation (whole scenario): 1m49s
- adaptation (resulting 419 instances): 2m15s
- TokenMetrics: 4m29s



I did parallelize adaptation and token metrics in #789 so data augmentation could be a significant fraction of the time.

Good point about data augmentation being CPU-bound though.  I'm curious what happens with ProcessPoolExecutor...let me try a small example.

Synonyms is now thread-safe. Should we close this?

> Synonyms is now thread-safe. Should we close this?

Yes, closing it.

@dtsip When I run with parallelism > 5, I get the following error:

```
  File "/Users/tonyhlee/research/mercury/benchmarking/src/benchmark/augmentations/data_augmenter.py", line 27, in process
    perturbed_instance: Instance = perturbation.apply(instance)
  File "/Users/tonyhlee/research/mercury/benchmarking/src/benchmark/augmentations/perturbation.py", line 41, in apply
    instance, input=self.perturb(instance.input, rng), references=references, perturbation=self.description,
  File "/Users/tonyhlee/research/mercury/benchmarking/src/benchmark/augmentations/synonym_perturbation.py", line 95, in perturb
    for base in wordnet._morphy(word.lower(), wordnet_pos):  # _morphy returns the base form of a word
  File "/Users/tonyhlee/research/mercury/benchmarking/venv/lib/python3.9/site-packages/nltk/corpus/util.py", line 121, in __getattr__
    self.__load()
  File "/Users/tonyhlee/research/mercury/benchmarking/venv/lib/python3.9/site-packages/nltk/corpus/util.py", line 95, in __load
    args, kwargs = self.__args, self.__kwargs
AttributeError: 'WordNetCorpusReader' object has no attribute '_LazyCorpusLoader__args'
```

`parallelism=1` still works.

Hm, that's weird. From main changing [this line](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/augmentations/data_augmenter.py#L55) to `parallelism = 10` runs fine.

Do you get this error consistently?

> this line

It happens to me every time. I tried 10 too. What version of NLTK are you using?

```
 % pip show nltk
Name: nltk
Version: 3.7
Summary: Natural Language Toolkit
Home-page: https://www.nltk.org/
Author: NLTK Team
Author-email: nltk.team@gmail.com
License: Apache License, Version 2.0
Location: /Users/tonyhlee/research/mercury/benchmarking/venv/lib/python3.9/site-packages
Requires: regex, tqdm, click, joblib
Required-by: summ-eval, rouge-score, cleantext, blanc, crfm-benchmarking
```

Ok, I think I found the issue, we need to make sure that wordnet is loaded before splitting into threads. Submitted https://github.com/stanford-crfm/benchmarking/pull/813.

@teetone you can quickly test by adding `wordnet.ensure_loaded()` after the `except LookupError:`.