Repository: stanford-crfm/helm
Issue: Missing runs in v0.2.0
Root cause: External Dependency Breakage

```
copyright:datatag=n_books_1000-extractions_per_book_1-prefix_length_125,model=anthropic_stanford-online-all-v4-s3,num_outputs=10
copyright:datatag=n_books_1000-extractions_per_book_1-prefix_length_125,model=together_t0pp,num_outputs=10,stop=hash
copyright:datatag=n_books_1000-extractions_per_book_1-prefix_length_125,model=together_yalm,num_outputs=10
copyright:datatag=oh_the_places,model=anthropic_stanford-online-all-v4-s3,num_outputs=1
copyright:datatag=oh_the_places,model=anthropic_stanford-online-all-v4-s3,num_outputs=10
copyright:datatag=oh_the_places,model=together_t0pp,num_outputs=1,stop=hash
copyright:datatag=oh_the_places,model=together_t0pp,num_outputs=10,stop=hash
copyright:datatag=popular_books-prefix_length_125.json,model=anthropic_stanford-online-all-v4-s3,num_outputs=10
copyright:datatag=popular_books-prefix_length_125.json,model=together_t0pp,num_outputs=1,stop=hash
copyright:datatag=popular_books-prefix_length_125.json,model=together_t0pp,num_outputs=10,stop=hash
the_pile:subset=ArXiv,model=cohere_large-20220720
the_pile:subset=ArXiv,model=cohere_medium-20220720
the_pile:subset=ArXiv,model=cohere_medium-20221108
the_pile:subset=ArXiv,model=cohere_small-20220720
the_pile:subset=ArXiv,model=cohere_xlarge-20220609
the_pile:subset=ArXiv,model=cohere_xlarge-20221108
the_pile:subset=Github,model=cohere_large-20220720
the_pile:subset=Github,model=cohere_medium-20220720
the_pile:subset=Github,model=cohere_medium-20221108
the_pile:subset=Github,model=cohere_small-20220720
the_pile:subset=Github,model=cohere_xlarge-20220609
the_pile:subset=Github,model=cohere_xlarge-20221108
the_pile:subset=PubMed Central,model=cohere_large-20220720
the_pile:subset=PubMed Central,model=cohere_medium-20220720
the_pile:subset=PubMed Central,model=cohere_medium-20221108
the_pile:subset=PubMed Central,model=cohere_small-20220720
the_pile:subset=PubMed Central,model=cohere_xlarge-20220609
the_pile:subset=PubMed Central,model=cohere_xlarge-20221108
```

We don't have the requests in the cache for these runs, either.

- YALM and t0pp aren't currently on Together, so we can't rerun those
- We can rerun medium-20221108 and xlarge-20221108, the other Cohere models have been [deprecated](https://docs.cohere.ai/changelog/model-sizing-update-improvements) (#1345); also the windowing needs to be fixed first in #1371.
- We don't have access to Anthropic currently.

The list for v0.2.1 is shorter (only Anthropic and Cohere and YALM):

```
copyright:datatag=n_books_1000-extractions_per_book_1-prefix_length_125,model=anthropic_stanford-online-all-v4-s3,num_outputs=10
copyright:datatag=n_books_1000-extractions_per_book_1-prefix_length_125,model=together_yalm,num_outputs=10
copyright:datatag=oh_the_places,model=anthropic_stanford-online-all-v4-s3,num_outputs=1
copyright:datatag=oh_the_places,model=anthropic_stanford-online-all-v4-s3,num_outputs=10
copyright:datatag=popular_books-prefix_length_125.json,model=anthropic_stanford-online-all-v4-s3,num_outputs=10
the_pile:subset=ArXiv,model=cohere_large-20220720
the_pile:subset=ArXiv,model=cohere_medium-20220720
the_pile:subset=ArXiv,model=cohere_medium-20221108
the_pile:subset=ArXiv,model=cohere_small-20220720
the_pile:subset=ArXiv,model=cohere_xlarge-20220609
the_pile:subset=ArXiv,model=cohere_xlarge-20221108
the_pile:subset=Github,model=cohere_large-20220720
the_pile:subset=Github,model=cohere_medium-20220720
the_pile:subset=Github,model=cohere_medium-20221108
the_pile:subset=Github,model=cohere_small-20220720
the_pile:subset=Github,model=cohere_xlarge-20220609
the_pile:subset=Github,model=cohere_xlarge-20221108
the_pile:subset=PubMed Central,model=cohere_large-20220720
the_pile:subset=PubMed Central,model=cohere_medium-20220720
the_pile:subset=PubMed Central,model=cohere_medium-20221108
the_pile:subset=PubMed Central,model=cohere_small-20220720
the_pile:subset=PubMed Central,model=cohere_xlarge-20220609
the_pile:subset=PubMed Central,model=cohere_xlarge-20221108
```

The newer runs also have failing runs... we just have to accept missing / failing runs as a fact of life. The mean win-rate metric is somewhat resistant to this.