Repository: stanford-crfm/helm
Issue: AI21 + NaturalQA failing
Root cause: External Dependency Breakage

  Error when running natural_qa:model=text,mode=openbook_longans,data_augmentation=canonical:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 98, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 63, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 92, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 130, in run_one
    scenario_state = self.executor.execute(scenario_state)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/executor.py", line 97, in execute
    request_states = parallel_map(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/general.py", line 177, in parallel_map
    results: List = list(tqdm(map(process, items), total=len(items)))
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/executor.py", line 107, in process
    raise ExecutorError(result.error + f"Request: {state.request}")
benchmark.executor.ExecutorError: Failed to make request to ai21 after retrying 5 times. Error: AI21 completion API error - Detail: Request too many tokens: 1949 + 100 > 2047Request: Request(model='ai21/j1-grande', prompt='passage: \n\nno.\noverall\nno.   in\nseason\ntitle\noriginal   air date\nviewers\n(millions)\n\n\n45\n1\n"forever  family"\nnovember  7,  2017 (2017-11-07)\n2.95[69]\n\n\na diver is sent   to the bottom  of   borehole  c-1.   mud  is   recovered.  a   nail  is foudn in an   old   spoils pile from  gal-1. a  copper coin  is  foudn  in  the  woods   near   issac\'s  point.\n\n\n46\n2\n"dead   man\'s  chest"\nnovember 14, 2017  (2017-11-14)\n2.98[70]\n\n\nthe   sea   chest   of a captain who lived   in  nova   scotia  in  the  1780s is examined. salt   water  starts   coming  out the top of   borehole   c-1.  another copper   coin is   found   on the   island.\n\n\n47\n3\n"obstruction"\nnovember   21, 2017  (2017-11-21)\n2.85[71]\n\n\ntwo old   british  copper coins   are   found   in  the  woods.  the province of  nova  scotia requests that an   archaeologist  be  on   site   on oak   island  as  an   adviser to the treasure hunters. part  of   a metal   spoon   and  round   rocks  are   discovered under tree stumps. based  on  the   rocks  and   the   spoon   pieces,   marty lagina concludes  that "something  must\'ve   happened   here".  a   small   piece   of scrap metal   is found   in   the  mud brought up  from one of   the  new  boreholes   being  drilled  in the   "money   pit" area.\n\n\n48\n4\n"close  call"\nnovember  28,  2017  (2017-11-28)\n3.36[72]\n\n\ndrilling  below bedrock, the team  discovers a   void   in  the rock.  a hose on the drill   detaches  resulting  in a worker   falling  backward  and suffering   a minor  injury to   his wrist.   drilling  is   halted  for over  a week.   a   toy gun is  found  in  the   woods   and  the  team   launches an   investigation of it.   lee   lamb   and  richard  restall   visit the island.  richard   restall   is  identified  conclusively   as the owner  of   the   toy gun.\n\n\n49\n5\n"bone   dry"\ndecember  5,  2017   (2017-12-05)\n3.16[73]\n\n\nan  iron  spike is found  on   the beach. some small pottery  fragments  are brought up from  one   of  the boreholes. two human   bone  fragments   were   recovered   anbd  sent   in for testing.\n\n\n50\n6\n"remains of  the  day"\ndecember  12, 2017  (2017-12-12)\n3.01[74]\n\n\na   piece of  leather   is   found.  rick lagina   has headaches and  a  rash.   he\'s   later  diagnosed as   having  lyme  disease.   the   team intends   to   have  tje  previously  discovered human bone fragments  carbon  dated.\n\n\n51\n7\n"the lot  thickens"\ndecember  19,   2017   (2017-12-19)\n3.13[75]\n\n\na   theory  is presented   tyhat  sir  francis   drake  is buried  on  oak  island  in  a  vault with  gold  treasure.  it\'s suggested   that thge lack  of  any   records  documenting drake travelling  to canada  is proof thast   he must\'ve   travelled to canada. a theory  is  presented  that tjhe  scrap leather piece found is   from a book binding.   a   patch   of   ground  is   excavated  and  nothing   is   foudn.\n\n\n52\n8\n"dan\'s   breakthrough"\njanuary   2,   2018  (2018-01-02)\n3.31[76]\n\n\nthe  team discovers   a  horseshoe,   a piece  of  shot  and   another coin. it\'s   discovered   that   one  of the   words on the   zena  halpern  map   is   the   name   of a  french family  (rochefoucauld)  who  participated in the  crusades. the  team   presents  an   unsupported   theory that   samuel   de   champlain   weas  ordered   to  incorrectly   map nova  scotia  to hide oak   island by   pierre  dugua,   sieur de mons   at the direction  of rochefoucauld family.   the team decides to  send several  members  to france for  furuther investigation  of  the  theory.   the testing on  the   human   bone   fragments (found   in   episode   5)  indicated that tyhe  dated in a  range  from  1682 ad   to   1764   ad.\n\n\n53\n9\n"the french   connection"\njanuary   9, 2018 (2018-01-09)\n3.02[77]\n\n\na  small pottery  fragment   is found on  lot  24.  it\'s determined   that  teh  rocks found  in   season  five   episode  3 aren\'t  consistent   with   thge foundation   of  a   housr.  the  team theorizes,  without  evidence,  that   there   must\'ve been   a  "watchtower" on lot   24. in france,  the  team   establishes  that there\'s no  connection  between the  rochefoucauld family   and   the  knights   templar. a   zena halpern map is  examined   again  and   it\'s  determined   that   the english   translations of   the  map done  to  present   have  been  incorrect.   the team looks at   cave carvings in france   and consults  an expert   in   the   knights  templar.   the  team   observes a   pattern   in  part   of   a cave   carving   that   they   claim   is   roughly   similar to  an upside-down   image of  the   rochefoucauld   family   coat  of arms   and conclude that   the  family  must be linked  to the knights templar.   they  find a   carving  of  a   multi-armed  cross  andd offer   an  unsupported   theory  that   it\'s   a  representation   of the   "tree of life" and that  it\'s the   same  as  nolan\'s  cross on  the  island.   a   theory is presented  that   the cave   carving may represent an   "earlier   version"   of   the "tree  of   life".  based  on  the   information   discovered  in  france,  the   team concludes   that   the oak  island swamp   should be excavated agina.\n\n\n54\n10\n"the   signs of   a  cross"\njanuary 16,   2018 (2018-01-16)\n3.34[78]\n\n\neight   pages  of   what\'s  alleged   to be  a  transcript  of  an  incomplete log of   an   unnamed   ship are presented.   it\'s claimed, without any  evidence, that   the  log is from  a french   ship  which  participated  in the   duc', temperature=0.0, num_completions=1, top_k_per_token=1, max_tokens=300, stop_sequences=['\n'], echo_prompt=False, top_p=1, presence_penalty=0, frequency_penalty=0, random=None)


Is this because of tokenization/is it related to known issues with ai21 tokenizer? 

I isolated the problem, and I think it's a bug with their API. I contacted their support through Discord.

```python
import requests

prompt = 'passage: \n\nno.\noverall\nno.   in\nseason\ntitle\noriginal   air date\nviewers\n(millions)\n\n\n45\n1\n' \
       '"forever  family"\nnovember  7,  2017 (2017-11-07)\n2.95[69]\n\n\na diver is sent   to the bottom  of   ' \
       'borehole  c-1.   mud  is   recovered.  a   nail  is foudn in an   old   spoils pile from  gal-1. a  ' \
       'copper coin  is  foudn  in  the  woods   near   issac\'s  point.\n\n\n46\n2\n"dead   man\'s  chest"\n' \
       'november 14, 2017  (2017-11-14)\n2.98[70]\n\n\nthe   sea   chest   of a captain who lived   in  nova   ' \
       'scotia  in  the  1780s is examined. salt   water  starts   coming  out the top of   borehole   c-1.  ' \
       'another copper   coin is   found   on the   island.\n\n\n47\n3\n"obstruction"\nnovember   21, 2017  ' \
       '(2017-11-21)\n2.85[71]\n\n\ntwo old   british  copper coins   are   found   in  the  woods.  the ' \
       'province of  nova  scotia requests that an   archaeologist  be  on   site   on oak   island  as  an   ' \
       'adviser to the treasure hunters. part  of   a metal   spoon   and  round   rocks  are   discovered ' \
       'under tree stumps. based  on  the   rocks  and   the   spoon   pieces,   marty lagina concludes  that ' \
       '"something  must\'ve   happened   here".  a   small   piece   of scrap metal   is found   in   the  mud ' \
       'brought up  from one of   the  new  boreholes   being  drilled  in the   "money   pit" area.\n\n\n48\n4\n"' \
       'close  call"\nnovember  28,  2017  (2017-11-28)\n3.36[72]\n\n\ndrilling  below bedrock, the team  discovers ' \
       'a   void   in  the rock.  a hose on the drill   detaches  resulting  in a worker   falling  backward  and ' \
       'suffering   a minor  injury to   his wrist.   drilling  is   halted  for over  a week.   a   toy gun is  ' \
       'found  in  the   woods   and  the  team   launches an   investigation of it.   lee   lamb   and  richard  ' \
       'restall   visit the island.  richard   restall   is  identified  conclusively   as the owner  of   the   ' \
       'toy gun.\n\n\n49\n5\n"bone   dry"\ndecember  5,  2017   (2017-12-05)\n3.16[73]\n\n\nan  iron  spike is ' \
       'found  on   the beach. some small pottery  fragments  are brought up from  one   of  the boreholes. two ' \
       'human   bone  fragments   were   recovered   anbd  sent   in for testing.\n\n\n50\n6\n"remains of  the  ' \
       'day"\ndecember  12, 2017  (2017-12-12)\n3.01[74]\n\n\na   piece of  leather   is   found.  rick lagina   ' \
       'has headaches and  a  rash.   he\'s   later  diagnosed as   having  lyme  disease.   the   team intends   ' \
       'to   have  tje  previously  discovered human bone fragments  carbon  dated.\n\n\n51\n7\n"the lot  thickens"' \
       '\ndecember  19,   2017   (2017-12-19)\n3.13[75]\n\n\na   theory  is presented   tyhat  sir  francis   drake  ' \
       'is buried  on  oak  island  in  a  vault with  gold  treasure.  it\'s suggested   that thge lack  of  any   ' \
       'records  documenting drake travelling  to canada  is proof thast   he must\'ve   travelled to canada. a ' \
       'theory  is  presented  that tjhe  scrap leather piece found is   from a book binding.   a   patch   of   ' \
       'ground  is   excavated  and  nothing   is   foudn.\n\n\n52\n8\n"dan\'s   breakthrough"\njanuary   2,   ' \
       '2018  (2018-01-02)\n3.31[76]\n\n\nthe  team discovers   a  horseshoe,   a piece  of  shot  and   another ' \
       'coin. it\'s   discovered   that   one  of the   words on the   zena  halpern  map   is   the   name   of a  ' \
       'french family  (rochefoucauld)  who  participated in the  crusades. the  team   presents  an   unsupported   ' \
       'theory that   samuel   de   champlain   weas  ordered   to  incorrectly   map nova  scotia  to hide oak   ' \
       'island by   pierre  dugua,   sieur de mons   at the direction  of rochefoucauld family.   the team decides ' \
       'to  send several  members  to france for  furuther investigation  of  the  theory.   the testing on  the   ' \
       'human   bone   fragments (found   in   episode   5)  indicated that tyhe  dated in a  range  from  1682 ad   ' \
       'to   1764   ad.\n\n\n53\n9\n"the french   connection"\njanuary   9, 2018 (2018-01-09)\n3.02[77]\n\n\na ' \
       ' small pottery  fragment   is found on  lot  24.  it\'s determined   that  teh  rocks found  in   season  ' \
       'five   episode  3 aren\'t  consistent   with   thge foundation   of  a   housr.  the  team theorizes,  ' \
       'without  evidence,  that   there   must\'ve been   a  "watchtower" on lot   24. in france,  the  team   ' \
       'establishes  that there\'s no  connection  between the  rochefoucauld family   and   the  knights   ' \
       'templar. a   zena halpern map is  examined   again  and   it\'s  determined   that   the english   ' \
       'translations of   the  map done  to  present   have  been  incorrect.   the team looks at   cave carvings ' \
       'in france   and consults  an expert   in   the   knights  templar.   the  team   observes a   pattern   ' \
       'in  part   of   a cave   carving   that   they   claim   is   roughly   similar to  an upside-down   ' \
       'image of  the   rochefoucauld   family   coat  of arms   and conclude that   the  family  must be linked  ' \
       'to the knights templar.   they  find a   carving  of  a   multi-armed  cross  andd offer   an  ' \
       'unsupported   theory  that   it\'s   a  representation   of the   "tree of life" and that  it\'s the   ' \
       'same  as  nolan\'s  cross on  the  island.   a   theory is presented  that   the cave   carving may ' \
       'represent an   "earlier   version"   of   the "tree  of   life".  based  on  the   information   ' \
       'discovered  in  france,  the   team concludes   that   the oak  island swamp   should be excavated ' \
       'agina.\n\n\n54\n10\n"the   signs of   a  cross"\njanuary 16,   2018 (2018-01-16)\n3.34[78]\n\n\neight   ' \
       'pages  of   what\'s  alleged   to be  a  transcript  of  an  incomplete log of   an   unnamed   ship are ' \
       'presented.   it\'s claimed, without any  evidence, that   the  log is from  a french   ship  which  ' \
       'participated  in the   duc'

response = requests.post(
    "https://api.ai21.com/studio/v1/tokenize",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"text": prompt},
).json()
print(len(response["tokens"]))  # 1747

request = {
    "prompt": prompt,
    "temperature": 0,
    "numResults": 1,
    "topKReturn": 1,
    "topP": 1,
    "maxTokens": 300,
    "stopSequences": ["\n"],
}

response = requests.post(
    "https://api.ai21.com/studio/v1/j1-large/complete",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json=request,
).json()
print(response["detail"])   # Request too many tokens: 1948 + 100 > 2047

```

https://discord.com/channels/874538902696914944/874539313889673216

> https://discord.com/channels/874538902696914944/874539313889673216

AI21 is looking into it and will update us.

@teetone Any updates from AI21, since looks this is still an issue? Making this a P2, since if its just a them-issue, well its not critical for release.

> @teetone Any updates from AI21, since looks this is still an issue? Making this a P2, since if its just a them-issue, well its not critical for release.

I asked again, and no one is actively looking into it. I think we just won't have NaturalQA results for the AI21 models and maybe we can revisit after they resolve the issue.

ok sounds great - lets make p3 then, since its their problem

AI21 fixed the issue for us and replied to me on Discord. We have results: https://nlp.stanford.edu/helm/current/?suite=v7&group=natural_qa_openbook_longans