Repository: stanford-crfm/helm
Issue: off by one error in codex
Root cause: Algorithmic Error

In the web interface, try the third example query (or anything with codex).  The last token of the prompt seems to be repeated in the first token of the completion.

@percyliang Closing since not sure what this means but the codex scenarios seem fine (probably this was for the crfm web interface since this was in december). 
https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*code.*

Let's not close yet since this was a bug, and we should verify that this is not an issue.  Might affect correctness.

@lxuechen Since you implemented the code scenarios, could you check if this was a problem?

Sorry for the delay. Looking at this now. 

@lxuechen Could you update on whether this is resolved (and close this issue if so)?

Sorry for the super huge delay. Will go over this again tonight. 

I took a look at the web interface query example. The repetition behaviour isn't consistent (see examples below). In the first example, the ending word is "likelihood"; this word isn't repeated, and moreover, the start of the completion doesn't overlap in any way with subword decompositions of the word "likelihood." The second example has the ending word "who," which is repeated in the completion, even though this repetition makes the completion totally unnatural and ungrammatical. 

Can any of you point me to the code that's responsible for the web interface? I didn't implement this, but may be able to help and gain a better understanding by parsing the implementation. @teetone @rishibommasani @percyliang 

<img width="1042" alt="Screen Shot 2022-07-26 at 12 50 31 PM" src="https://user-images.githubusercontent.com/12689993/181100033-4ad0d0be-90ed-4c35-b3e8-3462a50c9a89.png">
<img width="976" alt="Screen Shot 2022-07-26 at 12 51 50 PM" src="https://user-images.githubusercontent.com/12689993/181100038-d11fcabe-c19f-4356-8ef0-f545849ed73d.png">



For the frontend, check out: https://github.com/stanford-crfm/benchmarking/blob/main/src/proxy/static/index.js#L238

But could you check first if the backend is behaving properly?

@percyliang  I dug deeper and discovered the bug. To summarize, 

1. Does the bug affect the CodeScenario (i.e., HumanEval and APPS)? No. 
2. Can the bug with the web interface be fixed? Yes. 

Here's the detailed analysis:

1. The bug doesn't affect HumanEval or APPS, since the end of the prompt is always the newline char '\n' for all examples in the two tasks. Even in the case of repetition, this doesn't change the semantics of the generated code (hence the metric functions are also invariant). 
2. It seems the bug is caused by the codex model running some underlying partial-token completion algorithm before the actual autoregressive token generation(?). 
    - Note each completion request to OpenAI API results in (i) the full text completion as a string (e.g., fetched by `output['choices'][index]['text']`), and (ii) a list of tokens with their corresponding logprobs (e.g., fetched by `output['choices'][index]['logprobs']['tokens']`)
      - See the code example below for what the variables refer to. 
    - **(i) and (ii) could be different in that (ii) contains the 'completed' first token, whereas (i) seems to be based on string indexing.** This creates the issue you see in the web interface which is based on joining together the individual tokens returned from OpenAI, **as opposed to using the text completion string returned by OpenAI**.

To reproduce, below's a complete analysis I ran with 3 different prompts with their corresponding outputs. You can see this discrepancy issue in the second and third prompt examples. 

What's a quick fix for the web interface issue? Well, one naive sol'n is to not use `completion.tokens` but rather use the full text completion returned by OpenAI. But this breaks the logprob reporting when a cursor is hovered over. A full sol'n requires more care but should be doable. 

Code: 
```python
import os

import openai

openai.api_key = os.getenv("OPENAI_API_KEY")

for idx, prompt in enumerate((
        "Write a function that prints Hello",  # Doesn't repeat.
        "Write a function that prints hi",  # Repeats hi.
        "A lawyer is someone who"
)):
    output = openai.Completion.create(
        model="code-davinci-001",
        prompt=prompt,
        max_tokens=20,
        temperature=0.7,
        logprobs=1,
    )
    print(f'prompt:\n{prompt}\n')

    print(f"completion:\n{output['choices'][0]['text']}\n")

    print(f"tokens:\n{output['choices'][0]['logprobs']['tokens']}\n")
``` 

Outputs: 
```bash
prompt:
Write a function that prints Hello

completion:
, World!."

def hello_world():
    print("Hello, World!")

tokens:
[',', ' World', '!', '."', '\n', '\n', 'def', ' hello', '_', 'world', '():', '\n', '   ', ' print', '("', 'Hello', ',', ' World', '!"', ')']

prompt:
Write a function that prints hi

completion:
stogram of a given list
def histogram(list):
    for i in list:

tokens:
[' hist', 'ogram', ' of', ' a', ' given', ' list', '\n', 'def', ' hist', 'ogram', '(', 'list', '):', '\n', '   ', ' for', ' i', ' in', ' list', ':']

prompt:
A lawyer is someone who

completion:
 knows the law and can't be sued.",
    "A leak always gets bigger than the

tokens:
[' who', ' knows', ' the', ' law', ' and', ' can', "'t", ' be', ' sued', '.",', '\n', '   ', ' "', 'A', ' leak', ' always', ' gets', ' bigger', ' than', ' the']
```

Thanks for troubleshooting this!  Agreed this is a frontend issue.  This issue of tokens being different from the original text shows up in Jurassic too - perhaps @YianZhang can take a look?

@percyliang I tried a few examples and didn't run into the error. Can you share a failing example? Thanks!

Obsolete because all OpenAI codex models are deprecated.