Repository: stanford-crfm/helm
Issue: Long instances are truncated improperly
Root cause: Algorithmic Error

Currently, we truncate from the top. We should figure out how we want to truncate (e.g. we could try to keep any instructions + prefixes and then truncate from the bottom; this ensures models will see the question/recent context for QA, for example). 

@percyliang @tonywu95 @dtsip @YianZhang thoughts? (Since this probably effects the scenarios from your teams more than anyone, and may involve some relationship with tokenization and language modeling.)

This is particularly problematic for tasks like QA where we will either will miss the Question if we truncate from the bottom or miss potentially important context (without any real control over it) if we truncate from the top. Also, whether a passage fits in-context is model-dependent which is not great.

Overall, my personal take is that we should remove these inputs from the scenarios. Is there a way to measure how many inputs we would be missing?

I am ok with removing them as long as its a small fraction and well-documented. Though arguably models should just get them wrong if their context window is too small (or, symmetrically, models that can handle longer context should have the opportunity to get them right). 

As any rate, we should make a clear and uniform decision that we explicitly justify in the paper. 

Ok, so one thing that makes some sense is:
- Truncate from the top (i.e., skip things at the start of the prompt) so that we can still see questions and later content. (Is there any scenario family that does not like this?)
- Mark truncated instances.
- Compute metrics based on all or untruncated instances.

How does that sound?

I don't think we should remove anything and we should report metrics on all instances, since otherwise it gets more complex.  We can truncate heuristically - maybe just truncate the suffix of each input / reference?

@dtsip Can you just be the dictator and resolve what we should do, since it seems like we are going back and forth but should decide since it will influence the results? 

So, I agree with the sentiment that we should not remove inputs. We are benchmarking models and if the model cannot fit a use case into its context that it should lose points.

At the same time, we should definitely make sure that we are providing a valid input, i.e., for QA, the questions _has_ to be part of the prompt.

I think the best way it to **truncate from the left**, that is remove a prefix from each instance to make it fit. This way we can get the question inside the prompt. Is this breaking any other scenarios?

Also, I feel like we should at least mark these inputs in the interface.

I am on board with everything you said, though maybe Percy is not. 

I am not aware of anything breaking but we can ask during the meeting, certainly amidst the general scenarios. 

Marking seems good; making we have a P2 visualization issue for this that ideally gets resolved in the same PR? 

Ok, will test and submit a PR and we can discuss during the meeting before merging.

Also added a comment under https://github.com/stanford-crfm/benchmarking/issues/169 since these seem tied implementation-wise.

Fantastic, thanks! 

Truncate from the right for now because that's implemented.  See if this is a problem first.

@dtsip I think it would be good to support this, though at this point I think it is too late to do for most models as we have launched the big runs (@teetone to confirm) and we will want to be uniform. We could re-run on these models (as the monetary cost is mainly on OpenAI which we have yet to run), but not too sure its worth.

Nonetheless, it probably would be good to have this supported in the codebase if it is the right thing. Moving to P2 for now as a result though.

Ok, I will still implement the method inside the tokenizer. But I don't think it should be affecting too many instances anyway.

@dtsip Just checking in on this, since we should either:
a. Do soon since it influences results.
b. Decide not to do and relegate to P3

I think we can skip this for now as a P1. I want to implement keeping track of truncated inputs (piggyback on https://github.com/stanford-crfm/benchmarking/pull/650/) and see if it is an issue overall. 

Created https://github.com/stanford-crfm/benchmarking/pull/741 to keep track of how many times this happens. We should look at the final results and see if any scenario-model pairs are heavily truncated.

Let me add something to the website to make this easier to see.

Though we should independently see the fraction of instances where this happens - i looked through just now, and seems fairly uncommon for the QA tasks I have looked at so far, but presumably should just be a focus for the tasks with longest inputs/outputs