Repository: stanford-crfm/helm
Issue: `MATHScenario`: turn examples in instructions to train instances
Root cause: Architectural Constraint

Should the examples in the instruction be train instances so we can control how many examples are included when constructing the prompts?

https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/math_scenario.py#L15

This is already possible by setting `use_official_prompt=False` in the task options. 
That flag is for reproducing the official MATH paper settings. If we want to vary the number of examples, then we can just set it to False. 

If we also want the examples to be train instances when using the official prompt because that would be the more principled design choice (as in, use the adapter to build the prompt rather than hardcoding the prompt), the problem is just that `sample_examples` in [adapter.py#L370](https://github.com/stanford-crfm/benchmarking/blob/f57cbf4ab1b3165e22d2a3d54a9ba68b8a231181/src/benchmark/adapter.py#L370) will reorder the examples from the official prompt. ([code](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/adapter.py#L307-L308))
One way could be to generalize `sample_examples` to `select_examples` and allow multiple example selection modes, set by the AdapterSpec, eg. `prioritized_sample` for the default current approach in [adapter.py#L370](https://github.com/stanford-crfm/benchmarking/blob/f57cbf4ab1b3165e22d2a3d54a9ba68b8a231181/src/benchmark/adapter.py#L370) and `ordered` to support this scenario. This seems relatively future-proof and backwards-compatible without being too great a change.

Or we could just not care about the ordering matching the MATH paper? @tonywu95 
I could make the relevant changes if either sounds like a reasonable fix.

Relevant prior discussion: https://github.com/stanford-crfm/benchmarking/pull/141#discussion_r846929574

@friedeggs Is this resolved then/what is the status?

I was waiting on @teetone for confirmation that the solution in 2nd paragraph would be okay but realized that was unclear - I can go ahead with implementation and submit a PR with this fix for review instead

I think if we want to do this and also support chain-of-thought prompting (as per @tonywu95) there's still some issues with the adaptation, like there isn't a way to format the instructions + examples correctly without eg. adding another field to AdapterSpec..

Right now with the adapter options we can format all strings of the form: ([adapter.py#L27](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/adapter.py#L27))

S#IO#IO#IO#IO
where
S = instructions
I = input prefix + input
O = output prefix + output
\# = instance prefix

but I need
SIO#IO#IO#IO
(ie. no \# after the instructions)
and the problem is the code necessarily includes the first \# because of `self.adapter_spec.instance_prefix.join(blocks)` ([adapter.py#L474](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/adapter.py#L474))

unless I'm missing something silly.

Following up from https://github.com/stanford-crfm/benchmarking/pull/141#discussion_r836861703 where changing this without backwards compatibility would blow away the cache




> I think if we want to do this and also support chain-of-thought prompting (as per @tonywu95) there's still some issues with the adaptation, like there isn't a way to format the instructions + examples correctly without eg. adding another field to AdapterSpec..
> 
> Right now with the adapter options we can format all strings of the form: ([adapter.py#L27](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/adapter.py#L27))
> 
> S#IO#IO#IO#IO where S = instructions I = input prefix + input O = output prefix + output # = instance prefix
> 
> but I need SIO#IO#IO#IO (ie. no # after the instructions) and the problem is the code necessarily includes the first # because of `self.adapter_spec.instance_prefix.join(blocks)` ([adapter.py#L474](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/adapter.py#L474))
> 
> unless I'm missing something silly.
> 
> Following up from [#141 (comment)](https://github.com/stanford-crfm/benchmarking/pull/141#discussion_r836861703) where changing this without backwards compatibility would blow away the cache

Does S#IO#IO#IO#IO vs. SIO#IO#IO#IO make a considerable difference in performance? I would think it's better to be consistent across all Scenarios.

@tonywu95 @friedeggs Could we make some decision on this and resolve?

@friedeggs we can close this now, right?

Yes, thanks!