Repository: UKGovernmentBEIS/inspect_ai
Issue: Is it possible use a different model for generate() and eval with a different model?
Root cause: Documentation Deficiency

Hi,

I notice that there is an option to pass the eval model with the `--model` flag. When I use `generate()` solver, it uses the model set by the same flag. I am looking for a way to do the generations using a different models but have the same model for evaluations. This way I can run generations across different models and evaluate them using a single powerful model.

Is this possible today?  

For now, I am able to get around this by setting a different model for `self_critique()`

```
            self_critique(
                model="openai/gpt-4o-mini",
            )
```

And iterating the eval model for running generations with different models. This unblocks me. But, please let me know if there's a built-in way to do this. 

hmm i think you can do this by switching the order - in the --model flag you define the 'generation' model i.e the model you want to perform the tasks, and then define the eval model seperartly, you can natively do this in the in-built scorers. 

In the below example I have passed a model name in the scorer its self, this way, when i run eval on a given model i.e --model bedrock/meta.llama3-1-8b-instruct-v1:0 it will use in this case openai/gpt-4 for the evaluation part, and llama3 for the generation part.
```
@task
def job_qa_task(
    shuffle=True,
    CoT=False,
    prompt_directory=r"./default_prompt.txt",
    dataset=r"./dataset.csv",
):
    with open(prompt_directory, "r") as f:
        PROMPT_TEMPLATE = f.read()

    plan = [
        prompt_template(PROMPT_TEMPLATE),
        generate(),
    ]

    if CoT:
        middle_index = len(plan) // 2
        plan = plan[:middle_index] + [CoT] + plan[middle_index:]
    return Task(
        dataset=csv_dataset(
            dataset,
            FieldSpec(input="JobDescription", metadata=["Question"]),
        ),
        shuffle=shuffle,
        plan=plan,
        scorer=model_graded_fact(
            model="openai/gpt-4",
        ),
        config=GenerateConfig(temperature=0),
    )

```

If you want to be able to change the generation model **and ** eval model in the CLI when using inspect eval then you can define it as a parameter in the  task and then access it through the T argument in the CLI, for example:

```diff
@task
def job_qa_task(
    shuffle=True,
    CoT=False,
    prompt_directory=r"./default_prompt.txt",
    dataset=r"./dataset.csv",
+ grader_model = 'openai/gpt-4'
):
    with open(prompt_directory, "r") as f:
        PROMPT_TEMPLATE = f.read()

    plan = [
        prompt_template(PROMPT_TEMPLATE),
        generate(),
    ]

    if CoT:
        middle_index = len(plan) // 2
        plan = plan[:middle_index] + [CoT] + plan[middle_index:]
    return Task(
        dataset=csv_dataset(
            dataset,
            FieldSpec(input="JobDescription", metadata=["Question"]),
        ),
        shuffle=shuffle,
        plan=plan,
        scorer=model_graded_fact(
-            model='openai/gpt-4',
+            model=grader_model ,
        ),
        config=GenerateConfig(temperature=0),
    )

```

then from the command line you can run:

`inspect eval --model 'model to test' T grader_model 'stronger model to use for evaluation'`


I ran the above and got this in the CLI you can see in the top line the task name, and next to it the model i want to test (llama3-1-8b). you can see under the total time header that this model ran, and also gpt-4. in the logs gpt-4 is the evaluator.


Amazing! This is helpful. Thank you