Repository: promptfoo/promptfoo
Issue: Feature request: integrate file_ids for evaluating assistants
Root cause: Unimplemented Feature Gap

hi there! just checking if there is a way to test an openAI Assistant adding file_ids as parameters in the config file.
As of today there is no way to pass attachments files (see [open AI assistants API V2 ](https://platform.openai.com/docs/assistants/migration)). I tried adding them to the yaml structure but without success.

Just checking if you had this feature on roadmap by any chance!
thanks a lot and nice project!

+1 this feature request. Would be a big upgrade.

@gonzab @marcus-stromeyer How do you guys want to use `file_ids` in the config?  Have you already uploaded the files?  Do you want to reference them in your provider configuration, or for a specific test case?

For example, do you want to specify files like this that apply to all test cases?
```yaml
providers:
- id: openai:assistant:asst_uTqyTUDLdFTXWfDQYWFsI8ye
  config:
    file_ids: [abc123, def456] # or `attachments` for assistants v2
```

Or something like this, where you have test cases that depend on individual files
```yaml
providers:
- openai:assistant:asst_uTqyTUDLdFTXWfDQYWFsI8ye

scenarios:
  - config:
      - vars:
          file_id: file-wRiQEtyoqbiwwVwhIBN8XORW
          expected: foo
      - vars:
          file_id: file-d4C0SNBpu11MJszzUNg1tnEW
          expected: bar

tests:
  - assert:
    - type: icontains
      value: {{expected}}
```

Hi @typpo thanks for the response! By design the assistants api have file access throught tools (file search and code interpreter) at:
- assistant level
- thread level
- message level (updates thread)

Would be great to attach the file ids that must been added to the thread at the config level. Then would be part of the prompt and the tests to ask questions to the files and prepare the expected assertion!

Thanks a lot!

Thanks @typpo! Agreed with @gonzab here. 

But, _**if I had to only choose one**_, I'd love to first have it at the config level. 

```
providers:
- id: openai:assistant:asst_uTqyTUDLdFTXWfDQYWFsI8ye
  config:
    file_ids: [abc123, def456] # or `attachments` for assistants v2

```

I should mention, promptfoo is incredible. Such a cool tool. 

The relevant code for integrating `file_ids` is located in [openai.ts](https://github.com/promptfoo/promptfoo/blob/main/src/providers/openai.ts#L575-L643). I believe this feature would be relatively straightforward to implement. Would either of you (@gonzab or @marcus-stromeyer) be interested in developing this feature in promptfoo? If not, @typpo or I will get to it this week. 

Thanks @mldangelo! Would love to help but this is a bit beyond my expertise (not a TypeScript expert by any means)  

#1080 implements support for `attachments`.  You would use like so, consistent with [Assistant API v2](https://platform.openai.com/docs/assistants/migration):

```yaml
providers:
- id: openai:assistant:asst_uTqyTUDLdFTXWfDQYWFsI8ye
  config:
    attachments:
    - file_id: "file-123"
      tools:
      - type: "file_search"
      - type: "code_interpreter"
```

Thanks @typpo and @mldangelo for pushing this! I will check the developer process to see if i can help with some issues or implementations!