Repository: promptfoo/promptfoo
Issue: Question: Future support for AWS Bedrock Batch Inference?
Root cause: Unimplemented Feature Gap

Hi,

I'm using `promptfoo` to interact with one of Amazon Bedrock models. The interaction is handled as part of On Demand pricing model. The same model can be used with Batch pricing model, which would reduce the cost by 50%. The Batch inference require to upload a batch of >100 requests as `.jsonl` file(s) to S3 and trigger batch inference job. Once the job is complete the LLM output is stored in another S3 bucket. 

I couldn't find any trace of Batch inference for Amazon Bedrock in `promptfoo` docs, so I assumed that this pricing model (and interaction type) not currently supported. Are there any plans to add this as an option in the future?

From [AWS docs](https://aws.amazon.com/bedrock/pricing/):

> With Batch mode, you can provide a set of prompts as a single input file and receive responses as a single output file, allowing you to get simultaneous large-scale predictions. The responses are processed and stored in your Amazon S3 bucket so you can access them at a later time. Amazon Bedrock offers select foundation models (FMs) from leading AI providers like Anthropic, Meta, Mistral AI, and Amazon for batch inference at a 50% lower price compared to on-demand inference pricing. Please refer to model list [here](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference-supported.html).

You're correct, batch inference is not supported right now.  We don't have
specific plans to implement this, although we would gladly accept a PR.

On Mon, Dec 16, 2024 at 4:53 AM Kamil Bonna ***@***.***>
wrote:

> Hi,
>
> I'm using promptfoo to interact with one of Amazon Bedrock models. The
> interaction is handled as part of On Demand pricing model. The same model
> can be used with Batch pricing model, which would reduce the cost by 50%.
> The Batch inference require to upload a batch of >100 requests as .jsonl
> file(s) to S3 and trigger batch inference job. Once the job is complete the
> LLM output is stored in another S3 bucket.
>
> I couldn't find any trace of Batch inference for Amazon Bedrock in
> promptfoo docs, so I assumed that this pricing model (and interaction
> type) not currently supported. Are there any plans to add this as an option
> in the future?
>
> From AWS docs <https://aws.amazon.com/bedrock/pricing/>:
>
> With Batch mode, you can provide a set of prompts as a single input file
> and receive responses as a single output file, allowing you to get
> simultaneous large-scale predictions. The responses are processed and
> stored in your Amazon S3 bucket so you can access them at a later time.
> Amazon Bedrock offers select foundation models (FMs) from leading AI
> providers like Anthropic, Meta, Mistral AI, and Amazon for batch inference
> at a 50% lower price compared to on-demand inference pricing. Please refer
> to model list here
> <https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference-supported.html>
> .
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/2419>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJUEUWTBXBJLGZK6X3T2F3EMHAVCNFSM6AAAAABTWCQLH2VHI2DSMVQWIX3LMV43ASLTON2WKOZSG42DEMZRG4ZDQNY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


OpenAI also has a batch mode, again saving 50% of inference cost: <https://platform.openai.com/docs/guides/batch>

Batch mode is really useful for "running the kitchen sink," nightly/weekly runs, and evaluating new models, saving $$$ and maybe even making it faster (due to higher rate limits).

I'd consider coding this if we can agree on what should be done 😄

The MVP version is to batch everything up and leave the CLI running, polling every 60 seconds for batch completion. I can probably do that. Specifically, for OpenAI & Azure OpenAI, but generically enough that perhaps AWS can fit in, @kamilbonna.

Long term, but not what I'd have time for this go-around, is to somehow mark evals as "pending" in the UI and update the results whenever it finishes -- perhaps even sending notification emails.