Hey swyk! I have a similar perspective when it comes to model routing and even LLM frameworks that centralize the schema when almost every provider already supports OpenAI's standard.

1) The goal isn't often model switching but rather model-to-prompt optimization, balancing performance, quality, safeguards & cost at scale. Especially in a situation where an AI app requires multiple models, which oftentimes does exist from the users we talk to. I have seen applications that use llama 3.2 1b on Groq for quick data cleaning of simple unstructured data. At the same time, using GPT4 with streaming for front-end user chat use cases and separately using Gemini for processing large context-length PDF docs for queries tends to be a lot more accurate than the GPT suite. On average, we do see an application using 3 to 4 different models for their use case in significantly different ways.

2) There are hundreds of LLM routers in the market, and the goal isn't to add another one. Here are a few pain points we know exist that LLM routers in the market don't solve:

- On average, an AI app uses 3-4 models, meaning finding the best 3-4 for that application and sticking to it. In the article, we explain how our model scores the output and ranks the model over a few runs.

- Performance-Cost-Quality Ratio, while the best model in the market could execute almost every prompt. The best could also mean slow performance and high cost at scale, but this might not be the case in small apps that use Claud 3.5 sonnet, which should be good for all situations.

- Upgrading is not one of the biggest points, but AI is moving fast, just yesterday, Groq x Meta released llama 3.3 70b, which benchmarks close to the older 3.2 405b at a significantly lower cost with a huge performance increase with language support. Great "free" upgrade, but most apps using 3.2 405b/3.2 70b won't be able to make this quick change without expecting breaking changes on the prompt. We do this under the hood without the users having to change anything.

- Safeguards, with the AI world moving so quickly, safeguards are still far behind, and we're catching up. It's challenging for many companies to handle this at scale across multiple models while keeping up with all the new "hacks" and allowing users to choose the degree of guarding they would like in their application. This is something we can keep consistent across all models. Still pretty new for us too, but it's one of our main focus points. This is especially important when part of the prompt execution is dynamic variables that handle user inputs.

- The last and my favorite point is great DX, things from prompt caching at the API layer, prompt management, and storage for repeating execution. Dynamic variable management with smart caching. Internet access to LLMs with built-in safeguards. Etc...

Prompt Engine focuses on prompt management with smart routing rather than on the easy/constant switching of models.

3) This is a huge question mark for me, and I'm not sure how great this would be to develop as a product internally. We see in the SaaS-cloud world that multi-cloud is a norm now, it started with AWS being dominant, then providers like GCP & Azure took market share, and now we have layer-2 products like Supabase & Vercel. I bet that the same thing will happen in the LLM space, and we'll see apps building different pieces of their AI stack with ~2-3 different LLM providers that are stronger at different things.

Thanks for the super valid points! I love this kind of criticism, as it helps us be less idealistic and build better products :) I hope the points above answer some of the concerns, always happy to dive deeper over a chat!