Repository: promptfoo/promptfoo
Issue: custom prompt with custom arguments
Root cause: Unimplemented Feature Gap

I've been enjoying using promptfoo. However, one issue I run into is to use python custom prompt, it has to take in `context` as the argument, and then we extract actual inputs needed for the function from context["vars"]. This is fairly awkward. Now I have a lot of boiler code encoding and decoding `context`. In addition, if all my prompt function takes in a generic dict `context`, it is really hard to enforce type safety.  Is there any recommendation around this?

Hi @lilipads,

Thank you for your feedback and for highlighting the issues with `context["vars"]` in custom Python prompts.  I am all for type safety. Here are a few things that I think work reasonably well to improve the handling of `context["vars"]`. Here's a simple RAG example.

### 1. TypedDict

Using `TypedDict` provides a way to define the structure of a dictionary with type hints. This approach is compatible with Python 3.8 and later.

```python
from typing import TypedDict, List, Optional

class PromptVars(TypedDict, total=False):
    question: str
    retrieved_contexts: List[str]
    user_id: Optional[str]

def custom_prompt_typed_dict(context: dict) -> str:
    vars: PromptVars = context["vars"]
    retrieved_info = " ".join(vars.get("retrieved_contexts", []))
    prompt = (
        f"User {vars.get('user_id', 'unknown')} asks: {vars['question']}. "
        f"Here's some information that might help: {retrieved_info}"
    )
    return prompt
```

### 2. Dataclass

For a cleaner and more type-safe structure, consider using Python's `dataclass`. This approach is compatible with Python 3.7 and later.

```python
from dataclasses import dataclass, field
from typing import List, Optional

@dataclass
class PromptVars:
    question: str
    retrieved_contexts: List[str] = field(default_factory=list)
    user_id: Optional[str] = None

def custom_prompt_dataclass(context: dict) -> str:
    vars = PromptVars(**context["vars"])
    retrieved_info = " ".join(vars.retrieved_contexts)
    prompt = (
        f"User {vars.user_id or 'unknown'} asks: {vars.question}. "
        f"Here's some information that might help: {retrieved_info}"
    )
    return prompt
```

### 3. Pydantic

Pydantic offers advanced data validation and parsing capabilities.

```python
from pydantic import BaseModel, Field
from typing import List, Optional

class PromptVars(BaseModel):
    question: str = Field(..., description="Question to ask the model")
    retrieved_contexts: List[str] = Field(default_factory=list, description="List of retrieved information snippets")
    user_id: Optional[str] = Field(None, description="ID of the user asking the question")

def custom_prompt_pydantic(context: dict) -> str:
    vars = PromptVars(**context["vars"])
    retrieved_info = " ".join(vars.retrieved_contexts)
    prompt = (
        f"User {vars.user_id or 'unknown'} asks: {vars.question}. "
        f"Here's some information that might help: {retrieved_info}"
    )
    return prompt
```

If you want to get wild, you can add validations on the return type as well. I'd love to hear your thoughts on these suggestions. If you're interested, I would happily accept a PR in the examples folder around improving type safety, or hop on a call to discuss any further improvements you have in mind.

Thank you for your thorough answer!

I was using some of these solutions myself as well, so great to have validation! They do solve the type safety issues, though not addressing the boilerplate issue. I would need to convert all my existing functions to take in `context`; I imagine it is a common problem for users who adopt promptfoo later on in an existing codebase. That being said, I don't have a better solution to offer.

@lilipads you're welcome! You can always unpack vars as kwargs if you have existing functions you want to integrate. The mapping seems like a great task for an LLM! Please let us know if you run into any other issues.