Repository: promptfoo/promptfoo
Issue: Can't get the 'assert' index from within a custom_metric script (and hence can't access the current assert variables)
Root cause: Documentation Deficiency

I am passing several extra parameters (such as threshold, rouge_mode etc) when declaring my custom metrics in the promptfooconfig.yml, and I'm trying the retrieve them from the custom metric script.

I discovered that I can get access to the extra variables through `config['test']['assert'][assert_index]['my_variable']`. But this dictionary contains all the asserts, not specifically the one being called and I have no mean to know the 'assert_index' being currently evaluated.

So at first I wrote a script that looped over the `config['test']['assert']` and compare  `config['test']['assert'][assert_index]['value']` with the custom_script being called. And it works !

Well it works as long as I don't declare several metrics using the same script.

Let's suppose I want to declare 3 different versions of my custom nlp_rouge_score.py script :

    - type: python
      value: file://../metrics/nlp_rouge_score.py
      threshold: 0.002
      metric: ROUGE Score Default

    - type: python
      value: file://../metrics/nlp_rouge_score.py
      threshold: 0.003
      rouge_type: rougeL
      rouge_mode: recall
      metric: ROUGE Score RL Recall

    - type: python
      value: file://../metrics/nlp_rouge_score.py
      threshold: 0.004
      rouge_type: rouge1
      rouge_mode: precision
      metric: ROUGE Score R1 Precision

Well in this case, my loop stop on the fist metric containing "nlp_rouge_score.py" and hence all my tests use the very first threshold found (0.002).

So I need a way to know which assert is being called (a simple index being passed as `context['test']['current_index']` would solve the problem).
Am I missing something already existing ?

- Promptfoo version: 0.104.4


Hey @Valkea, what you're looking for already exists - the `config` property on assertions.

Instead of putting custom properties directly on the assertion like this:

```yaml
- type: python
  value: file://../metrics/nlp_rouge_score.py
  threshold: 0.002
  rouge_type: rougeL  # This gets stripped
  rouge_mode: recall  # This too
```

Use the `config` property:

```yaml
- type: python
  value: file://../metrics/nlp_rouge_score.py
  config:
    rouge_type: rougeL
    rouge_mode: recall
    my_threshold: 0.003
  metric: ROUGE Score RL Recall

- type: python
  value: file://../metrics/nlp_rouge_score.py
  config:
    rouge_type: rouge1
    rouge_mode: precision
    my_threshold: 0.005
  metric: ROUGE Score R1 Precision
```

Then in your Python script, access them via `context['config']`:

```python
def get_assert(output, context):
    config = context.get('config', {})
    rouge_type = config.get('rouge_type', 'rouge1')
    rouge_mode = config.get('rouge_mode', 'fmeasure')
    threshold = config.get('my_threshold', 0.5)

    # Your scoring logic here...
```

Each assertion's `config` is passed independently, so the same script works with different configurations. No need to track assertion indices.

This is documented here: https://www.promptfoo.dev/docs/configuration/expected-outputs/python/#external-py

Note: The built-in `threshold` property on assertions is still useful - it's automatically applied to your score. So if your script returns `{'pass': True, 'score': 0.8}` and the assertion has `threshold: 0.9`, it will fail. You can use both together.