Repository: promptfoo/promptfoo
Issue: [HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout (Redteam test approach)
Root cause: External Dependency Breakage

**Describe the bug**  
When running dynamic prompt generation in Promptfoo (version 0.111.1), the process fails partway through with HTTP 504 errors for certain plugins. This prevents the generation of all test cases.

----------

**To Reproduce**

1.  Run `promptfoo test` or a similar command that triggers dynamic test case generation.
    
2.  Wait for plugin-based test generation to begin.
    
3.  Observe the following output:
    
```
[index.js:345] Test Generation Summary:
• Total tests: 15
• Plugin tests: 15
• Plugins: 3
• Strategies: 1
• Max concurrency: 1

[index.js:393] Validating plugins...
[index.js:418] Checking Promptfoo API health at https://api.promptfoo.app/health...
[apiHealth.js:16] [CheckRemoteHealth] Checking API health: {"url":"https://api.promptfoo.app/health","env":{"nodeTls":"0"}}
[apiHealth.js:35] [CheckRemoteHealth] Making fetch request: {"url":"https://api.promptfoo.app/health","options":{"headers":{"Content-Type":"application/json"}},"timeout":5000,"nodeVersion":"v23.10.0"}
[apiHealth.js:54] [CheckRemoteHealth] API health check response: {"data":{"status":"OK"}}
[index.js:424] API health check passed
[index.js:445] Extracting system purpose...
[index.js:452] Extracting entities...
[index.js:457] System purpose: Evaluate and test the responses of a language model application to various prompts
[index.js:467] Generating tests for harmful:radicalization...
[index.js:471] Generating tests for harmful:radicalization...
[promptfoo.js:35] [HarmfulCompletionProvider] Calling generate harmful API (https://api.promptfoo.app/api/v1/task/harmful) with body: {"email":"example@example.com","harmCategory":"harmful:radicalization","n":5,"purpose":"Evaluate and test the responses of a language model application to various prompts","version":"0.110.0"}
Open URL in browser? (y/N): [promptfoo.js:54] [HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 1
[promptfoo.js:35] [HarmfulCompletionProvider] Calling generate harmful API (https://api.promptfoo.app/api/v1/task/harmful) with body: {"email":"example@example.com","harmCategory":"harmful:radicalization","n":5,"purpose":"Evaluate and test the responses of a language model application to various prompts","version":"0.110.0"}
[promptfoo.js:54] [HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 1
[promptfoo.js:35] [HarmfulCompletionProvider] Calling generate harmful API (https://api.promptfoo.app/api/v1/task/harmful) with body: {"email":"example@example.com","harmCategory":"harmful:radicalization","n":5,"purpose":"Evaluate and test the responses of a language model application to various prompts","version":"0.110.0"}
[promptfoo.js:54] [HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 2
[promptfoo.js:35] [HarmfulCompletionProvider] Calling generate harmful API (https://api.promptfoo.app/api/v1/task/harmful) with body: {"email":"example@example.com","harmCategory":"harmful:radicalization","n":5,"purpose":"Evaluate and test the responses of a language model application to various prompts","version":"0.110.0"}
[promptfoo.js:54] [HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 2
[promptfoo.js:35] [HarmfulCompletionProvider] Calling generate harmful API (https://api.promptfoo.app/api/v1/task/harmful) with body: {"email":"example@example.com","harmCategory":"harmful:radicalization","n":5,"purpose":"Evaluate and test the responses of a language model application to various prompts","version":"0.110.0"}
[promptfoo.js:54] [HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 3
[generation.js:52] Sampling 5 items from array of length 0
[index.js:484] Failed to generate tests for harmful:radicalization
[index.js:502] Generated 0 tests for harmful:radicalization
[index.js:504] Added 0 harmful:radicalization test cases
[redteam.js:76] Error running redteam: Error: Operation cancelled
[promptfoo.js:54] [HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 3
[generation.js:52] Sampling 5 items from array of length 0
[index.js:484] Failed to generate tests for harmful:radicalization
[index.js:502] Generated 0 tests for harmful:radicalization
[index.js:504] Added 0 harmful:radicalization test cases
[index.js:467] Generating tests for harmful:self-harm...
[index.js:471] Generating tests for harmful:self-harm...
[promptfoo.js:35] [HarmfulCompletionProvider] Calling generate harmful API (https://api.promptfoo.app/api/v1/task/harmful) with body: {"email":"example@example.com","harmCategory":"harmful:self-harm","n":5,"purpose":"Evaluate and test the responses of a language model application to various prompts","version":"0.110.0"}
[promptfoo.js:54] [HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 1
[promptfoo.js:35] [HarmfulCompletionProvider] Calling generate harmful API (https://api.promptfoo.app/api/v1/task/harmful) with body: {"email":"example@example.com","harmCategory":"harmful:self-harm","n":5,"purpose":"Evaluate and test the responses of a language model application to various prompts","version":"0.110.0"}
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #1), retrying: TypeError: fetch failed (Cause: HeadersTimeoutError: Headers Timeout Error)
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #2), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo EAI_AGAIN api.promptfoo.app)
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #3), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo EAI_AGAIN api.promptfoo.app)
[promptfoo.js:54] [HarmfulCompletionProvider] Error: Request failed after 2 retries: TypeError: fetch failed (Cause: Error: getaddrinfo EAI_AGAIN api.promptfoo.app)
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 2
[promptfoo.js:35] [HarmfulCompletionProvider] Calling generate harmful API (https://api.promptfoo.app/api/v1/task/harmful) with body: {"email":"example@example.com","harmCategory":"harmful:self-harm","n":5,"purpose":"Evaluate and test the responses of a language model application to various prompts","version":"0.110.0"}
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #1), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo EAI_AGAIN api.promptfoo.app)
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #2), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo EAI_AGAIN api.promptfoo.app)
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #3), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo EAI_AGAIN api.promptfoo.app)
[promptfoo.js:54] [HarmfulCompletionProvider] Error: Request failed after 2 retries: TypeError: fetch failed (Cause: Error: getaddrinfo EAI_AGAIN api.promptfoo.app)
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 3
[generation.js:52] Sampling 5 items from array of length 0
[index.js:484] Failed to generate tests for harmful:self-harm
[index.js:502] Generated 0 tests for harmful:self-harm
[index.js:504] Added 0 harmful:self-harm test cases
[index.js:467] Generating tests for harmful:sexual-content...
[index.js:471] Generating tests for harmful:sexual-content...
[promptfoo.js:35] [HarmfulCompletionProvider] Calling generate harmful API (https://api.promptfoo.app/api/v1/task/harmful) with body: {"email":"example@example.com","harmCategory":"harmful:sexual-content","n":5,"purpose":"Evaluate and test the responses of a language model application to various prompts","version":"0.110.0"}
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #1), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo EAI_AGAIN api.promptfoo.app)
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #2), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo EAI_AGAIN api.promptfoo.app)
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #3), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
[promptfoo.js:54] [HarmfulCompletionProvider] Error: Request failed after 2 retries: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 1
[promptfoo.js:35] [HarmfulCompletionProvider] Calling generate harmful API (https://api.promptfoo.app/api/v1/task/harmful) with body: {"email":"example@example.com","harmCategory":"harmful:sexual-content","n":5,"purpose":"Evaluate and test the responses of a language model application to various prompts","version":"0.110.0"}
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #1), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #2), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #3), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
[promptfoo.js:54] [HarmfulCompletionProvider] Error: Request failed after 2 retries: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 2
[promptfoo.js:35] [HarmfulCompletionProvider] Calling generate harmful API (https://api.promptfoo.app/api/v1/task/harmful) with body: {"email":"example@example.com","harmCategory":"harmful:sexual-content","n":5,"purpose":"Evaluate and test the responses of a language model application to various prompts","version":"0.110.0"}
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #1), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #2), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
[fetch.js:208] Request to https://api.promptfoo.app/api/v1/task/harmful failed (attempt #3), retrying: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
[promptfoo.js:54] [HarmfulCompletionProvider] Error: Request failed after 2 retries: TypeError: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
[generation.js:32] Added 0 unique items. Total: 0
[generation.js:35] No new unique items. Consecutive retries: 3
[generation.js:52] Sampling 5 items from array of length 0
[index.js:484] Failed to generate tests for harmful:sexual-content
[index.js:502] Generated 0 tests for harmful:sexual-content
[index.js:504] Added 0 harmful:sexual-content test cases
```

An other example without debug mode. 

```
Generating test cases...
Cache is disabled
Synthesizing test cases for 1 prompt...
Using plugins:

harmful:hate (5  tests)
harmful:self-harm (5  tests)

Using strategies:



Test Generation Summary:
• Total tests: 10
• Plugin tests: 10
• Plugins: 2
• Strategies: 1
• Max concurrency: 1

Extracting system purpose...
Extracting entities...
Generating tests for harmful:hate...
[HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
[HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
[HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
Failed to generate tests for harmful:hate
Generated 0 tests for harmful:hate
Generating tests for harmful:self-harm...
[HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
[HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
[HarmfulCompletionProvider] Error: API call failed with status 504: upstream request timeout
Failed to generate tests for harmful:self-harm
Generated 0 tests for harmful:self-harm

Test Generation Report:
┌─────┬──────────┬────────────────────────────────────────┬────────────┬────────────┬──────────────┐
│ #   │ Type     │ ID                                     │ Requested  │ Generated  │ Status       │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 1   │ Plugin   │ harmful:hate                           │ 5          │ 0          │ Failed       │
├─────┼──────────┼────────────────────────────────────────┼────────────┼────────────┼──────────────┤
│ 2   │ Plugin   │ harmful:self-harm                      │ 5          │ 0          │ Failed       │
└─────┴──────────┴────────────────────────────────────────┴────────────┴────────────┴──────────────┘
No test cases generated. Please check for errors and try again.
No test cases generated. Skipping scan. 
```
    
Several plugins (e.g., harmful:chemical-biological-weapons) fail to generate their respective tests.

----------

**Expected behavior**  
All plugin-based tests should be generated without errors, assuming a valid internet connection and no local misconfiguration.

**System information:**

-   Promptfoo version: `0.111.1`
-   Promptfoo version: `0.110.0`
    
**Additional context**

-   Issue seems isolated to specific harmful content-related plugins. (https://api.promptfoo.app/api/v1/task/harmful)
-   System has a stable internet connection; other plugin tests generate correctly,  **but if I use the harmfull plugin at a given point in my tests, it breaks the generation of the other prompts, and therefore of my tests (timeout after several minutes)**. 
-   Possibly a timeout issue on the plugin provider side.

✅ Update
The [HarmfulCompletionProvider] endpoint is working again.
Looks like the issue was indeed API-related ( possibly due to an update on https://api.promptfoo.app/api/ yesterday?)


For the future, having a status page would be super helpful to alert everyone when the API (or parts of it) are down.
If that’s too long to implement, even a quick update on X (Twitter) would do the trick — just to keep the community informed in real-time.

I'm closing the GitHub issue.


*Nothing to do with the subject above, but thank you very much for this project, which I think is one of the most successful I've ever seen in all areas of IT.* 👏