Repository: promptfoo/promptfoo
Issue: Facing isuue with API call for LLM model hosted in private VPC.
Root cause: External Dependency Breakage

**Describe the bug**
I am trying to run `promptfoo eval` command and getting the below error.

`API call error: Error: Request failed after 4  API call error: Error: Request failed after 4 retries: fetch failed.`

We have hosted the "huggingface:text-generation:llama3-8b-instruct" LLM model in our private VPC.


**To Reproduce**
I am running promptfoo as a container using below command:

`docker run --rm -it --entrypoint /bin/sh -p 3000:15500 ghcr.io/promptfoo/promptfoo:0.96.2`

below is my "promptfooconfig.yaml":

```
description: 'HF private inference endpoint'

prompts:
  - write few lines about {{artist}} & summarize it.

providers:
  - id: huggingface:text-generation:llama3-8b-instruct
    config:
      apiEndpoint: https://api_endpoint_of_LLM_model

tests:
  - vars:
      artist : "jordi"
```
same `promptfooconfig.yaml` is working fine when I do `promptfoo eval` with different image mentioned below:

`docker run --rm -it --entrypoint /bin/sh -p 3000:15500 ghcr.io/promptfoo/promptfoo:0.93.3`

1. Run '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**


**System information (please complete the following information):**

- Promptfoo version: 0.96.2

**Additional context**
Add any other context about the problem here.


Can you try running with --verbose or `LOG_LEVEL=debug` environment variable and share that output?

```
`Running migrations from /usr/local/lib/node_modules/promptfoo/dist/drizzle...
Drizzle: Query:
			CREATE TABLE IF NOT EXISTS "__drizzle_migrations" (
				id SERIAL PRIMARY KEY,
				hash text NOT NULL,
				created_at numeric
			)

Drizzle: Query: SELECT id, hash, created_at FROM "__drizzle_migrations" ORDER BY created_at DESC LIMIT 1
Drizzle: Query: BEGIN
Drizzle: Query: COMMIT
Running migrations from /usr/local/lib/node_modules/promptfoo/dist/drizzle...
Drizzle: Query:
			CREATE TABLE IF NOT EXISTS "__drizzle_migrations" (
				id SERIAL PRIMARY KEY,
				hash text NOT NULL,
				created_at numeric
			)

Drizzle: Query: SELECT id, hash, created_at FROM "__drizzle_migrations" ORDER BY created_at DESC LIMIT 1
Drizzle: Query: BEGIN
Drizzle: Query: COMMIT
Migrated results from file system to database
Server running at http://localhost:15500 and monitoring for new evals.
Open URL in browser? (y/N): Drizzle: Query: select "id" from "evals" order by "evals"."created_at" desc limit ? -- params: [1]
Drizzle: Query: select "id", "created_at", "author", "description", "results", "config", "prompts" from "evals" where "evals"."id" = ? -- params: ["eval-ODW-2024-11-15T11:38:22"]
Drizzle: Query: select "dataset_id" from "evals_to_datasets" where "evals_to_datasets"."eval_id" = ? limit ? -- params: ["eval-ODW-2024-11-15T11:38:22", 1]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-ODW-2024-11-15T11:38:22"]
Drizzle: Query: select "id" from "evals" order by "evals"."created_at" desc limit ? -- params: [1]
Drizzle: Query: select "id", "created_at", "author", "description", "results", "config", "prompts" from "evals" where "evals"."id" = ? -- params: ["eval-ODW-2024-11-15T11:38:22"]
Drizzle: Query: select "dataset_id" from "evals_to_datasets" where "evals_to_datasets"."eval_id" = ? limit ? -- params: ["eval-ODW-2024-11-15T11:38:22", 1]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-ODW-2024-11-15T11:38:22"]
Drizzle: Query: select "evals"."id", "evals"."created_at", "evals"."description", json_array_length("evals"."results"->'table'->'body'), "evals_to_datasets"."dataset_id", json_type("evals"."config", '$.redteam') IS NOT NULL from "evals" left join "evals_to_datasets" on "evals"."id" = "evals_to_datasets"."eval_id" where not "evals"."results" = ? order by "evals"."created_at" desc limit ? -- params: ["{}", 100]
Drizzle: Query: select "evals"."id", "evals"."created_at", "evals"."description", COUNT(DISTINCT "eval_results"."test_idx") as "numTests", "evals_to_datasets"."dataset_id" from "evals" left join "evals_to_datasets" on "evals"."id" = "evals_to_datasets"."eval_id" left join "eval_results" on "evals"."id" = "eval_results"."eval_id" where "evals"."results" = ? group by "evals"."id" order by "evals"."created_at" desc limit ? -- params: ["{}", 100]
listPreviousResults execution time: 9.64ms
listPreviousResults execution time: 28.15ms
Drizzle: Query: select "id" from "evals" order by "evals"."created_at" desc limit ? -- params: [1]
Drizzle: Query: select "id", "created_at", "author", "description", "results", "config", "prompts" from "evals" where "evals"."id" = ? -- params: ["eval-ODW-2024-11-15T11:38:22"]
Drizzle: Query: select "dataset_id" from "evals_to_datasets" where "evals_to_datasets"."eval_id" = ? limit ? -- params: ["eval-ODW-2024-11-15T11:38:22", 1]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-ODW-2024-11-15T11:38:22"]
Drizzle: Query: select "evals"."id", "evals"."created_at", "evals"."description", json_array_length("evals"."results"->'table'->'body'), "evals_to_datasets"."dataset_id", json_type("evals"."config", '$.redteam') IS NOT NULL from "evals" left join "evals_to_datasets" on "evals"."id" = "evals_to_datasets"."eval_id" where not "evals"."results" = ? order by "evals"."created_at" desc limit ? -- params: ["{}", 100]
Drizzle: Query: select "evals"."id", "evals"."created_at", "evals"."description", COUNT(DISTINCT "eval_results"."test_idx") as "numTests", "evals_to_datasets"."dataset_id" from "evals" left join "evals_to_datasets" on "evals"."id" = "evals_to_datasets"."eval_id" left join "eval_results" on "evals"."id" = "eval_results"."eval_id" where "evals"."results" = ? group by "evals"."id" order by "evals"."created_at" desc limit ? -- params: ["{}", 100]
listPreviousResults execution time: 5.55ms
listPreviousResults execution time: 5.08ms
Running migrations from /usr/local/lib/node_modules/promptfoo/dist/drizzle...
Drizzle: Query:
			CREATE TABLE IF NOT EXISTS "__drizzle_migrations" (
				id SERIAL PRIMARY KEY,
				hash text NOT NULL,
				created_at numeric
			)

Drizzle: Query: SELECT id, hash, created_at FROM "__drizzle_migrations" ORDER BY created_at DESC LIMIT 1
Drizzle: Query: BEGIN
Drizzle: Query: COMMIT
Reading prompts from "write few lines about {{metal}} & summarize it."
Drizzle: Query: insert into "evals" ("id", "created_at", "author", "description", "results", "config", "prompts") values (?, ?, ?, ?, ?, ?, null) -- params: ["eval-NMf-2024-11-18T06:52:50", 1731912770495, null, "HF private inference endpoint", "{}", "{\"defaultTest\":{},\"description\":\"HF private inference endpoint\",\"env\":{},\"prompts\":[{\"raw\":\"write few lines about {{metal}} & summarize it.\",\"label\":\"write few lines about {{metal}} & summarize it.\"}],\"providers\":[{\"id\":\"huggingface:text-generation:llama3-8b-instruct\",\"config\":{\"apiEndpoint\":\"api_endpoint_of_LLM_model/8b/llm/\"}},{\"id\":\"huggingface:text-generation:llama3-70b-instruct\",\"config\":{\"apiEndpoint\":\"api_endpoint_of_LLM_model/70b/llm/\"}}],\"scenarios\":[],\"tests\":[{\"vars\":{\"metal\":\"gold\"}}],\"writeLatestResults\":true,\"sharing\":true}"]
Drizzle: Query: insert into "prompts" ("id", "created_at", "prompt") values (?, CURRENT_TIMESTAMP, ?) on conflict do nothing -- params: ["692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614", "write few lines about {{metal}} & summarize it."]
Drizzle: Query: insert into "evals_to_prompts" ("eval_id", "prompt_id") values (?, ?) on conflict do nothing -- params: ["eval-NMf-2024-11-18T06:52:50", "692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614"]
Inserting prompt 692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614
Drizzle: Query: insert into "datasets" ("id", "tests", "created_at") values (?, ?, CURRENT_TIMESTAMP) on conflict do nothing -- params: ["5c7265b5f01b631faf3e86191ae72fa863dafb2c605248769928c529bd71de3d", "[{\"vars\":{\"metal\":\"gold\"}}]"]
Drizzle: Query: insert into "evals_to_datasets" ("eval_id", "dataset_id") values (?, ?) on conflict do nothing -- params: ["eval-NMf-2024-11-18T06:52:50", "5c7265b5f01b631faf3e86191ae72fa863dafb2c605248769928c529bd71de3d"]
Inserting dataset 5c7265b5f01b631faf3e86191ae72fa863dafb2c605248769928c529bd71de3d
Running 2 concurrent evaluations with 4 threads...
Huggingface API request: api_endpoint_of_LLM_model/8b/llm/ {"inputs":"write few lines about gold & summarize it.","parameters":{"return_full_text":false}}
Huggingface API request: api_endpoint_of_LLM_model/70b/llm/ {"inputs":"write few lines about gold & summarize it.","parameters":{"return_full_text":false}}
Drizzle: Query: select "id" from "evals" order by "evals"."created_at" desc limit ? -- params: [1]
Drizzle: Query: select "id", "created_at", "author", "description", "results", "config", "prompts" from "evals" where "evals"."id" = ? -- params: ["eval-NMf-2024-11-18T06:52:50"]
Drizzle: Query: select "dataset_id" from "evals_to_datasets" where "evals_to_datasets"."eval_id" = ? limit ? -- params: ["eval-NMf-2024-11-18T06:52:50", 1]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-NMf-2024-11-18T06:52:50"]
Drizzle: Query: select "evals"."id", "evals"."created_at", "evals"."description", json_array_length("evals"."results"->'table'->'body'), "evals_to_datasets"."dataset_id", json_type("evals"."config", '$.redteam') IS NOT NULL from "evals" left join "evals_to_datasets" on "evals"."id" = "evals_to_datasets"."eval_id" where not "evals"."results" = ? order by "evals"."created_at" desc limit ? -- params: ["{}", 100]
Drizzle: Query: select "evals"."id", "evals"."created_at", "evals"."description", COUNT(DISTINCT "eval_results"."test_idx") as "numTests", "evals_to_datasets"."dataset_id" from "evals" left join "evals_to_datasets" on "evals"."id" = "evals_to_datasets"."eval_id" left join "eval_results" on "evals"."id" = "eval_results"."eval_id" where "evals"."results" = ? group by "evals"."id" order by "evals"."created_at" desc limit ? -- params: ["{}", 100]
listPreviousResults execution time: 4.97ms
listPreviousResults execution time: 5.05ms
Drizzle: Query: select "id", "created_at", "author", "description", "results", "config", "prompts" from "evals" order by "evals"."created_at" desc limit ? -- params: [100]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-NMf-2024-11-18T06:52:50"]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-ODW-2024-11-15T11:38:22"]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-DbI-2024-11-15T11:30:11"]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-rCb-2024-11-15T11:21:21"]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-FEq-2024-11-15T11:20:07"]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-ofY-2024-11-15T11:15:35"]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-eXQ-2024-11-15T11:05:13"]
Running migrations from /usr/local/lib/node_modules/promptfoo/dist/drizzle...
Drizzle: Query:
			CREATE TABLE IF NOT EXISTS "__drizzle_migrations" (
				id SERIAL PRIMARY KEY,
				hash text NOT NULL,
				created_at numeric
			)

Drizzle: Query: SELECT id, hash, created_at FROM "__drizzle_migrations" ORDER BY created_at DESC LIMIT 1
Drizzle: Query: BEGIN
Drizzle: Query: COMMIT
Reading prompts from "write few lines about {{metal}} & summarize it."
Drizzle: Query: insert into "evals" ("id", "created_at", "author", "description", "results", "config", "prompts") values (?, ?, ?, ?, ?, ?, null) -- params: ["eval-RYq-2024-11-18T06:54:10", 1731912850219, null, "HF private inference endpoint", "{}", "{\"defaultTest\":{},\"description\":\"HF private inference endpoint\",\"env\":{},\"prompts\":[{\"raw\":\"write few lines about {{metal}} & summarize it.\",\"label\":\"write few lines about {{metal}} & summarize it.\"}],\"providers\":[{\"id\":\"huggingface:text-generation:llama3-8b-instruct\",\"config\":{\"apiEndpoint\":\"api_endpoint_of_LLM_model/8b/llm/\"}},{\"id\":\"huggingface:text-generation:llama3-70b-instruct\",\"config\":{\"apiEndpoint\":\"api_endpoint_of_LLM_model/70b/llm/\"}}],\"scenarios\":[],\"tests\":[{\"vars\":{\"metal\":\"gold\"}}],\"writeLatestResults\":true,\"sharing\":true}"]
Drizzle: Query: insert into "prompts" ("id", "created_at", "prompt") values (?, CURRENT_TIMESTAMP, ?) on conflict do nothing -- params: ["692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614", "write few lines about {{metal}} & summarize it."]
Drizzle: Query: insert into "evals_to_prompts" ("eval_id", "prompt_id") values (?, ?) on conflict do nothing -- params: ["eval-RYq-2024-11-18T06:54:10", "692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614"]
Inserting prompt 692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614
Drizzle: Query: insert into "datasets" ("id", "tests", "created_at") values (?, ?, CURRENT_TIMESTAMP) on conflict do nothing -- params: ["5c7265b5f01b631faf3e86191ae72fa863dafb2c605248769928c529bd71de3d", "[{\"vars\":{\"metal\":\"gold\"}}]"]
Drizzle: Query: insert into "evals_to_datasets" ("eval_id", "dataset_id") values (?, ?) on conflict do nothing -- params: ["eval-RYq-2024-11-18T06:54:10", "5c7265b5f01b631faf3e86191ae72fa863dafb2c605248769928c529bd71de3d"]
Inserting dataset 5c7265b5f01b631faf3e86191ae72fa863dafb2c605248769928c529bd71de3d
Running 2 concurrent evaluations with 4 threads...
Huggingface API request: api_endpoint_of_LLM_model/8b/llm/ {"inputs":"write few lines about gold & summarize it.","parameters":{"return_full_text":false}}
Huggingface API request: api_endpoint_of_LLM_model/70b/llm/ {"inputs":"write few lines about gold & summarize it.","parameters":{"return_full_text":false}}
[136aa99e-3ffd-4676-931f-ee7c1f964ee9] 0/2
Eval #2 complete (1 of 2)
[391fb17e-fb49-4625-9767-95ee38fca81f] 0/2
Eval #2 complete (1 of 2)
Drizzle: Query: insert into "eval_results" ("id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata") values (?, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, ?, ?, ?, ?, ?, ?, ?, ?, null, ?, ?, ?, ?, ?, ?, null) returning "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" -- params: ["6f51b069-59e3-42f9-8e71-ef11e2e460db", "eval-NMf-2024-11-18T06:52:50", 1, 0, "{\"vars\":{\"metal\":\"gold\"},\"assert\":[],\"options\":{},\"metadata\":{}}", "{\"raw\":\"write few lines about gold & summarize it.\",\"label\":\"write few lines about {{metal}} & summarize it.\"}", "692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614", "{\"id\":\"huggingface:text-generation:llama3-70b-instruct\",\"label\":\"\",\"config\":{\"apiEndpoint\":\"api_endpoint_of_LLM_model/70b/llm/\"}}", 83886, "{\"error\":\"API call error: Error: Request failed after 4 retries: fetch failed. Output:\\nundefined\"}", "API call error: Error: Request failed after 4 retries: fetch failed. Output:\nundefined", 0, 0, null, "{}"]
Drizzle: Query: insert into "eval_results" ("id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata") values (?, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, ?, ?, ?, ?, ?, ?, ?, ?, null, ?, ?, ?, ?, ?, ?, null) returning "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" -- params: ["7f4bb6bf-75b7-4066-affd-c7842f7ab7a5", "eval-RYq-2024-11-18T06:54:10", 1, 0, "{\"vars\":{\"metal\":\"gold\"},\"assert\":[],\"options\":{},\"metadata\":{}}", "{\"raw\":\"write few lines about gold & summarize it.\",\"label\":\"write few lines about {{metal}} & summarize it.\"}", "692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614", "{\"id\":\"huggingface:text-generation:llama3-70b-instruct\",\"label\":\"\",\"config\":{\"apiEndpoint\":\"api_endpoint_of_LLM_model/70b/llm/\"}}", 4197, "{\"error\":\"API call error: Error: Request failed after 4 retries: fetch failed. Output:\\nundefined\"}", "API call error: Error: Request failed after 4 retries: fetch failed. Output:\nundefined", 0, 0, null, "{}"]
[136aa99e-3ffd-4676-931f-ee7c1f964ee9] 0/2
Eval #1 complete (2 of 2)
[391fb17e-fb49-4625-9767-95ee38fca81f] 0/2
Eval #1 complete (2 of 2)
Drizzle: Query: insert into "eval_results" ("id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata") values (?, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, ?, ?, ?, ?, ?, ?, ?, ?, null, ?, ?, ?, ?, ?, ?, null) returning "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" -- params: ["8c31292e-03f7-4e59-a0ee-379c2ff49c91", "eval-NMf-2024-11-18T06:52:50", 0, 0, "{\"vars\":{\"metal\":\"gold\"},\"assert\":[],\"options\":{},\"metadata\":{}}", "{\"raw\":\"write few lines about gold & summarize it.\",\"label\":\"write few lines about {{metal}} & summarize it.\"}", "692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614", "{\"id\":\"huggingface:text-generation:llama3-8b-instruct\",\"label\":\"\",\"config\":{\"apiEndpoint\":\"api_endpoint_of_LLM_model/8b/llm/\"}}", 85355, "{\"error\":\"API call error: Error: Request failed after 4 retries: fetch failed. Output:\\nundefined\"}", "API call error: Error: Request failed after 4 retries: fetch failed. Output:\nundefined", 0, 0, null, "{}"]
Drizzle: Query: insert into "eval_results" ("id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata") values (?, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, ?, ?, ?, ?, ?, ?, ?, ?, null, ?, ?, ?, ?, ?, ?, null) returning "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" -- params: ["8257d8e3-17fa-4ab8-a1a3-44f16f68de0a", "eval-RYq-2024-11-18T06:54:10", 0, 0, "{\"vars\":{\"metal\":\"gold\"},\"assert\":[],\"options\":{},\"metadata\":{}}", "{\"raw\":\"write few lines about gold & summarize it.\",\"label\":\"write few lines about {{metal}} & summarize it.\"}", "692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614", "{\"id\":\"huggingface:text-generation:llama3-8b-instruct\",\"label\":\"\",\"config\":{\"apiEndpoint\":\"api_endpoint_of_LLM_model/8b/llm/\"}}", 5664, "{\"error\":\"API call error: Error: Request failed after 4 retries: fetch failed. Output:\\nundefined\"}", "API call error: Error: Request failed after 4 retries: fetch failed. Output:\nundefined", 0, 0, null, "{}"]
Drizzle: Query: update "evals" set "prompts" = ? where "evals"."id" = ? -- params: ["[{\"raw\":\"write few lines about {{metal}} & summarize it.\",\"label\":\"write few lines about {{metal}} & summarize it.\",\"id\":\"692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614\",\"provider\":\"huggingface:text-generation:llama3-8b-instruct\",\"metrics\":{\"score\":0,\"testPassCount\":0,\"testFailCount\":1,\"assertPassCount\":0,\"assertFailCount\":0,\"totalLatencyMs\":85355,\"tokenUsage\":{\"total\":0,\"prompt\":0,\"completion\":0,\"cached\":0},\"namedScores\":{},\"namedScoresCount\":{},\"cost\":0}},{\"raw\":\"write few lines about {{metal}} & summarize it.\",\"label\":\"write few lines about {{metal}} & summarize it.\",\"id\":\"692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614\",\"provider\":\"huggingface:text-generation:llama3-70b-instruct\",\"metrics\":{\"score\":0,\"testPassCount\":0,\"testFailCount\":1,\"assertPassCount\":0,\"assertFailCount\":0,\"totalLatencyMs\":83886,\"tokenUsage\":{\"total\":0,\"prompt\":0,\"completion\":0,\"cached\":0},\"namedScores\":{},\"namedScoresCount\":{},\"cost\":0}}]", "eval-NMf-2024-11-18T06:52:50"]
Drizzle: Query: update "evals" set "prompts" = ? where "evals"."id" = ? -- params: ["[{\"raw\":\"write few lines about {{metal}} & summarize it.\",\"label\":\"write few lines about {{metal}} & summarize it.\",\"id\":\"692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614\",\"provider\":\"huggingface:text-generation:llama3-8b-instruct\",\"metrics\":{\"score\":0,\"testPassCount\":0,\"testFailCount\":1,\"assertPassCount\":0,\"assertFailCount\":0,\"totalLatencyMs\":5664,\"tokenUsage\":{\"total\":0,\"prompt\":0,\"completion\":0,\"cached\":0},\"namedScores\":{},\"namedScoresCount\":{},\"cost\":0}},{\"raw\":\"write few lines about {{metal}} & summarize it.\",\"label\":\"write few lines about {{metal}} & summarize it.\",\"id\":\"692002674b677a1eb6742846ba06d8ac372322e5fbefd6b77db9ac893223b614\",\"provider\":\"huggingface:text-generation:llama3-70b-instruct\",\"metrics\":{\"score\":0,\"testPassCount\":0,\"testFailCount\":1,\"assertPassCount\":0,\"assertFailCount\":0,\"totalLatencyMs\":4197,\"tokenUsage\":{\"total\":0,\"prompt\":0,\"completion\":0,\"cached\":0},\"namedScores\":{},\"namedScoresCount\":{},\"cost\":0}}]", "eval-RYq-2024-11-18T06:54:10"]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-NMf-2024-11-18T06:52:50"]
[136aa99e-3ffd-4676-931f-ee7c1f964ee9] Complete
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-RYq-2024-11-18T06:54:10"]
[391fb17e-fb49-4625-9767-95ee38fca81f] Complete
Drizzle: Query: select "id" from "evals" order by "evals"."created_at" desc limit ? -- params: [1]
Drizzle: Query: select "id", "created_at", "author", "description", "results", "config", "prompts" from "evals" where "evals"."id" = ? -- params: ["eval-RYq-2024-11-18T06:54:10"]
Drizzle: Query: select "dataset_id" from "evals_to_datasets" where "evals_to_datasets"."eval_id" = ? limit ? -- params: ["eval-RYq-2024-11-18T06:54:10", 1]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-RYq-2024-11-18T06:54:10"]
Drizzle: Query: select "evals"."id", "evals"."created_at", "evals"."description", json_array_length("evals"."results"->'table'->'body'), "evals_to_datasets"."dataset_id", json_type("evals"."config", '$.redteam') IS NOT NULL from "evals" left join "evals_to_datasets" on "evals"."id" = "evals_to_datasets"."eval_id" where not "evals"."results" = ? order by "evals"."created_at" desc limit ? -- params: ["{}", 100]
Drizzle: Query: select "evals"."id", "evals"."created_at", "evals"."description", COUNT(DISTINCT "eval_results"."test_idx") as "numTests", "evals_to_datasets"."dataset_id" from "evals" left join "evals_to_datasets" on "evals"."id" = "evals_to_datasets"."eval_id" left join "eval_results" on "evals"."id" = "eval_results"."eval_id" where "evals"."results" = ? group by "evals"."id" order by "evals"."created_at" desc limit ? -- params: ["{}", 100]
listPreviousResults execution time: 4.96ms
listPreviousResults execution time: 4.77ms
`
```
Please find the above logs with `LOG_LEVEL=debug` environment variable.

Hi @typpo any update on the issue I am facing.

Hi, @typpo I am also facing same issue.

Hard to say.  The `fetch failed` error @sagarghutke is getting is caused by things like: invalid hostname, unable to resolve IP address, firewall blocking the request, connection timeout, proxy needs to be set up, etc.

Are you able to `curl` the endpoint from within your docker image?

Hi @typpo I am able to curl to the endpoint from both below mentioned docker container.

```
docker run --rm -it --entrypoint /bin/sh -p 3000:15500 ghcr.io/promptfoo/promptfoo:main
docker run --rm -it --entrypoint /bin/sh -p 3000:15500 ghcr.io/promptfoo/promptfoo:0.96.2
```
But with the first container I mentioned here we are able to do promptfoo eval to same the endpoint without any change in the config.
I have mentioned the logs from the first container below:

```
Running migrations from /usr/local/lib/node_modules/promptfoo/dist/drizzle...
Drizzle: Query:
CREATE TABLE IF NOT EXISTS "__drizzle_migrations" (
id SERIAL PRIMARY KEY,
hash text NOT NULL,
created_at numeric
)
Drizzle: Query: SELECT id, hash, created_at FROM "__drizzle_migrations" ORDER BY created_at DESC LIMIT 1
Drizzle: Query: BEGIN
Drizzle: Query: COMMIT
Reading prompts from "write something about {{city}}"
Drizzle: Query: insert into "evals" ("id", "created_at", "author", "description", "results", "config", "prompts") values (?, ?, null, ?, ?, ?, null) -- params: ["eval-SEx-2024-11-26T07:05:17", 1732604717661, "HF private inference endpoint", "{}", "{\"defaultTest\":{},\"description\":\"HF private inference endpoint\",\"env\":{},\"prompts\":[{\"raw\":\"write something about {{city}}\",\"label\":\"write something about {{city}}\"}],\"providers\":[{\"id\":\"huggingface:text-generation:llama3-8b-instruct\",\"config\":{\"apiEndpoint\":\"https://LLM_edpoint\",\"use_cache\":false,\"wait_for_model\":true}}],\"scenarios\":[],\"tests\":[{\"vars\":{\"city\":\"canada\"}}],\"writeLatestResults\":true,\"sharing\":true}"]
Drizzle: Query: insert into "prompts" ("id", "created_at", "prompt") values (?, cast(unixepoch() as int), ?) on conflict do nothing -- params: ["2de797bc74f810631a3454211ea4b6fccab152c8266229bef3490585c2bbc082", "write something about {{city}}"]
Drizzle: Query: insert into "evals_to_prompts" ("eval_id", "prompt_id") values (?, ?) on conflict do nothing -- params: ["eval-SEx-2024-11-26T07:05:17", "2de797bc74f810631a3454211ea4b6fccab152c8266229bef3490585c2bbc082"]
Inserting prompt 2de797bc74f810631a3454211ea4b6fccab152c8266229bef3490585c2bbc082
Drizzle: Query: insert into "datasets" ("id", "tests", "created_at") values (?, ?, cast(unixepoch() as int)) on conflict do nothing -- params: ["d742081a5464da672838bbcab9a876b52dada1cd9e47ecc03d15a80ce81ac7d7", "[{\"vars\":{\"city\":\"canada\"}}]"]
Drizzle: Query: insert into "evals_to_datasets" ("eval_id", "dataset_id") values (?, ?) on conflict do nothing -- params: ["eval-SEx-2024-11-26T07:05:17", "d742081a5464da672838bbcab9a876b52dada1cd9e47ecc03d15a80ce81ac7d7"]
Inserting dataset d742081a5464da672838bbcab9a876b52dada1cd9e47ecc03d15a80ce81ac7d7
Huggingface API request: https://LLM_edpoint {"inputs":"write something about canada","parameters":{"return_full_text":false,"use_cache":false,"wait_for_model":true}}
Returning cached response for https://LLM_edpoint: [{"generated_text":"\nCanada is a country located in North America, bordered by the United States to the south and the Atlantic Ocean to the east. It is the second-largest country in the world by land area, after Russia, and is known for its vast wilderness, diverse wildlife, and natural beauty. Canada is also a bilingual country, with both English and French being official languages.\nCanada has a rich history, dating back to the arrival of indigenous peoples thousands of years ago. The country was later colonized by European"}]
Huggingface API response: [{"generated_text":"\nCanada is a country located in North America, bordered by the United States to the south and the Atlantic Ocean to the east. It is the second-largest country in the world by land area, after Russia, and is known for its vast wilderness, diverse wildlife, and natural beauty. Canada is also a bilingual country, with both English and French being official languages.\nCanada has a rich history, dating back to the arrival of indigenous peoples thousands of years ago. The country was later colonized by European"}]
[101dc658-5dcd-4bde-a3cc-a54499b7cd0c] 0/1
Eval #1 complete (1 of 1)
Drizzle: Query: insert into "eval_results" ("id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "provider_id", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata") values (?, cast(unixepoch() as int), cast(unixepoch() as int), ?, ?, ?, ?, ?, ?, ?, null, ?, null, ?, null, ?, ?, ?, ?, null) returning "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "provider_id", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" -- params: ["a724576d-2778-4811-b81e-8176805d6705", "eval-SEx-2024-11-26T07:05:17", 0, 0, "{\"vars\":{\"city\":\"canada\"},\"assert\":[],\"options\":{},\"metadata\":{}}", "{\"raw\":\"write something about canada\",\"label\":\"write something about {{city}}\"}", "2de797bc74f810631a3454211ea4b6fccab152c8266229bef3490585c2bbc082", "{\"id\":\"huggingface:text-generation:llama3-8b-instruct\",\"label\":\"\",\"config\":{\"apiEndpoint\":\"https://LLM_edpoint\",\"use_cache\":false,\"wait_for_model\":true}}", 3, "{\"output\":\"\\nCanada is a country located in North America, bordered by the United States to the south and the Atlantic Ocean to the east. It is the second-largest country in the world by land area, after Russia, and is known for its vast wilderness, diverse wildlife, and natural beauty. Canada is also a bilingual country, with both English and French being official languages.\\nCanada has a rich history, dating back to the arrival of indigenous peoples thousands of years ago. The country was later colonized by European\"}", 1, 1, "{\"pass\":true,\"score\":1,\"reason\":\"No assertions\",\"tokensUsed\":{\"total\":0,\"prompt\":0,\"completion\":0,\"cached\":0},\"assertion\":null}", "{}"]
Drizzle: Query: update "evals" set "prompts" = ? where "evals"."id" = ? -- params: ["[{\"raw\":\"write something about {{city}}\",\"label\":\"write something about {{city}}\",\"id\":\"2de797bc74f810631a3454211ea4b6fccab152c8266229bef3490585c2bbc082\",\"provider\":\"huggingface:text-generation:llama3-8b-instruct\",\"metrics\":{\"score\":1,\"testPassCount\":1,\"testFailCount\":0,\"assertPassCount\":0,\"assertFailCount\":0,\"totalLatencyMs\":3,\"tokenUsage\":{\"total\":0,\"prompt\":0,\"completion\":0,\"cached\":0},\"namedScores\":{},\"namedScoresCount\":{},\"cost\":0}}]", "eval-SEx-2024-11-26T07:05:17"]
Drizzle: Query: select "id", "provider_id", "options" from "providers" where "providers"."id" = ? -- params: ["d03979eacdf49e0dd4d0cf24206c93650d2458b1d796a313c4d89070e41f0c42"]
Drizzle: Query: insert into "evals_to_providers" ("provider_id", "eval_id") values (?, ?) on conflict do nothing -- params: ["d03979eacdf49e0dd4d0cf24206c93650d2458b1d796a313c4d89070e41f0c42", "eval-SEx-2024-11-26T07:05:17"]
[101dc658-5dcd-4bde-a3cc-a54499b7cd0c] Complete
Drizzle: Query: select "id" from "evals" order by "evals"."created_at" desc limit ? -- params: [1]
Drizzle: Query: select "id", "created_at", "author", "description", "results", "config", "prompts" from "evals" where "evals"."id" = ? -- params: ["eval-SEx-2024-11-26T07:05:17"]
Drizzle: Query: select "dataset_id" from "evals_to_datasets" where "evals_to_datasets"."eval_id" = ? limit ? -- params: ["eval-SEx-2024-11-26T07:05:17", 1]
Drizzle: Query: select "id", "created_at", "updated_at", "eval_id", "prompt_idx", "test_idx", "test_case", "prompt", "prompt_id", "provider", "provider_id", "latency_ms", "cost", "response", "error", "success", "score", "grading_result", "named_scores", "metadata" from "eval_results" where "eval_results"."eval_id" = ? -- params: ["eval-SEx-2024-11-26T07:05:17"]
Drizzle: Query: select "evals"."id", "evals"."created_at", "evals"."description", json_array_length("evals"."results"->'table'->'body'), "evals_to_datasets"."dataset_id" from "evals" left join "evals_to_datasets" on "evals"."id" = "evals_to_datasets"."eval_id" where not "evals"."results" = ? order by "evals"."created_at" desc limit ? -- params: ["{}", 100]
Drizzle: Query: select "evals"."id", "evals"."created_at", "evals"."description", COUNT(DISTINCT "eval_results"."test_idx") as "numTests", "evals_to_datasets"."dataset_id" from "evals" left join "evals_to_datasets" on "evals"."id" = "evals_to_datasets"."eval_id" left join "eval_results" on "evals"."id" = "eval_results"."eval_id" where "evals"."results" = ? group by "evals"."id" order by "evals"."created_at" desc limit ? -- params: ["{}", 100]
listPreviousResults execution time: 3.34ms
listPreviousResults execution time: 2.93ms
```
Where as I am facing issue with second error as I mentioned.
ERROR:
`API call error: Error: Request failed after 4 retries: fetch failed. Output:undefined`

Hey @sagarghutke, @Bunny15738 - good catch on this one.

This was a real regression. When we moved from `node-fetch` to native fetch in v0.96.x, the proxy handling broke for private endpoints. The `proxy-agent` library we were using wasn't compatible with how native fetch handles agents, which explains why v0.93.3 worked and v0.96.2 didn't.

We shipped a fix in January 2025 (PR #2737) that switches to `undici`'s global dispatcher pattern, which handles TLS and proxy connections properly. This has been in all releases since then (current version is 0.120.x).

Upgrading should fix it. If you're working with self-signed certs or custom CAs, these env vars might help:

```
PROMPTFOO_INSECURE_SSL=true
PROMPTFOO_CA_CERT_PATH=/path/to/your/ca.crt
```

Closing this as resolved. If you run into anything else on a recent version, open a new issue.