Repository: promptfoo/promptfoo
Issue: Promptfoo crashes due to memory limit
Root cause: Resource Mishandling

**Describe the bug**
When running the promptfoo viewer when there is a large test dataset, the programs runs for a few seconds and then crashes with the error message `FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory`.

**To Reproduce**

Run `promptfoo eval` against a prompt with 100 `tests` that each test has large dataset (in my example, full HTML of a website), and then run `promptfoo view`

**System information (please complete the following information):**
Node v20.0.5
macOS 14.5
Apple M2 Pro
32GB RAM

Promptfoo v0.76.1

**Additional context**
<details>
  <summary>Full crash log</summary>
```
<--- Last few GCs --->

[28421:0x128008000]     6916 ms: Mark-Compact 4009.7 (4143.9) -> 4009.7 (4143.9) MB, 31.88 / 0.00 ms  (average mu = 0.550, current mu = 0.100) allocation failure; scavenge might not succeed
  [28421:0x128008000]     6951 ms: Mark-Compact 4009.9 (4144.0) -> 4009.9 (4144.3) MB, 34.71 / 0.00 ms  (average mu = 0.341, current mu = 0.017) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
1: 0x104168588 node::Abort() [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
2: 0x104168770 node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
3: 0x1042ea33c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
4: 0x1044be9f4 v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
5: 0x1044bd4d0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
6: 0x1044b3ce8 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
7: 0x1044b4548 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
8: 0x104498e10 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
9: 0x10448f51c v8::internal::MaybeHandle<v8::internal::SeqTwoByteString> v8::internal::FactoryBase<v8::internal::Factory>::NewRawStringWithMap<v8::internal::SeqTwoByteString>(int, v8::internal::Map, v8::internal::AllocationType) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
10: 0x1045c3704 v8::internal::JsonParser<unsigned short>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
11: 0x1045c1898 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::JsonParser<unsigned short>::ParseJsonValue<false>(v8::internal::Handle<v8::internal::Object>) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
12: 0x1045bf2c8 v8::internal::JsonParser<unsigned short>::ParseJson(v8::internal::Handle<v8::internal::Object>) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
13: 0x1045bf1e8 v8::internal::JsonParser<unsigned short>::Parse(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::Object>) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
14: 0x104378680 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
15: 0x104be0b24 Builtins_CEntry_Return1_ArgvOnStack_BuiltinExit [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
16: 0x104b583e4 Builtins_InterpreterEntryTrampoline [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
17: 0x109ea45d0
18: 0x104c07fb4 Builtins_ArrayReduce [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
19: 0x104b583e4 Builtins_InterpreterEntryTrampoline [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
20: 0x104b583e4 Builtins_InterpreterEntryTrampoline [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
21: 0x104c03ee4 Builtins_ArrayMap [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
22: 0x104b583e4 Builtins_InterpreterEntryTrampoline [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
23: 0x104b583e4 Builtins_InterpreterEntryTrampoline [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
24: 0x104b583e4 Builtins_InterpreterEntryTrampoline [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
25: 0x104b583e4 Builtins_InterpreterEntryTrampoline a[/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
26: 0x109f66a1c
27: 0x104b583e4 Builtins_InterpreterEntryTrampoline [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
28: 0x104b583e4 Builtins_InterpreterEntryTrampoline [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
29: 0x109f66a1c
30: 0x109f68ec8
31: 0x109f63e70
32: 0x109f65e98
33: 0x109f9fcb4
34: 0x109f66a1c
35: 0x109f6e1a0
36: 0x109f68f50
37: 0x109f63e70
38: 0x109f65e98
39: 0x109fa03b4
40: 0x109f66a1c
41: 0x109f6e1a0
42: 0x109f68f50
43: 0x109f63e70
44: 0x109f65e98
45: 0x109f74870
46: 0x109f66a1c
47: 0x109f6e1a0
48: 0x109f68f50
49: 0x109f63e70
50: 0x109f65e98
51: 0x109f93b8c
52: 0x109f74d7c
53: 0x109f7498c
54: 0x109f77328
55: 0x109f74a34
56: 0x109f74bd8
57: 0x109f66a1c
58: 0x109f6e1a0
59: 0x109f68f50
60: 0x109f63e70
61: 0x109f65e98
62: 0x109f735cc
63: 0x109f66a1c
64: 0x109f6e1a0
65: 0x109f68f50
66: 0x109f63e70
67: 0x109f65e98
68: 0x109f74ec8
69: 0x109f66a1c
70: 0x109f6e1a0
71: 0x109f68f50
72: 0x109f63e70
73: 0x109f65e98
74: 0x109f75758
75: 0x109f76738
76: 0x109f758e4
77: 0x109f779b0
78: 0x109f7a504
79: 0x109f26400
80: 0x109f91fa8
81: 0x109f72e38
82: 0x104b5650c Builtins_JSEntryTrampoline [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
83: 0x104b561f4 Builtins_JSEntry [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
84: 0x10442bfa4 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
85: 0x10442b3f0 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
86: 0x104305cc8 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
87: 0x10418b8bc node::(anonymous namespace)::Parser::Proxy<int (node::(anonymous namespace)::Parser::*)(), &node::(anonymous namespace)::Parser::on_headers_complete()>::Raw(llhttp__internal_s*) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
88: 0x10508741c llhttp__internal__run [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
89: 0x105082aac llhttp__internal_execute [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
90: 0x10418a990 node::(anonymous namespace)::Parser::Execute(char const*, unsigned long) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
91: 0x10418a7d0 node::(anonymous namespace)::Parser::OnStreamRead(long, uv_buf_t const&) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
92: 0x104236ba0 node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
93: 0x104237324 node::LibuvStreamWrap::ReadStart()::$_1::__invoke(uv_stream_s*, long, uv_buf_t const*) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
94: 0x104b3ff48 uv__stream_io [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
95: 0x104b4784c uv__io_poll [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
96: 0x104b35d38 uv_run [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
97: 0x10409d754 node::SpinEventLoopInternal(node::Environment*) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
98: 0x1041a8984 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
99: 0x1041a872c node::NodeMainInstance::Run() [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
100: 0x1041344b0 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResultImpl const*) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
101: 0x1041347d0 node::Start(int, char**) [/Users/{{user}}/Library/Application Support/fnm/node-versions/v20.5.0/installation/bin/node]
102: 0x19941e0e0 start [/usr/lib/dyld]
ELIFECYCLE  Command failed.
```

</details>

A few notes:
1. I set `PROMPTFOO_CACHE_MAX_SIZE` to `300000000` and `PROMPTFOO_CACHE_MAX_FILE_COUNT` to `50000`
2. Running `promptfoo cache clean` doesn't help
3. **Removing `~/.promptfoo/promptfoo.db` fixes the issue.**
4. My `~/.promptfoo/promptfoo.db` weights 1.3GB

Hello, thank you for raising this. I am also facing this issue. 🙏

We have a potential fix in #1439. It will be in the next release.

@typpo Thanks Ian! I have another issue related to the sharing feature, and I hope this PR will fix it as well. Otherwise, I'll document the issue and post it here.

Hi @MatanYadaev and @kaiktl,

I believe @typpo has addressed the issue you reported. Please try running the latest version of promptfoo. If the problem persists, feel free to re-open this issue or comment here. Thank you both for your contributions and feedback!