Repository: promptfoo/promptfoo
Issue: Unable to view result for large dataset due to `RangeError: Invalid string length at JSON.stringify`
Root cause: Resource Mishandling

**Bug description**

I'm running promptfoo with a larger dataset:
- model: `anthropic:messages:claude-3-5-sonnet-20240620`
- 500 test cases
- prompt generated using Python function
  - prompt consists of: 1 system message, 1 assistant message and 1 user message (with 1 text block and 20 image blocks)
  - due to image size each prompt can have ~ 1.5MB size

Sample prompt returned from Python function:

```json
[
  {
    "role": "system",
    "content": "<system_prompt>"
  },
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "<user_prompt>"
      },
      {
        "type": "image",
        "source": {
          "type": "base64",
          "media_type": "image/png",
          "data": "iVBORw0KGgo..."
        }
      }
      // 19 more images
    ]
  },
  {
    "role": "assistant",
    "content": "{"
  }
]
```  

`promptfoo eval` runs as expected and returns

```
✔ Evaluation complete.

» Run promptfoo view to use the local web viewer
» Run promptfoo share to create a shareable URL
» This project needs your feedback. What's one thing we can improve? https://forms.gle/YFLgTe1dKJKNSCsU7
======================================================================================================================================================================
Successes: 500
Failures: 0
Errors: 0
Pass Rate: 100.00%
Total tokens: 940,994 / Prompt tokens: 779,812 / Completion tokens: 161,182 / Cached tokens: 0
Done.
```

However, when I try to inspect results using `promptfoo view` I'm getting:

```
Migrated results from file system to database
Server running at http://localhost:15500 and monitoring for new evals.
Open URL in browser? (y/N): /Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/socket.io-parser/build/cjs/index.js:92
            str += JSON.stringify(obj.data, this.replacer);
                        ^

RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at Encoder.encodeAsString (/Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/socket.io-parser/build/cjs/index.js:92:25)
    at Encoder.encode (/Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/socket.io-parser/build/cjs/index.js:68:22)
    at Adapter._encode (/Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/socket.io-adapter/dist/in-memory-adapter.js:167:45)
    at Adapter.broadcast (/Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/socket.io-adapter/dist/in-memory-adapter.js:119:37)
    at BroadcastOperator.emit (/Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/socket.io/dist/broadcast-operator.js:169:26)
    at Namespace.emit (/Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/socket.io/dist/namespace.js:309:73)
    at Server.<computed> [as emit] (/Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/socket.io/dist/index.js:798:33)
    at StatWatcher.<anonymous> (/Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/dist/src/server/server.js:177:16)

Node.js v20.17.0
```

> [!NOTE]
> The breaking point is somewhere between 250 and 350 files. I ran the same pipeline for 250 test cases and both `eval` and `view` worked as expected.

**To Reproduce**

Run `promptfoo eval` and `promptfoo view` on a larger dataset consisting of >350 test cases and prompts with file size ~1.5MB.

**Expected behavior**

`promptfoo view` displaying analysis result as expected.

**System information:**

`promptfoo debug` output:

```json
{
  "version": "0.102.4",
  "platform": {
    "os": "darwin",
    "release": "23.6.0",
    "arch": "arm64",
    "nodeVersion": "v20.17.0"
  },
  "env": {},
  "configInfo": {...}
}
```

- Promptfoo version: `0.102.4`


Hi @kamilbonna, thanks for reporting. #2509 adds an environment variable `PROMPTFOO_STRIP_PROMPT_TEXT` that should fix this issue for you when set.

Hi @typpo, thank you for quick response and adding environment variables to control inclusion of memory-heavy fields. 

I used `PROMPTFOO_STRIP_PROMPT_TEXT=true` and it resolved the issue, but I ran into another memory issue when I further increased the size of the dataset by including higher resolution images (n=500 samples, each prompt file ~4-8MB):

```
<--- Last few GCs --->

[52499:0x150008000]    27867 ms: Mark-Compact (reduce) 4107.2 (4113.8) -> 4107.2 (4113.8) MB, 16.83 / 0.00 ms  (average mu = 0.924, current mu = 0.390) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x1007fb5a4 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
 2: 0x10098404c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
 3: 0x100b58720 v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
 4: 0x100b571fc v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
 5: 0x100b4da14 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
 6: 0x100b4e274 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
 7: 0x100b32ba8 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
 8: 0x100b2930c v8::internal::MaybeHandle<v8::internal::SeqTwoByteString> v8::internal::FactoryBase<v8::internal::Factory>::NewRawStringWithMap<v8::internal::SeqTwoByteString>(int, v8::internal::Map, v8::internal::AllocationType) [/Users/kbonna/.nvm/versions/no]
 9: 0x100b34f7c v8::internal::Factory::NewStringFromUtf8(v8::base::Vector<unsigned char const> const&, unibrow::Utf8Variant, v8::internal::AllocationType) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
10: 0x100b35130 v8::internal::Factory::NewStringFromUtf8(v8::base::Vector<char const> const&, v8::internal::AllocationType) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
11: 0x1009a62b0 v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
12: 0x1281dfc20 Data::GetValueJS(v8::Isolate*, sqlite3_stmt*, int, bool) [/Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/better-sqlite3/build/Release/better_sqlite3.node]
13: 0x1281dbd0c Data::GetRowJS(v8::Isolate*, v8::Local<v8::Context>, sqlite3_stmt*, bool, char) [/Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/better-sqlite3/build/Release/better_sqlite3.node]
14: 0x1281da7e8 Statement::JS_all(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/kbonna/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/better-sqlite3/build/Release/better_sqlite3.node]
15: 0x1009f1bc8 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8:]
16: 0x1009f12c0 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
17: 0x101278b24 Builtins_CEntry_Return1_ArgvOnStack_BuiltinExit [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
18: 0x10668a28c 
19: 0x1011f03e4 Builtins_InterpreterEntryTrampoline [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
20: 0x106819270 
21: 0x1011f03e4 Builtins_InterpreterEntryTrampoline [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
22: 0x10683620c 
23: 0x1012d66d8 Builtins_PromiseResolveThenableJob [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
24: 0x101216d80 Builtins_RunMicrotasks [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
25: 0x1011ee3f4 Builtins_JSRunMicrotasksEntry [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
26: 0x100ac5ba0 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
27: 0x100ac608c v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
28: 0x100ac6268 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
29: 0x100aed434 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
30: 0x100aedbd0 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
31: 0x1009f1bc8 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8:]
32: 0x1009f12c0 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
33: 0x101278b24 Builtins_CEntry_Return1_ArgvOnStack_BuiltinExit [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
34: 0x1067b2aa8 
35: 0x1011ee50c Builtins_JSEntryTrampoline [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
36: 0x1011ee1f4 Builtins_JSEntry [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
37: 0x100ac5bc8 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
38: 0x100ac5014 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
39: 0x10099f904 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
40: 0x100724d24 node::InternalCallbackScope::Close() [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
41: 0x100724fe4 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
42: 0x10073b068 node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
43: 0x10081d324 node::(anonymous namespace)::Parser::OnStreamRead(long, uv_buf_t const&) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
44: 0x1008ce620 node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
45: 0x1008ced54 node::LibuvStreamWrap::ReadStart()::$_1::__invoke(uv_stream_s*, long, uv_buf_t const*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
46: 0x1011d9d24 uv__stream_io [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
47: 0x1011e1628 uv__io_poll [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
48: 0x1011cfb14 uv_run [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
49: 0x1007256f0 node::SpinEventLoopInternal(node::Environment*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
50: 0x10083b3f0 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
51: 0x10083b104 node::NodeMainInstance::Run() [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
52: 0x1007c2fa0 node::Start(int, char**) [/Users/kbonna/.nvm/versions/node/v20.17.0/bin/node]
53: 0x186693154 start [/usr/lib/dyld]
[1]    52499 abort      promptfoo view
```

I was able to get around that by increasing max memory size of V8's old memory section by specifying:

```sh
export NODE_OPTIONS="--max-old-space-size=16384"
```

but this "fix" won't work for even larger datasets as I'd inevitably run out of memory. I'm planning to run 5000 test cases for ~10MB prompt files, so I'd need a different solution. 

Is there another way to deal with this memory problem that I could try?

We're running in to the same error with large datasets. Would be great with a solution!