Repository: promptfoo/promptfoo
Issue: Only most recent Eval is viewable in the WebViewer - Failed to execute 'setItem' on 'Storage'
Root cause: Resource Mishandling

# Description
Only the most recent evaluation is able to be loaded in the WebViewer.  Each time a new eval is run, any previous evals that have been run do not load in the 'Evals' tab of the WebViewer.

## Steps to reproduce:
1.  Run an eval
2. View it in the 'Evals' tab of the WebViewer
3. Run a second eval
4. Attempt to load the eval from step 1 by selecting it in the 'Progress' tab

**NOTE:** the size of the eval data (prompt, context, response, # of test cases) may be the issue.  I am running large evals comparing 3 different models

<img width="1720" alt="Screenshot 2024-05-20 at 12 33 52 PM" src="https://github.com/promptfoo/promptfoo/assets/95300041/13cc520c-e8af-45bd-8904-3b4730ab1079">

## Additional Context
Google Chrome (Version 124.0.6367.91 (Official Build) (arm64)) on an M1 Macbook Pro.

Error in console of Chrome:

> Uncaught (in promise) DOMException: Failed to execute 'setItem' on 'Storage': Setting the value of 'ResultsViewStorage' exceeded the quota.

I believe this issue originates from the code within `web/nextui/src/app/eval/store.ts`:
```typescript
export const useStore = create<TableState>()(
  persist(
    (set, get) => ({
      evalId: null,
      setEvalId: (evalId: string) => set(() => ({ evalId })),
      ........
    }),
    {
      name: 'ResultsViewStorage',
      storage: createJSONStorage(() => localStorage),
    },
  ),
);
```



Are your evals particularly large and/or do you have a lot of them?

On Mon, May 20, 2024 at 9:41 AM Connor ***@***.***> wrote:

> Description
>
> Only the most recent evaluation is able to be loaded in the WebViewer.
> Each time a new eval is run, any previous evals that have been run do not
> load in the 'Evals' tab of the WebViewer.
> Steps to reproduce:
>
>    1. Run an eval
>    2. View it in the 'Evals' tab of the WebViewer
>    3. Run a second eval
>    4. Attempt to load the eval from step 1 by selecting it in the
>    'Progress' tab
>
> Screenshot.2024-05-20.at.12.33.52.PM.png (view on web)
> <https://github.com/promptfoo/promptfoo/assets/95300041/13cc520c-e8af-45bd-8904-3b4730ab1079> Additional
> Context
>
> Google Chrome (Version 124.0.6367.91 (Official Build) (arm64)) on an M1
> Macbook Pro.
>
> Error in console of Chrome:
>
> Uncaught (in promise) DOMException: Failed to execute 'setItem' on
> 'Storage': Setting the value of 'ResultsViewStorage' exceeded the quota.
>
> I believe this issue originates from the code within
> web/nextui/src/app/eval/store.ts:
>
> export const useStore = create<TableState>()(
>   persist(
>     (set, get) => ({
>       evalId: null,
>       setEvalId: (evalId: string) => set(() => ({ evalId })),
>       ........
>     }),
>     {
>       name: 'ResultsViewStorage',
>       storage: createJSONStorage(() => localStorage),
>     },
>   ),);
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/802>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJSZU67O2H46G3FY5PLZDIRUXAVCNFSM6AAAAABIAA7XOWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGMYDMMZWGQZTQOA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


I am running 130 evals over 3 different providers to compare 3 different OpenAI models.  Each eval contains a fairly large context window.  To answer your question more directly - yes, I am running large evals and I am running a lot of them.  Would something like IndexedDB be more suited or is the WebViewer not meant to support this large of a test suite?  I would be willing to help fix if need be.

I would love for the viewer to support this use case... IndexedDB is
probably a solution that makes sense here!

On Mon, May 20, 2024, 11:22 AM Connor ***@***.***> wrote:

> I am running 130 evals over 3 different providers to compare 3 different
> OpenAI models. Each eval contains a fairly large context window. To answer
> your question more directly - yes, I am running large evals and I am
> running a lot of them. Would something like IndexedDB be more suited or is
> the WebViewer not meant to support this large of a test suite? I would be
> willing to help fix if need be.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/802#issuecomment-2120972241>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJSQHW7GW5O2LYQXJ5LZDI5OPAVCNFSM6AAAAABIAA7XOWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQHE3TEMRUGE>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


I would be happy to submit a PR for this.  I quickly tried getting everything running locally this morning but ran out of time.  It looks like the development documentation is a little out of date.  Would you mind updating?

I went ahead and included this in 0.63.0 (I think the dev documentation has been updated as well)