Repository: explodinggradients/ragas
Issue: periodically save current progress and restart from last save
Root cause: Unimplemented Feature Gap

Both the evaluation and testset generation are computationally and monetarily expensive.

It is very painful to experience a crash after several expensive hours of evaluation or generation, only to have nothing to show for it.  If a project's budget is spent on a long run, and it crashes near the end, it may be impossible for the project to conclude without shifting to a different platform.

Could you please add optional, configurable support (ideally, enabled by default) to periodically save current progress to disk, and resume progress from saved state?  After using this tool for a while, several objects seem like they could be saved to disk periodically and restored, if they exist:

- Embedding vectors
- Vectorized document stores
- Generated questions
- Evaluated questions

@trevorbowen I'm sorry for the hard time and this is something we will improve

but today you do have a few options `raise_exceptions=False` which is the default should keep catch all exceptions while running evaluations or testset generation - if you find any please do share here (for v0.2+, if using v0.1 do migrate - https://docs.ragas.io/en/stable/howtos/migrations/migrate_from_v01_to_v02/)

You can also save testeset generation too. The process is divided into 2 steps

1. generate the transforms - which will produce a knowledge graph you can save
2. Do the generations

so the 3 points you mentioned that needs to be stored can already be stored - we will improve the docs so that it's more apparent.


if you have some specific examples do share them too, we would love to build out more options to make it easier to run these exps for you. I hope this helps 🙂 

I am trying to generate a test-set of 200 questions, covering about 100 documents.  Currently, this will take 3-days because of rate limitations with OpenAI.  If anything glitches during that time period, I have zero questions and must start all over.

I am asking for the generation process to be enhanced to periodically dump its state, so it can resume if I lose power, internet, unexpected program error, etc.

I don't think this is possible today without enhancements to the code.

@trevorbowen actually there are ways we can work around this with the existing tools. I would love to go over them with you, to see if it helps

any chance me might be able to get on a call?

Sure, @jjmachan.  I will email you.

awesome 🙂, if we figure it out we can add it back into the docs too

@jjmachan, email sent.  If you don't receive it, please let me know. ... Thanks!

we now have a guide for a more broken down view of testset generation that allows you to save: https://docs.ragas.io/en/latest/getstarted/rag_testset_generation/#a-deeper-look

@trevorbowen I did get your mail but we haven't scheduled a call right?

@jjmachan, sorry, I have been so busy, I haven't had time for a call.  What would I like to see is `diskcache` placed in front of every llm and embedding call, so it should be added to the lowest base classes, although new optional parameters might need to added to the inheriting classes.  This would save a lot of money and time.  There's a similar project that has a similar request and example here:

- https://github.com/confident-ai/deepeval/issues/775
- https://pypi.org/project/diskcache/
- https://grantjenks.com/docs/diskcache/tutorial.html

hey @trevorbowen caching is something we should do - would get some easy wins - will get this sorted for you next week 🙂 

@trevorbowen we have merged something in 🙂 - do check it out if you get the chance

It seems the issue was answered, closing this now.