Repository: stanford-crfm/helm
Issue: Error in running code on Azure ml
Root cause: Environment Incompatibility

Command:

helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10

Error
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rash-new-ci/code/Users/Rashmi.Soni/HELM_LLM/helm-venv/lib/python3.8/site-packages/sqlitedict.py", line 414, in run
    cursor.execute(req, arg)
sqlite3.OperationalError: database is locked

This can happens if the Sqlite request cache is used with high concurrency. Could you try re-running with `--num-threads 1` and see if this fixes it for you? Also, are you running multiple processes of `helm-run` at once? 

Hi Yifan, I tried to run helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10 --num-threads 1, still getting same issue. I am only running 1 process of helm-run at a time.

Error:
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rash-new-ci/code/Users/Rashmi.Soni/HELM_LLM/helm-venv/lib/python3.8/site-packages/sqlitedict.py", line 414, in run
cursor.execute(req, arg)
sqlite3.OperationalError: database is locked

I'm facing the same exact error

@yifanmai pls help

Could you try moving or deleting your SQLite files? They are located at `prod_env/cache/*.sqlite`

@sahil18930691 Regarding your pre-edit comment - if you can identify which commit needs to be reverted, let me know and I can take a look.

The same error persists even after deleting the sqlite file! It's regenerated with the new run.

@impact-rm Hi, where are you trying to run the code ?....I am trying it on Azure ml 

@yifanmai still facing same issue...

It is getting timeout I think.........I have tried to increase the time but still getting same error.
In file sqlitedict.py, line 414....               
 try:
cursor.execute(req, arg) -------- no response after this, it says database is locked.




@sahil18930691  I'm trying to run it on Azure ml too! I also tried to increase the time inside the sqlite shell to 10 minutes or so, but as soon as I close the shell  the timeout is reset again to 0.... have you noticed that?

@impact-rm okay, for me it is running till the time I set but after that it gives the same issue "database is locked".

Would either of you have a non Azure ML platform that you could try this out on? Seems like this is an Azure ML specific issue.

My suspicion is that the `./prod_env/` folder in Azure ML is some sort of network disk, which SQLite does not work well on.

As a temporary workaround, I can add an flag to disable the SQLite usage for now.

For me Azure ML platform is the only available platform I can work on currently! Adding this flag would be quite helpful, so could you please notify us when it's available?! thanks

@yifanmai Yes, it is only happening in AML environment. Problem is that we have all our base code in Azure, so need to run it on Azure only.

Closing due to staleness, but please feel free to reopen if you have further questions.

We now have a `--disable-cache` flag that will disable use of SQLite. Please try this out and see if it works for you.

Got another report of this.

```
Traceback (most recent call last):
  File "/path/to/bin/helm-run", line 8, in <module>
    sys.exit(main())
  File "/path/to/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/path/to/site-packages/helm/benchmark/run.py", line 341, in main
    run_benchmarking(
  File "/path/to/site-packages/helm/benchmark/run.py", line 117, in run_benchmarking
    runner: Runner = runner_cls(
  File "/path/to/site-packages/helm/benchmark/runner.py", line 156, in __init__
    self.executor = Executor(execution_spec)
  File "/path/to/site-packages/helm/benchmark/executor.py", line 84, in __init__
    self.service = ServerService(
  File "/path/to/site-packages/helm/proxy/services/server_service.py", line 71, in __init__
    self.accounts = Accounts(accounts_path, root_mode=root_mode)
  File "/path/to/site-packages/helm/proxy/accounts.py", line 129, in __init__
    with SqliteDict(self.path) as cache:
  File "/path/to/site-packages/sqlitedict.py", line 230, in __init__
    self.conn.commit()
  File "/path/to/site-packages/sqlitedict.py", line 672, in commit
    self.select_one(_REQUEST_COMMIT)
  File "/path/to/site-packages/sqlitedict.py", line 662, in select_one
    return next(iter(self.select(req, arg)))
  File "/path/to/site-packages/sqlitedict.py", line 654, in select
    self.check_raise_error()
  File "/path/to/site-packages/sqlitedict.py", line 606, in check_raise_error
    reraise(e_type, e_value, e_tb)
  File "/path/to/site-packages/sqlitedict.py", line 47, in reraise
    raise value
```

Apparently the issue is that `helm-run` attepmts to open the SQLite accounts file event though it does not need to.