Repository: UKGovernmentBEIS/inspect_ai
Issue: VSCode Inspect View doesnt load logs

No log ever renders, the inspect view panel just spins forever. Running inspect view in VSCode or CLI still works and the log can be viewed in an external web browser. I have tried the troubleshooting steps in the documentation, but did not resolve the problem. The output in VSCode suggests it should be working.

Hi I'm sorry that you've run into this! 

Could you share what version of the inspect (you can type `inspect --version` in the terminal) you're using as well as confirm that you're using the latest version of the vscode extension (it pretty much auto-updates, so I doubt this is it, but just wanting to check it off as a possibility).

No worries, I only recently installed Inspect using the pip package, it currently shows: 0.3.17
The VSCode extension, I think is 0.3.24 -- but as you said, this should be auto updating I suppose.

Should they align? and/or which versions are most recent?

These are pretty much the most recent versions (and while they once aligned, they have since diverged and the VSCode extension should generally be fine with whatever version of Inspect you have installed). A couple more things to try, if you wouldn't mind:

- See if this also reproduces when using `inspect view` directly. There are slight but important differences in how the extension loads log files, so it might only occur in the vscode extension.

- Previously, I fixed an issue related to special character in folder paths- is there anything unusual about the log directory that you are using (e.g. an unusual name)?

- Try to determine if there is a specific log file which reproduces this (I'm guessing that the issue is caused by a log file perhaps throwing an error at an unexpected place for us while attempting to load). If you have a set of log files in a directory that is causing this, you could try to search for the file causing the error (by binary searching / removing logs) or could also try with an empty log dir to see if a fresh eval works (confirming it is caused by an older log file). 

inspect view works from vscode and other cli, I can view the logs in an external browser no problem.

It seems like all logs are not working in VSCode, I have cleared the log directories several times while troubleshooting.

Im on windows (gasp), but only have underscores present in some of the directories in the path. I will try creating a project with a very short path and no underscores and report back.

Running from C:\test, successful log creation, not loading in VSCode extension. Works in external browser with inspect view.

OK thanks of the diligence here! Let me check on my local Windows machine (I'm working daily on a Mac) - it's possible that there is a simple path issue related to the path separators... (I hope I didn't make that mistake!)

For what it's worth, I started trying out Inspect on my work mac, and had the same issue- but have since moved everything onto my personal workstation. Here is the output, in VSCode, which seems like it should work:

2024-07-12 10:58:10.980 [info] Inspect view path: C:\ProgramData\anaconda3\Lib\site-packages\inspect_ai\_view\www
2024-07-12 10:58:13.072 [info] Watching for evaluation logs
2024-07-12 10:58:24.415 [info] New log: logs\2024-07-12T10-58-22-04-00_hellaswag_LV8Ja7xWwotQQjzN5P8SZY.json

Interesting - I guess I still haven't quite pinpointed the issue. What I'm seeing would explain the issue on a Windows machine - there is a comparison operation we're doing to determine whether the requested file is available on that appears to be happening with inconsistent path separators on windows. But on Mac this should not be an issue.

I've rewritten some of the file loading to improve support for large evaluation files in the upcoming release and that does appear to have fixed the issue in my testing. You could try installing from this repo (which is newer than the PyPi release) to see if that fixes the issue `pip install git+https://github.com/UKGovernmentBEIS/inspect_ai` though please be warned this will emit log files with a V2 format to support new features, which will mean that old versions of Inspect will fail to launch the viewer since they can't render these V2 log files.

We're planning on releasing this to PyPi in the coming days, so you may want to wait for that. I am going to see if I can further pinpoint this as well, to be sure I understand why the changes in the upcoming release fix the issue...

Appreciate the reply, I will wait on the new release and stick to inspect view in an external browser for now. Feel free to close the issue :)

Small update, I gave in installed the developer build and you are correct, the VSCode viewer now works.

Closing as this fix is now published to PyPI