Repository: UKGovernmentBEIS/inspect_ai
Issue: `inspect view` chokes on large logs ("out of memory" in console)
Root cause: Resource Mishandling

If you download these files and assemble them with `cat 2025-02-24T04-06-18+00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.part* | tar -xzvf -` and view with `inspect view`, I see
[2025-02-24T04-06-18+00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partaa.tar.gz](https://github.com/user-attachments/files/18936276/2025-02-24T04-06-18%2B00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partaa.tar.gz)

[2025-02-24T04-06-18+00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partab.tar.gz](https://github.com/user-attachments/files/18936278/2025-02-24T04-06-18%2B00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partab.tar.gz)

[2025-02-24T04-06-18+00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partac.tar.gz](https://github.com/user-attachments/files/18936279/2025-02-24T04-06-18%2B00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partac.tar.gz)

[2025-02-24T04-06-18+00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partad.tar.gz](https://github.com/user-attachments/files/18936281/2025-02-24T04-06-18%2B00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partad.tar.gz)

[2025-02-24T04-06-18+00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partae.tar.gz](https://github.com/user-attachments/files/18936286/2025-02-24T04-06-18%2B00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partae.tar.gz)

[2025-02-24T04-06-18+00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partaf.tar.gz](https://github.com/user-attachments/files/18936287/2025-02-24T04-06-18%2B00-00_coq-to-lean_S8wjYGvVdM3nRuCRSGXuw6.eval.tar.gz.partaf.tar.gz)

![Image](https://github.com/user-attachments/assets/124fb8ba-774b-42ae-a7ec-e781833371f2)

![Image](https://github.com/user-attachments/assets/93ab844c-f63f-4eb5-8a0e-7ce40d6e60bd)

It would be nice if inspect could handle large logs better (though perhaps 123M .eval file which decompresses to a 968M .json file is a bit much to ask --- but even in this case it would be nice to have a program that can process these .eval files into smaller eval files with some parts elided, which can be viewed)

Sorry you ran into this issue- the viewer is generally pretty scalable and handles files of this size (and even larger) without too much problem. In this case, the contents of this eval file are extremely unusual. This log file contains an error during the run (before any samples were logged) and the header information that is summarizing the evaluation run is itself more than 1GB uncompressed. From poking around, it appears that the error itself logged the contents of a large binary file in the traceback, which is the root of the issue.

We currently definitely depend upon loading the whole header into memory in order to render the viewer, which is the root of the issue. I'm not sure I can think of a good way detect a condition like this which explodes the size of the header (and where we would 'draw the line'). Is it expected for you to have such large traceback messages? 



I do not want to have such large tracebacks, but it's not trivial to avoid them, and having the viewer crash makes it quite hard to debug what's going wrong in the trace.  Can the logging infrastructure do more of the preprocessing and save only the minimal information needed to render to the header in the header json, and store the rest in the log for this particular sample?  (It seems fine to split the json files for "successful samples" and "unsuccessful samples" or something.).

We do store only minimal information in the header (splitting sample data into per sample files) - the issue is that error message + traceback is a part of the header (so that we can display the eval failure). In the case of this log file, no samples were completed due to the error afaict. In any event, we're going to have a challenge anywhere we put a 1GB traceback in the log file, as it will be contained in a JSON object that the viewer needs to load.

We could just truncate the traceback, but I'm thrilled with this approach since we don't actually know a lot of context (other than that it is long). 

If I'm recalling correctly, interrupted samples (often? always?) are lacking information about messages that were sent before interruption.  Would it be possible to have a separate json file of incomplete samples, and display the traceback as part of the body rather than the header, with only an abbreviated summary of the error as part of the header?

As far as long tracebacks go, if the traceback exceeds a certain length, could be be stored off to the side, to be loaded only when the user says "I want to see the details of this traceback in particular"?

I'm wondering what the source of the 500MB traceback is. I know you had previously asked about showing locals in the stack frame (https://github.com/UKGovernmentBEIS/inspect_ai/issues/673). Are locals somehow enabled and that is where the data is coming from. If not, I'm wondering how a stack frame could ever get that large.

I just checked and rich traceback defaults to truncating locals at 80 characters, so it doesn't seem like that's the problem even if show locals is somehow enabled. Any insight you can provide on how a 500MB stack frame could come to be would be helpful.

If there are no locals in play and/or they are being truncated to 80 characters as per the default then I'd be inclined to truncate the traceback at some reasonable but still ample size (e.g. 1MB). Since tracebacks are most recent call last we could truncate from the end.

I've added a fix which truncates the traceback after 1MB. In this case, where an object is being used as the key to a dict, the contents of the dict are printed, and in this case that dict is very large. The truncation will prevent breakage like this in the viewer.

If you need to see the complete traceback for a case like this, you could catch KeyError and write the contents to a file yourself.

Regarding the source: I think it might be that I'm hitting an assertion error that is dumping the repr of a very large object