Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature Request] Support virtual machines as a sandbox option
Root cause: Unimplemented Feature Gap

### Executive summary
1) Many threat model relevant tasks require a GUI OS to complete
2) Benchmarks for desktop agents already exist
3) Desktop agents are a realistic threat model
4) Desktop agents require a virtual machine (ie. VirtualBox)

Quick example of desktop agents: see the [video + graphical abstract of OSWorld](https://os-world.github.io/)

### Main arguments

**1) Many relevant tasks require a GUI OS**
- You can't pass many forms of CAPTCHA without a GUI
- Many websites block headless browsing eg. I think you can't create a Facebook account headless
- Tasks like graphics design require a GUI (relevant for an building online storefront and certain freelancing jobs)
- Many desktop apps do not have a CLI version

**2) Benchmarks for desktop agents already exist**  
- [OSWorld](https://os-world.github.io/) benchmarked GPT, Gemini and Claude models against 369 open-ended computer tasks:
  - The best model scaffold tested achieved 12.24% completion on the benchmark
  - Example of a successfully completed task instruction: "I downloaded an episode of Friends to practice listening, but I don't know how to remove the subtitles. Please
help me remove the subtitles from the video and export it as "subtitles.srt" and store it in the same directory as the video." Taken from Fig 9 of the [paper](https://arxiv.org/pdf/2404.07972)
- [CRAB](https://crab.camel-ai.org/) benchmarked GPT, Gemini and Claude models against 100 cross-environments tasks (Ubuntu desktop and Android smartphone)
  - The best model scaffold tested achieved 14% completion on the benchmark
  - Example of a successfully complete task instruction: " Please open the X app on my phone, search for CAMEL-AI.org, check the latest post, summarize them, and then send the summary to Tianqi Xu on Slack from my PC." Taken from a [demo video](https://crab.camel-ai.org/static/videos/demo4_X_to_slack.mp4) on the CRAB website

**3) Desktop agents are a realistic threat model**
- Desktop agents to will plausibly have faster adoption as the median consumer will find desktop agents to be much more intuitive and useful than say pure CLI agents
- The "observation space" of a desktop agent is a superset of a pure CLI or browser agent
- Big companies are working on desktop agents (or similar):
  - Salesforce Research was involved with OSWorld
  - Microsoft has [UFO: A UI-Focused Agent for Windows OS Interaction](https://github.com/microsoft/UFO)

**4) Desktop agents require a virtual machine**
- [OSWorld's codebase](https://github.com/xlang-ai/OSWorld/tree/main/desktop_env/providers) currently supports local VMs via VirtualBox and VMWare, and cloud VMs via AWS, GCP and Azure
- [CRAB's codebase](https://github.com/camel-ai/crab/tree/main/crab-benchmark-v0) currently supports local VMs via Linux KVM, and cloud VMs via GCP. The Android environment is provided by Andrdoid Studio

### Next steps
If there's sufficient agreement behind this feature, I'm happy to write up more technical details about how OSWorld implemented their desktop environments. I'm too capacity constrained at the moment to work on this myself, and am still very unfamiliar with Inspect on the development side.

Closed in favor of https://github.com/UKGovernmentBEIS/inspect_ai/issues/290