Repository: rust
Issue: `Command::spawn` on a newly-written file can fail with ETXTBSY due to racing with itself on Unix
Labels: C-bug,T-libs,A-process

The following code:

```rs
fn main() {
    thread::spawn(|| loop {
        process::Command::new("/bin/true").status().unwrap();
    });

    for _ in 0..20 {
        let mut file = File::create("/tmp/executable.sh").unwrap();
        file.write_all(b"#!/bin/true\n").unwrap();

        let mode = 0b111_101_101;
        file.set_permissions(Permissions::from_mode(mode)).unwrap();

        drop(file);

        process::Command::new("/tmp/executable.sh")
            .status()
            .unwrap();
    }
}

use std::fs::File;
use std::fs::Permissions;
use std::io::Write as _;
use std::os::unix::prelude::PermissionsExt as _;
use std::process;
use std::thread;
```

Will almost immediately panic (on Linux) with:

```
$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s
     Running `target/debug/example`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 26, kind: ExecutableFileBusy, message: "Text file busy" }', src/main.rs:17:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
$ rustc --version --verbose
rustc 1.71.0 (8ede3aae2 2023-07-12)
binary: rustc
commit-hash: 8ede3aae28fe6e4d52b38157d7bfe0d3bceef225
commit-date: 2023-07-12
host: x86_64-unknown-linux-gnu
release: 1.71.0
LLVM version: 16.0.5
```

(verison info is included for completeness, but this error will occur on all existing versions of Rust).

This is due to the following sequence of events occuring:
1. The main thread opens the file descriptor for writing into `/tmp/executable.sh`.
2. The spawned thread calls `fork(2)`, inheriting the file descriptor that has write access to `/tmp/executable.sh`.
3. The main thread closes its copy of the file descriptor — but note that it is **not** closed in the child process.
4. The main thread then attempts to execute the file. However, because there exists a process with a file descriptor open that has write access to the file, it fails with ETXTBSY and the process panics.

After this, the spawned process will run `execve` and the presence of `O_CLOEXEC` on the file descriptor will cause it to be closed as we desire. However, that is too late: the damage has already been done.

## An “ideal” solution: `O_CLOFORK`

The ideal solution here is for the kernel to support a file-opening flag analogous to `O_CLOEXEC`, `O_CLOFORK`, which causes the file descriptor to be closed when `fork(2)` is called, preventing it from beïng leaked to child processes. The feature has been [accepted into POSIX](https://www.austingroupbugs.net/view.php?id=1318) and [allegedly](https://lwn.net/Articles/785962/) exists on AIX, *BSD, Solaris and macOS. However, it is not in Linux despite multiple suggestions:

- [A patch from 2011](https://lkml.org/lkml/2011/5/7/5): Simply fizzled out
- [An email on LKML from 2017](https://lkml.org/lkml/2017/10/18/604): No responses
- [A series of 4 patches in April 2020](https://lkml.org/lkml/2020/4/20/113), [further pursued in May](https://lkml.org/lkml/2020/5/15/821): Rejected on the basis of performance concerns

## Userspace Solutions

### Wrapping calls to `Command::spawn` in a mutex

One simple userspace solution to this is to wrap all calls to `Command::spawn` in a mutex. This prevents the bug by ensuring that calls to `Command::spawn` cannot occur in between another process calling `spawn` and `execve`, as `Command::spawn` only returns after the cloexec event has occurred. However, this can be quite invasive to an existing codebase, and implementing this in `Command::spawn` itself may lose performance.

A potential alternative is to build in to `Command::spawn` a special synchronization primitive that allows for a parallelism-supporting fast-path, only falling back to the slow path when an `ETXTBSY` error is encountered.

### The `flock`ing algorithm

An alternative solution that only affects the parts of the codebase that write to a file and then execute it is to follow the following algorithm:

1. Open the file with write permissions and write the contents
2. Lock the file with exclusive access (note that because of how `flock(2)` works, the resulting lock guard attached to the file descriptor is shared with the file descriptor in any child process; this means that there is no race condition even if the fork happens between the previous step and this one)
3. Close the file
4. Reopen the file with read permissions
5. Lock the file with exclusive or shared access — if any existing writeäble file descriptor is currently open, this will wait for it to close
6. Close the file again
7. Call `Command::spawn`. A fork may still hold an open file descriptor to the file, but it is guaranteed to be read-only and so cannot cause `ETXTBSY` errors

<details><summary>Implementation of the `flock` algorithm</summary>

Note that for simplicity error handling is omitted; in a real implementation you’d check for errors after calling `flock`.

```rs
fn main() {
    thread::spawn(|| loop {
        process::Command::new("/bin/true").status().unwrap();
    });

    for _ in 0..20 {
        let mut file = File::create("/tmp/executable.sh").unwrap();
        file.write_all(b"#!/bin/true\n").unwrap();

        let mode = 0b111_101_101;
        file.set_permissions(Permissions::from_mode(mode)).unwrap();

        std::thread::sleep(std::time::Duration::from_micros(2));
        unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX) };

        drop(file);

        let file = File::open("/tmp/executable.sh").unwrap();
        unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_SH) };
        drop(file);

        process::Command::new("/tmp/executable.sh")
            .status()
            .unwrap();
    }
}

use std::fs::File;
use std::fs::Permissions;
use std::io::Write as _;
use std::os::fd::AsRawFd as _;
use std::os::unix::prelude::PermissionsExt as _;
use std::process;
use std::thread;
```
</details>

Given that this algorithm will be needed in relatively rare scenarios, it is unlikely we would want to add it to the standard library anywhere. However, it may be useful to add a note about this in the documentation.

### Waiting for an anonymous pipe to close

A somewhat simplified version of the above algorithm involves:
1. Opening an anonymous `O_CLOEXEC` pipe before opening the file.
2. Closing the writer end of the pipe between closing the file and running `Command::spawn`.
3. Waiting for an EOF on the reader end of the pipe directly after that.

Since the pipe writer file descriptor’s lifetime definitely exceeds the lifetime of the file’s file descriptor, any `fork(2)` call that inherits the file’s file descriptor will also inherit the pipe writer. Thus, by refusing to continue until the pipe writer has been closed we also refuse to continue until the file’s file descriptor has been closed.

Note that this trick works based on the assumption that the pipe EOF delivery will happen-after all the CLOEXECs have finished taking effect. I am uncertain of whether this is a guarantee made by Linux.

<details><summary>Implementation of the anonymous pipe trick</summary>

```rs
fn main() {
    thread::spawn(|| loop {
        process::Command::new("/bin/true").status().unwrap();
    });

    for _ in 0..100 {
        let [reader, writer] = pipe().unwrap();
        let mut reader = File::from(reader);

        let mut file = File::create("/tmp/executable.sh").unwrap();
        file.write_all(b"#!/bin/true\n").unwrap();

        let mode = 0b111_101_101;
        file.set_permissions(Permissions::from_mode(mode)).unwrap();

        drop(file);
        drop(writer);

        loop {
            match reader.read(&mut [0]) {
                Ok(0) => break,
                Err(e) if e.kind() == io::ErrorKind::Interrupted => {}
                _ => panic!(),
            }
        }

        process::Command::new("/tmp/executable.sh")
            .status()
            .unwrap();
    }
}

fn pipe() -> io::Result<[OwnedFd; 2]> {
    let mut fds = [0; 2];
    if unsafe { libc::pipe2(fds.as_mut_ptr(), libc::O_CLOEXEC) } != 0 {
        return Err(io::Error::last_os_error());
    }
    Ok(fds.map(|fd| unsafe { OwnedFd::from_raw_fd(fd) }))
}

use std::fs::File;
use std::fs::Permissions;
use std::io;
use std::io::Write as _;
use std::os::fd::FromRawFd as _;
use std::os::fd::OwnedFd;
use std::os::unix::prelude::PermissionsExt as _;
use std::process;
use std::thread;
use std::io::Read as _;
```
</details>

### Sleeping or spinlooping on `ETXTBSY`

One possible, but hacky, solution is to simply sleep when `Command::spawn` encounters an `ETXTBSY` error. The sleeping version was [used by the `go` command from 2012](https://github.com/golang/go/commit/a4b2c5efbc259c7d23159d304f9cb4266cd64643) and [removed after it was no longer necessary in 2017](https://go-review.googlesource.com/c/go/+/71571). [The spinning version has been used elsewhere in Go since 2022](https://go-review.googlesource.com/c/go/+/458016).

### Performing the write inside a fork

This is an easy and safe solution more oriënted for usage by applications. Simply perform all the writing in a child process: as long as the main process never opens a file descriptor to the executable file, nothing can go wrong.

```rs
let mut writer = process::Command::new("sh")
    .args(["-c", "cat >/tmp/executable.sh && chmod +x /tmp/executable.sh"])
    .stdin(process::Stdio::piped())
    .spawn()?;
writer.stdin.take().unwrap().write_all(b"#!/bin/sh\n")?;
writer.wait()?.exit_ok()?;
```

## Other languages with this problem

- Java (2014): [**(process) Executing recently modified file results in ETXTBSY**](https://bugs.openjdk.org/browse/JDK-8068370) — remains unresolved with no activity in 7 years
- Go (2017): https://github.com/golang/go/issues/22315 — issue remains open