Difference of FICLONE vs FICLONERANGE vs copy_file_range (for copy-on-write support)
Tags: linux
Published: 2021-09-02 06:43:31

This was original a [StackOverflow question by me](https://stackoverflow.com/questions/65505765/difference-of-ficlone-vs-ficlonerange-vs-copy-file-range-for-copy-on-write-supp), although it was maybe way too specific for StackOverflow, and too long. I then later also answered it myself. So the answer comes below.
I feel this might be interesting for some developers, so this is why I post it here.

---

I wonder about an efficient way to copy files (on Linux, on a FS which supports [**copy-on-write (COW)**](https://en.wikipedia.org/wiki/Copy-on-write)).
Specifically, I want that my implementation uses copy-on-write if possible, but otherwise falls back to other efficient variants. Specifically, I also care about **server-side copy** (supported by [SMB](https://wiki.samba.org/index.php/Server-Side_Copy), [NFS](https://linux-nfs.org/wiki/index.php/Server_Side_Copy) and [others](https://serverfault.com/questions/566708/which-file-transfer-protocols-support-server-side-copying)), and also [**zero-copy**](https://en.wikipedia.org/wiki/Zero-copy) (i.e. bypassing the CPU or memory if possible).

(This question is not really specific to any programming language. It could be C or C++, but also any other like Python, Go or whatever has bindings to the OS syscalls, or has any way to do a syscall. If this is confusing to you, just answer in C.)

It looks like [`ioctl_ficlonerange`, `ioctl_ficlone` (i.e. `ioctl` with `FICLONE` or `FICLONERANGE`)](https://man7.org/linux/man-pages/man2/ioctl_ficlonerange.2.html) support copy-on-write (COW). Specifically `FICLONE` is used by GNU `cp` ([here](https://github.com/coreutils/coreutils/blob/00ea4bacf6063ccc125209d5186f8f2382c6f0d4/src/copy.c#L408), via `--reflink`).

Then there is also [`copy_file_range`](https://man7.org/linux/man-pages/man2/copy_file_range.2.html), which also seems to support COW, and server-side-copy.
([LWN about copy_file_range](https://lwn.net/Articles/659523/).)

It sounds as if `copy_file_range` is more generic (e.g. it supports server-side-copy; not sure if that is supported by `FICLONE`).

However, `copy_file_range` seems to have some issues. 
E.g. [here](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24399), Paul Eggert comments:

> [copy_file_range]'s man page 
> says it uses a size_t (not off_t) to count the number of bytes to be 
> copied, which is a strange choice for a file-copying API.

Are there situations where `FICLONE` would work better/different than `copy_file_range`?

Are there situations where `FICLONE` would work better/different than `FICLONERANGE`?

Specifically, assuming the underlying FS supports this, and assume you want to copy a file. I ask about the support of these functions for the functionality of:

* [Copy-on-write](https://en.wikipedia.org/wiki/Copy-on-write) support
* Server-side copy support
* [Zero-copy](https://en.wikipedia.org/wiki/Zero-copy) support

Are they (`FICLONE`, `FICLONERANGE`, `copy_file_range`) always performing exactly the same operation? (Assuming the underlying FS supports copy-on-write, and/or server-side copy.)

Or are there situations where it make sense to use `copy_file_range` instead of `FICLONE`? (E.g. COW only works with `copy_file_range` but not with `FICLONE`. Or the other way around. Or can this never happen?)

Or formulating the same question differently: Would `copy_file_range` always be fine, or are there situations where I would want to use `FICLONE` instead?

Why does GNU `cp` use `FICLONE` and not `copy_file_range`? (Is there a technical reason, or is this just historic?)

Related: GNU `cp` originally did not use `reflink` by default (see [comment by the GNU coreutils maintainer Pádraig Brady](https://unix.stackexchange.com/a/152639/2954)).
However, that was changed recently ([this commit](https://github.com/coreutils/coreutils/commit/25725f9d41735d176d73a757430739fb71c7d043), [bug report 24400](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24400)), i.e. COW behavior is the default now (if possible) (`--reflink=auto`).

Related [question about Python for COW support](https://stackoverflow.com/questions/65492317/copy-file-in-python-for-zfs-or-other-fs-with-cow-copy-on-write-support/).

Related [discussion about FICLONE vs copy_file_range by Python developers](https://bugs.python.org/issue37157). I.e. this seems to be a valid question, and it's not totally clear whether to use `FICLONE` or `copy_file_range`.

Related [Syncthing documentation about the choice of methods for copying data between files](https://docs.syncthing.net/advanced/folder-copyrangemethod.html), and
[Syncthing issue about `copy_file_range` and others for efficient file copying, e.g. with COW support](https://github.com/syncthing/syncthing/issues/4271).
It also suggests that it is not so clear that `FICLONE` would do the same as `copy_file_range`, so their solution is to just try all of them, and fallback to the next, in this order:
ioctl (with FICLONE), copy_file_range, sendfile, duplicate_extents, standard.

Related [issue by Go developers on the usage of `copy_file_range`](https://github.com/golang/go/issues/36817).
It sounds as if they agree that `copy_file_range` is always to be preferred over [`sendfile`](https://man7.org/linux/man-pages/man2/sendfile.2.html).

---

The answer:

See the [Linux vfs doc](https://github.com/torvalds/linux/blob/fcadab740480e0e0e9fa9bd272acd409884d431a/Documentation/filesystems/vfs.rst) about `copy_file_range`, `remap_file_range`, `FICLONERANGE`, `FICLONE` and `FIDEDUPERANGE`.

Then see
[`vfs_copy_file_range`](https://github.com/torvalds/linux/blob/0adc313c4f20639f7e235b8d6719d96a2024cf91/fs/read_write.c#L1506). This first tries to call `remap_file_range` if possible.

`FICLONE` calls `ioctl_file_clone` ([here](https://github.com/torvalds/linux/blob/fcadab740480e0e0e9fa9bd272acd409884d431a/fs/ioctl.c#L714)),
and `FICLONERANGE` calls `ioctl_file_clone_range`.
`ioctl_file_clone_range` calls the more generic `ioctl_file_clone` ([here](https://github.com/torvalds/linux/blob/fcadab740480e0e0e9fa9bd272acd409884d431a/fs/ioctl.c#L259)).
`ioctl_file_clone` calls `vfs_clone_file_range` ([here](https://github.com/torvalds/linux/blob/fcadab740480e0e0e9fa9bd272acd409884d431a/fs/ioctl.c#L239)).
`vfs_clone_file_range` calls `do_clone_file_range` and that calls `remap_file_range` ([here](https://github.com/torvalds/linux/blob/74f602dc96dd854c7b2034947798c1e2a6b84066/fs/remap_range.c#L406)).

I.e. that answers the question. `copy_file_range` is more generic, and anyway tries to call `remap_file_range` (i.e. the same as `FICLONE`/`FICLONERANGE`) first internally.

I think the `copy_file_range` syscall is slightly newer than `FICLONE` though, i.e. it might be possible that `copy_file_range` is not available in your kernel but `FICLONE` is.

In any case, if `copy_file_range` is available, it should be the best solution.

The order done by Syncthing (ioctl (with FICLONE), copy_file_range, sendfile, duplicate_extents, standard) makes sense.