I agree that it’d be a cool hack to be able to somehow reuse all the existing BitTorrent software and network to sync up arbitrary pieces of files that people just-to-happen to have. I’ve been thinking about this for a while, but I haven’t come up with any good solutions.

The main problems are that:

1. torrents have different “piece sizes” (i.e. the torrent creator chooses a power-of-two size for the partitions of the torrent’s data that get their hashes stored in the torrent file); and

2. multi-file torrents produce “pieces” that are an arbitrary split of the concatenation of their files, as if they were actually a torrent of an archive file—just a virtual one synthesized on the spot by the torrent client, as if it was using FUSE to translate the filesystem it sees into a packfile to share.

The “solution” to problem #1, if you can call it that, is to walk your files and generate (log N) different torrents per file, for each piece size. Then—at least for single-file torrents—you could in theory build a tracker (and client-API-hooked search engine) that exposes “virtual torrents” for every possible combination of pieces (when fed this sort of... torrent-data overlapped Merkle tree.)

I don’t think there’s really a more clever solution to problem #2 than the one torrents already use, though. It really does make sense to pack files together into single large pieces, when files are small and numerous, because the metadata of sharing a single file on the network (or, to a lesser extent, of storing a single file separately on disk) has overhead, and so people don’t tend to like seeding tons of tiny files, preferring to instead seed archives of them. (Or, even worse, when a file is seen by most as useless except in the context of being a “resource file” of a particular application, people will only ever bother to seed it in the form of the installation archive of said application.)

To find a particular tiny file on the network, then, you need an index mapping the metadata (and/or hash) of the file you want, to the metadata and hash of an archive/packfile that contains it, and which is tracked by the network. Which is... the thing torrent search engines—usually built on the backs of torrent trackers—give us: the ability to plug in a filename, and find torrents that list that name in their manifests.

(IPFS doesn’t solve problem #2 either, if you’re curious. It just keeps everything as individual file hashes, and then allows you to retrieve directory manifests by their hashes—but you can’t predict a directory manifest hash from a file hash, in order to discover the directories that “make use of” a given file, and might therefore have a more-well-pinned packfile equivalent representation that includes that file.)

I think we could solve #2 if we had some sort of system of hashing—not necessarily cryptographic hashing—that enabled us to ask, in O(1) time, “given that the hash of a packfile of bytes of size Xsz is Xh, does the packfile contain a file whose size is Ysz≤Xsz but otherwise arbitrary, and whose hash is Yh; and if so, at what range within the packfile can we find the file?” (The “but otherwise arbitrary” part is important; we can’t just pre-chunk up the packfile and hash all possible chunks of it, like we can with #1, since the files in it might be in any arbitrary positions with any arbitrary lengths, potentially even straddling a piece boundary.)

If we had such a hash algorithm, then any file hash of this type that a tracker received in a request could be used as, essentially, a zero-knowledge proof of what packfiles contain that file (since you can just iterate all the packfiles you have and check whether they contain it, and use that to build an inverted index from segment hashes to packfile hashes.) But I don’t think a hashing scheme like this currently exists. (At least, in this use-case... YouTube’s ContentID algorithm is kind of an implementation of this for audio fingerprinting. Or maybe not; maybe it’s just pre-generating comparator fingerprints for every possible slice of each claimed audio track!)