Ok, wow, that's really irritating: what I had been staring at was the output from subsequent connections to different hosts (such as between the backends, or between the backends and the webhooks)...

# ./password_db_launcher 012345678901 127.0.0.1:3232 2>&1 | grep Received
[127.0.0.1:34651:1] Received payload: '{"password": "012345678901", "webhooks": []}'
[127.0.0.1:39664:1] Received payload: '{"password_chunk": "012"}'
[127.0.0.1:34991:1] Received payload: '{"password_chunk": "345"}'
[127.0.0.1:42667:1] Received payload: '{"password_chunk": "678"}'
[127.0.0.1:33434:1] Received payload: '{"password_chunk": "901"}'

...but it turns out that to a single host source port assignment is still both sequential and mediated by the same global sequential counter...

[127.0.0.1:35502:23] Received payload: '{"password": "112345678901", "webhooks": []}'
[127.0.0.1:38011:23] Received payload: '{"password_chunk": "112"}'
[127.0.0.1:35504:24] Received payload: '{"password": "112345678901", "webhooks": []}'
[127.0.0.1:38013:24] Received payload: '{"password_chunk": "112"}'

The algorithm for determining the ephemeral source port apparently has this behavior because it involves 1) entropy that is generated once at boot, 2) an md5sum of that entropy with the destination/source address and destination port, and 3) a hashtable insertion with linear probing.

As #1 is at boot, it doesn't affect the result in a way that really matters, but as #2 is based on other properties of the connection, we get behavior that looks random between backends and webhooks, but is in fact deterministic to the webhook.

What bugs me, however, is that hashtable: it is treated as a hashtable with open addressing and linear probing; the hash value that it starts with is the aforementioned md5sum: it really shouldn't end up allocating a sequential port to outgoing connections across hosts (it should only do so per host).

Except, I would assume to make it more efficient (as otherwise the hash that is going into it is going to keep returning the same thing if you are making lots of connections to one place), there is a "hint", and the hint is a static integer that is incremented for each port skipped.

So, you could seriously delete that one line of code from the kernel, then, and "solve" this at some minor performance expense. Alternatively, you could add the high bits of port_offset instead of 1 and get a quite reasonable middle ground.

Regardless, there: that's the story of why I've been operating under the assumption that Linux has not had this problem for a very long time, and exactly why I was wrong (down to the individual line of code that screwed me: inet_hashtables.c, line 521).

That said, it only somewhat changes what I was saying: I am curious if the level was designed and thought about on a device that had truly sequential port numbering. It definitely is incredibly obvious once you run it on, say, a Mac OS X laptop, that the port numbers are a side channel.

However, if you run it on Linux, you are going to have to sift through the sea of port numbers and notice that the connections to individual servers is incrementing even though, for the most part, the port numbers you are seeing look mostly like garbage.

(By the way: except for the massive difficulty cliff at level 8, this contest was amazingly well put together. I believe I said something similar already about the previous one, but I'll also say it again: that one was also amazing. I'm really curious if you have some kind of need to hire an entire army of security people, given the effort you are putting into this ;P.)