Here are my thoughts: I managed to scrape through level 4 so saw them all - this CTF was fun and educational. Level 4 in particular felt like a real education (for me at least), and a great way to learn. Unlike many others I was fond of level 0 because it brought up questions that even beginner programmers should address - how to organise your data so that you can search through it easily, which was in some ways a preparation for the other levels. For the more expert than me it probably felt tedious or inconsequential, but it was quick to solve, and I thought inviting for beginners and open ended.

Lessons learned for me

Distributed consensus was really interesting; I'd never heard of it and enjoyed reading about Paxos and Raft, even if I don't fully understand them yet. Distributed searches can actually be faster than searches in one process - I didn't believe this but as resources were limited on the server it was true in some cases (depends how fast you can make your search, mine wasn't as fast as Henrik's :).

Big scala projects take an aaage to compile - is it normally this bad in real projects? Anyone care to comment? The remote was pretty slow, so that might be why.

Golang needs package versioning! On level 4 there was a versioning bug with differing versions of the go-raft library being fetched locally and on the server which led to problems setting the heartbeat timeout of raft. That was confusing and annoying.

A few notes for the CTF authors

The dramatic variation between resources on remote and locally was a problem in tests - I felt like that needed to be adjusted for as the local test results were meaningless compared to the server. This particularly made level 3 and 4 difficult.

Code examples are great to get you started, but it'd be nice if there was also a clearly specified API for each one which you were required to conform to, to make it easier to use another language, and if the build scripts simply launched one binary at all times - level 3 had two different styles of starting the servers for remote and local for example, that was frustrating.

The varying test cases were sometimes frustrating, and were too varied and randomly chosen. I completely agree with Henrik that you need small tests. Each run should have say 5 tests which can be run individually (for speed), or as a suite. The downloading of tests broke a few times for me and left me stranded and constantly re-downloading new test data/tests seemed a bit pointless. Proper logging output when tests fail is really important - level3 was particularly bad for this as the diffs were useless; I had to edit the test harness to get decent output.

The use of the JVM on level 3 was pretty painful for me - compile times on remote were in the minutes, and I couldn't even get it to compile locally even after installing another JVM, nor on remote without compiling all the sbt stuff - this one was probably my fault, and I just rewrote with golang in the end. Explicit language independence for all the problems would be great.

Unix sockets felt like an unnecessary distraction which caused pain on level 4, it would have been nice if it used IPs, but perhaps for reasons to do with Octopus that wasn't possible, and on a deeper level, being given broken code to fix just felt a bit wrong and distracting - having slow code you have to speed up is one thing, but having code which feels like it has been sabotaged is another.

Don't bother changing the deadline even if people whine - the contest isn't remotely fair anyway or an objective test, so it's better to have a deadline and stick to it.

Oh, and thanks again Stripe, it was a blast.

The original code and some more writups:

https://github.com/ctfs/write-ups/tree/master/stripe-ctf3

http://blog.joneisen.me/post/75008410654