First of all, massive kudos for the Stack Exchange-like technical transparency. Definitely consider a massive upgrade album like http://blog.serverfault.com/2015/03/05/how-we-upgrade-a-live... and http://imgur.com/a/X1HoY!

GitLab is awesome. I'm really sad that in the past two or three months I've only found one GitLab link on HN to click. There really needs to be more. (I'm not sure if this is because I'm browsing in AEDT or if GitLab isn't used a lot on here.)

I wondered about how you guys might do advertising to get more mindshare, and then I realized one possible explanation about why you're doing this: getting technical advice from the community means everyone's had a part to play, and they're likely to remember that. Good move ;)

---

In my case, I have little (okay, 0) practical experience; a lot of the following is mentioned experimentally, to see how these ideas would handle the described environment. It's pretty much all stuff I've read online.

I welcome replies that shoot down any of these ideas.

> Disk

Disks can be slow so we looked at improving latency. Higher RPM hard drives typically come in GB instead of TB sizes. Going all SSD is too expensive. To improve latency we plan to fit every server with an SSD card. On the fileservers this will be used as a cache. We're thinking about using Bcache for this.

There's already been another brief comment (https://news.ycombinator.com/item?id=13153317) about ZFS.

So, I'll ask. Why not ZFS? You don't have to run FreeBSD anymore to get a stable implementation.

You can put both the L2ARC and ZIL on SSDs. You can even use striping with them. Don't quote me on this but I think there MAY be some recovery capabilities built into these layers for if the power goes out (either it didn't use to be possible and now it is, or it's architecturally impossible, I hilariously cannot remember which).

---

> In general 1GB of memory per TB of raw ZFS disk space is recommended.

This is ONLY if you have dedupe switched on. If you have dedupe off you can run systems in just 4GB. A lot of home server enthusiasts do this.

There are a lot of unfortunate and widespread misconceptions about ZFS.

---

(This bit's somewhat anecdotal and is more informational than actionable. It's worth noting if you're interested in disks.)

> Every node can fit 3 larger (3.5") harddrives. We plan to purchase the largest one available, a 8TB Seagate with 6Gb/s SATA and 7.2K RPM.

Technically, the largest one available (on Amazon and presumably elsewhere) right now is 10TB, but its price/capacity ratio is atrocious compared to the rest of the market ($450-$520 per disk).

I've heard that Seagate Enterprise Capacity drives either die within the first 2-4 weeks or last 20 years. They have 5 year warranties in any case. I haven't heard anything else about other disks.

Very interestingly, 8TB seems to be the current market leader. Here are a bunch of prices I took straight off Amazon, as guides:

#3: 4TB: $170 (13 disks for 52TB = $2040)

#2: 5TB: $200 (10 disks for 50TB = $2000)

#4: 6TB: $239 (9 disks for 54TB = $2151)

#1: 8TB: $360 (7 disks for 56TB = $1673)

#5: 10TB: $450 (5 disks for 50TB = $2250)

A little while ago 5TB was the leader, and I was going to argue for more disks.

---

(Hitting add comment now instead of waiting so I can keep up with the discussion)