Wow - still a bit painful to remember my first go around with this. What i am about to admit is damning enough to keep me unemployed for the forseeable future so hopefully no hiring managers are reading this...

I was a sysadmin and did the datacenter buildouts for the company. The task was to take the dev setup in Softlayer (on bare metal servers) and reproduce it in our racks.

The error was to take this somewhat literally. The bare metal servers that we leased from Softlayer were 2u boxes with redundant power supplies, Raid controllers and 10g networking.

I found a Supermicro configuration that almost identically matched and ordered 30 datanodes; 2 name nodes (this was back when Hadoop used the curiously named primary and secondary name nodes); and a server to launch jobs from.

We already had a Netezza in place and since that used Bladenetwork switches I decided to use the same as TOR switches with a beefier on as the Aggregation switch.

We decided to use Ubuntu as the OS and the Cloudera packages - but without the support or the console.

Every single one of those choices were mistakes. It is remarkable in hindsight that it worked at all.

The mistakes:

1) starting with the power supplies. Since they were redundant it dictated an A/B power setup in the racks. What this means is that you cannot use more than 50% of the power density because the entire rack is set to fail over. Each PDU has to be able ot keep the whole rack up so it alarms at 40% capacity. My using redundant powersupplies I was more than halving the amount of power i had at my disposal.

2) RAID. I hadnt yet read the excellent Hadoop Operations O'Reilly book which tells you why RAID is a bad idea for hdfs. Further buildouts included ripping out the RIAD card and JBODing the drives allowing Hadoop to properly use the raw disks.

3) 10G networking - because more is better right? Later after we hired talented networking folks I learned a bit about how important caching is in switches - particulary for Hadoop. My expensive monster Bladenetwork switches were falling over because the bursty traffic would saturate the switches. In addition I had the networking setup for a much more conventional network and we were doing the switch aware stuff in the config but had a wide open /16 as the network. I learned about properly segmenting to avoid uneccesary cross talk.

4) Ubuntu - never listen to developers.
;)
Actually it was our default OS. CentOS works better for Hadoop I would later find out.

5) Lack of support. Cloudera is expensive but - unless you are already an expert - worth it. We should have gotten help early on.

6) Hardware choices. Supermicro was a bad choice. Penny wise / Pound foolish. They changed hardware config and it was hard to get replacemnts, etc. Cant say enough good things about working with PSSClabs for hardware though.
I already mentioned the switches. I learned about Arista switches which are amazing for this application.

By the time I left that company, I learned an enormous amount from really smart engineers that knew gobs more about Hadoop and netwrking than I did.

I think the number of datanodes was up over 200 and we had a lean 1u datanode from PSSCLabs with single powersupplies 12 drives JBOD with onboard flash for the OS. Bonded 1G networking running up to Arista TORs and mutliple Arita Agg switches in a leaf / spine topology.

It was a thing of beauty.