> It's strange that you disregard that in an aside, as it's his entire point.

Perhaps I misunderstood as to what his point was.

If it is his entire point, I'd wish he'd not add in the bit about serialization. It just confuses the point.

To be honest, what the point of the article is strikes me as confused, because he's crossing the gamut of a variety of topics even in his summary at the end. Maybe you can help me understand my reading comprehension problem. Looking at the summary I see a variety of points that aren't really related:

1) Don't use the implementation of Object.hashCode() for your distributed hash table. Okay, this should not be news to anyone who is at all proficient with Java. It also doesn't require a long article. I'm not sure if another article needs to be written. If one must, you can just say, "Object.equals() doesn't provide magical equivalence powers; it's a place holder for when you implement equivalence. BTW, as you'd expect, the Java standard library documents its equivalence semantics whenever it overrides the method itself." You don't even have to mention hashCode().

2) Your serialization framework ought to be how you implement equivalence. Umm, no. That couples the concepts serialization with equivalence. It is essentially arguing that you ought to implement equals() by first serializing both objects and then comparing their output! I'd argue they are related in the same manner as the articles description of the relationship between Object.hashCode()'s implementation and doing good distributed partition. Serialized equivalence is correlated with logical equivalence just enough to create a huge mess. Hadoop wanders about as close to this as one should by providing hooks for doing comparisons of objects without deserializing them. One should really, really NOT go down this road.

3) Since Object.hashCode()'s implementation doesn't provide equivalence, you should not use overrides of that implementation for your partitioning. That's just stupid, even beyond the confusion of interface and implementation. You're going to need equivalence logic for your keys. Are you really going to create your own "T.equivalent(T)" method whose behaviour is distinct from "T.equals(T)"? What does it mean when those two don't have the same value? This suggests a lack of understanding as to why Object.hashCode(Object) is there in the first place.

4) Your distributed system needs its own hash function, but not necessarily a secure one. Talk about getting it backwards. Built in hash mechanisms generally work fine for short circuiting equivalence tests and already make a pretty reasonable effort to deal with poor programming. In the end, using a different hash function might only help if there is a lot of entropy in the key that is being lost by the existing one, which tends to only occur with large numbers of reasonably large keys (ironically, String.hashCode() makes some pretty good trade offs here as compared to a generic hash function). This tends to be an issue that you never encounter, and to the extent that you do, it's just an optimization issue, rather than the logical one the article described. The only major hurdle that is (wisely) left on the table by built in hash tables is the security one. For tackling that you are best off using an HMAC (designed to address that space of problem) instead of a bare hash. MD5 really isn't the right choice for any case where you have a choice about your hash function, and SHA-1 really isn't any more either, so mentioning them in the article is just silly.

5) Implementations of hashCode() generally work great for in-process hash tables, but are lousy for distributed systems. For the vast majority of hashCode() implementations, if it provides a good in-process partition input, it provides a good out-of-process partition input. Along with its corresponding equals(Object) method, the implementation of Object.hashCode() is, by design, an exceptional case and shouldn't normally get invoked.

6) Form parameters sent to Java web servers are a great example of this problem, and they wisely avoid the DoS problem by limiting the number of keys. Ummm no. First, the form parameters sent to a Java web server are no more a distributed systems problem than any other user provided data shoved in to a lookup table. It's a pure trusted systems problem. The only thing 'distributed' about it is that it tends to involve a network somewhere. In fact, a web server ought to be worried about DoS problems that have nothing to do with hash tables (like, the memory footprint of the form data). Given their solution to these DoS exposures and the context, they really ought to just shove the parameters in to some kind of efficient tree.