If you don't think ahead, it's easy to expose an API with implicit implications of poor or at least limited performance. The rest of your code base becomes dependent on it, and the month before shipping when you start really focusing on optimizing you realize it's too late to do much of anything to fix it. Sometimes it can be fixed but only by adding a ton of behind-the-scenes code complexity to that API to try to speculatively optimize for anticipated usage patterns.

A simple, innocent-looking example is a string data type that offers (and implicitly encourages) use of random access. If you're using a fixed byte-width for characters that's unproblematic, but it's an issue if you later replace it with a variable width representation like UTF-8. If all your string processing client code is written to rely on random access rather than incremental iterators, you have a problem. You could choose to store an index array alongside the character array. That would roughly double your string storage requirements, but at least you could still offer O(1) random access.

Or you could add code complexity by having each string remember the last index it looked up (sometimes called a finger), and if the next requested index is near by, it uses simple O(distance) forward or backward movement to return the nearby character's value. That seems like a good compromise. But what about multithreading? It looks like we need this cached index to be thread local. Not too bad. But what if we're interleaving accesses to the same string within a single thread? Now the different accesses are fighting over the same cached index. It looks like we need a multiple-entry cache to support this usage pattern.

If you're a good engineer you can probably find the right compromise along the above lines that works for your code base. But it would have been much better if your API design hadn't constrained your future opportunities for changing internal representations and optimizing performance.

Here's another example: OpenGL's immediate mode graphics interface where you can specify vertex data piecemeal. Because everything written in that style is code driven rather than data driven, the driver can't just allocate a single dynamic vertex buffer with a fixed vertex format and push attributes onto it in response to glColor(), glVertex(), etc. What you can do (and what the good OpenGL drivers do) is to speculatively keep around a bunch of staging buffers with different vertex formats behind the scenes that are first created dynamically in response to recurring vertex and primitive patterns. You can see how it goes: It's a lot like the implicit caching in the string random access example, and it has all the same problems, and a whole lot of new ones.

These kinds of late-in-the-day performance fixes for early API design decisions (even though the early API is perfectly clean and modular, in fact maybe too modular) are doable but they are costly in complexity and programmer time. They are also fragile. In my two examples, if you're in the sweet spot of the behind-the-scenes caching system, you'll be seeing good performance. But change one small thing that from your point of view shouldn't impact performance, and you might suddenly fall off a performance cliff because the cache is now too small, or whatever. Incidentally, this problem of fragile automatic optimization isn't restricted to library design. It's also a very serious issue with compilers. Modern compilers rely way too much on a fragile form of "crystal ball optimization", as one of my coworkers likes to call it. It makes it very difficult to write code with predictable high performance and even harder to maintain.

Damn, that was a lot of text.