The point you're missing though, is that even in your example, the backwards case is still physically valid. It is true that no macroscopic interaction is reversible in practice, but this ceases to be the case for a fully defined system where there is no increase in the entropy, and you could argue it is because these do not exist, but there is no physical reason that they CANNOT exist. For one, if you could model all of the energy losses due to vibrations effectively, there's no reason to think that physically one could not use a very precise arrangement of speakers to cause compression waves to converge to the point of contact at the exact instant of the collision and revert the system to the initial conditions. Just because it's backwards doesn't make it physically invalid, except for the friction (and thus heat/entropy thermodynamic part). The backwards scenario still fully obeys the laws of physics. What you're doing here is essentially begging the question by saying that thermodynamics (and thus experience) tells us that these things (elastic collisions) don't happen because they don't happen. What you haven't established, however, is why. The point of this article is that from a physical perspective, there is no reason to believe that the laws of thermodynamics exist. But there are less complicated examples as well.

For instance, if you have an electron placed some distance away from another negative charge source, you could very easily model its dynamics. But there is no fundamental difference between this electron being repelled from the other negative charge compared to a positron that is being attracted to the charge, but moving backwards relative to our perception of time. So it is a forward moving electron or a time-backwards positron? There's no difference, they're both completely physical systems.

Boltzmann's big accomplishment, one that was at his time rejected until Einstein's description of Brownian motion, was his completion of the H-theorem (which Claude Shannon subsequently named his information-theoretical counterpart for), which describes that physical systems under very simple assumptions (uncorrelated position and momentum vectors) will 'relax' to a distribution of energies given by the maxwell-boltzmann distribution. It turns out, however, that these assumptions are not completely valid for physical systems, so the H-theorem is not really considered a 'true' explanation. I highly suggest reading the wikipedia article on Boltzmann's H-theorem that goes into the mathematical derivation of the positive-definite nature of time derivative of entropy of a closed system.

<wild speculation>
So what are the real answers as to the arrow of time? It's an unsolved problem, but I have guesses. The most obvious answer for things like this is based on the anthropic principle. In an informational world, one could assume that everything that we interact with all moves in the same time direction because DNA is effectively some Turing-equivalent string of 1s and 0s that is modified by the 'computation' of evolution. Memory of the timeline is encoded into DNA, which is what we are from an information perspective.

Chances are, however, that things like this will never be proven, possibly because in this case the proof will be non-computable by Turing machines like us. It isn't even unreasonable to think that the 2nd law of thermodynamics and P!=NP are not equivalent statements and equally unprovable. Am I far out there enough yet? </speculation>