> Its very sensitive to fat tails and skew. That's the reason robust testing and estimation is a thing.

Yeah, I shouldn't have said "extremely". Its much more robust than is commonly perceived, but that does not make it "extremely" robust. Thank you for the correction. But please note that I specified equal sample sizes, so the fact that "there are general conditions under which these methods can provide poor control over the probability of a type I error" is not really a refutation—I already implied that the t-test is not generally (= in all cases) robust. I also mentioned non-parametric tests as a potential alternative. I do not wish to imply that the t-test is always the right choice. But I stand by the assertion that approximate knowledge of the distribution, such as obtained by inspecting a histogram, is perfectly adequate to choose a test. The main point remains — list what you know about your data, and pick a test that tells you what you want to know with a tolerable error level for your application.

You can spend an awful lot of time picking the "correct" statistical test (if there is such a thing; tradeoffs exist) with little gain. Worse, making a lot of decisions about what test to use after looking at the data leads to p-hacking, potentially leaving you with more bias than if you naively used a slightly-wrong test from the start.

To elaborate on the t-test discussion:

From your ref [1], "If sampling is from nonnormal distributions that are absolutely identical, so in particular the variances are equal, the probability of a Type I error will not exceed 0.05 by very much, assuming the method is applied with the desired probability of a Type I error set at α = 0.05. These two results have been known for some time and have been verified in various studies conducted in more recent years" (page 79). I think Wilcox makes this statement under the equal sample size caveat, but the text is unclear on this point. This does not guarantee robustness under nonnormality, but it does refute the idea that nonnormality is always fatal to the t-test.

Yes, a t-test can be affected by skew even with equal sample sizes. Whether this is a problem depends on the application. The example in [1] uses a lognormal distribution and ends up with actual α = 0.15 for n = 20, desired α = 0.05. Which isn't great, but is somewhat tolerable. Also, a lognormal distribution is strongly skew right and left-truncated, and this is easily noticeable on a plot and can be transformed to a normal distribution. So this does not refute the idea that a sort-of-normal distribution is fatal to the t-test.

You can look up the potential α level and β level errors from violating the assumptions of a particular test—they're tabulated. E.g., applying the t-test to a Pearson distribution produces a typical α level error of 0.005 at desired α = 0.05 [2]. That's definitely tolerable. This is why I stated that sort-of-normal is good enough for a t-test. It's also fairly straightforward to calculate the expected errors yourself for a particular situation, if reassurance is needed. If your statistics are intended to support a high-stakes decision this may be a good use of time. Working to validate your finding by other means is better, though.

I'm not sure what your ref [0] (the "Robustness of Standard Tests" book chapter) is arguing for or against from the abstract alone. I don't have a copy of that book. The abstract mentions most flavors of maximum likelihood ratio tests, which is too broad a set of topics to discuss effectively. The implication seems to be that null hypothesis statistical tests are bad and something else (Bayesian analysis? visual inspection?) is better, which I don't necessarily disagree with. If you could please clarify its contents and if it is worth tracking down, I would appreciate it.

[0] Wilcox, Robustness of Standard Tests

[1] Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy

[2] Posten, H.O. (1984). Robustness of the Two-Sample T-Test. In Robustness of Statistical Methods and Nonparametric Statistics, D. Rasch, and M.L. Tiku, eds. (Springer, Dordrecht), pp. 92–99.