The way I understand this is that a lot of it comes down to articulating useful prior distributions over some of the outcomes to be predicted.

The kind of backtesting that this post explores, which is basically the bread and butter of "quantamental" asset managers (lolz CRSP Fama/French data), is almost always performed in a frequentist set up where you plan to regress outcomes (portfolio returns) on a simple set of "easy to understand" predictors, like price momentum over some trailing time period.

You might have to do a few cross-sectional or time-series adjustments, like the Fama/MacBeth regression and some poor man's time series smoothing of coefficients, but it's basically just a really large, really crappy OLS model, where the plan is to just torture the vendor data sets until you get out whatever kind of statistical significance you need for a marketing whitepaper and/or pitchbook entry to shop around to unsophisticated board members in client organizations.

This is the whole "which 1000 tests did I do before I showed you the results" thing from this article, and it just amounts to simple p-hacking (or t-stat hacking) which is a well-known Bad Thing To Do.

Clients are already very sensitive to things like transaction costs, so the clients probably would have demanded that the analysis be carried out net of transaction costs in the first place. But transaction costs are easy for a paying client to get their head around... for non-quantitative client board members, getting their head around nuances of p-values and regression models is actually quite hard, even if these are statistics 101 for the finance professionals working with them.

But the amazing thing is that while everyone's busy worrying about p-hacking (and never Bonferroni-correcting reported results, if they even know how many researcher degrees of freedom [1] are involved in the first place), no one is talking about the fact that all of this is in the service of a frequentist-style analysis of Prob(Data | market hypothesis), rather than a Bayesian-style policy question that seeks to analyze Prob(market hypothesis | Data) and necessitates some use of expert-derived priors about the probability of some market hypothesis in general.

In the case of a momentum strategy like the one studied, the market hypothesis would need to be about modern market conditions, the causal underpinnings of price momentum as a predictive factor, and whether or not its efficacy will improve or decrease when you condition on the modern market situation.

And crucially, this is the part that active managers are (like everyone else) very bad at. They can't articulate accurate or informative priors about the current status of something like the causal underpinnings of price momentum any better than anyone else.

If you are seeking to pay someone for a policy analysis, e.g. what should I think about the posterior probability Prob(market hypothesis | Data), then you shouldn't be paying them for the easy part of that analysis -- the part where you write down some silly likelihood function due to heroic OLS assumptions and optimize it for parameters and update a portfolio position based on those parameters. That part is borderline trivial if you have decent software engineers to make sure the code is not excessively buggy (yet most quantamental shops don't, and they're still letting portfolio managers do this shit directly with SLOPE in Excel and then making Ivy MBAs translate it into poor Python or MATLAB code for "production") -- the hard part is actually articulating informative priors.

I think overall, it higlights a need for client organizations to become a bit more sophisticated about the sorts of forecasting models that quantamental managers use. Then they can begin applying market pressure by firing managers that do bang-simple things like OLS in a crappy dev environment, and hopefully create competitive pressure to make quant shops shift to more rigorous Bayesian methods, or at least make the rampant researcher degrees of freedom and p-hacking problems transparent so they can be mitigated.

Of course, a lot of this is moot because a very large amount of the business done with quantamental asset managers is based on nepotism and cover-your-ass blame insurance. For example, suppose you are a non-technical board member of the entity charged with managing the pension plans for some state's retired firefighters. You are paid money not just based on whether the pension plan's assets improve, but also on whether you did due diligence when hiring an active manager (read: you hired a fancy consulting firm who told you what you wanted to hear), and whether you are in a position to fire firms if they seem to do a bad job, or create pockets of plausible deniability by crafting "data journalism" accounts about a market downturn that "no one could have avoided."

Because these board members are the ones with most of the interaction with quantamental shops, most quantamental shops end up turning into shoddy data journalism mills -- need a story to explain yesterday's bad returns to the rest of your board? We're on it. Whadya want? Something about oil & natural gas? Something data mined about some random earnings reports? Just let us know. We're your one stop shop to rationalizing away yesterday's returns. Just don't ask us what will happen tomorrow.

In this sense, clients rarely punish quantamental shops. In fact, many of them are handsomely rewarded for doing p-hacking and shoddy data journalism as long as it covers the ass of some client board member who doesn't want to lose his bonus because of a downturn in the S&P 500 and needs to sell a story to his bosses fast.

If that kind of stuff doesn't go away, there will be no actual prediction-based incentive for the asset managers themselves to ever improve. Why work hard to write good code and do proper statistics when you can make bank by selling shoddy data journalism?

[1] http://andrewgelman.com/2012/11/01/researcher-degrees-of-fre...