> Obsessions with code history are a little disconcerting.

Wanting to see code history is not anywhere close to an "obsession" -- that's just a loaded word that does nothing but distract from the issue.

> It's interesting to see, but has little relation to the quality of the paper (or the code base, for that matter).

This is just completely wrong. I don't want to see code history because I care about the author's earlier mistakes, previous implementation attempts, refactorings, etc., from any sort of judgemental, coding skill point of view.

The reason it's important is that maybe they used version 3 when they ran some part of the study and generated some of their data. They figured it was good enough, but then later they updated some code to version 4. They didn't need to re-run things for that older data, but they needed version 4 for something else.

But, version 4 makes a change that causes an error, or an undefined behavior, or a change to some statistical degrees of freedom calculation or something, that would imply the data from version 3 also needs updated.

So the authors are using something that came out of version 3, looked good, and they thought never needed revisiting. Then they are also using something from version 4 that, overlooked by them, invalidates or changes something from version 3.

All of this gets whitewashed because they go through and manually finalize the code, and upload it to a public repo.

Now, you as the reader are not even aware that there ever even was a version 3, let alone that some of the data came from version 3, some came from version 4, and the mixup is based on an oversight or something.

The code history really matters. If you cannot reproduce the results immediately with the tarball the authors release as the "good" version, and there is no version history, then, effectively, there is zero reproducibility. You have to push back on the authors, who weren't making their changes under version control when the version 3 to version 4 issue arose, and so good luck ever getting an answer.

I've actually experienced this first hand. I had to reproduce some quant finance research on country-based beta calculations. It turned out that in an early draft of the work, the team had used a completely different data set and in the earlier data set, the country code for Sweden had been set to "SW" even though the ISO country code for Sweden is "SE". They got some intermediate statistics out of that data set and saved them for later usage.

Then later on they used another data set and someone had the bright idea of using "SW" for Switzerland, even though that country code is "CH".

But because all that remained from the first data set was aggregate statistics, and no version history of the code that had loaded the spreadsheet and made these assumptions at that time, there was no way to figure this out.

In the final result, Swiss and Swedish data was being aggregated together and throwing numbers off. And those two countries both have very small market caps, so this is hard to spot and track down.

After a ton of outrageous debugging, we just happened to get lucky and notice that the difference was almost equal to the largest Swiss company in that time period (Nestle I think), and then began to wonder about the "SW" issue and then tried all permutations of separating the two data sets into Swiss and Swedish data until things matched.

If we could have bisected the original source revisions, it would have saved weeks worth of tedious, expensive data drilldown nonsense to replicate a result.