Bayesianism is a 'grand unified theory of reasoning' that all of science be should be based on assigning (and updating) probabilities for a list of possible outcomes; the probabilities are supposed to indicate your subjective degree of confidence that a given outcome will occur.

Contrast this with an alternative conception of rationality as espoused by David Deutsch.

David Deutsch in his superb books, 'The Fabric Of Theory' and 'The Beginning Of Infinity', argued for a different theory of reasoning than Bayesianism. Deutsch (correctly in my view) pointed out that real science is not based on probabilistic predictions, but on explanations. So real science is better thought of as the growth or integration of knowledge, rather than probability calculations.

So what's wrong with Bayesianism?

Probability theory was designed for reasoning about external observations - sensory data. (for example, "a coin has a 50% chance of coming up heads"). In terms of predicting things in the external world, it works very well.

Where it breaks down is when you try to apply it to reasoning about your own internal thought processes. It was never intended to do this. As statistician Andrew Gelman correctly points out, it is simply invalid to try to assign probabilities to mathematical statements or theories, for instance.

Can an alternative mathematical framework be developed, one more in keeping with the ideas of David Deutsch and the coherence theory of knowledge?

I believe the answer is yes, and I am going to sketch the basic ideas for such a framework.

The basic idea is to separate out levels of abstraction when reasoning (or equivalently, levels of recursion). In my proposed framework, there are 3 levels, and each level gets its own measure of 'truth-value'. All reasoning must terminate in a Boolean truth value (True/False) at the base level but the idea is that different forms of reasoning correspond to different levels of abstraction.

1st level: Boolean logic (True/False)

2nd level: Probability value (0-1)

3rd level:
Conceptual coherence (categorization measure)

For full reflection, you need three different numbers: a Boolean value (T/F) at the base, a probability value (0-1) at the next level of abstraction, and an entirely new measure called conceptual coherence at the highest of abstraction.

As a rough working definition of conceptual coherence, I would define it thusly;

"The degree to which a concept coheres with (integrates with) the overall world-model."

It should now be clear what's wrong with Bayesianism! It only gets us to the 2nd level abstraction! There is not just uncertainty about our own knowledge of the world (probability), there is another meta-level of uncertainly; uncertainty about our own reasoning processes, or logical uncertainty. Bayesianism can't help us here. Conceptual coherence can. Lets see how:

All statements of the form:

‘outcome x has probability y’

can be converted into statements about conceptual coherence, simply by redefining ‘x’ as a concept in a world-model. Then the
correct form of logical expression is:

‘concept x has coherence value y’.

The idea is that probability values are just special cases of coherence (the notion of coherence is more general than the notion of probabilities).

To conclude, conceptual coherence is the degree with which a concept is integrated with the rest of your world-model, and I think it accurately captures in mathematical terms the ideas that Deutsch was trying express, and is a more powerful method of reasoning than Bayesianism.