> There is similar issue with even ZFC and PA. It’s not really a dealbreaker imo.

We obtain PA from our basic intuitions about how the standard integers work, derived from empirical evidence. Everything past that involves increasing levels of intuition. So to continue it indefinitely, you must postulate an infinite amount of correct intuition, in some magical fashion that can never be captured in a computer. You can claim unlimited ingenuity all you want, but there's no a priori reason that it should indefinitely yield the truth, especially when it goes far, far past what our finite empiricism can provide.

We just haven't hit these limits yet, since very weak inductive theories are still sufficient for proving BB(5): we don't even need the full power of PA yet for our non-halting proofs. Thus why it looks like it should be so easy.

> If you fix an axiomatic theory, it’s already known that the theory has a limit.[1]

Not quite. Fix some consistent axiomatic theory T which proves PA. Then there will be infinitely many TMs which do not halt (in the standard model), but T cannot prove that they cannot halt, due to incompleteness. (Therefore T cannot settle the BB(n) question past a certain point, as Aaronson correctly says.)

But for every TM that does halt (in the standard model), T can prove that it halts, and the proof is to list out a trace of the TM's execution. Thus, every halting machine of every length has a halting proof in T.

The only benefit of a more powerful theory T is that it can "compress" this maximal BB(n)-sized proof into something more physically managable. But once we fix a certain T, we find (by my earlier argument) that it can only compress the proof so far, and the compressed size still must be an uncomputable function.

We can also see this by a forward argument, instead of by contradiction. Suppose that we'll accept any halting proof in a theory T. Then we can write a TM that lists through all proofs in T that are smaller than some bound N. (Notice that this is a finite set, since I've put an upper bound on it!) Then, for every proof that is a valid halting proof, the TM runs the corresponding machine. Then, the TM will halt, and its halting time will be greater than the halting time of any machine that can be proven to halt in T within N symbols. Set N to Graham's number (which is easily definable), and now the halting proof of the TM in T will not fit in our light cone.

(Notice how our TM clearly halts if T is Σ₁-sound! But since T cannot prove its own Σ₁-soundness, it doesn't have any way to prove our TM halting other than by the brute-force method.)

> In fact for similar reasons I believe a Halt program, which has infinite states but which works for all TMs with finite states, platonically exists. It’s an emulator and an infinitely long busy beaver number lookup table. The diagonalization argument doesn’t apply, since the infinite Halt doesn’t accept itself as input.

In that case, you just end up with the well-known oracle halting problem, where you equip a TM with access to this "infinite-state" machine. Then the problem is that you have a more powerful model of computation, but still with no way of solving its own halting problem. Much like how a consistent theory can only prove the consistency of weaker theories, not its own consistency.

> So it’s not clear that f(n) is uncomputable. If f(n) is the symbol count and not the binary encoded length of the symbols, it even seems that it’s trivially bounded by some constant for all n. The proof could be one symbol the meaning of which is encoded in the theory.

Of course I'm fixing a particular theory and a particular alphabet of constant size, the alternative would be absurd. The important part is about the ultimate behavior as n varies.