RE the incompleteness theorems, first let's lay out why I think it's a death blow for formalism if you believe in a single true set of natural numbers and you believe that these natural numbers are a mathematical object.

The basic argument of formalism is that it is the symbols and manipulations that are the essential bedrock of mathematics and everything else is just an application of those symbols. Yet at the same time, the natural numbers as they truly exist in our universe seems to be a valid, if not foundational, object of mathematical study and they occupy a curious epistemological state. On the one hand, they are abstract entities that do not exist in the same way that physical objects exist in our universe. On the other hand, they seem to have an objective truth value. It is false to say that 2 + 3 = 4 if we interpret 2, 3, 4, and + in the usual way, in a stronger way than just symbols on a piece of paper would imply. If 2 + 3 = 4, then we just simply say that the symbols 2, 3, and 4 must not represent natural numbers, or else 4 really is just a weird way of writing five. This is of course just rehashing the difference between numerals and natural numbers, but I think most mathematicians would agree they are interested in studying the latter, and not the former except as a tool to understand the latter.

But again we hit this weird issue, where it seems like mathematics now is trying to study some objective thing that exists in our universe, because after all consequences of the natural numbers have real ramifications for physical behavior in our universe. However, by Godel's incompleteness theorems we cannot capture the natural numbers in their entirety with symbols. There will always be a new axiom you need to add. And most importantly, if there's truly a single set of natural numbers, that axiom is determined. Unlike an arbitrary axiom system, you can't willy-nilly just add the opposite of the correct axiom because then you aren't describing the natural numbers anymore.

Now you can say that the "natural numbers" as they exist in our universe are not a proper object of mathematical study. Rather our limited formalization of them in say the first order Peano axioms is the proper object of mathematical study. And since we're talking about definitions here, there's no way to say that's wrong: one can always draw the lines in mathematics wherever one prefers. But then the question arises, what field is responsible for studying the "true" natural numbers, rather than the pale reflection of them we see in the Peano axioms? If not math, then what? Physics? And yet natural numbers seem far too abstract a thing to study in physics. And it sounds so weird that something as foundational as the natural numbers is not an object of mathematical study.

And moreover it's fairly clear if you talk to mathematicians that the "true" natural numbers hold primacy over the Peano Axioms. If we found something about the natural numbers that contradicted the Peano Axioms, we'd throw out the latter and keep the former, not the other way around. That is we use the Peano Axioms to study the natural numbers, which necessarily exist outside of symbols by Godel's incompleteness theorems, not the other way around.

Indeed the very idea of there being objectively "correct" rather than only "useful" or "applicable" axioms is a very different intuition of how to do mathematics than what is implied by formalism. And it's very hard to escape that there are objectively "correct" axioms that can only be accessed through some form of informal intuition (since by nature of being axioms they cannot be accessed by formal proofs) if you believe that there is a single true set of natural numbers and take Godel's incompleteness theorems into account. (Note that this is not just hypotheticals, if you read the papers of set theorist giants such as Hugh Woodin, they are content to declare certain axioms as "true" or "false" based on certain argumentation). In that world those informal lines of arguments are extremely important and the centerpiece of mathematics and the best that formal arguments can do is serve as a backstop to make sure we aren't making egregious mistakes. That is in the case of natural numbers our main mission is to make a variety of mathematical, but ultimately informal arguments to declare the validity of a new axiom, and then just keep an eye out that we haven't introduced inconsistency accidentally (note that often, except with rather conservative extensions, we can't even formally check whether we've preserved consistency, even if we assume our original axioms were consistent, by Godel's incompleteness theorems! We've really got our hands tied if we only accept formal arguments)

Now if I put my formalist hat on, how would I respond? Well first, let's consider plausible discoveries that would seriously cast doubt on a single "true" set of natural numbers.

Most mathematicians think that something like the Continuum Hypothesis doesn't have an absolute truth value (although most set theorists probably would disagree). It is an empirical observation that mathematicians are broadly in agreement that every arithmetic statement about the natural numbers so far has an absolute truth value. However, the potential remains that someone could come up with an arithmetic statement independent of PA that large portions of the mathematical community view in the same way as the Continuum Hypothesis. The mathematical realist would argue that this is not possible, but it's not clear to me a priori that that's the case (although the fact that it has not yet shown up after so much scrutiny about the natural numbers is perhaps a good indication that it is the case). If such a statement were to be found, that would be very strong empirical evidence that the notion of "one true set of natural numbers" is wrong.

Another, perhaps more likely, possibility is if through various exotic physical processes we had access to nonstandard natural numbers, i.e. if we found evidence that the physical Church-Turing thesis was wrong and that there are physical computations which cannot be simulated by symbolic ones. This would be a philosophical triumph for Godel's completeness theorem and by extension formalism (formalism bears a close relationship to semantic completeness, stated roughly as the idea that if a formal axiom system is consistent, there is some conceivable world where that axiom system is applicable) and would also be a strong case against the thesis that there is one true set of natural numbers. Simultaneously though it would jeopardize the primacy of symbolic computation and the same arguments would probably be rehashed about the equivalent "hyper natural numbers."

So as a formalist my first instinct would be to question the idea of "one true set of natural numbers" and to say that in fact natural numbers in the objective sense do not exist. That they seem to exist is only an artifact of limited human experience. (to be fair this not completely absolve the formalist. An intuitionist could cut in at this point and say "I believe everything you've said so far, but I'm still not a formalist")

But anyways, that's enough for now. I'll return if there's interest.

EDIT: Another possibility is to deny the reality of the natural numbers as an entity altogether to various different degrees which is roughly what something like finitism or ultrafinitism is. In this world, you would basically deny that statements such as "for all natural numbers" or "there exists a natural number" have a coherent physical interpretation or at least that portions of those statements don't make sense.