I agree with what you're saying, actually. I think these questions require drilling down to philosophical questions such as "what are numbers", "what is the essence of 'number'", "are numbers even real", etc. So I'll give it a go and be shamelessly long-winded about it :)

I think that numbers, set theory, programming languages, etc, are all abstractions. Forming abstractions is a mental tool associated with our ability for language: we spot a pattern and we give that pattern a name. Once we've given it a name, we can talk _about_ the abstract concept as a subject of our statements.

Perhaps the simplest form of abstraction we're capable of is "categorization". Ie we see a pointy-eared small animal that makes meowing sounds, we another one, and another one, ... and even though none of them are identical, our brain picks up on certain patterns in these creatures. We know that when we next see one of them, we can expect certain behaviors. So we give it a name, let's say, "cat", and now we can talk about the abstract concept of "cat" without referring to any specific individual cat. To see how big a deal this is, I find it useful to remember that cats aren't explicitly defined in the fabric of the universe, like they would be in a computer program (where we'd have a "Cat" data structure or object, usually), they're just emergent behavior of interacting molecules.

But we can do more, we can also abstract out properties of objects. We can see that some frogs have a similar color to the leaves of most trees, so we can give this leaf-color a name, "green", and talk about the abstract concept of "green" without referring to a specific green thing. You can just say "I like green" (no one will ask you "a green what?"), or "mixing green and blue makes cyan" (no one will ask "mixing green and blue what makes cyan what?").

And then we also have "chunking", where we can talk about sets of objects as objects themselves. So we can talk about "a herd of sheep" as a singular thing. Our ability to abstract out properties of objects also applies to these chunked concepts. One property of a set of objects is "quantity". Speculation ahead. Initially we probably only had fairly vague terms to talk about quantity, perhaps individual words for small quantities (one, two and three, maybe), and after that it's just "many". But it seems we figured out pretty fast that counting sheep, counting apples, or counting pebbles are really in some sense "the same thing", so we might have first learned to count through some unary "notation" even though we had no names for quantities. We'd make carvings into bone or carry a bag of pebbles, one carving or one pebble for each sheep in our herd. Then we notice that certain operations on these quantities are also all "the same": it doesn't matter whether you're adding pebbles, sheep, carvings, ... you're always doing the same thing. So after a while we start using these operations to define bigger numbers (the French still call the number 80 "quatre-vingts" or "four twenties"). Now all numbers become nameable, and eventually we streamline and standardize these names.

So now we have a name for every number, and we can talk about "thirty-two" without getting back a confused reply "thirty-two what?". And we can talk about operations on these numbers, like addition, multiplication, etc, but our hunger for abstraction isn't satiated at all. Since now we can talk about operations, we can also talk about properties of operations, so we notice that you can flip the arguments of additions and multiplications, and we give that property a name, commutativity, and we go on to invent abstract algebra.

After doing maths for a few thousand years we start looking into patterns into the different, disparate branches of maths at the time (geometry, number theory, logic, etc), and notice that the common pattern is that all of them are doing deductive reasoning, and that should be modellable by logic. So we abstract out the common bits and come up with set theory and proof theory and the like. And in that process we notice that number theory has unnecessarily many axioms and Peano figures out we can just use induction to define the set of natural numbers.

So yes, our intuition for numbers precedes Peano's definition by millennia or far more than millennia. But I'd say that's how it goes for _every_ definition. We always start by noticing some pattern, which initially we can't quite put our finger on (for example, Leibniz had a vague intuition that a lot of work in mathematics was in some sense so mechanical that one should be able to have a machine do it; he couldn't quite define computation yet but his hunch was spot-on), and then we try to refine our thoughts and come up with a definition.

In that sense, I don't think it's more meaningful to worry about the "true nature" of a number (the Peano definition, or some other notation?) than it is to worry about the "true nature" of the color red (is it its wavelength on the electromagnetic spectrum? but a wavelength can be expressed as a number; so is the essence of red just a number? or its RGB value, which is linked to how humans perceive color?). Both numbers and colors are abstractions, and abstractions are mental tools. They are a means to an end: to make sense of the world and communicate about it to other humans. And we use just use the tool that is most helpful given the problem at hand. For most practical applications we're going to use a notation that's compact and easily manipulated (Arabic numerals), but when we're studying the foundations of mathematics, it's more sensible to reduce everything to its smallest, most elegant definition possible. Neither view is "more true" than the other, in both cases you're doing no more than naming patterns in herds of sheep and piles of apples.