I'm using the term "mathematics" in a deliberately hand-wavy fashion because people keep wanting to poke at this from various angles and even though the underlying theory is the same, it can appear that you are receiving different answers simply because you are asking the question from different epistemological frameworks. There's a reason Wittgenstein was the man and I'm not.

If you like, we can assume that different numbers have a concrete, discrete meaning. We can then add commutation and association, maybe induction. At some point, however far down the rabbit hole we want to go, these numbers in a table represent some physical thing: six apples or whatever. These diagrams and other visuals represent some tangible, agreed-upon thing: a diagram of an enzyme, perhaps.

The mapping of various calculii to either physical things or commonly-held concepts is not on the table here. The point is that the concepts themselves can have subtly-different meanings for any two practitioners or observers. A doctor in the middle ages may check your humeurs and a modern nurse may check you blood pressure. Imagining for a second that both operations look similar, even though you may have the same number, you have a completely different understanding of what those numbers mean.

So whatever formal rules of mathematics you'd like to have, and whatever visuals or measurements you'd like to take or produce, the end goal is analysis or language creation around a shared interest. If I hold up a coconut and want to trade it, then you make some odd sound, it means nothing. But then if you hold up two bananas, we may be beginning to converse and exchange meaning. (In this case by way of commerce, but this is just one example of thousands) The visuals enable higher-bandwidth conversations. Do I then think that 1=2, since I had one coconut and you had 2 bananas? I might. I might not. If we're from completely different cultures we have a lot more work to do.

That's an obvious example, the deeper and much more profound truth is that the same thing can happen with a vapor trail in a particle accelerator. I'm reminded of John Wheeler's idea that maybe there's only one electron in the entire universe. Once the web of meaning reaches some degree of complexity, the human brain shuts down and stops evaluating all the possible alternative meaning paths; we are not aware of this. In our mind we've thought through everything and are sitting on top of thousands of years of received wisdom. It has always been like this.

So yes, they are much, much more important than text or formulas, but they're more important because they assist us in the drive for common language creation. Frankly, many times they do a much better job than the others. But the job in both cases is the same.

I will make another stupid analogy. Ever see the scintillating grid optical illusion?[1] It looks like you can see every other black dot except the one you're looking directly at. Meaning is similar in that whatever the concept under observation, it appears like some analysis can work out the problems. The other ones, farther away, don't need any work. They're all set in stone. But then, given time, when you look at those, you realize that no, actually there's some problems over here. That area you were looking at before? That's all fixed now.

The lesson here is that there is a cognitive limit to the things that you can simultaneously ponder about their meaning, relationships, and relevance to any one situation. After that limit, it's all just "common knowledge" or received wisdom. It has to be this way. (no time to go into why). But then you realize that it's all a web, we're all living in our own constructed grid that's different, and the goal is to align both the concepts under observation and the "given" concepts among several of us such that we can erect a language scaffold sturdy enough to make progress towards some common goal.

I know that sucks, but that's the best I've got in ten minutes. Hope it helped. There's an entirely other conversation about how we construct these grids, how we share them, and more importantly why this is a feature of sentience and not just a stupid primate trick. No time for that.

1. https://www.illusionsindex.org/i/scintillating-grid