It's definitely not necessary to have taxes or a 21st-century-level GDP to do mathematics. Current world GDP is about US$17500 per capita PPP, three times the income I've been living on for years, and 1500 times the world GDP in the time of Archimedes. Computers do help, but they aren't essential, and most of the economy isn't devoted to producing computers or other mathematically useful goods anyway; most of it is devoted to things like the steak I just ate, tearing down buildings and putting them back up, killing people in senseless wars, destroying the atmosphere, team sports, junk food flavorings, surveillance capitalism like Facebook, literal prostitution, disposable fashion, toys, furniture, second cars, daytime TV, and so on.

Admittedly, a significant fraction of world GDP is devoted to essentials like child care, elder care, primary food production, housing, and education. But an enormous amount of it is just wasted and counterproductive effort.

However, you weren't talking about those parts; you were talking about truly essential services. Garbage collection and water treatment are essential but are not significant fractions of the GDP. Let's consider garbage collection, using US statistics. The US has 118520 garbage collectors, including the people who work at recycling plants and dumps, out of a population of 328 million people. If this difficult, dangerous, and sometimes unpleasant work were shared equitably among everyone, it would average 52 seconds per person per week, ¾ hour per year. In practice, even with good coordination, you'd probably lose an order of magnitude in efficiency to the lack of specialization, so probably sharing it equitably would mean that everyone would spend an hour or so every couple of months sorting the recycling at the recycling plant. This would have several salutary effects: people would learn not to put milk bottles in the recycling bin without rinsing them, the social stigma attached to garbage collection would disappear, and garbage collection contracts would no longer be an appealing way for mafias to dispose of clandestine murder victims' bodies.

As for construction work, although I'm not a journeyman electrician or anything, I've poured concrete a few times, I've bent and tied rebar, and I've repaired roofs; and when I was volunteering at the Casa de María in the slums near here, almost everybody I knew there had built their own house (or married or been born to someone who had.) With modern materials, you can build a perfectly adequate and livable house in a person-month, and it can house a family for several lifetimes. Even inefficient construction techniques like adobe, which I've also done, require less than a person-year per family dwelling. There is absolutely no need for people to specialize in hot-tar roofing as a career. That's senselessly exploitative oppression, and it destroys people's physical health as well as their intellectual potential.

Maybe you think that, if some people don't spend 40–50 hour weeks destroying their health breathing in construction dust, then other people will have to go homeless, because there won't be enough housing. The simple ratio I gave above should clarify why that isn't the case: one person-month of construction work per lifetime is more than enough to supply everyone with adequate housing, and that's even without 3-D printing in concrete to build new houses. The reason people go homeless is that they're not allowed to build houses, or the government bulldozes their houses (this happened to a group of my friends), or they're socially isolated and have sufficient mental or physical health problems that they can't take care of themselves.

In the US, about 3% of the population raises food full-time. If this labor were distributed across the population as a whole, it would amount to an hour or two per week. Gardening for an hour or two per week is not enough to seriously impact your mathematical productivity, whether you're driving a combine or tying tomato plants to the fence.

The big time commitments — which mathematicians often do end up doing — are child care, elder care, and education. The kibbutzim have shown that it's possible to collectivize these in a useful and humane way; school and nursing-home systems in places like the US have shown that it's possible to collectivize them in destructive and unbelievably cruel ways. The economy and government the non-mathematicians have constructed makes it very difficult to evade these curses, whether or not you are a mathematician.

As for slavery, slavery does not help education; not only is it inhumane and degrading to both the slave and the owner, it's an inefficient way to satisfy material needs. If the serfs had been liberated before Bruno (who had no patron) and Gauss were born, they would have had greater opportunities to educate themselves, not less.

You say that students of mathematics, physicists, engineers, and so on are "non-mathematicians". I'm not entirely in agreement with that but I'm willing to stipulate it for the sake of this conversation. I still don't think there's any reason for mathematicians to care about their opinion. If a student of mathematics thinks they have squared the circle or that differential forms are useless, [other] mathematicians have no reason to care, except that it might be a fun conversation with someone who might grow into a future collaborator, or that the conversation might sharpen their own understanding of differential forms or whatever. Students of mathematics, physicists, and engineers have motives of their own for caring about mathematics, though. I don't think [other] mathematicians have any responsibility for convincing them of that, nor any real benefit in doing so, except, again, insofar as it might result in interesting new mathematics. But if physicists by and large don't care about category theory (though I've met at least one exception), that doesn't really affect the category theorists, much less the consistency or beauty of category theory. Certainly category theorists wouldn't be doing themselves any favors by writing grants that pretend they're doing some kind of industrial R&D.

I don't think of mathematicians as saint-like figures levitating above the unwashed masses. I think of the unwashed masses as totally depraved screaming degraded chimpanzees throwing turds, and the mathematicians as slightly less depraved, screaming a bit less, slightly less degraded, but still basically chimpanzees.