> why would I even want to talk to a physicist

I dunno.

> degrade towards lower entropy

Entropy is increasing, justifying our everywhere-observed second law of thermodynamics. The observable universe is the ultimate closed system, much more so than any laboratory, or our planet, even or our galactic cluster (which, after all, shines brightly). The expansion of the universe is as far as we can measure an adiabatic process.

Although when one thinks of increasing entropy one usually thinks about structures dissipating away from something well-ordered, like a book, to something much less well-ordered like ashes and gas when you burn the book, the universe's entropy is growing somewhat differently. Visible structure is mostly increasing in complexity in the few billion years on either side of "today": gas clouds are collapsing into stars, some with planetary systems, and so on, even as the universe's entropy is growing enormously. How can that be?

The largest sources of entropy growth in the known universe are empty space and black holes. Space has enormous entropy: you can take a million cubic centimetres of space (one cubic metre) and rearrange them in an enormous number of practically identical ways. By comparison, if we start moving cubic centimetres around within you, like a cm^3 of air in your lung and a cm^3 of your heart, most of the time that kills you fairly quickly. Entropy is thus a relationship between the contents of a volume and the number of ways we can rearrange those contents and not make a significant difference: the more ways we can safely rearrange, the higher the entropy.

Modern CPUs are very low-entropy; a sandy beach is much higher entropy. The space between Earth and the Moon is much higher entropy still; and the space between galaxy clusters has enormous entropy. The metric expansion of space creates more space, and that space is also emptier. Thus, most entropy in the universe is being created far from galaxies.

Black holes hide what's inside them according to the no hair conjecture; even if the conjecture is not precisely true, it does an excellent job of hiding what has fallen into a black hole over billions of years, and likely will keep hiding everything but its macrostates of mass and angular momentum (the various possible charges, like the electric charge, tending to neutralize quickly) for trillions of years to come. Looking at a black hole we cannot tell if it was formed from just hydrogen, or hydrogen with some helium, or hydrogen and helium and other elements. We can't tell if it was just gas, or gas and dust that fell in. Because we can make so many changes to what has fallen into a black hole without changing the macrostates of the black hole, the black hole has very very high entropy. The bigger the black hole, the higher its entropy, and probably all black holes are currently growing.

By comparison with a growing black hole or expanding empty space, the change in entropy of the Earth over time is small. Likewise, while the black holes in the Milky Way have enormous and growing entropy, the overall entropy of the galaxy changes very slowly: there is a lot of mass in gas and dust that has yet to collapse into structures like binary star systems, much less into black holes.

If we take the second law of thermodynamics seriously at a cosmic scale and consider any stage of the earlier universe, it must have had lower entropy than later stages. Indeed, we can get that just by removing some of the expanded space, and from shrinking black holes (and removing black holes that formed well after our star did). But in the much denser and hotter universe, the entropy would in this view be very very small.

Items with tiny entropy are pretty rare. Human brains are much higher entropy than a comparable volume in the early universe. Things with higher entropy are much more common than things with lower entropy, in our universe. Grains of sand are much more common than human brains or microchips.

This leads us to fluctuation theory. Even very high entropy systems have small fluctuations to lower entropy within them. Statistical mechanics lets us define a probability of fluctuating from entropy n to lower entropy m. Such a fluctuation is a sudden spontaneous local increase in complexity. This probability falls off very quickly when the difference in n and m grows: a fluctuation into something simple is enormously more likely than a fluctuation into something complex.

Fluctuating from practically empty space to space with a low energy photon in it is of much higher probability than fluctuating from practically empty space with a proton in it. An atom or molecule has even less entropy, so the probability of fluctuating from practically empty space to a gas of atomic or molecular hydrogen is much smaller than fluctuating into a gas of photons. Fluctuating into a computer or into a brain (not necessarily a human one!) is extremely improbable by comparison, because those have lower and lower entropy still. But the entropy of the chip or brain is huge compared to the entropy of the extremely orderly hot dense phase of the universe some 13.5+ billion years ago.

Consequently, theories that cause universes to appear by fluctuating out of a high-entropy state -- say, a new baby universe arising in the extremely far future of our own universe when there is almost nothing but empty space everywhere -- have to suppress fluctuations into relatively high entropy configurations like a handful of photons, or sparse atomic/molecular gas. Human brains are still relatively high entropy compared to the hot dense early universe.

More generally, a proposed cosmology that suffers "an invasion of Boltzmann brains" is probably unphysical, because such a cosmology would also suffer an even more obvious invasion of much higher entropy dusts and gases, and we do not see that happening in intergalactic space.

One could easily substitute "apples" or "Apple Computers" or just random hydrocarbon molecules. Brains are just catchier, because brains are obviously much more complicated -- and thus much lower entropy -- than soot, although both are almost the same entropy compared to the early conditions of our own universe.

Arguments about what a Boltzmann brain thinks are misplaced. The point is that there isn't Boltzmann soot, much less Boltzmann brains, so we can bet that our early universe likely did not appear as an extremely low entropy fluctuation in an extremely high entropy vacuum.
We can also bet that our future universe is extremely unlikely to "give birth" to anything as low-entropy as its own early phase. It will be cold and sparse and fluctuations in the increasingly empty space probably won't even rise to the level of ELF radio, much less soot or brains.

> fine tuning

is a somewhat different concept. There is a measure of entropy even given only one physical universe to examine; it's the log relationship between macrostates like galaxy clusters and microstates like the contents of cubic micrometres throughout the universe. There is no measure of fine tuning within one physical universe. If you had lots of universes to look at you could build a distribution of some parameter and consider whether that parameter in one chosen universe was finely tuned compared to the others.

> the universe is stable

It's not. It has a finite age with extremely distinct eras including one dominated by radiation, one dominated by matter, and later ones where the matter on the largest scales is increasingly diluted away. On smaller scales, matter is condensing chemically and collapsing gravitationally. The universe of the past looks remarkably different from the universe close to now, and the universe of the future is virtually certain to look even more different.

> the rules could change at any moment

Some rules, like those of electrodynamics or the Standard Model are known to change at extremes of energy already; we can show this in laboratories and have ample observational data from supernovae and their remnants. That fluctuations from ordinary energies to extremes of energies don't seem to happen anywhere we have looked is an argument against Boltzmann brains, true, but doesn't really say anything about the global applicability of our most fundamental theories (General Relativity and the Standard Model).

> You are a Boltzmann brain floating in mid space arguing for empiricism

I think you have to add the constraint that the brain in question is hallucinating that it is something else.

However, the exact nature and experience of a Boltzmann brain is not really physically (as opposed to philosophically) relevant. What's relevant to phsyical cosmologists is that a Boltzmann brain is much lower entropy than empty space (or even giant molecular clouds), but much much much higher entropy than the earliest part of our universe reasonably describable with General Relativity.