I've toyed with a sci-fi novel, and while "worldbuilding" what quickly becomes clear if you're going to do "space opera" type environments that are even somewhat realistic is that you have relatively few sets of choices that drastically constrain your setting.

You're right there's a lack of imagination there, but it's also limited by the setting.

E.g. you have to choose FTL or no-FTL. If you choose no-FTL you're basically limited to a single solar system at most, so whether or not you believe FTL may become possible, for a space sim it's pretty much a requirement.

You need to rule out the singularity or people opting to "upload" to a computer in any significant proportion of the population.

You also have to decide if you believe scarcity will still exist. If you decide we'll all automate everything, then the only route to scarcity is that someone is hoarding everything or is running out of raw materials.

That's a potentially interesting setting, but has significant impacts on space sim type trade: Nobody would be hauling [insert whatever] manufactured goods from Earth to Mars if it can be manufactured by robots right there. Refined materials, maybe, or people. Mining asteroids, certainly. But not shipping foods of all things, for example. But such a setting would also need e.g. significant costs of raw materials and difficulty obtaining tech to automate manufacturing, or the chokehold would slip fairly quickly.

If you rule out automation you've made a choice that's decidedly anti-tech: it presupposes technological stagnation at some point, or that it is slow enough that scarcity is still there at whatever point in time you've chosen, in which case you accordingly need to reflect that in an environment that does not have all kinds of super-fancy technology.

Most space sims end up in that last category: Automation, if it exists, is limited and doesn't affect manufacture and trade enough to be worth mentioning to the point where there are still arbitrage opportunities for small ships; FTL exists, but most other tech seems to have largely stagnated.

Part of the problem, I think, is that these choices makes it easiest to create a frontier feel, and space sims tends to very much be westerns in space (incidentally, "space opera" is derived from "horse opera": a term for a very cliched western).

Exceptions mostly come when you add story elements that let you add e.g. large capital ships and the like and armed forces, such as games like Terminus, that lets you join military units or be a rebel etc.

Thinking outside that box and still have a compelling story is difficult, because it means overcoming a lot of barriers. E.g. let's say you drop FTL. Now we're stuck, unless you find some way of setting your game in the kind of accelerated timeframe that it would take for non-FTL travel to be viable, but hen you also need a setting where that "works" which would be decidedly non-human.

Let's say you admit to people "uploading". Now you need a compelling reason why people stay human, or you'll have a Clifford D. Simak "City" situation on your hands (a bunch of dogs tell each other stories about humans, who have long since left Earth behind, not to "upload", but the effect being the same).

No scarcity? Now you need another mechanism than basic commodities trade.

But the "trade plus occasional piracy" model is very simple. Thinking up variations that doesn't immediately seem flawed and idiotic that solves the above issues is a lot harder.

Don't blame these games for it; blame sci-fi authors who also have not explored this very well, with a few exceptions.

E.g. Iain M. Banks is one of the few authors of what you describe with the "fully automated luxury gay space communism" meme. I'd love to see someone adapt ideas from that into a game. But it also presents a lot of difficulties because his solution to make it interesting is alternatively go up in scale (e.g. the Idiran war, described with destruction of entire habitats housing billions) and down in scale, focusing on specific conflicts involving specific people that might suit e.g. a more complex graphical adventure type game, but that'd be hard to adapt to a space sim. Notably his books have a lot more scenes of people conspiring or doing stuff on the surface than they have of people flying space ships. Because why would you fly a space ship when you can hitch a ride on a Mind controlled ship the size of a city in according comfort? There are reasons, but again they require a lot more thinking and making that work in an open world game as opposed to a story led one would probably be tricky.

One interesting other approach is Frederik Pohl's Gateway. It's set on an alien space station far from earth in a future where humanity does not have FTL, but this alien race that mysteriously left thousands of years ago did have FTL. The lack of FTL + Gateway's distance means there's a lot of scarcity on Gateway irrespective of conditions on Earth. At the same time, Gateway is the literal gateway to a massive number of alien locations via FTL ships that humans haven't figured out to control other than to start them, wait for them to reach their destination, and trigger a return trip. Which means it's basically prospecting where the lucky ones bring back untold fortunes in alien artifacts.

Creative thinking on introducing constraints the way Gateway did is, I think, something that has great potential for more interesting variety in space sims (since I mentioned Banks above, two of his other novels demonstrates this: In "the Algebraist" the setting is a solar system that has been cut off from its surroundings from a century after their local jump gate was destroyed; they're waiting for a replacement sent from the nearest system at sublight speeds - the tech is implied to use entanglement, so they can't just build a new local gate; in "Against a Dark Background" the system is isolated far away from any nearby galaxies, basically crushing any fantasy of interstellar travel)