While I can't claim to be an expert in _why_ the FAA chose to structure the regs the way they do, I have some personal thoughts on the matter. dweekly has a great comment explaining Part 91, 135, and 121 operation rules that I'll reference here.

A Part 91 flight conducted by a PPL with passengers certainly isn't the safest activity the pilot or passengers could partake in. But its certainly enjoyable. Let's take a similar example. A hobbyist motorcycle racer owns a private track, and a friend wants a joy-ride. Similarly to a passenger on a Part 91 flight, a passenger on a high speed sports bike is at higher risk of fatality than a person who decided to spend the day relaxing at the beach. But doing so on private property with little/no risk to non-participants is not illegal. Many/Most high-risk activities are not illegal provided that the risk is only to the willing participants. While I can't be sure that's the reason the FAA allows low-hour non-CPL pilots to carry passengers, I would imagine it is a large contributing factor considering the statistical probability of fatality is with the pilot and passengers, not people in other planes or on the ground (to the best of my knowledge).

Part 135 and 121 operations have the license requirements such that people purchasing services have a reasonable expectation that the pilots involved have been properly trained, and are VERY safe.

Now to the kicker. Why are private pilots not allowed to accept ANY compensation? Really, NO COMPENSATION. Although I haven't personally heard of anyone getting their license revoked for something as little as paying the full cost of expenses, it is against the regulations for your passengers to pay more than half the pro-rata share of expenses (rent, fuel, etc...). This is a hotly debated item, that no one seems to have a good answer for. The most convinving argument I've heard is that it goes back to the late '20s when the regs for commercial pilots started going into place. The history around this is shaky, but supposedly the commercial license came about as a way to track interstate and foreign commerce. Supposedly the government at the time was scared of the ability to circumvent taxes since they couldn't as easily stop and check trucks or trains crossing borders. So they created the commercial license as a way of regulating inter-border trade to trusted pilots. I don't have a source on that unfortunately - more of an oral tradition story. But it makes the most sense to me of everything I've heard.

Another rabbit-hole I won't go too deep into is the issue of insurance. Getting insurance on an aircraft is much more expensive if it will be used for compensation purposes (Part 135 or 121). Even if a private pilot were able to get compensation from passengers or for cargo, the insurance rate would be astronomical(as a commercial rotor wing pilot, I can't find a job doing anything but instructing until I hit ~1000 hours due to insurance rates, if an insurance company would even let me fly on a commercial bird before then). And if you got in an accident while providing paid-for services without proper (for comp)insurance, it would not pay-out anything.

tl;dr You can have PPL w/ passengers because you're mostly only endangering yourself and willing passengers. Commercial operations are limited to using CPL/ATPL pilots such that consumers have a reasonable guarantee of safety. And nobody knows why private pilots can't be compensated (although if someone has a source with concrete answer I would love to hear it). Even if they could legally, they financially couldn't due to insurance costs.