It isn't always speculation and doesn't have to be speculation. Here's a model of what that might look like:

I buy a house for $500,000. I know property taxes + repairs/depreciation + the amount of time I spend interacting with tenets will cost ~3-4% the cost of my property per year. I know with the stock market, I can average returns of around 8%. So, in order to make the real estate property attractive, I need to rent it out for ~12% of the purchase price per year or since we usually discuss rent in a monthly rate, I need to charge tenets about 1% per month. In this example, $5,000 per month.

Where I live, apartments are selling for $500,000 but rent for similar apartments are closer to $2,000-$2500. Not $5,000. So, the owners of the property are getting 6% of the property value in rent per year which minus the 3-4% costs gives them a net profit of 2-3% the cost of the property. Why would anyone do this? Some might have bought the property at a lower value but they could always sell and invest in something more lucrative if they wanted. From an opportunity cost perspective, they aren't much different than someone buying in at the higher rate. So, from an actually logical viewpoint (let's not claim real estate investors are just too stupid to realize what they're doing), I see 2 possibilities:

1. They're speculating that the price will go up. My area has seen housing prices go up an average of 5% per year for the last 30+ years. If you are getting 2-3% ROI from rent and 5% ROI from housing price increase, that's 7-8% ROI which isn't bad. You're claiming real estate isn't speculative so let's take a look at the other option I can imagine to see if it paints a better picture.

2. They're highly leveraged. If you have 20x leverage at an interest rate of 2.5%, and subtract your interest payments from the 3% you made, you get 0.5% of the property value in profit when you only invested 5% giving you a 10% ROI. Again, not too bad a deal for the landlord. Different risk profile than stocks but about the similar returns. Unfortunately, this situation isn't very flattering to landlords IMO. In that case they're basically taking advantage of the fact that they have access to cheap credit and their tenets don't. That cheap credit by the way only existed because of the federal reserve and we'll all be paying it back in some form or another (possibly through high inflation rates possibly some other way).

I'm not saying landlords are evil or anything. This is the game that our government created and landlords are just playing the game. I am however saying that I don't think this game is good or sustainable and maybe our government should change the game. If that requires screwing over the people that have bought housing in the last few years, so be it. I'm not going to feel too bad for either the person that has been speculating on housing or the person that over leveraged themselves. If you aren't either of those people, housing prices collapsing shouldn't scare you that much. If you noticed, my argument is based on rent being too low in comparison to house prices. I can absolutely see a situation where home prices drop a significant amount but rent prices stay about the same. That scenario shouldn't hurt you if you aren't in either of the camps I listed above.