Question: why did ogres leave the horde?
Answer: Various reasons, for starters, the ogres were used as a kind of military police to force order on the Orcs and keep them in line by Guldan. After thrall made his reforms the ogres weren't welcome. As for the Trolls, after the old horde was broken they saw it in their best interests to run and abandon the horde.