There is a significant risk in general, but it depends on your housing type. Radon's half-life is 3.8 days, so roughly only the radon produced by the uranium in your house in the last week or two is of concern; it can't build up over longer periods of time. But if there are areas that are sealed well enough that radon can accumulate for days, but not sealed so well that radon is sealed inside the wall until it decays, you could have a high level in the air.

There are some minerals that have relatively high concentrations of uranium, but no rocks are completely devoid of it. Consequently any rock, sand, dirt, or concrete has some amount of U-238, which decays (through the four or five intermediate steps of the so-called radium series) to radium-226, which produces radon-222. Radium-226 has a half-life of 1600 years, so if you were to remove all the radium from a rock, it would stop emanating radon and take millennia to ramp back up to its previous radon emanation level. But the reason we use these natural minerals for construction in the first place is that they're cheap, and this is not cheap, so we don't do it.

Radon decays through a variety of seven radioactive steps, six of which are ridiculously more radioactive than radon itself (radioisotopes of lead, bismuth, and polonium). The reason we worry about the radon instead of its even more radioactive products is that it's chemically inert, so it just floats around as a gas instead of staying put in the rock or settling out of the air, and so when it finally breaks down into polonium, it might happen to be inside human lungs, and that's where the polonium atom is going to stay, giving off three alpha rays and two beta rays over the next hour or so, until it decays into comparatively harmless radioactive lead.

Alpha rays are normally absolutely harmless to the humans because a few microns of anything solid will stop them, and the human skin is covered in tens of microns of epidermis cells that have no nuclei or blood supply and therefore can't get cancer. And beta rays are only a little more penetrating. But inside the lungs it's a different story, and that's why so much lung cancer is caused by radon.

Because radon's half-life is so short, and the resulting decay chain effectively multiplies its radioactivity by 5, it's almost inconceivably more radioactive than the radium that emanates it: a mole of radium-226 would be 226 grams, containing 6×10²³ atoms, and if I'm doing my calculation right that's 8×10¹² becquerels (decays per second). A mole of radon-222 weighs 222 grams and by the same calculation produces 1.3×10¹⁸ becquerels itself, 6.4×10¹⁸ becquerels if you include the next four steps in the decay chain. (Calculation in units(1): avogadro mol / (3.8 days / ln(2))). That's nearly a million times as much radioactivity per gram, and radium is already no slouch in that department.

Of course, in any given year, the amount of radon that decays in your house is almost exactly the same as the amount of radium that decays (since almost every radium atom that decays becomes a radon atom, which decays within a week or two), so the radon only produces 5 times as much radioactivity as the radium. If it stayed inside the wall instead of gadding about in human lungs it wouldn't be an issue.