You'd need an awful lot of Dyson Spheres, and they'd all be in places you wouldn't expect suitable stars at all, and then you'd have to hide them from searches for brown dwarfs and black holes as components of Dark Matter. See https://en.wikipedia.org/wiki/Massive_compact_halo_object for a brief overview.

The killer problem is that stars bright enough to gather up energy via Dyson Sphere mechanisms aren't seen around any of the many thousands of galaxies we've looked at. Moreover, in order to account for an appreciable fraction of the halo masses, there would have to be at least hundreds of millions of Dyson Spheres for every galaxy in the sky. The Dyson Spheres would also have to be very very old because, thanks to gravitational lensing by foreground galaxy clusters, we have decent enough views of galaxies billions of light-years away that we would see large numbers of not-yet-Dyson-Sphered stars well outside galaxies. The most distant spiral-type galaxies in particular would look very different than much closer ones, and that is not the case. Have a look for yourself: http://legacysurvey.org/viewer#IC%203556

Bright stars only form in regions where the dust and gas is sufficiently dense to collapse gravitationally; those regions are generally found only within galaxies or in "low points" of gravitational potential within clusters of galaxies (the dust and gas gathering in such points may form a new galaxy). Dust that doesn't fall into galaxies or into cluster "low points" becomes too diffuse to form even the dimmest stars.

By comparison, dark matter tends not to fall into galaxies or cluster low points because collisions between dark matter particles, or between them and ordinary matter, does not produce radiation. Radiation from particle collisions carries off the angular momentum that keeps the particles on higher orbits. Ordinary dust and gas collisions release light (and radio and so on) which enables gravitational collapse into stars. We can see the light (and radio and microwaves and so on) with telescopes.

This tendency of dark matter to stay on high orbits well above the luminous matter in galaxies leads to a greater angular momentum than expected in the outer reaches of the luminous matter. That angular momentum excess is how dark matter was discovered. Why don't the fast-moving stars and dust fly away from their galaxies? Because dark matter throughout the galaxy adds more mass than is seen electromagnetically, so there is a sufficient tendency to fall inwards counteracting the excess momentum. Why doesn't the fast-moving luminous matter collide more often, throw off more radiation, and collapse into the centres of galaxies more rapidly? Because the shell of dark matter well outside the bright parts of galaxies holds it up, somewhat like a scaffold. It's a balance, but not an especially fine one; it seems to be pretty generic for a collisionless, cold, non-self-interacting (except by gravitation), non-radiating, microscopic dust in which the visible components of galaxies are immersed.

Let's ignore the other responders' concerns about not being able to keep the Dyson Spheres as cool as the background to all observers (how would a Dyson Sphere builder in a galaxy 100 million light years away know that we might be looking from our galaxy, and so decide to cloak themselves only from us?). Let's also ignore whether such things would be transparent, or would occlude optical emissions from the galaxies they surround. We can't however ignore their gravitational influence: these Dyson Spheres have to "hold up" the non-Dyson-Sphered stars they surround, to stop their host galaxies from collapsing. You can't do that with only a few Dyson Spheres, and as you add more, you'd expect to see more small gravitational lensing distortions.

Although our telescopes aren't good enough to probe many galaxies, there are no such (gravitational lensing) distortions around our own galaxy, or Andromeda, and yet both galaxies' luminous matter's behaviour essentially requires an abundance of dark matter.

(There also isn't a dust eclipsing distant sources of light, which we would expect for a very dim but not exactly invisible Dyson Sphere would; there also isn't a large number of pointlike sources of deep infrared or radio in our or Andromeda's galactic halos, which one would expect if a Dyson Sphere builder were extremely energy-efficient. We also don't see excited dusts and gas around galaxies suggesting accidental "painting" with lasers or the like being shot out of Dyson Spheres if their operators decided not to radiate unused energy in every direction).

Finally, there is a hierarchical problem here: not only does dark matter surround galaxies and affect their structure gravitationally, it also surrounds whole clusters of galaxies. Without lots of dark matter hanging around waaaaay outside galaxies within a galaxy-cluster, clustering would be very different from what we see in the sky. Moreover, we know from Einstein lensing that the bulk of the magnification done by clusters is not traceable to the luminous matter in the galaxy cluster.

So you'd need to have lots and lots and lots of invisible Dyson Spheres many tens of thousands of light years outside galaxies. By lots, for a cluster similar to ours, this would mean a few hundred trillion Dyson Spheres, all very carefully engineered and positioned to mimic cold dark matter in areas where stars are not naturally seen. Also all evidence of towing stars, or generating new stars from scratch, has to be hidden from view in tens of thousands of galaxies over a span of a billion or more years, and somehow no galaxies known are older than the completion of the Dyson Sphere project, or somehow undisturbed early galaxies look identical to much more recent galaxies.

The engineering challenges would be outright cosmological, not just astronomical.

Wanna scare yourself with sci-fi thoughts? If this type of engineering is all done by a single culture, what would make them want to hide so carefully?