There are a few elements here, one of which geoelectric addresses in an earlier comment: does the crowd have expert knowledge.

Another is whether or not the question at hand requires expert knowledge. There are times when you're better off handing off the helm or pilot's seat to a qualified pilot than trying to average inputs of a large crowd, or to allow an electrician or plumber to address a problem within their skill scope.

A third aspect though might concern what the negative cost functions of a crowd might be.

The wisdom of crowds concept generally assumes the larger the crowd, the better it will be at arriving at some truth. In reality, various biases, distortions, and manipulations can emerge, to the point that the crowd's view is far worse than other options. Aristotle drinks hemlock. Trump is presumptive nominee. Brexit.

An element of networked systems, including decisionmaking systems, is what their cost functions are, in the sense of imposing negative results on those participating. I've been arguing for a year or two now that there is such a cost function, and that you can estimate that by noting the maximum size an effective network can grow to. Conversely, you can increase (or decrease) the effective size of a network by addressing that cost function. Increase it and you'll make large-scale aggregation less viable. Decrease it, and you can increase the size of effective aggregation.

As examples, a village is constrained in total size not only by its ability to secure necessary inputs (especially food and water), but in its ability to dispose of wastes and noxious emissions. London of the late 18th century had a mortality rate above its natural birthrate, and the only way the city could maintain its population was through net in-migration from the countryside (or foreign lands). This wasn't materially addressed until revolutions in water provision and sanitation, including the first modern sewerage system around 1850, addressed such concerns as cholera epidemics which were killing as many as 50,000 people a year.

In programming, Fred Brooks' The Mythical Man Month notes that few programming teams scale well beyond about 6-12 developers. The inter-personal communications costs make larger groups not only inefficient, but less effective, net than smaller ones. To produce larger teams, you've effectively got to split them into smaller ones. That's among the things that a highly modularised development process as is common in Free Software projects achieves -- see Apache, the Linux Kernel, or the Debian Project as examples (Gabriella Coleman, now of McGill University, wrote her dissertation on this topic, it's fascinating reading).

Computer chip design essentially removes the space and resistance costs of crowding high densities of electronic gates in small spaces. Again, the cost function is reduced.

In email and traditional (POTS and mobile) phone service, increasing amounts of spam are increasing cost functions, reducing the appeal and utility of the network to all involved. POTS has been shedding subscribers for some time, my expectation is that mobile phone service itself will be as well, more especially if interconnects, and filtering of VOIP alternatives (including iChat, Google voice chat, Skype, etc.) are further developed. Those networks, as a Long Island friend of mine some time back said, "gotta leahn to tawk to each othah!