A larger random sample reduces error by the square root of the sample.

If the sampling is biased, however, all bets are off.

A 100,000 observation poll has 100 times the costs of a 1,000 poll (data must be collected from 100,000 times more samples), but offers only 10x greater accuracy.

You'll find this mathematically in the definition of standard deviation, which divides by the square root of the sample size: sqrt(1000) ~= 31.6, sqrt(100000) ~= 316.

Larger random samples are useful where you're exploring many variables, or very small portions of the population. They afford greater precision. The accuracy however is dictated by the randomness.

I had a strong lesson in this a few years back when the question of active user participation in Google+ came up. I'd had one too many hand-wavey assertions that the site was far more active than was generally claimed in the press. I'd realised that G+ had a set of sitemaps files, and included in those were sitemaps of individual Google+ user profiles. Present on the web results for that URL was an indication of whether the profile had never posted publicly at all, or, if it had, what the most recent publicly-posted content was.

Each sitemap file had roughly 50,000 entries. There were something on the order of 40--50,000 profile sitemap files, about 25 GB in all.

I took a gamble and made the assumption (later tested and largely validated) that profiles listed within a given sitemap were themselves a random assortment. This checked out on any number of eyeball analysis (creation dates, user names, global regions, and activity status all seemed both random and uniform over a few tested files). So I selected one sitemap file (itself at random) and over the course of a few days using a pretty modest laptop and broadband connection pulled down and web-scraped some 50,000 entries. A simple pattern match told me whether or not the profile was active, and when.

Within the first 100 profiles viewed, the trend was very clear. Only about 9% of profiles seemed to have ever posted any content. That percentage varied between roughly 7--12% initially, but rapidly converged as my dataset grew.

I let the run continue regardless. I resampled my sample subsetting it variously ("Monte Carlo estimation") to see if the values varied (I believe either 60 or 100 record subsamples), and again, the same 7--12% or so was returned for each. Looking at recent activity (within the month during which the analysis was performed), was 0.3%. The analysis also revealed just how much the forced integration of YouTube and G+ had inflated G+ activity numbers (a bit over 1/3 of all most-recent activity).

This generated some blow-up on G+ among members there, and I got called a few things, as happens. Google themselves never formally responded (I did hear from a few Googlers who contested findings or methods.) A few months later, an Internet marketing group, Stone Temple Consulting, re-ran the analysis based on my methodology but on a 10x larger sample (500k profiles), selected from across a much larger set of the sitemaps. They fully confirmed my own headline numbers, though could (thanks to their larger sample) offer more precise insights on smaller groups within the overall population. I had absolutely no participation in the follow-up study, and was unaware of it until it was made public.

Stealth edit/update: Then as now, what annoyed me most about the whole episode was that Google were so obviously dissembling, making up numbers and/or outright lying about Google+ activity, and the press were largely lapping it up, when a very modest investment of time and effort would put paid to the lie. And after I'd done the analysis, armchair warriors continued whinging even as I'd fully documented methodology and tools used, enabling anyone to replicate and confirm or deny findings. Eric Enge of Stone Temple was the only one to do that. I don't generally hold marketers in high regard, but he earned respect from me in doing that.

https://ello.co/dredmorbius/post/naya9wqdemiovuvwvoyquq

https://blogs.perficient.com/2015/04/14/real-numbers-for-the...

(Stone Temples has since been aquired by Perficient.)