I'd just commented on this at HN a few days ago[1], noting both that a fair amount of "big data" is actually surprisingly small, and that sampling-based methods are very, very, very often not only good enough, but tremendously reduce costs and complexity. There are other factors, some of which favour the smaller-is-better approach, some of which favour more comprehensive approaches.

Background: though my pseudonymity precludes giving specifics, I've worked in and around data analytics for much of three decades, across various fields. What was "big data" at the onset of my career -- when multiple departments of over thirty analysts shared access to a few GB of storage distributed over multiple cluster systems -- is now something I can trivially tackle on a modest and decade-and-a-half-old desktop system. Newer boxen, with yet more memory and SSD or hybrid drives are even more capable.

Under my present 'nym, I've done analysis especially of Google+ since 2015 (largely out of personal interest and realising methods were available to me). Much of that has been based on sampling methods, some has relied on more comprehensive analysis, though again, with an eye to limiting total data processing.

In the case of Google+, the questions of how many actively-posting users and how many active Communities exist have come up. Sampling methods have been reasonably useful in assessing these.

Google's sitemaps files (from https://plus.google.com/robots.txt) supplied total counts of profiles, communities, and several other categories. This itself takes some work -- there are 50,000 Profile sitemap files, and 100 Communities files, containing 3.4 billion and 8.1 million entries, respectively. Even downloading just the Profile sitemaps itself is roughly 37 GB of data.

For a rough estimate of active profiles, it only took about 100 randomly sampled observations to come up with pretty solid value: about 9% of profiles had posted publicly ever (a ratio which remains true). Finer-level detail for more frequently posting accounts takes a far larger sample because those highly-active profiles are so much rarer. I'd done my first analysis based on about 50,000 records, while Stone Temple Consulting independently confirmed and extended my analysis to find an estimate of profiles posting 100+ times per month (about 50,000 across the full site). That is, 0.0015625% of all G+ profiles extant at the time (about 3.2 billion).

When I first tackled the G+ Communities question, I was trying to get a sense of:

1. How many there were.

2. What typical membership was.

3. How many were "reasonably active", based on ... somewhat contrived measures.

I adopted a sampling approach -- it was easy to pull a full listing of Communities, but web-scraping these would take time (at about 1.5s per URL). So my first approach utilised a 12,000 record sample. That gave a good overall view, but turned out to be thin on the very largest communities, so I ran a second pull based on 36,000 records, also randomly sampled. (A test of sampling: how many communities should be in both samples. The expected and actual result matched: 53.)

Even then, representation from the very largest communities was thin. Fortunately, I found someone able to sample all 8.1 million communities rapidly, and from this compiled a list of 105,000 communities, with 100+ members and posting activity within the 31 previous days (based on Jan 5-6, 2019).

Note that this highlights another aspect of media: of all 8.1 million communities, only 1.3% fell into the active list. That is media and community activity tend to focus most ACTIVITY on a VANISHINGLY small subset of total members, groups, posts, or other entities. That is, attention is highly rivalrous, and is exceedingly unequally distributed.

The challenge is in finding the active entities. Given my approach, via sitemaps, this was difficult (though a few options may have been available).

The upshot is that in the case of both users and communities, the really active and interesting set is, by modern standards, small data. There are 50,000 to a few million active users, a few thousands (and far fewer than 100,000) truely active communities. Even the total post volume of G+ communities is, by modern standards, strikingly small -- from January 2013 to January 2019, about 300 GB of post text was submitted. Image content and the surrounding Web payload (800 kiB per post) inflate that considerably, but yes, over half a decade of a reasonably large social media's community discussions could sit comfortably on most modern hard drives, or even much mobile storage.

I'll skip over the questions of truly social analysis of such data, with connections between users and other entities, though I'll note that even for very large systems, managing these at scale is difficult and the results may not be particularly usefully interpreted. Combinatorial maths remains challenging, even with beefy iron.

________________________________

Notes:

1. See: https://news.ycombinator.com/item?id=19294024 and https://news.ycombinator.com/item?id=19294004, on the often surprisingly small scale of "big data" and the value of sampling, respectively.