Online activities can be associated with dramatic offline effects, such as COVID19 misinformation leading to panic buying of toilet paper 1 , online narratives incorrectly attributing Australia's "Black Summer" bushfires to arson amplifying attention to it in the media [1] , and attempts to influence domestic and foreign politics [2, 3] . For researchers to successfully analyse online activity and provide advice about protection from such events, they must be able to reliably analyse data from online social networks (OSNs).
Social Network Analysis (SNA) facilitates exploration of social behaviours and processes. OSNs are often considered convenient proxies for offline social networks, because they seem to offer a wide range of data on a broad spectrum of individuals, their expressed opinions and inter-relationships. It is assumed that the social networks present on OSNs can inform the study of information dissemination and opinion formation, contributing to an understanding of offline community attitudes. Though such claims are prevalent in the social media literature, there are serious questions about their validity due to an absence of SNA theory on online behaviour, the mapping between online and offline phenomena, and the repeatability of such studies. In particular, the issue of reliable data collection is fundamental. Collection of OSN data is often prone to inaccurate boundary specifications due to sampling issues, collection methodology choices, as well as platform constraints.
Previous work has considered the question of data reliability from a sampling perspective [4] [5] [6] [7] , biases [8] [9] [10] [11] and the danger of making invalid generalisations using "big data" approaches lacking nuanced interpretation of the data [12, 13] . Analyses of incomplete networks exist [14] , but this paper specifically considers the questions of data reliability for SNA, considering not only the significance of online interactions to discover meaningful social networks, but also how sampling and boundary issues can complicate analyses of the networks constructed. Through an exploration of modelling and collection issues, and a measurement study examining the reliability of simultaneously collected, or parallel, datasets, this multidisciplinary study addresses the following research question:
How do variations in collections affect the results of social network analyses?
Our work makes the following contributions:
1. Discussion of the challenges mapping OSN data to meaningful social and information networks; 2. A methodology for systematic dataset comparison;
3. Recommendations for the use and evaluation of social media collection tools; and 4. Two original social media datasets collected in parallel, and relevant analysis code 2 .
This paper continues with four sections: 1) challenges obtaining and modelling social networks from OSN data for SNA; 2) systematic parallel dataset comparison methodology; 3) results from using our methodology in a two-phase case study; and 4) recommendations for social media researchers and analysts, and directions for future research.
Using SNA to explore social behaviours and processes from OSN data presents many challenges. Most easily accessible OSN data consists of timestamped interactions, rather than details of long-standing relationships, which form the basis of SNA theory. Additionally, although interactions on different OSNs are superficially similar, how they are implemented may subtly alter their interpretation. They offer a window onto online behaviour only, and any implications for offline relations and behaviour are unclear. Beyond modelling and reasoning with the data is the question of collection -accessing the right data to construct meaningful social networks is challenging. OSNs provide a limited subset of their data through a variety of mechanisms, balancing privacy and competitive advantage with openness and transparency.
SNA provides concepts and tools to model social networks among actors and using network-based techniques to study social behaviours and processes [15] . Given the availability, nature and structure of much OSN data, they offer unique opportunities to apply SNA. Relationships between social network actors must be stable, though they may be dynamic [16] [17] [18] . On social media, accounts can easily fulfil the role of actors, but precisely what constitutes a relationship is unclear. A common semi-permanent candidate is the friend or follower relationship common to most OSNs, but, due to how OSNs present their specific features to users, each online community develops its own interaction culture. Therefore, such connections do not necessarily easily translate between OSNs. Is a Facebook friend ship really the same as a follow on Twitter, even if reciprocated? And how do each relate to offline friendships?
OSNs offer ways to establish and maintain relations with others. Specific interactions may be visible to different accounts, intentionally or incidentally (cf. replying versus using a hashtag). Exploration of these differences may lead to an understanding of the author's intent and the identity of the intended audience. Is replying to a politician's Facebook post a way to connect directly with the politician, or is it a way to engage with the rest of the community replying to the post, either by specifically engaging with dialogue or merely signalling one's presence with a comment of support or dismay? A reply could be all of these things but, in particular, it is evidence of engagement at a particular time and indicates information flow between individuals [19] . Since most interactions are directed they offer an opportunity to study the flow of information and influence. On the other hand, although friend and follower connections can indicate community membership, they obscure the currency of that connection. Through their dynamic interactions, a user who liked a Star Wars page ten years ago can be distinguished from one who not only liked it, but posted original content to it on a monthly basis. Therefore, we specifically focus on interactions rather than membership relations in this study.
Relationships between individuals in a social network may last for extended periods of time, vary in strength, and be based upon a variety of factors, not all of which are easily measurable. Because of the richness of the concept of social relationships, data collection is often a qualitative activity, involving directly surveying community members for their perceptions of their direct relations and then perhaps augmenting that data with observation data such as recorded interactions (e.g., meeting attendance, emails, phone calls). It is tempting to believe that this richness should be discoverable in the vast amount of interaction data available from OSNs, but there are issues to consider:
1. links between social media accounts may vary in type and across OSNsit is unclear how they contribute to any particular relationship;
2. what is observed online is only a partial record of interactions in a relationship, where interactions may occur via other OSNs or online media, or entirely offline; and 3. collection strategies and OSN constraints may also hamper the ability to obtain a complete dataset.
Although many interactions seem common across OSNs (e.g., a retweet on Twitter, a repost on Tumblr and a share on Facebook), nuances in how they are implemented and how data retrieved about them is modeled (beyond questions of semantics) may confound direct comparison. For example, a Twitter retweet refers directly to the original tweet, obscuring any chain of accounts through which it passed to the retweeter [8] . There are efforts to probabilistically regenerate such chains [20, 21] , but, in any case, is one account sharing the post further evidence of a relationship? What if it is reciprocated once, or three times? What if the reciprocation occurs only over some interval of time? These questions require careful consideration before SNA can be applied to OSN data.
Social media data is typically accessed via an OSN's Application Programming Interfaces (APIs), which place constraints on how true a picture researchers can form of any relationship. Via its API an OSN can control: how much data is available, through rate limiting, biased or at least non-transparent sampling, and temporal constraints; what types of data are available, through its data model; and how precisely data can be specified, through its query syntax. Many OSNs offer commercial access, which provides more extensive access for a price, though use of such services in research raises questions of repeatability [8] . This is done to protect users' privacy but also to maintain competitive advantage. Researchers must often rely on the cost-free APIs, which present further issues. Twitter's 1% Sample API has been found to provide highly similar samples to different clients and it is therefore unclear whether these are truly representative of Twitter traffic [6, 7] . Studying social media data therefore raises questions about the "the coverage and representativeness" [5, p.17] of the sample obtained and how it therefore "affects the networks of communication that can be reconstructed from the messages sampled" [5, p.17] .
Empirical studies have compared the inconsistencies between collecting data from search and streaming APIs using the same or different lists of hashtags. Differences have been discovered between the free streaming API and the full (commercial) "firehose" API [4] . There is general agreement in the literature that the consistency of networks inferred from two streaming samples is greater when there is a high volume of tweets even when the list of hashtags is different [5] . More concerning is the ability to tamper with Twitter's sample API to insert messages [10] , introducing unknown biases at this early stage of data collection [9, 11] .
Taking a purely "big data" approach can also lead to inappropriate generalisations and conclusions [12, 13] . This is well illustrated, for example, by the range of motivations behind retweeting behaviour including affirmation, sarcasm, disgust and disagreement [13] . Similarly, in the study of collective action, there are important social interactions that occur offline [22] . Furthermore, relying solely on observable online behaviours risks overlooking passive consumers, resulting in underestimating the true extent to which social media can influence people [23] .
OSN APIs provide data by streaming it live or through retrieval services, both of which make use of OSN-specific query syntaxes. Conceptually, therefore, there are two primary collection approaches to consider: 1) focusing on a user or users as seeds (e.g., [24, 25] ) using a snowball strategy [26] and 2) using keywords or filter terms (e.g., [2, 3, 25, 27] ). Focusing on seeds can reveal the flow of information within the communities around the seeds, while a keywordbased collection provides the ebb and flow of conversation related to a topic. These approaches can be combined, as exemplified by Morstatter et al. [25] in their study of the 2017 German election: an initial keyword-based collection was conducted for eleven days to identify the most active accounts, the usernames of which were then used as keywords in a six week collection.
Once a reasonable dataset is obtained, there may be benefit in stripping junk content included by automated accounts such as bots [28] . The question of whether to remove content from social bots (bots that actively pretend to be human) depends on the research question at hand; because humans are easily fooled by social bots [27] , their contribution to discussions may be valid (unlike, e.g., that of a sport score announcement bot).
Our initial hypothesis was that if the same collection strategies were used at the same time, then each OSN would provide the same data, regardless of the collection tool used. Consequently, social networks built from such data using the same methodology should be highly similar, in terms of both network and node level measurements. Our methodology consisted of these steps:
1. Conduct simultaneous collections on an OSN using the same collection criteria with different tools.
2. Compare statistics across datasets.
3. Construct sample social networks from the data collected and compare network-level statistics.
4. Compare the networks at the node level.
Compare the networks at the cluster level.
The scope of this work includes datasets obtained via streaming APIs rather than those aggregated through progressively expanding searches, such as follower networks. Such collections (especially follower networks) often require data that is prohibitive to obtain, is immediately out of date, and provides no real indication of strength of relationship, as discussed in Section 2.1. Additionally, in the absence of a domain-focused research question, no particular accounts would make sensible seeds, so we rely on keyword-based collections.
Twitter was chosen as the source OSN due to the availability of its data, the fact that the data it provides is highly regular [6] , and because it has similar interaction primitives to other major OSNs. In the interests of comparing a proprietary collection tool with a baseline two tools were chosen: Twarc 3 is an open source library which wraps Twitter's API, and provided the baseline.
RAPID (Real-time Analytics Platform for Interactive Data Mining) [29] is a social media collection and data analysis platform for Twitter and Reddit. It enables filtering of OSN live streams, as well as dynamic topic tracking, meaning it can update filter criteria in real time, adding terms popular in recent posts and removing unused ones.
Both tools facilitate filtering Twitter's Standard live stream 4 with keywords, providing datasets of tweets as JSON objects.
A social network is constructed from dyads of pair-wise relations between nodes, which in our case are Twitter accounts. The node ties denote intermittent relations between accounts, inferred from observed interactions [17, 18] . Here, we consider three social networks built from interaction types common to many OSNs: 'mention networks', 'reply networks', and 'retweet networks' (e.g., retweets are analogous to Facebook shares or Tumblr reposts, and replies analogous to comments on posts on Reddit). We define a social network, G=(V, E), of accounts u ∈ V linked by directed, weighted edges (u i , u j ) ∈ E based on the criteria below.
Mention Networks. Twitter users can mention one or more other users in a tweet. In a mention network, an edge (u i , u j ) exists iff u i mentions u j in a tweet, and the weight corresponds to the number of times u i has mentioned u j .
Reply networks. A tweet can be a reply to one other tweet. In a reply network, an edge (u i , u j ) exists iff u i replies to a tweet by u j , and the weight corresponds to the number of replies u i has made to u j 's tweets.
Retweet networks. A user can repost or 'retweet' another's tweet on their own timeline (visible to their followers). Though retweets are not necessarily direct interactions [8] , they can be used to determine an account's reach. In a retweet network, an edge (u i , u j ) exists iff u i retweets a tweet by u j , and its weight corresponds to the number of u j 's tweets u i has retweeted.
Examining networks built from the same dataset, replies (Figure 1b) are the least common of the three interaction types, and all are dominated by a single large component. Mention networks exhibit relatively high cohesiveness. The similarity between retweets ( Figure 1c ) and mentions (Figure 1a) is because the data model of a retweet includes a mention of the retweeted account.
Other network types can be constructed based on direct or inferred relations, including shared use of hashtags or URLs, reciprocation or minimum levels of interaction, or friend/follower connections. As our focus is on the social relationships implied by direct interactions we will focus only on the above three types of network construction here, however similar issues will certainly arise in other applications, e.g., narrative analysis [30] .
To compare the parallel datasets we examined the following features:
Absolute counts: accounts, tweets, retweets, quotes, replies, URLs, hashtags, and mentions; and Highest counts: tweeting account, mentioned account, retweeted tweet, repliedto tweet, used hashtags, and used URLs.
Based on these figures, we account for major discrepancies, which can guide post-processing (e.g., spam filtering).
The following network statistics are used to assess differences in the constructed networks: number of nodes, edges, average degree, density, mean edge weight, component count and the size and diameter of the largest, Louvain [31] cluster count and the size of the largest, reciprocity, transitivity, and maximum k-cores.
Centrality measures offer a way to consider the importance of individual nodes within a network. The centrality measures considered here include: degree centrality, indicating how many other nodes one node is directly linked to; betweenness centrality, referring to the number of shortest paths in the network that a node is on and thus to what degree the node acts as a bridge between other nodes; closeness centrality, which provides a sense of how topologically close a node is to the other nodes in a network; and eigenvector centrality, which measures how connected a node is to other highly-connected nodes.
Only centrality measures for mention and reply networks are considered, as edges in retweet networks are not direct interactions [8] .
Given the set of nodes in each corresponding pair of networks is not guaranteed to be identical, it is not possible to directly compare the centrality values of each node, so instead we rank the nodes in each network by the centrality values, take the top 1, 000 from each list, further constrain the lists to only the nodes common to both lists, and then compare the rankings. We initially compare the rankings visually using scatter plots, where a node's rank in the first and second list is shown on the x and y axes, respectively. A statistical measure of the similarity of the two rankings (of common nodes) is obtained with the Kendall τ coefficient. A τ score of 0.4 is considered to indicate reasonable similarity [32] . We also calculate Spearman's ρ coefficient as a confirmation measure.
The final step is to consider the clusters discoverable in the mention, reply and retweet networks and compare their membership. We first compare the distribution of the sizes of the twenty largest Louvain clusters [31] visually (chosen because it works well with large and small networks [33] ).
We use the Adjusted Rand index [34] to compare membership. This considers two networks of the same nodes that have been partitioned into subsets. When considered in pairs, there are nodes that appear in the same subset in both partitions (a), and there are (many) pairs of nodes that do not appear in the same subsets in either partition (b), and the rest appear in the same subset in one of the partitions but not in the other. Defining the total of possible pairings of the n nodes ( n(n−1)
2 ) as c, the Rand index, R, is simply R = a+b c . The Adjusted Rand index (ARI) corrects for chance and provides a value in the range [−1, 1] where 0 implies that the two partitions are random with respect to one another and 1 implies they are identical.
To obtain a moderately active portion of activity, we collected data from Twitter's Standard live stream 5 relevant to an Australian television panel show that invites its viewers to participate in the discussion live 6 . A particular broadcast in 2018 was chosen due to the expectation of high levels of activity given the planned discussion topic. As a result, the filter keywords used were 'qanda' 7 and two terms that identified a panel member (available on request). We collected two parallel datasets:
Part 1: Four hours starting 30 minutes before the hour-long programme, to allow for contributions from the country's major timezones; and Part 2: From 6am to 9pm the following day, capturing further related online discussions.
Twarc acted as the baseline collection as it provides direct access to Twitter's API, while RAPID was configured to use co-occurrence keyword expansion [29] , meaning it would progressively add keywords to the original set if they appeared sufficiently frequently (five times in ten minutes). This expanded dataset was 5 All data were collected, stored, processed and analysed according to two ethics protocols #170316 and H-2018-045, approved by the University of Adelaide's human research and ethics committee. 6 The Australian Broadcasting Commission's "Q&A" observes #QandA. 7 The '#' was omitted to catch mentions of '@qanda', the programme's Twitter account.
referred to as 'RAPID-E' and was filtered back to just the tweets containing the original keywords (and labelled 'RAPID') for comparison with the 'Twarc' dataset. We expected the moderate activity observed would not breach rate limits, and thus RAPID should capture all tweets captured by Twarc. This was not the case (Table 1) . 
The first striking difference between the datasets was the number of tweets collected and the effect on the number of contributors (Table 1) . RAPID collected fewer tweets by fewer accounts, but the datasets were close to subsets of the Twarc datasets. Between 26% and 42% of the tweets collected by Twarc were missed by RAPID, but the proportion of retweets in each part is similar (52% and 55% for Part 1 and 69% and 71% for Part 2). In both parts, very few accounts appear in only the RAPID collections. Discussions with RAPID's developers revealed it dumps tweets that miss the filter terms from the textual parts of tweets (e.g., the body, the author's screen name, and the author's profile description). The extra tweets RAPID collected were relevant and in English 8 (based on manual inspection) but posted by different accounts (unique to RAPID-E). Of the tweets that RAPID collected which contained the keywords, they were posted by almost the same accounts as Twarc, but simply did not contain the same tweets. The benefit of topic tracking via keyword expansion is yet to be strongly evaluated, but this study indicates there are benefits (relevant tweets) as well as costs (missed matching tweets). The rest of this analysis explores how much of a difference it makes with regard to SNA. Table 2 reveals that although feature counts vary significantly, many of the most common values are the same (e.g., most retweeted tweet, most mentioned account, most used hashtags). Many are approximately proportional to corpus size (Twarc is 1.72 and 1.32 times larger than RAPID for Parts 1 and 2, respectively), but with notable exceptions and no apparent pattern. Some values are remarkably similar, despite the size of the corpora they arise from being so different. For example, Twarc picked up nearly 8, 000 more hashtag uses than RAPID in Part 1, but fewer than 200 more in Part 2. Both Part 1 datasets have the top ten hashtags, in slightly different order. Approximately 5, 000 of the extra hashtag uses are of '#qanda'. In Part 2, again, the top ten hashtags are nearly the same, but this time the usage counts are similar, except for '#auspol' being used 22% more often in RAPID (1, 652 times compared with 1, 349), which would account for the overall difference of 190 uses when combined with the noise of lesser used hashtags.
Given the differences in datasets, we expect differences in the derived social networks (Tables 3 and 4 ) [14] . Each network is dominated by a single large component, comprising over 90% of nodes in the retweet and mention networks, and around 70% in the reply networks. The distributions of component sizes appear to follow a power law, resulting in corresponding high numbers of detected clusters. Structural statistics like density, diameter (of the largest component in dis- connected networks), reciprocity and transitivity may offer insight into social behaviours such as influence and information gathering. The high component counts in all networks lead to low densities and correspondingly low transitivities, as the potential number of triads is limited by the connectivity of nodes. That said, the largest components were consistently larger in the Twarc datasets, but the diameters of the corresponding largest components from each dataset were remarkably similar, implying that the extra nodes and edges were in the components' centres rather than on the periphery. This increase in internal structures improves connectivity and therefore the number of nodes to which any one node could pass information (and therefore influence) or, at least, reduces the length of paths between nodes so information can pass more quickly. The similarities in transitivity imply the increase may not be significant, however, with networks of these sizes. Reciprocity values may provide insight into information gathering, which often relies on patterns of to-and-fro communication as a person asks a question and others respond. Interestingly, the only significant difference in reciprocity is in the Part 1 retweet networks, with the Twarc dataset having a reciprocity nearly double that of the RAPID dataset (though still small). The Twarc dataset includes 60% more retweets than the corresponding RAPID dataset and 40% more accounts (Table 1) , which may account for the discrepancy. Given the network sizes, the reciprocity values indicate low degrees of conversation, mostly in the reply networks. Interestingly, mean edge weights are very low (1.3 at most), implying that most interactions between accounts in all networks happen only once, despite these being corpora of issue-based discussions.
Next we look at two major categories of network analysis: indexing, for the computation of node-level properties, such as centrality; and grouping, for the computation of specific groups of nodes, such as clustering.
Centrality measures can tell us about the influence an individual has over their neighbourhood, though the timing of interactions should ideally be taken into account to get a better understanding of their dynamic aspects (e.g., [35] ). If networks are constructed from partial data, network-level metrics (e.g., radius, shortest paths, cluster detection) and neighbourhood-aware measures (e.g., eigenvector and Katz centrality) may vary and not be meaningful [14] . We compare centralities of corresponding networks using scatter plots of node rankings, as per Section 3.4 ( Figure 2 ). The symmetrical structures come from corresponding shifts in order: if an item appears higher in one list, then it displaces another, leading to the evident fork-like patterns. There is considerable variation in most centrality rankings for both mention and reply networks in Part 1 (Figure 2a ) but much less in Part 2 ( Figure 2b) . Furthermore, the relatively few common nodes in Part 1's Twarc mention networks (521 to 585) and greater edge count (Tables 3) could indicate that the extra edges significantly affect the node rankings. However, Part 2's Twarc mentions networks also had many more edges, but many more nodes in common (approximately 900). Thus it must have been how the mentions were distributed in the datasets that differed, rather than simply their number. It is not clear that Part 1's four hour duration (cf. Part 2's 15 hours) explains this. Instead, if we look at the 11, 480 tweets unique to Twarc in Part 1 (cf. fewer than 4, 000 are unique to Twarc in Part 2, Table 1 ), only 622 are replies, whereas 6, 915 include mentions. There are also 34% more unique accounts in the Part 1 Twarc dataset, but only 19% more in the Part 2 Twarc dataset (Table 1) . Each mention refers to one of these accounts and forms an extra edge in the mention network, thus altering the network's structure and the centrality values of many of its nodes; this is likely where the variation in rankings originates.
The Kendall τ and Spearman's ρ coefficients were calculated comparing the corresponding lists of nodes, each pair ranked by one of the four centrality measures ( Figure 3 ). Although somewhat proportional, it is notable how different the coefficient values are, especially in Part 2. While Twarc produced more tweets than RAPID (Table 1) , and more unique accounts, the corresponding mention and reply node counts are not significantly higher (Tables 3 and 4 ). In fact, the node counts in the Part 1 reply networks are correspondingly lower than in the Part 2 reply networks, even though both Part 2 datasets were smaller. Edge counts in the mention networks were very different (Twarc had many more) but were quite similar in the reply networks.
The biggest variation was in the mention networks from Part 1 (Figure 2a  and Table 3 ), due to the large number of extra mentions from Twarc. It is notable that the Kendall's τ was low for all mention networks (Figure 3 ), especially for degree and closeness centrality. It is worth noting the minor differences in the degree and immediate neighbours of nodes impacts degree and closeness centralities significantly, and, correspondingly, their relative rankings. In contrast, rankings for betweenness and eigenvector centrality, which rely more on global network structure, remained relatively stable.
We finally compare the networks via largest clusters (Figure 4 ). The reply network clusters are relatively similar, and the largest mention and reply clusters differ the most. The ARI scores confirm that the reply clusters were most similar for Parts 1 and 2 (0.738 and 0.756, respectively), possibly due to the small size of the reply networks. The mention and retweet clusters for Part 2 were more similar than those of Part 1 (0.437 and 0.468 compared to 0.320 and 0.350), possibly due to the longer collection period. In Part 1, there is a chance the networks are different due to RAPID's expansion strategy. Changes to filter keywords may have collected posts of other vocal accounts not using the original keywords, at the cost of the posts which did. 
RAPID and Twarc provided very different views of the Q&A Twitter discussion. This manifested as variations in collection statistics, and network-level and node-level statistics for various social networks built from the collected data. The extra tweets collected by Twarc appear to have resulted in more connections in the largest components. This may affect analysis of diffusion, as reachability correspondingly increases. Deeper study of reply content is required to inform discussion patterns. How reliable social media can be as a source for research without deep knowledge of the effects of collection tools on analyses is an open question. If a tool adds value through analytics, what is the nature of the effect? This paper provides a methodology to explore those effects.
We recommend the following to those using OSN data:
• Be aware of tool biases and their effects.
• Take care to specify filtering conditions with keywords that capture relevant data and avoid irrelevant data. Beware of short terms and ones that are meaningful in non-target languages.
• Check the integrity of data. We observed gaps in another case studies due to connection failures as well as the appearance of duplicate tweets, identical in data and metadata.
Key for future social media research is to develop processes for repeatable analysis. Currently, OSN terms and conditions often hamper researchers exchanging datasets, so new techniques cannot easily be evaluated on the same data, raising questions of fair comparison.
Finally: Does it matter? As long as a researcher makes clear how they conducted a collection and using what tools and configuration, does it not still result in an analysis of behaviour that occurred online? Answer: It very much depends on the conclusions being drawn. Yes, the collection represents real activity that occurred, but it may not be complete, and conclusions drawn from it may be unintentionally misinformed and lacking in nuance. Variations in collections may affect centrality analyses and mislead influential account identification, which, depending on who uses the results of these analyses, may waste effort attending to the wrong topics and/or accounts. A firm understanding of the data and how it was obtained is therefore vital.