# **Data Access**

Due to the confidential nature of mobile data, data access is perhaps the most daunting and time-consuming task of most mobile data-based projects. For CDRs or any mobile network- generated data, there are several ways to approach data access: _a) Directly contacting with MNOs:_ Depending on the relationship and connections, as well as the MNO’s familiarity with the topic, some teams have reported success in accessing data by directly contacting MNOs. This approach is as simple as it sounds: it simply involves directly reaching out to one or more MNOs and requesting their willingness to collaborate on a project.

_b) Leveraging the convening power of telecom regulators:_ In most countries, there is a regulatory body that oversees the MNOs’ work and sets the rules to which MNOs have to adhere, which in theory means the MNOs are obliged to work closely with the regulators and be on their good side. Most of the organizations that have had success with accessing mobile data have done so by going through regulators. The advantage of using regulators is that it is relatively easy to deal with all the MNOs in a country at once rather than communicating with separately with many MNOs.

_c) Third–party organizations:_ In this context, we use the term _third-party organization_ to refer to a private organization or firm which deals with mobile data. Examples of such firms include Flowminder, OPAL, and the University of Tokyo. These organizations often have, on their own initiative or through being contracted by other organizations, worked with MNOs and/or regulators to access mobile data. They often play the role of a consulting firm that handles the negotiations with either MNOs and/or regulators to access data. They also provide technical support to store, process and analyze the data. The way one would use these third-party organizations to access data is by engaging them as consultants who do all the work of negotiating with either MNOs and/or regulators in a specific country to access the data and carry out the analysis.

# **Technologies and Platforms**

Since data generated from mobile phones are large due to the high frequency and scale on which the data are generated, it falls into the category of big data, and dealing with it requires non-traditional technologies and frameworks. In this regard, it is worthwhile describing best practices in terms of the tools and techniques required to store, process and analyze big data.

_Data storage:_ With the advent of cloud computing, handling large scale data sets is not as daunting as it used to be for example 20 years ago. Now, organizations can choose to store and process their data on their premises or store it in the cloud. The advantage of cloud storage is that organizations do not have to pay for maintenance costs, and the processing and storage is highly elastic in that it can easily be scaled up or down as need be. This is not the case with on-premise infrastructure, which is static and inflexible. However, for some countries, shipping off their data to the cloud is not acceptable due to security concerns. There are many cloud computing service providers to choose from. Luckily, all the technology solutions we discuss here are agnostic to a specific cloud solution and so would work equally well on any cloud platform. For both on-premise and cloud storage, the state-of-the-art software framework for handling large-scale data sets is the Apache Hadoop software library (or in short Hadoop). Hadoop is an open-source framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. One of the core modules for Hadoop is the Hadoop Distributed File System (HDFS); a distributed file system that provides high-throughput access to application data. There are several programming models which enables developers to interact with data stored in HDFS; these include, Apache Spark, Hive, Pig and more. Since Hadoop is an open-source platform, it can be downloaded for free and used as needed. However, in practice, it is easier to use Hadoop with third-party distributions such as _Hortonworks HDP_ . When you install Hortonworks HDP, you gain access to the whole suite of Hadoop tools, including Apache Spark.

_d) Data processing and analysis:_ We mentioned that Hadoop is probably the most commonly used platform for b i g data storage, processing and analysis. Within the Hadoop ecosystem, there are several solutions for data processing and analysis. In this work, we used Apache Spark as the big data processing engine because it is very fast and fault-tolerant among many other advantages. More importantly, with Spark, you can use your 19