# Open Data

## Summary


The idea of open data is to share data freely with others. Openness also allows others to modify, reuse or redistribute the data. Openness has two facets: legal and technical. **Legal openness** is about applying a suitable open license to the data. **Technical openness** is about removing technical barriers and making it easy to access, read, store or process the data. 

By opening up data, others can unlock value in the form of information and knowledge. For example, in the travel sector, locations, images, prices and reviews are data that can help us plan a holiday. Information is data within a given context. Knowledge personalizes information and helps us make decisions. 

Many organizations worldwide promote open data. Open licenses, datasets and tools are available. Governments are releasing open data that citizens can use.

## Discussion

### Could you describe open data?

Database is really about structure and organization of data, also known as the database model. This is generally covered by copyright. In the context of open data, we're more concerned with the contents of the database, which we simply call data. 

Data can mean a single item or an entire collection. Particularly for factual databases, a collection can be protected but not individual items. For example, a protected collection may be about the melting point of various substances but no one can be prevented from stating a particular item, such as element E melts at temperature T. 

To understand the meaning of openness, we can refer to the *Open Definition* that states, "Open means anyone can freely access, use, modify, and share for any purpose (subject, at most, to requirements that preserve provenance and openness)." 

Open data should be accessible at a reasonable cost if not free. It should be available in bulk. There shouldn't be restrictions on who or for what purpose they wish to use the data. Tools to use the data should be freely available and not proprietary. Data shouldn't be locked up behind passwords or firewalls. 


### Where could open data be useful?

Open data can make governments more transparent. Citizens will have confidence that their money is being spent as budgeted or in implementing the right policies. For example, one activist noted that Canadian citizens used open data to save their government $3.2bn in fraudulent charitable donations. In Brazil, DataViva provides millions of interactive visualizations based on open government data. 

New business opportunities are possible. For example, once Transport for London opened their data, developers used it to build apps. Likewise, Thomson Reuters uses open data to provide better services to its customers. OpenStreetMap and Copernicus are examples that enable new GIS applications. 

In research, open data is a part of what is called Open Science. It leads to reproducible research and faster advancements. Open data also enables researchers to revalidate their own findings. 

Open data can be used to protect the environment. Some apps that do this include mWater, Save the Rain and Ecofacts. 


### Could you mention some sources of open data?

There are many curated lists on the web for open data. From some of these, we mention a few useful sources of open data by category:   

    + **General**: DBpedia, Datasets Subreddit, Kaggle, FiveThirtyEight, Microsoft Marco
    + **Government**: Data.gov, Data.gov.uk, European Union Open Data Portal
    + **Economy**: World Bank Open Data, Global Financial Data, International Monetary Fund
    + **Business**: OpenCorporates, Yellowpages, EU-Startups, Glassdoor
    + **Health & Science**: World Health Organization, HealthData.gov, NHS Digital, Open Science Data Cloud, NASA Earth Data, LondonAir
    + **Research**: Google Scholar, Pew Research Center, OpenLibrary Data Dumps, CERN Open Data
    + **Environment**: Climate Data Online, IEA Atlas of Energy

### Which are some organizations working with or for open data?

We mention a few organizations:

    + Open Data Institute: Works with companies and governments to build an open, trustworthy data ecosystem, where people can make better decisions using data and manage any harmful impacts.
    + Open Data Commons: Provides a set of legal tools to publish and use open data. They've published open licenses applicable to data.
    + Open Knowledge Foundation: A worldwide network of people passionate about openness, using advocacy, technology and training to unlock information and enable people to work with it to create and share knowledge. It was briefly called Open Knowledge International. The Open Definition is one of their projects.
    + Open Data Charter: A collaboration between governments and organisations working to open up data based on a shared set of principles.

### Why and what aspects of open data should we standardize?

Data is more valuable if we can combine two different datasets to obtain new insights. **Interoperability** is the key. Given diverse systems, tools and data formats, interoperability can be almost impossible. 

Without standards, it becomes more difficult for us to publish, access or share data effectively. Standards also make it easier to repeat processes, compare results and reach a shared understanding. Moreover, we need **open standards** that are available to public and are defined through collaboration and consensus. 

Open standards should define a common data model. The data pipeline should be streamlined. It should be easy to combine data. It should promote common understanding. 

Open Data Institute's page on standards is a useful resource to learn more. A checklist for selecting a suitable standard looks at the licensing, project needs, maintenance of the standard, and guidance on usage. 


### What sort of licensing should I adopt when opening my data?

Releasing your data without a license creates uncertainty. In some jurisdictions, data lacking explicit permission may be protected by intellectual property rights. It's therefore better to attach a license. 

Aspects that define a license include public domain, attribution, share-alike, non-commercial, database only and no derivatives. 

There are a number of licenses that conform to Open Definition. *Creative Commons CC0* is an example. It releases the data into public domain. Anyone can copy, modify and distribute, even for commercial purposes, without asking permission. A similar license is *Open Data Commons Public Domain Dedication and Licence (PDDL)*. Other conformant but less reusable licenses are data licenses from governments (Germany, UK, Canada, Taiwan). UK's Open Government Licence is an example.

Two other licenses worth looking at come from the Linux Foundation: CDLA–Sharing-1.0 and CDLA-Permissive-1.0, where CDLA refers to Community Data License Agreement. 

*Open Data Commons Open Database License (ODC-ODbL)* is seen as a "viral license". Any changes you make to the dataset, you're required to release the same under this license.  


### What are some challenges with open data?

Publishers continue to use customized licenses. This makes it hard to reuse data. It makes licenses incompatible across datasets. Instead, they should use standardized open licenses. Ambivalent or redundant clauses cause confusion. Licenses often are not clear about the data to which they apply. Data is often not linked to legal terms. 

Data is hard to find. Sometimes their locations change. A platform such as CKAN might help. 

Data could be misinterpreted, which could result in a wrong decision. This creates a fear of accountability and prevents producers from opening their datasets. 

Quality of data is another concern. Data should be machine-readable and in raw form. Publishing data in HTML or PDF is not research friendly. For context and interpretation, metadata should be shared. 

Raw data is good but also requires advanced technical skills and domain knowledge. We therefore need to build better data literacy. AI algorithms are being used to analyse data but many are black-box models. They also promote data centralization and control, which are at odds with the open data movement. 

Data infrastructure must of consistent quality. Data can also be biased by gender or against minority groups. 

## Milestones

1942

The concept of open data starts with Robert King Merton, one of the fathers of the sociology of science. He explains how freely sharing scientific research and results can stimulate growth and innovation. 

1994

In the US, the Government Printing Office (GPO) goes online and opens a few government-related documents. This is an early example of **open government data** at a time when the Internet was becoming popular. Historically and legally, the idea can be traced back to the the Freedom of Information Act of 1966. 

2005

The Open Knowledge Foundation creates the **Open Definition**. This is based on established principles from the open source movement for software. This definition is later translated into more than 30 languages. In November 2015, Open Definition 2.1 is released. 

Feb  
2006

At a TED talk, Hans Rosling presents compelling visuals of global trends in health and economics. Using publicly available datasets from different sources, he debunks myths about the developing world. He makes a case for governments and organizations to open up their data. Data must be enabled using design tools. His mantra is to animate and liberate data from closed databases. Data must also be searchable. 

Dec  
2007

Thirty individuals meet at Sebastopol, California to discuss open public data. Many of them come from the culture of free and open source software movement. This event can be seen as a convergence of many past US and European efforts in open data. They identify **eight principles**: complete, primary, timely, accessible, machine processable, non-discriminatory, non-proprietary, and license-free. 

Feb  
2009

At a TED talk, Tim Berners-Lee gets people to chant "Raw data, now." He makes reference to Rosling's talk of 2006 and adds that it's important to link data from different sources. He calls this **Linked Data**. He mentions *DBpedia*, which takes data from Wikipedia and connects them up. 

May  
2009

The US government launches **Data.gov** with 47 datasets. About five years later, it has about 100,000 datasets. 

2010

Many governments attempt to open their data to public but there are concerns about privacy. It's only in 2015 that privacy principles become an essential part of discussions on open data. It's become clear that providing raw data is not always possible. Governments must balance potential benefits to public against privacy rights of individuals. 

2012

Guillermo Moncecchi of Open Knowledge International writes that beyond transparency, open data is also about building a **public data infrastructure**. While the focus during 2007-2009 was on data transparency, during 2010-2016 focus shifts to public infrastructure. Consider data about street light locations. Citizens can use this data to solve problems on their own. Metrics change from number of published datasets to APIs and reuse. 

Jun  
2017

Open Knowledge Foundation publishes the **Global Open Data Index (GODI)**. This shows that only 38% of government data is really open. This is based on Open Definition 2.1. A later update available online shows that only 166 of 1410 datasets (11%) are open. The report gives a number of recommendations to governments and policy makers to make their data more open. 

Jan  
2018

Open Data Charter replaces their earlier ambitious call of "open by default" with a more practical "publish with purpose". Rather than getting governments to open up as much data as possible quickly, the intent is to get them to take small steps in that direction. Governments can open datasets with a clear view of tangible benefits to citizens. 

Feb  
2018

Using open data exposed by Here.com via an API, Tjukanov uses visualization to show average traffic congestion in many of the world's largest cities. Interesting patterns emerge, plotted within a radius of 50km from the city center. He uses QGIS, an open source GIS platform.  This is an example of the value that open data plus open source can unlock.

Jan  
2020

Using open data collated by the Institute for Government from data.gov.uk, Peter Cook produces interesting **organograms**, which are visualizations of organization structures. He does this for many UK government departments. Clicking on individual data points shows name, job title, salary range and other details. This sort of data has been available (though patchy) since March 2011.