Source: [as of 5.01.2018]. As shown in the table above, metadata management can be grouped in four different categories regarding the of metadata management – metadata handling, metadata authority, relationship to statistical cycle/process and users. For each principle, a common set of indicators may be constructed, depending on the project being implemented. As written in the introduction, metadata may managed in three different ways – active, semi-active and passive. Depending on the metadata store that must be implemented, we can observe that most of the metadata management is performed in passive way, which means that metadata repository is stored in external information system. This allows managing metadata for the whole statistical data and allows better integration of the data. Dealing with metadata management is important to increase the reliability and clearance of the data being processed. Because data quality is a part of the WP8 Quality Report, we will concentrate how to manage and deal with the quality of the metadata. In Table 2. we included metadata quality issues that should be regarded when collecting and processing the information. Table 2. Metadata quality evaluation Quality Dimension Factors to consider Complexity Technical constraints Whether structured or unstructured Readability Presence of hierarchies and nesting Completeness Whether the metadata is available, interpretable and complete Usability Resources required to import and analyse Risk analysis Time-related factors Timeliness Periodicity Changes through time Linkability Presence and quality of linking variables Linking level Coherence - consistency Standardisation Metadata available for key variables (classification variables, construct being measured) Validity Transparency of methods and processes Soundness of methods and processes Source: A Suggested Framework for the Quality of Big Data, Deliverables of the Big Data Quality Task Team, December, 2014, [as of 5.01.2018]. Every factor listed in table above can have a set of indicators. It is important to know that selected factors may not be relevant for various datasets, so the decision of using specific set of indicators may depend on the data source type (e.g., structured or unstructured) and its origin (e.g., administrative data source or web data). Discussion Metadata management is strictly related to data processing. However it is not possible to ensure accurate data management without knowing the quality dimension of the metadata. Having a
2 Collect information from public profiles, also very specific such as photos metatags Mostly present information, typical no more than dozens of requests Population Account and API code needed 3 Looking for any kind of objects (e.g., hotels), verification of addresses, monitoring the traffic on specific Free up to 2.5 requests per day. $0.50 USD / 1 thous. additional requests, up to 100 thous. daily, if billing is enabled. Tourism Google account and API code needed 4 Can be used to search through one website, with modifications it will search for a keywords in the whole Internet; can be used to find a URL of the specific enterprise JSON/Atom Custom Search API provides 100 search queries per day for free. Additional requests cost $5 per 1000 queries, up to 10k queries per day. Business Google account and API code needed 5 Finding specific URL of the enterprise 7 queries per second (QPS) per IP address Business needed 6 Collect news articles and comments from Guardian website Free for use. Up to 12 calls per second, Up to 5,000 calls per day, Access to article text, Access to over Population, Social Statistics Registered account needed commercial