Statistical information is usually presented as data. This can appear in digital format as a data set, in hard copy as a set of tables and graphs, or even as a combination of the two, where computer software enables a data user to customise tables, graphs and maps.
Data is often accompanied by metadata, although this sometimes appears as an afterthought: shoddily combined, hidden at the end of the data and inaccessible to all but a few specialists. In modern statistical practice, however, metadata is increasingly being elevated to central importance, and the construction, maintenance and management of metadata is receiving more and more attention in national statistical agencies and international bodies that rely on statistical data.
This is the context in which a work session on statistical metadata was recently hosted by the UN's Economic Commission for Europe, the EU's Eurostat agency and the Organisation for Economic Co-operation and Development's statistics directorate. Held in Geneva from April 3 to April 5, the work session explored the functions of metadata in statistical production, metadata interchange, models and terminology, and ways of using metadata to find statistical data on websites.
Metadata is defined as data about data. According to Statistics New Zealand, metadata "refers to the definitions, descriptions of procedures, system parameters, and operational results that characterise and summarise statistical programs. Metadata may be passive [descriptive], that is, the form of documentation, which is used by agency staff, or it may be active [prescriptive], determining the actions of automated survey processes."
Metadata is essential because there is no such thing as a perfect or absolutely accurate dataset. There is no simple one-to-one correlation between statistics and the social or economic world that statistical data seek to represent.
One function of metadata is to document data structure, assumptions and limitations so that data users can easily understand limitations on use, and avoid the pitfalls of misuse, of the data. To facilitate appropriate use, a standard format is required for metadata that fits the data content and the needs of authors, information providers and end users.
Statistical agencies that are committed to transparency are responsible for informing users of the concepts, classifications and methodologies used in collecting, processing and analysing data, the quality of those data, and all other relevant features of the statistics.
As part of a data management and information delivery project, Stats SA has been shifting metadata and its management to centre stage. Elements of metadata management include: understanding the nature and scope of metadata; establishing responsibility for maintaining and regulating metadata; ensuring that users have easy access to up-to-date data and metadata; determining appropriate storage environments; and populating the storage system with metadata items.
Procedural or methodological metadata, which relate to the procedures by which data are collected and processed, such as sampling, collection methods and editing processes.
Metadata usually includes information describing important features about metadata documentation such as its author, technical form and publisher. Metadata about metadata is needed, for example, to document important details, and to keep a record of metadata updates and changes.
Metadata occupies an important role in every phase of the statistical value chain, enhancing the quality of data and the statistical output process. A central metadata store is key in maintaining and improving data quality by ensuring a consistent interpretation of standards and definitions, and by enforcing compliance with workflow controls and procedures.
Metadata are best registered, stored and accessed according to a common standard, as this makes information sharing more reliable and universal, and facilitates comparisons. The use of metadata standards enables producers to describe datasets fully and coherently, and facilitates data discovery, retrieval and use. If a standard is used, it is far easier to find information in a metadata record.
However, adopting or developing a metadata standard is a complex process, and there is often significant overlap as one standard is extended into the territory of another. A number of bodies are involved in generating standards, including the International Standards Organisation (ISO), the International Monetary Fund and the SA Bureau of Standards.
Stats SA is evaluating metadata standards and systems from the ISO (ISO/IEC 11179), the SA Spatial Metadata Standard (SANS 1878, which is a profile of the international standard ISO 19115), and Statistics Sweden's metadata system, SCBDOK, a documentation model that specifies the basis for the different statistical production methods.
Metadata and its management is not a simple technical issue. It sets out how reliable data are and what they can legitimately be used for. Viewed in this way, metadata form the platform on which data use is built, and from which appropriate and inappropriate use of data can best be established.