Evaluating the quality of statistics is no easy matter and is often an arena of contestation. Statistics that measure progress in implementing policy, for example, are often challenged, not just because of accuracy, but because of the competing interests represented in any democratic political order. This occurs in similar ways in other areas, where there is competition over resources, positioning and influence, depending on the agendas of competing socio-economic interests.
This explains why it is often so difficult to evaluate specific criticisms of particular statistics. It is because criticism of statistics without exploring the basis of their collection rarely goes to the heart of statistical quality.
Evaluating the quality and accuracy of statistical measurement necessitates examining the infrastructure that underpins collection and processing of data. This includes both physical and conceptual or intellectual infrastructure.
To be usefully comparable over time, statistics need to be collected and analysed according to coherent and consistent classifications, standards, concepts and definitions. This is part of the intellectual infrastructure of statistical practice.
The way this form of statistical infrastructure is captured and stored, and the procedures for its updating, amendment and retrieval, are important indicators of statistical quality.
For some time, Statistics SA has been developing a central system for intellectual infrastructure such as classifications, definitions, concordances and code files.
This uses database technology to provide centralised storage, maintenance and access facilities for classification data used both in the development and processing of surveys, and in the subsequent analysis and evaluation of the data.
In addition, it contributes to improved data quality by using computerised data documentation, or metadata, to support the application of standard data item definitions and classifications.
It facilitates the comparison and analysis of data by storing comparisons between different versions of classifications and different classifications, and facilitates the creation of code files (a comprehensive list of probable survey responses and the categories to which they are coded) used in processing results.
Without this centralised infrastructure, accurate measurement can be compromised.
Consider, for example, tourism as a contributor to economic activity. The definition of a tourist is central to the accuracy of this measurement, yet those collecting data on tourism have historically defined it in different ways.
A person travelling to a place other than that of his or her usual environment for less than 12 months and whose main purpose of the trip is other than the exercise of an activity remunerated from within the place visited.
A standard definition of tourism used by all parties measuring the contribution of tourism to the economy is clearly needed to compare and evaluate the quality of data.
Integrity (institutional arrangements that ensure professionalism, transparency and ethical standards on the part of the agency collecting data and producing statistics).
Stats SA's Data Management and Information Delivery is the vehicle though which the components of statistical infrastructure are brought into one system, enabling consistent and standardised capture, storage, amendment and retrieval of the elements required for sound evaluation of statistical quality. This infrastructure will allow for more informed evaluation of the accuracy of statistical measurement.