People’s main searches, such as of depression Mobile data content: text messages (1) Mobile phone usage, analysis of the profile of public administration customers E-Mail (1) Using new technologies in everyday life Others (3) Process-mediated data (Traditional Business systems and Websites): 28 ideas Data produced by Public Agencies: 14 Medical records (1) People’s health Others (13) Data produced by businesses: 14 Commercial transactions (5) Population and Social conditions – how much people on specific services/products, e.g., education records (1) Tourism – data on travelling E-commerce (6) Population – ICT skills to purchase products Credit (2) Tourism – data on travelling, people’s budget Machine-generated data (Automated Systems): 37 ideas Data from sensors: 29 Fixed sensors: 15 Home automation (2) Population – time budget Weather/pollution sensors (2) Influence of changing weather/pollution on travelling Traffic sensors/webcam (6) Tourism – internal and external border crossing Scientific sensors (1) Population – habits and behaviour Security/surveillance videos/images (3) Population – migration Others (1) Mobile sensors (tracking): 13 Person (Mobile phone location) (5) Tourism – travelling, border crossing, internal migration; Population - daily and night population. Road (Car, Trucks) (4) Tourism – travelling, border crossing Air (planes) (2) Tourism – travelling, border crossing Nautical (Ships) (2)
Mobile sensors (tracking) - Satellite images Data produced by Public Agencies During brainstorming only Satellite imagery was indicated as a potential data source to use. Pilot surveys conducted in Poland and numerous elaborations on using satellite images allows recommending the following data types and pilot surveys by potential of their usage: Data produced by Public Agencies Mobile sensors (tracking) - Satellite images Extracting of particular cultivation and area coverage, including recognition of selected cultivations based on satellite images. To obtain hands-on experience for using Big Data to official statistics purposes, WP7 is planning to carry out three use cases (the table below). It will cover the entire process of generating statistical output from a specific Big Data source from the start, obtaining the data, till the end, the production of statistical estimates. Domain Population Agriculture Tourism/Border Crossing Name of the use case Everyday citizen satisfaction Estimation of Agricultural statistics – pilot case study on crop types based on satellite data Border movement Big Data source Social portals Satellite images Traffic sensors Responsibility UK – coordinator (SGA-1) RSO Poznań/Bydgoszcz Department of Agriculture, RSO Olsztyn + IE RSO Rzeszów, Department of Social Survey + NL Brief overview of the methodology Webscraping Data/Text/Web mining Machine learning combining data – data fusion on radar and optical remote sensing data; data comparison with traditional surveys e.g. FSS; combining data – administrative data sources with satellite data. Intertemporal disaggregation and interpolation, Latent variable models, Cross entropy econometrics.
Mobile sensors tracking - Satellite images 3/6 50 5/6 83 5/13 38 3 Pictures: Instagram, Flickr, Picasa 1/4 25 4/4 100 4/12 33 4 Data produced by businesses - E-commerce 1/5 20 4/5 80 8/13 62 10 Data from sensors - Fixed sensors - Security/surveillance videos/images 1/4 25 4/4 100 10/11 91 4 Data produced by businesses - Banking/stock records 1/3 33 3/3 100 11/13 85 1 E-Mail 1/3 33 2/3 67 11/13 85 2 Part of the data sources was indicated only as to be used after 2018. It is worth mentioning a few of them because of the easy access: Blogs and comments; Data from computer systems – Web logs; Videos: Youtube etc.; Data from sensors – Fixed sensors – Weather/pollution sensors. 3. Q8: Please indicate which Big Data sources are used currently or are planned to be used before or after 2018 in your country: Source Up to 2018 After 2018 Difficult to access Brainstorming Resp. % Resp. % Resp. % Response User generated maps 0/4 0 4/4 100 6/12 50 0 Mobile sensors tracking – Cars 0/4 0 4/4 100 8/12 67 6 Blogs and comments 0/3 0 3/3 100 2/14 14 3 Mobile data content: text messages 0/3 0 3/3 100 11/12 85 3 Data from computer systems – Web logs 0/3 0 3/3 100 4/11 36 5 Videos: YouTube etc. 0/2 0 2/2 100 4/12 33 5 Data from sensors - Fixed sensors - Weather/pollution sensors 0/2 0 2/2 100 4/16 25 4 Data from sensors - Fixed sensors - Scientific sensors 0/2 0 2/2 100 6/11 55 1 Data from computer systems - Logs 0/2 0 2/2 100 5/11 45 5 Personal documents 0/1 0 1/1 100 10/12 83 0 Due to the fact that the number of answers from Q8 did not include all of the data sources classified in UNECE typology, some of them were indicated in the opening of Q9: AIS data; AIS data of ships;
22 Data produced by businesses Business Registers (to match enterprises) 1 2210 Commercial transactions Commercial transactions Scanner data (partnership agreement) Scanner data for CPI Scanner data 4 3111 Home automation Smart-meter data 1 3121 Mobile sensors (tracking) - Person (Mobile location) Mobile phone data Call detail records 2 3122 Mobile sensors (tracking) - Road (Car, Trucks) road sensors Traffic sensor data; to be taken to the production in planning: traffic data for road freight statistics 3 3125 Mobile sensors (tracking) - Nautical (Ships) AIS data AIS (from public agency) 2 If in Q16 respondents marked “yes”, in Q19 they were to specify rate the quality of BD sources used in their office. In the area of Clarity, BD sources were assessed very low (probably because of the data from websites). It is important to precise all definitions and metadata that are related to acquired data. In other areas the data quality assessment is on medium level, apart from Timeliness and frequency which is on high level. 7. Q19: If yes q 16, rate the quality of Big Data sources used in your office: Answer options Low Low % Medium Medium % High High % Response count Complexity -refers to the lack of simplicity and uniformity in the data 2 18 8 73 1 9 11 Completeness - the extent to which metadata are available for a proper understanding and use of data. It refers to the exhaustiveness of the descriptions available for the input data (i.e. covering all the required aspects mentioned in the hyper-dimensions Source and Data as well as the level of detail of descriptions 3 30 7 70 0 0 10 Usability - is the extent to which the NSO will be able to work with and use the data without the employment of specialized resources or place significant burden on existing resources; and the ease with which it can be integrated with existing systems and standards 4 36 7 64 0 0 11 Timeliness and frequency 0 0 3 30 7 70 10 Accuracy - is the degree to which the information correctly describes the phenomena it was designed to measure 3 30 6 60 1 10 10 Coherence - is the extent to which the dataset follows standard conventions, is internally consistent, is consistent over 3 33 5 56 1 11 9
Administrative data 8 44 10 56 18 Non administrative data 4 25 12 75 16 Sample survey 11 58 8 42 19 Data from Agricultural Census 0 0 15 100 15 Data from the Population and Housing Census 3 20 12 80 15 Reporting 1 8 12 92 13 Others 4 29 10 71 14 answered question 19 skipped question 0 Table 9. Q23: Are you planning to combine Big Data sources with existing sources? Answer options Yes Yes % No No % Response count Administrative data 17 89 2 11 19 Non administrative data 10 71 4 29 14 Sample survey 17 94 1 6 18 Data from Agricultural Census 4 33 8 67 12 Data from the Population and Housing Census 10 71 4 29 14 Reporting 3 30 7 70 10 Others 7 58 5 42 12 answered question 19 skipped question 0 Q24 shows the biggest potential of BD sources for official statistics for the following surveys: tourism, prices, followed by labour, transport, population and migration. Only 39% of respondents see the potential for agriculture surveys. However, it is quite clear that BD can be beneficial for nearly all of the statistical domains. Q25 allowed to some concerns: For the time being, only methodological pilots are carried out with the aim to investigate the potential use of BD sources for the development, production and dissemination of official statistics. For web scraping, we face several technical issues with websites, finding the most suitable IT tool for web scraping is challenging. A fully-fledged consideration of use of BD for concrete statistical operations is still pending. We are currently concentrated on the human population census and tourism statistics. Concerning questions 12, 19 and 24 that were intentionally left blank: There are ongoing pilot studies at the Federal Statistical Office as well as within the ESSnet projects, so that for many aspects, e.g. for the quality of BD sources, it is too early to provide an assessment. In case we can access the above mentioned BD sources, we will use them for all statistical domains that could possibly apply. At this the questionnaire is very difficult to be answered correctly. Summary