Use Case Description Maintain data “as-is”. No access and no data analytics for 75 years. Preserve the data at the bit-level. Perform curation, which includes format transformation if necessary. Provide access and analytics after nearly 75 years. Current Solutions Compute(System) Linux servers Storage NetApps, Magnetic tapes. Networking Software Big Data Characteristics Data Source (distributed/centralized) Centralized storage. Volume (size) 380 Terabytes. Velocity (e.g. real time) Static. Variety (multiple datasets, mashup) Scanned documents Variability (rate of change) None Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Cannot tolerate data loss. Visualization TBD Data Quality Unknown. Data Types Scanned documents Data Analytics Only after 75 years. Big Data Specific Challenges (Gaps) Preserve data for a long time scale. Big Data Specific Challenges in Mobility TBD Security & Privacy Requirements Title 13 data. Highlight issues for generalizing this use case (e.g. for ref. architecture) . More Information (URLs) Government Operation NBD(NIST Big Data) Requirements WG Use Case Template Use Case Title National Archives and Records Administration Accession NARA Accession, Search, Retrieve, Preservation Vertical (area) Digital Archives Author/Company/Email Quyen Nguyen & Vivek Navale (NARA)
Netflix competition (to improve ratings by 10%) combined over 100 different algorithms. Big Data Specific Challenges (Gaps) Analytics needs continued monitoring and improvement. Big Data Specific Challenges in Mobility Mobile access important Security & Privacy Requirements Need to preserve privacy for users and digital rights for media. Highlight issues for generalizing this use case (e.g. for ref. architecture) Recommender systems have features in common to e-commerce like Amazon. Streaming video has features in common with other content providing services like iTunes, Google Play, Pandora and Last.fm More Information (URLs) by Xavier Amatriain Note: <additional comments> Commercial NBD(NIST Big Data) Requirements WG Use Case Template Use Case Title Web Search (Bing, Google, Yahoo..) Vertical (area) Commercial Cloud Consumer Services Author/Company/Email Geoffrey Fox, Indiana University gcf@indiana.edu Actors/Stakeholders and their roles and responsibilities Owners of web information being searched; search engine companies; advertisers; users Goals Return in ~0.1 seconds, the results of a search based on average of 3 words; important to maximize “precision@10”; number of great responses in top 10 ranked results Use Case Description .1) Crawl the web; 2) Pre-process data to get searchable things (words, positions); 3) Form Inverted Index mapping words to documents; 4) Rank relevance of documents: PageRank; 5) Lots of technology for advertising, “reverse engineering ranking” “preventing reverse engineering”; 6) Clustering of documents into topics (as in Google News) 7) Update results efficiently Current Solutions Compute(System) Large Clouds Storage Inverted Index not huge; crawled documents are petabytes of text – rich media much more Networking Need excellent external network links; most operations pleasingly parallel and I/O sensitive. High performance internal network not needed Software MapReduce + Bigtable; Dryad + Cosmos. Final step essentially a recommender engine Big Data Characteristics Data Source (distributed/centralized) Distributed web sites