Atmospheric Turbulence - Event Discovery and Predictive Analytics (Scientific Research: Earth Science) Michael Seablom, NASA HQ Pathology Imaging/digital pathology (Healthcare) Fusheng Wang, Emory University Genomic Measurements (Healthcare) Justin Zook, NIST Cargo Shipping (Industry) William Miller, MaCT USA Radar Data Analysis for CReSIS (Scientific Research: Polar Science and Remote Sensing of Ice Sheets) Geoffrey Fox, Indiana University Particle Physics: Analysis of LHC Large Hadron Collider Data: Discovery of Higgs particle (Scientific Research: Physics) Geoffrey Fox, Indiana University Netflix Movie Service (Commercial Cloud Consumer Services) Geoffrey Fox, Indiana University Web Search (Commercial Cloud Consumer Services) Geoffrey Fox, Indiana University NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Vertical (area) Author/Company/Email Actors/Stakeholders and their roles and responsibilities Goals Use Case Description Current Solutions Compute(System) Storage Networking Software Big Data Characteristics Data Source (distributed/centralized) Volume (size) Velocity (e.g. real time) Variety (multiple datasets, mashup) Variability (rate of change) Big Data Science (collection, curation, analysis, Veracity (Robustness Issues, semantics) Visualization Data Quality (syntax)
Current Solutions Compute(System) Linux servers Storage NetApps, Hitachi, Magnetic tapes. Networking Software Custom software, commercial search products, commercial databases. Big Data Characteristics Data Source (distributed/centralized) Distributed data sources from federal agencies. Current solution requires transfer of those data to a centralized storage. In the future, those data sources may reside in different Cloud environments. Volume (size) Hundred of Terabytes, and growing. Velocity (e.g. real time) Input rate is relatively low compared to other use cases, but the trend is bursty. That is the data can arrive in batches of size ranging from GB to hundreds of TB. Variety (multiple datasets, mashup) Variety data types, unstructured and structured data: textual documents, emails, photos, scanned documents, multimedia, social networks, web sites, databases, etc. Variety of application domains, since records come from different agencies. Data come from variety of repositories, some of which can be cloud-based in the future. Variability (rate of change) Rate can change especially if input sources are variable, some having audio, video more, some more text, and other images, etc. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Search results should have high relevancy and high recall. Categorization of records should be highly accurate. Visualization TBD Data Quality Unknown. Data Types Variety data types: textual documents, emails, photos, scanned documents, multimedia, databases, etc. Data Analytics Crawl/index; search; ranking; predictive search. Data categorization (sensitive, confidential, etc.) PII data detection and flagging. Big Data Specific Challenges (Gaps) Perform pre-processing and manage for long-term of large and varied data. Search huge amount of data. Ensure high relevancy and recall. Data sources may be distributed in different clouds in future.
Custom code to develop new patient properties from stored data. Big Data Specific Challenges (Gaps) For individualized cohort, we will effectively be building a datamart for each patient since the critical properties and indices will be specific to each patient. Due to the number of patients, this becomes an impractical approach. Fundamentally, the paradigm changes from relational lookup to semantic graph traversal. Big Data Specific Challenges in Mobility Physicians and patient may need access to this data on mobile platforms Security & Privacy Requirements Health records or clinical research databases must be kept secure/private. Highlight issues for generalizing this use case (e.g. for ref. architecture) Data integration: continuous values, ontological annotation, taxonomy Graph Search: indexing and searching graph Validation: Statistical validation More Information (URLs) Note: <additional comments>
Netflix competition (to improve ratings by 10%) combined over 100 different algorithms. Big Data Specific Challenges (Gaps) Analytics needs continued monitoring and improvement. Big Data Specific Challenges in Mobility Mobile access important Security & Privacy Requirements Need to preserve privacy for users and digital rights for media. Highlight issues for generalizing this use case (e.g. for ref. architecture) Recommender systems have features in common to e-commerce like Amazon. Streaming video has features in common with other content providing services like iTunes, Google Play, Pandora and Last.fm More Information (URLs) by Xavier Amatriain Note: <additional comments>