Solutions Storage SAN and Direct Storage Networking Fiber, 10 gigabit Ethernet, Infiniband 40 gigabit. Software Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig Big Data Characteristics Data Source (distributed/centralized) Survey data, other government administrative data, web scrapped data, wireless data, e-transaction data, potentially social media data and positioning data from various sources. Volume (size) TBD Velocity (e.g. real time) TBD Variety (multiple datasets, mashup) Textual data as well as the traditionally defined strings and numerical fields. Data can be from multiple datasets mashed together for analytical use. Variability (rate of change) TBD. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues, semantics) Data must have high veracity and systems must be very robust. The semantic integrity of conceptual metadata concerning what exactly is measured and the resulting limits of inference remain a challenge Visualization Data visualization is useful for data review, operational activity and general analysis. It continues to evolve. Data Quality (syntax) Data quality should be high and statistically checked for accuracy and reliability throughout the collection process. Data Types Textual data, pre-defined ASCII strings and numerical data Data Analytics Analytics are required to create reliable estimates using data from traditional survey sources, government administrative data sources and sources from the digital economy. Big Data Specific Challenges (Gaps) Improving analytic and modeling systems that provide reliable and robust statistical estimated using data from multiple sources, that are scientifically transparent and while providing confidentiality safeguards that are reliable and publically auditable. Big Data Specific Challenges in Mobility Mobile access is important. Security & Privacy Requirements All data must be both confidential and secure. All processes must be auditable for security and confidentiality as required by various legal statutes. Highlight issues for generalizing this use Statistical estimation that provide more detail, on a more near real time basis for less cost. The reliability of estimated statistics from such “mashed up” sources still must be evaluated.
Data Quality The typical problem is visualization implying quality/accuracy not available in the original data. All data should include metadata for accuracy or circular error probability. Data Types Imagery (various formats NITF, GeoTiff, CADRG) Vector (various formats shape files, kml, text streams: Object types include points, lines, areas, polylines, circles, ellipses. Data Analytics Closest point of approach, deviation from route, point density over time, PCA and ICA Big Data Specific Challenges (Gaps) Indexing, retrieval and distributed analysis Visualization generation and transmission Big Data Specific Challenges in Mobility Visualization of data at the end of low bandwidth wireless connections. Security & Privacy Requirements Data is sensitive and must be completely secure in transit and at rest (particularly on handhelds) Highlight issues for generalizing this use case (e.g. for ref. architecture) Geospatial data requires unique approaches to indexing and distributed analysis. More Information (URLs) Applicable Standards: Geospatial Indexing: Quad Trees, Space Filling Curves (Hilbert Curves) – You can google these for lots of references. Note: The has been some work with in DoD related to this problem set. Specifically, the DCGS-A standard cloud (DSC) stores, indexes, and analyzes some big data sources. However, many issues still remain with visualization. Defense NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Object identification and tracking from Wide Area Large Format Imagery (WALF) Imagery or Full Motion Video (FMV) – Persistent Surveillance Vertical (area) Defense (Intelligence) Author/Company/Email David Boyd/Data Tactics/dboyd@data-tactics.com Actors/Stakeholders and their roles and responsibilities Civilian Military decision makers Intelligence Analysts Warfighters Goals To be able to process and extract/track entities (vehicles, people, packages) over time from the raw image data. Specifically, the idea is to reduce the petabytes of data generated by persistent surveillance down to a manageable size (e.g. vector tracks) Use Case Description
values could range from less than 100 (new patient) to more than 100,000 (long term patient) with typical patients composed of 100 CV values and 1000 continuous values. Most values are time based, i.e. a timestamp is recorded with the value at the time of observation. Variability (rate of change) Data will be updated or added during each patient visit. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Data are annotated based on domain ontologies or taxonomies. Semantics of data can vary from labs to labs. Visualization no visualization Data Quality Provenance is important to trace the origins of the data and data quality Data Types text, and Continuous Numerical values Data Analytics Integrating data into semantic graph, using graph traverse to replace SQL join. Developing semantic graph mining algorithms to identify graph patterns, index graph, and search graph. Indexed Hbase. Custom code to develop new patient properties from stored data. Big Data Specific Challenges (Gaps) For individualized cohort, we will effectively be building a datamart for each patient since the critical properties and indices will be specific to each patient. Due to the number of patients, this becomes an impractical approach. Fundamentally, the paradigm changes from relational lookup to semantic graph traversal. Big Data Specific Challenges in Mobility Physicians and patient may need access to this data on mobile platforms Security & Privacy Requirements Health records or clinical research databases must be kept secure/private. Highlight issues for generalizing this use case (e.g. for ref. architecture) Data integration: continuous values, ontological annotation, taxonomy Graph Search: indexing and searching graph Validation: Statistical validation More Information (URLs) Note: <additional comments> Healthcare and Life Science NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Statistical Relational AI for Health Care Vertical (area) Healthcare Author/Company/Email Sriraam Natarajan / Indiana University /natarasr@indiana.edu Actors/Stakeholders and their roles and responsibilities Researchers in Informatics, medicine and practitioners in medicine. Goals The goal of the project is to analyze large, multi-modal, longitudinal data. Analyzing different data types such as imaging, EHR, genetic and column
track finding algorithms. Produce “large” full physics files and stripped down Analysis Object Data (AOD) files that are ~10% original size Also Monte codes to produce simulated data to evaluate efficiency of experimental detection. Tier 2: Universities and laboratories. Note processing is compute and data intensive Physics Analysis Information to Knowledge/Discovery Disk Files of Information including accelerator and Monte data. Include wisdom from lots of physicists (papers) in analysis choices Use simple statistical techniques (like histogramming, multi-variate analysis methods and other data analysis techniques and model fits to discover new effects (particles) and put limits on effects not seen Data reduction and processing steps with advanced physics algorithms to identify event properties, particle hypothesis etc. For interactive data analysis of those reduced and selected data sets the classic program is Root from CERN that reads multiple event (AOD, NTUP) files from selected data sets and use physicist generated C++ code to calculate new quantities such as implied mass of an unstable (new) particle While the bulk of data processing is done at Tier 1 and Tier 2 resources, the end stage analysis is usually done by users at a local Tier 3 facility. The scale of computing resources at Tier 3 sites range from workstations to small clusters. ROOT is the most common software stack used to analyze compact data formats generated on distributed computing resources. Data transfer is done using ATLAS and CMS DDM tools, which mostly rely on gridFTP middleware. XROOTD based direct data access is also gaining importance wherever high network bandwidth is available. Physics discoveries and results are confidential until certified by group and presented at meeting/journal. Data preserved so results reproducible Figure 1: The LHC Collider location at CERN Figure 2: The Multi-tier LHC computing infrastructure Astronomy and Physics
forecasts vs. capacity, what-if analysis; Realtime information display; Apps with push notification of alerts Data Quality (syntax) Gaps in smart meters and weather data; Quality issues in sensor data; Rigorous checks done for “billing quality” meter data; Data Types Timeseries (CSV, SQL tuples), Static information (RDF, XML), topology (shape files) Data Analytics Forecasting models, machine learning models, time series analysis, clustering, motif detection, complex event processing, visual network analysis, Big Data Specific Challenges (Gaps) Scalable realtime analytics over large data streams Low-latency analytics for operational needs Federated analytics at utility and microgrid levels Robust time series analytics over millions of customer consumption data Customer behavior modeling, targeted curtailment requests Big Data Specific Challenges in Mobility Apps for engaging with customers: Data collection from customers/premises for behavior modeling, feature extraction; Notification of curtailment requests by utility/building managers; Suggestions on energy efficiency; Geo-localized display of energy footprint. Security & Privacy Requirements Personally identifiable customer data requires careful handling. Customer energy usage data can reveal behavior patterns. Anonymization of information. Data aggregation to avoid customer identification. Data sharing restrictions by federal and state energy regulators. Surveys by behavioral scientists may have IRB restrictions. Highlight issues for generalizing this use case (e.g. for ref. architecture) Realtime data-driven analytics for cyber physical systems More Information (URLs) Note: <additional comments>