Departments of Agriculture, Commerce, Defense, Energy, Census, Health and Human Services, Homeland Security, Transportation, Treasury, and Veterans Affairs. NIST would like to acknowledge the specific contributions to this volume, during Version 1 and/or Version 2 activities, by the following NBD-PWG members: Tsegereda Beyene Cisco Systems Deborah Blackstock MITRE Corporation David Boyd InCadence Strategic Services Scott Brim Internet2 Pw Carey Compliance Partners, LLC Wo Chang NIST Marge Cole SGT, Inc. Yuri Demchenko University of Amsterdam Safia Djennane Cloud-Age-IT Geoffrey Fox Indiana University Nancy Grady SAIC Jay Greenberg The Boeing Company Karen Guertler Consultant Keith Hare JCC Consulting, Inc. Babak Jahromi Microsoft Pavithra Kenjige PK Technologies Donald Krapohl Augmented Intelligence Luca Lepori Data Hold Orit Levin Microsoft Eugene Luster DISA/R2AD Ashok Malhotra Oracle Corporation Robert Marcus ET-Strategies Gary Mazzaferro AlloyCloud, Inc. William Miller MaCT USA Sanjay Mishra Verizon Doug Scrimager Slalom Consulting Cherry Tom IEEE-SA Wilco van Ginkel Verizon Timothy Zimmerlin Consultant Alicia Zuniga-Alvarado Consultant
Use Case Description 1) Crawl the web; 2) Pre-process data to get searchable things (words, positions); 3) Form Inverted Index mapping words to documents; 4) Rank relevance of documents: PageRank; 5) Lots of technology for advertising, “reverse engineering ranking” “preventing reverse engineering”; 6) Clustering of documents into topics (as in Google News) 7) Update results efficiently Current Solutions Compute(System) Large Clouds Storage Inverted Index not huge; crawled documents are petabytes of text – rich media much more Networking Need excellent external network links; most operations pleasingly parallel and I/O sensitive. High performance internal network not needed Software Map/Reduce + Bigtable; Dryad + Cosmos. PageRank. Final step essentially a recommender engine Big Data Characteristics Data Source (distributed/centralized) Distributed web sites Volume (size) 45B web pages total, 500M photos uploaded each day, 100 hours of video uploaded to YouTube each minute Velocity (e.g. real time) Data continually updated Variety (multiple datasets, mashup) Rich set of functions. After processing, data similar for each page (except for media types) Variability (rate of change) Average page has life of a few months Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Exact results not essential but important to get main hubs and authorities for search query Visualization Not important although page layout critical Data Quality A lot of duplication and spam Data Types Mainly text but more interest in rapidly growing image and video Data Analytics Crawling; searching including topic based search; ranking; recommending Big Data Specific Challenges (Gaps) Search of “deep web” (information behind query front ends) Ranking of responses sensitive to intrinsic value (as in Pagerank) as well as advertising value Link to user profiles and social network data Big Data Specific Challenges in Mobility Mobile search must have similar interfaces/results Security and Privacy Requirements Need to be sensitive to crawling restrictions. Avoid Spam results Highlight issues for generalizing this use case (e.g. for ref. architecture) Relation to Information retrieval such as search of scholarly works. More Information (URLs)
instruments. check data. Information: Offline Analysis L1B Transported Disks copied to (LUSTRE) File System Produce processed data as radar images Matlab Analysis code running in parallel and independently on each data sample ≈2500 cores running standard cluster tools N/A except results checked before release on CReSIS web site Information: L2/L3 Geolocation and Layer Finding Radar Images from L1B Input to Science as database with GIS frontend GIS and Metadata Tools Environment to support automatic and/or manual layer determination GIS (Geographical Information System). Cluster for Image Processing. As above Knowledge, Wisdom, Discovery: Science GIS interface to L2/L3 data Polar Science Research integrating multiple data sources e.g. for Climate change. Glacier bed data used in simulations of glacier flow Exploration on a cloud style GIS supporting access to data. Simulation is 3D partial differential equation solver on large cluster. Varies according to science use. Typically results open after research complete. See Figure 11: Radar Data Analysis for CReSIS Remote Sensing of Ice Sheets– Typical CReSIS radar data after analysis. See See Figure 13: Radar Data Analysis for CReSIS Remote Sensing of Ice Sheets – Typical echogram with detected boundaries. The upper (green) boundary is between air and ice layers, while the lower (red) boundary is between ice and terrain. Earth, Environmental and Polar Science> Use Case 44: UAVSAR Data Processing Use Case Title UAVSAR Data Processing, Data Product Delivery, and Data Services Vertical (area) Scientific Research: Earth Science Author/Company/Email Andrea Donnellan, NASA JPL, ; Jay Parker, NASA JPL, Actors/Stakeholders and their roles and responsibilities NASA UAVSAR team, NASA QuakeSim team, ASF (NASA SAR DAAC), USGS, CA Geological Survey Goals Use of Synthetic Aperture Radar (SAR) to identify landscape changes caused by seismic activity, landslides, deforestation, vegetation changes, flooding, etc.; increase its usability and accessibility by scientists. Use Case Description A scientist who wants to study the after effects of an earthquake examines multiple standard SAR products made available by NASA. The scientist may find it useful to interact with services provided by intermediate projects that add value to the official data product archive. Current Solutions Compute(System) Raw data processing at NASA AMES Pleiades, Endeavour. Commercial clouds for storage and service front ends have been explored. Storage File based.
Summary of Key Properties Information related to five key properties was extracted from each use case. The five key properties were three Big Data characteristics (volume, velocity, and variety), software related information, and associated analytics. The extracted information is presented in Table B-1. Table B-1: Use Case Specific Information by Key Properties Use Case Volume Velocity Variety Software Analytics 1 Census 2000 and 2010 380 TB Static for 75 years Scanned documents Robust archival storage None for 75 years 2 NARA: Search, Retrieve, Preservation Hundreds of terabytes, and growing Data loaded in batches, so bursty Unstructured and structured data: textual documents, emails, photos, scanned documents, multimedia, social networks, web sites, databases, etc. Custom software, commercial search products, commercial databases Crawl/index, search, ranking, predictive search; data categorization (sensitive, confidential, etc.); personally identifiable information (PII) detection and flagging 3 Statistical Survey Response Improvement Approximately 1 PB Variable, field data streamed continuously, Census was ≈150 million records transmitted Strings and numerical data Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig Recommendation systems, continued monitoring 4 Non-Traditional Data in Statistical Survey Response Improvement — — Survey data, other government administrative data, web-scraped data, wireless data, e-transaction data, (potentially) social media data and positioning data from various sources Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig New analytics to create reliable information from non-traditional disparate sources 5 Cloud Eco-System for Finance — Real time — Hadoop RDBMS XBRL Fraud detection
6 Mendeley 15 TB presently, growing about 1 TB per month Currently Hadoop batch jobs scheduled daily, real-time recommended in future PDF documents and log files of social network and client activities Hadoop, Scribe, Hive, Mahout, Python Standard libraries for machine learning and analytics, LDA, reporting tools for aggregating readership and social activities per document 7 Netflix Movie Service Summer 2012 – 25 million subscribers, 4 million ratings per day, 3 million searches per day, 1 billion hours streamed in June 2012; Cloud storage – 2 petabytes in June 2013 Media (video and properties) and rankings continually updated Data vary from digital media to user rankings, user profiles, and media properties for content-based recommendations Hadoop and Pig; Cassandra; Teradata Personalized recommender systems using logistic/linear regression, elastic nets, matrix factorization, clustering, LDA, association rules, gradient-boosted decision trees, and others; streaming video delivery 8 Web Search 45 billion web pages total, 500 million photos uploaded each day, 100 hours of video uploaded to YouTube each minute Real-time updating and real-time responses to queries Multiple media Map/Reduce + Bigtable; Dryad + Cosmos; PageRank; final step essentially a recommender engine Crawling; searching, including topic-based searches; ranking; recommending 9 Business Continuity and Disaster Recovery Within a Cloud Terabytes up to petabytes Can be real time for recent changes Must work for all data Hadoop, Map/Reduce, open source, and/or vendor proprietary such as AWS, Google Cloud Services, and Microsoft Robust backup 10 Cargo Shipping — Needs to become real time, currently updated at events Event-based — Distributed event analysis identifying problems 11 Materials Data 500,000 material types in 1980s, much Ongoing increase in new materials Many datasets with no standards National programs (Japan, Korea, and No broadly applicable analytics
Imaging results per 2D image, 1 TB raw image data + 1 TB analytical results per 3D image, 1 PB data per moderated hospital per year changed with spatial extension feature clustering and classification 18 Computational Bioimaging Medical diagnostic imaging around 70 PB annually, 32 TB on emerging machines for a single scan Volume of data acquisition requires HPC back end Multi-modal imaging with disparate channels of data Scalable key-value and object store databases; ImageJ, OMERO, VolRover, advanced segmentation and feature detection methods Machine learning (support vector machine [SVM] and random forest [RF]) for classification and recommendation services 19 Genomic Measurements >100 TB in 1 to 2 years at NIST, many PBs in healthcare community ≈300 GB of compressed data/day generated by DNA sequencers File formats not though some standards exist; generally structured data Open-source sequencing bioinformatics software from academic groups Processing of raw data to produce variant calls, clinical interpretation of variants 20 Comparative Analysis for Metagenomes and Genomes 50 TB New sequencers stream in data at growing rate Biological data that are inherently heterogeneous, complex, structural, and hierarchical; besides core genomic data, new types of omics data such as transcriptomics, methylomics, and proteomics Standard bioinformatics tools (BLAST, HMMER, multiple alignment and phylogenetic tools, gene callers, sequence feature predictors), Perl/Python wrapper scripts Descriptive statistics, statistical significance in hypothesis testing, data clustering and classification 21 Individualized Diabetes Management 5 million patients Not real time but updated periodically 100 controlled vocabulary values and 1,000 continuous values per patient, mostly time-stamped values HDFS supplementing Mayo internal data warehouse (EDT) Integration of data into semantic graphs, using graph traverse to replace SQL join; development of semantic graph-mining algorithms to identify graph patterns, index graph, and search graph; indexed Hbase; custom code to develop new
patient properties from stored data 22 Statistical Relational Artificial Intelligence for Health Care Hundreds of GBs for a single cohort of a few hundred people; possibly on the order of 1 PB when dealing with millions of patients Constant updates to EHRs; in other controlled studies, data often in batches at regular intervals Critical feature – data typically in multiple tables, need to be merged to perform analysis Mainly Java-based, tools to process the data Relational probabilistic models (Statistical Relational AI) learned from multiple data types 23 World Population-Scale Epidemiological Study 100 TB Low number of data feeding into the simulation, massive amounts of data generated by simulation Can be rich with various population activities, geographical, cultural variations Charm++, MPI Simulations on a synthetic population 24 Social Contagion Modeling for Planning Tens of terabytes per year During social unrest events, human interactions and mobility leads to rapid changes in data; e.g., who follows whom in Twitter Big issues – data fusion, combining data from different sources, dealing with missing or incomplete data Specialized simulators, open source software, proprietary modeling environments; databases Models of behavior of humans and hard infrastructures, models of their interactions, visualization of results 25 Biodiversity and LifeWatch N/A Real-time processing and analysis in case of natural or industrial disaster Rich variety and number of involved databases and observation data RDBMS Requires advanced and rich visualization 26 Large-Scale Deep Learning Current datasets typically 1 TB to 10 TB, possibly 100 million images to train a self-driving car Much faster than real-time processing; for autonomous driving, need to process thousands of high-resolution Neural net very heterogeneous as it learns many different features In-house GPU kernels and MPI-based communication developed by Stanford, C++/Python source Small degree of batch statistical all other data analysis performed by the learning algorithm itself
(six megapixels or more) images per second 27 Organizing Unstructured Collections of Consumer Photos 500+ billion photos on Facebook, 5+ billion photos on Flickr Over 500 million images uploaded to Facebook each day Images and metadata including EXIF (Exchangeable Image File) tags (focal distance, camera type, etc.) Hadoop Map/Reduce, simple hand-written multi-threaded tools (Secure Shell [SSH] and sockets for communication) Robust non-linear least squares optimization problem, SVM 28 Truthy Twitter Data 30 TB/year compressed data Near real-time data storage, querying and analysis Schema provided by social media data source; currently using Twitter only; plans to expand, incorporating Google+ and Facebook Hadoop IndexedHBase and HDFS; Hadoop, Hive, Redis for data management; Python: SciPy NumPy and MPI for data analysis Anomaly detection, stream clustering, signal classification, online learning; information diffusion, clustering, dynamic network visualization 29 Crowd Sourcing in Humanities GBs (text, surveys, experiment values) to hundreds of terabytes (multimedia) Data continuously updated and analyzed incrementally So far mostly homogeneous small data sets; expected large distributed heterogeneous datasets XML technology, traditional relational databases Pattern recognition (e.g., speech recognition, automatic audio-visual analysis, cultural patterns), identification of structures (lexical units, linguistic rules, etc.) 30 CINET for Network Science Can be hundreds of GBs for a single network, 1,000 to 5,000 networks and methods Dynamic networks, network collection growing Many types of networks Graph libraries (Galib, NetworkX); distributed workflow management (Simfrastructure, databases, semantic web tools) Network visualization 31 NIST Information Access Division >900 million web pages occupying 30 TB of storage, 100 million tweets, 100 million biometric Legacy evaluations mostly focused on retrospective analytics, newer evaluations focused on simulations of Wide variety of data types including textual search/extraction, machine translation, speech recognition, image and voice biometrics, object and person PERL, Python, C/C++, Matlab, R development tools; create ground-up test and measurement applications Information extraction, filtering, search, and summarization; image and voice biometrics; speech recognition and understanding; machine
Minuit2 38 Large Survey Data for Cosmology Petabytes of data from Dark Energy Survey 400 images of 1 GB in size per night Images Linux cluster, Oracle RDBMS server, Postgres PSQL, large memory machines, standard Linux interactive hosts, GPFS; for simulations, HPC resources; standard astrophysics reduction software as well as Perl/Python wrapper scripts Machine learning to find optical transients, Cholesky decomposition for thousands of simulations with matrices of order 1 million on a side and parallel image storage 39 Particle Physics at LHC 15 PB of data (experiment and Monte Carlo combined) per year Data updated continuously with sophisticated selection and test analysis but all analyzed "properly" offline Different format for each stage in analysis but data uniform within each stage Grid-based environment with over 350,000 cores running simultaneously Sophisticated specialized data analysis code followed by basic exploratory statistics (histogram) with complex detector efficiency corrections 40 Belle II High Energy Physics Experiment Eventually 120 PB of Monte Carlo and observational data Data updated continuously with sophisticated selection and test analysis but all analyzed "properly" offline Different format for each stage in analysis but data uniform within each stage DIRAC Grid software Sophisticated specialized data analysis code followed by basic exploratory statistics (histogram) with complex detector efficiency corrections 41 EISCAT 3D incoherent scatter radar system Terabytes/year (current), 40 PB/year starting ≈2022 Data updated continuously with real-time test analysis and batch full analysis Big data uniform Custom analysis based on flat file data storage Pattern recognition, demanding correlation routines, high-level parameter extraction 42 ENVRI Environmental Research Infrastructure Low volume (apart from EISCAT 3D given above), one system EPOS ≈15 TB/year Mainly real-time data streams Six separate projects with common architecture for infrastructure, data very diverse across projects R and Python (Matplotlib) for visualization, custom software for processing Data assimilation, (statistical) analysis, data mining, data extraction, scientific modeling and simulation, scientific
workflow 43 CReSIS Remote Sensing Around 1 PB (current) increasing by 50 to 100 TB per mission, future expedition ≈1 PB each Data taken in missions including test analysis and then later batch processing Raw data, images with final layer data used for science Matlab for custom raw data processing, custom image processing software, GIS as user interface Custom signal processing to produce radar images that are analyzed by image processing to find layers 44 UAVSAR Data Processing 110 TB raw data and 40 TB processed, plus smaller samples Data come from aircraft and so incrementally added, data occasionally get reprocessed: new processing methods or parameters Image and annotation files ROI_PAC, GeoServer, GDAL, tools; moving to clouds Process raw data to get images that are run through image processing tools and accessed from GIS 45 NASA LARC/GSFC iRODS MERRA collection (below) represents most of total data, other smaller collections Periodic updates every six months Many applications to combine MERRA reanalysis data with other reanalyses and observational data such as CERES SGE Univa Grid Engine Version 8.1, iRODS Version 3.2 and/or 3.3, IBM GPFS Version 3.4, Cloudera Version Federation software 46 MERRA Analytic Services 480 TB from MERRA Increases at ≈1 TB/month Applications to combine MERRA reanalysis data with other re-analyses and observational data Cloudera, iRODS, Amazon AWS CAaaS 47 Atmospheric Turbulence 200 TB (current), 500 TB within 5 years Data analyzed incrementally Re-analysis datasets are inconsistent in format, resolution, semantics, and metadata; interpretation/analysis of each of these input streams into a common product Map/Reduce or the like, SciDB or other scientific database Data mining customized for specific event types 48 Climate Studies Up to 30 PB/year from 15 end-to-end simulations at NERSC, more at other HPC 42 GB/second from simulations Variety across simulation groups and between observation and simulation National Center for Atmospheric Research (NCAR) PIO library and utilities NCL and NCO, Need analytics next to data storage
(~200 PB) over the next 7 years. In a given year, EOSDIS distributes a volume that is comparable to the overall cumulative archive volume. archive volume. disciplines are represented in EOSDIS. are being investigated. Python, Fortran, C languages. Visualization through tools such as Giovanni. filtering (4) analyzing time series. Web-Enabled Landsat Data (WELD) Processing The data represent the operational time period of 1984 to 2011 for the Landsat 4, 5, and 7 satellites and corresponds to 30PB of processed data through the pipeline (1PB inputs, 10PB intermediate, 6PB outputs) Data was collected over a period of 27 years and is being processed over a period of 5 years. Based on programmatic goals of processing several iterations of the final product over the span of the project, 150TB/day is processed per day during processing time periods. None. This use case basically deals with a single dataset. NEX science platform – data management, workflow processing, provenance capture; WELD science processing algorithms from South Dakota State University (SDSU), browse visualization, and time-series code; Global Imagery Browse Service (GIBS) data visualization platform; USGS data distribution platform. Custom-built application and libraries built on top of libraries. There are number of analytics processes throughout the processing pipeline. The key analytics is identifying best available pixels for composition and spatial aggregation processes as a part of the overall QA. The analytics algorithms are custom developed for this use case.
Use Case Requirements Summary Requirements were extracted from each version 1 use case (the version 2 use cases were not included) within seven characteristic categories introduced in Section 3.1. The number of requirements within each category varied for each use case. Table C-1 contains the use case specific requirements. Table C-1: Use Case Specific Requirements Use Case Data Sources Data Transformation Capabilities Data Consumer Security and Privacy Life Cycle Management Other 1 Census 2010 and 2000 1. Large document format from centralized storage -- 1. Large centralized storage (storage) -- 1. Title 13 data 1. Long-term preservation of data as-is for 75 years 2. Long-term preservation at the bit level 3. Curation process including format transformation 4. Access and analytics processing after 75 years 5. No data loss -- 2 NARA: Search, Retrieve, Preservation 1. Distributed data sources 2. Large data storage 3. Bursty data ranging from GBs to hundreds of terabytes 4. Wide variety of data formats including unstructured and 1. Crawl and index from distributed data sources 2. Various analytics processing including ranking, data categorization, detection of PII data 3. Data 1. Large data storage 2. Various storage systems such as NetApps, Hitachi, magnetic tapes 1. High relevancy and high recall from search 2. High accuracy from categorization of records 3. Various storage systems such as NetApps, 1. Security policy 1. Pre-process for virus scan 2. File format identification 3. Indexing 4. Records categorization 1. Mobile search with similar interfaces/ results from desktop
structured data 5. Distributed data sources in different clouds 4. Long-term preservation management of large varied datasets 5. Huge numbers of data with high relevancy and recall Hitachi, magnetic tapes 3 Statistical Survey Response 1. Data size of approximately one petabyte 1. Analytics for recommendation systems, continued monitoring, and general survey improvement 1. Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig 1. Data visualization for data review, operational activity, and general analysis; continual evolution 1. Improved systems that reduce costs and improve quality while providing confidentiality safeguards that are reliable and publicly auditable 2. Confidential and secure data; processes that are auditable for security and confidentiality as required by various legal statutes 1. High veracity on data and very robust systems (challenges: semantic integrity of conceptual metadata concerning what exactly is measured and the resulting limits of inference) 1. Mobile access 4 Data in -- 1. Analytics to create reliable estimates using data from 1. Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, 1. Data visualization for data review, 1. Confidential and secure data; processes that are 1. High veracity on data and very robust systems (challenges: semantic integrity of --
System 3. Various analytic processing techniques for consumer personalization 4. Robust learning algorithms 5. Continued analytic processing based on monitoring and performance results searches per day (DB) 5. Huge amounts of storage (2 PB) 6. I/O intensive processing 8 Web Search 1. Distributed data sources 2. Streaming data 3. Multimedia content 1. Dynamic fetching content over the network 2. Linking of user profiles and social network data 1. Petabytes of text and rich media (storage) 1. Search time of ≈0.1 seconds 2. Top 10 ranked results 3. Page layout (visual) 1. Access control 2. Protection of sensitive content 1. Data purge after certain time interval (a few months) 2. Data cleaning 1. Mobile search and rendering 9 Business Continuity and Disaster Recovery Within a Cloud -- 1. Robust backup algorithm 2. Replication of recent changes 1. Hadoop 2. Commercial cloud services -- 1. Strong security for many applications -- -- 10 Cargo Shipping 1. Centralized and real-time distributed sites/sensors 1. Tracking items based on the unique identification with its sensor information, GPS coordinates 1. Internet connectivity -- 1. Security policy -- --
(storage) 7. Scalable and object store (platform) 8. Data streams from peta/exascale centralized simulation systems 13 Large-Scale Geospatial Analysis and Visualization 1. Unique approaches to indexing and distributed analysis required for geospatial data 1. Analytics: closest point of approach, deviation from route, point density over time, PCA and ICA 2. Unique approaches to indexing and distributed analysis required for geospatial data 1. Geospatially enabled RDBMS, geospatial server/analysis software, e.g., ESRI ArcServer, Geoserver 1. Visualization with GIS at high and low network bandwidths and on dedicated facilities and handhelds 1. Complete security of sensitive data in transit and at rest (particularly on handhelds) -- -- 14 Object and Tracking 1. Real-time data FMV (30 to 60 frames/ second at full-color 1080P resolution) and WALF (1 to 10 frames/ second at 10,000 x 10,000 full-color resolution) 1. Rich analytics with object identification, pattern recognition, crowd behavior, economic activity, and data fusion 1. Wide range of custom software and tools including traditional RDBMSs and display tools 2. Several network requirements 3. GPU usage important 1. Visualization of extracted outputs as overlays on a geospatial display; links back to the originating image/video segment as overlay objects 1. Significant security and privacy issues; sources and methods never compromised 1. Veracity of extracted objects --
Imaging pathology images 2. Various image quality analyses algorithms 3. Various image data formats, especially BigTIFF with structured data for analytical results 4. Image analysis, spatial queries and analytics, feature clustering, and classification image analysis to extract spatial information 2. Spatial queries and analytics, feature clustering and classification 3. Analytic processing on huge multi-dimensional large dataset; correlation with other data types such as clinical data, omic data (computing cluster) 2. Huge legacy and new storage such as storage area network (SAN) or HDFS (storage) 3. network link (networking) 4. MPI image analysis, Map/Reduce, Hive with spatial extension (software packages) for validation and training protection for protected health information validation and rendering on mobile platforms 18 Bioimaging 1. Distributed multi-modal experimental sources of bioimages (instruments) 2. 50 TB of data in formats that include images 1. High-throughput computing with responsive analysis 2. Segmentation of regions of interest; crowd-based selection and extraction of features; object classification, and organization; and search 3. Advanced biosciences discovery through Big Data 1. ImageJ, OMERO, VolRover, advanced segmentation and feature detection methods from applied math researchers; scalable key-value and object store databases needed 2. NERSC’s Hopper infrastructure 3. database and 1. 3D structural modeling 1. Significant but optional security and privacy including secure servers and anonymization 1. Workflow components including data acquisition, storage, enhancement, minimizing noise --
techniques / extreme-scale computing; processing and analytics; machine learning (SVM and RF) for classification and recommendation services; advanced algorithms for massive image analysis; computational solutions 4. Massive data analysis toward massive imaging datasets. image collections 4. 10 GB and future 100 GB and advanced networking (software defined networking [SDN]) 19 Genomic 1. compressed data (300 GB/day) from various DNA sequencers 2. Distributed data source (sequencers) 3. Various file formats with both structured and unstructured data 1. Processing raw data in variant calls 2. Challenge: characterizing machine learning for complex analysis on systematic errors from sequencing technologies 1. Legacy computing cluster and other PaaS and IaaS (computing cluster) 2. Huge data storage in PB range (storage) 3. Unix-based legacy sequencing bioinformatics software (software 1. Data format for genome browsers 1. Security and privacy protection of health records and clinical research databases -- 1. Mobile platforms for physicians accessing genomic data (mobile device)
package) 20 Comparative Analysis for and Genomes 1. Multiple centralized data sources 2. Proteins and their structural features, core genomic data, new types of omics data such as transcriptomics, methylomics, and proteomics describing gene expression 3. Front real-time web UI interactive; backend data loading processing that keeps up with exponential growth of sequence data due to the rapid drop in cost of sequencing technology 4. Heterogeneous, complex, structural, and hierarchical 2. Scalable RDBMS for heterogeneous biological data 2. Real-time rapid and parallel bulk loading 3. Oracle RDBMS, SQLite files, flat text files, Lucy (a version of Lucene) for keyword searches, BLAST databases, USEARCH databases 4. Linux cluster, Oracle RDBMS server, large memory machines, standard Linux interactive hosts 5. Sequencing and comparative analysis techniques for highly complex data 6. Descriptive statistics 1. Huge data storage 1. Real-time interactive parallel bulk loading capability 2. Interactive Web UI, backend batch job computation submission from the UI. 3. Download of assembled and annotated datasets for offline analysis 4. Ability to query and browse data via interactive web UI 5. Visualize data structure at different levels of resolution; ability to view abstract of highly similar data 1. Login security: username and password 2. Creation of user account to submit and access dataset to system via web interface 3. Single capability (SSO) 1. Methods to improve data quality 2. Data clustering, classification, reduction 3. Integration of new data/content into the system’s data store and data annotation --
biological data 5. Metagenomic samples that can vary by several orders of magnitude, such as several hundred thousand genes to a billion genes 21 Diabetes 1. Distributed EHR data 2. Over 5 million patients with thousands of properties each and many more derived from primary values 3. Each record: a range of 100 to 100,000 data property values, average of 100 controlled vocabulary values, and average of 1,000 continuous values 4. No real-time, but data updated periodically; data timestamped with the time of observation (time 1. Data integration using ontological annotation and taxonomies 2. Parallel retrieval algorithms for both indexed and custom searches; identification of data of interest; patient cohorts, patients’ meeting certain criteria, patients sharing similar characteristics 3. Distributed graph mining algorithms, pattern analysis and graph indexing, pattern searching on RDF triple graphs 4. Robust statistical analysis tools to 1. data warehouse, open source indexed Hbase 2. supercomputers, cloud and parallel computing 3. I/O intensive processing 4. HDFS storage 5. custom code to develop new properties from stored data. 1. Efficient data visualization needed 1. Protection of health data in accordance with privacy policies and legal requirements, e.g., HIPAA. 2. Security policies for different user roles 1. Data annotated based on domain ontologies or taxonomies 2. Traceability of data from origin (initial point of collection) through use 3. Data conversion from existing data warehouse into RDF triples 1. Mobile access
the value is recorded) 5. Two main categories of structured data about a patient: data with controlled vocabulary (CV) property values and data with continuous property values (recorded/ captured more frequently) 6. Data consist of text and continuous numerical values manage false discovery rates, determine true sub-graph significance, validate results, eliminate false positive/false negative results 5. Semantic graph mining algorithms to identify graph patterns, index and search graph 6. Semantic graph traversal 22 Statistical Relational Artificial Intelligence for Health Care 1. Centralized data, with some data retrieved from Internet sources 2. Range from hundreds of GBs for a sample size to 1 PB for very large studies 3. Both constant updates/additions (to data subsets) and scheduled batch inputs 1. Relational probabilistic models/ probability theory; software that learns models from multiple data types and can possibly integrate the information and reason about complex queries 2. Robust and accurate learning methods to 1. Java, some in house tools, [relational] database and NoSQL stores 2. Cloud and parallel computing 3. computer, 48 GB RAM (to perform analysis for a moderate sample size) 1. Visualization of very large data subsets 1. Secure handling and processing of data 1. Merging multiple tables before analysis 2. Methods to validate data to minimize errors --
4. Large, longitudinal data 5. Rich relational data comprising multiple tables, different data types such as imaging, EHR, demographic, genetic, and natural language data requiring rich representation 6. Unpredictable arrival rates, often real time account for data imbalance (where large numbers of data are available for a small number of subjects) 3. Learning algorithms to identify skews in data, so as to not to (incorrectly) model noise 4. Generalized and refined learned models for application to diverse sets of data 5. Challenge: acceptance of data in different modalities (and from disparate sources) 4. Dlusters for large datasets 5. 200 GB–1 TB hard drive for test data 23 World Population Scale Study 1. File-based synthetic population, either centralized or distributed sites 2. Large volume of real-time output data 3. Variety of output datasets depending on the model’s 1. and computation, like supercomputer performance 2. Unstructured and irregular nature of graph processing 3. Summary of various runs of 1. Movement of very large volume of data for visualization (networking) 2. Distributed MPI-based simulation system (platform) 3. Charm++ on multi-nodes (software) 1. Visualization 1. Protection of PII on individuals used in modeling 2. Data protection and secure platform for computation 1. Data quality, ability to capture the traceability of quality from computation --
complexity simulation 4. Network file system (storage) 5. Infiniband network (networking) 24 Social Contagion Modeling for Planning 1. Traditional and new architecture for dynamic distributed processing on commodity clusters 2. Fine-resolution models and datasets to support Twitter network traffic 3. Huge data storage supporting annual data growth 1. Large-scale modeling for various events (disease, emotions, behaviors, etc.) 2. Scalable fusion between combined datasets 3. Multi-level analysis while generating sufficient results quickly 1. Computing infrastructure that can capture human-to-human interactions on various social events via the Internet (infrastructure) 2. File servers and databases (platform) 3. Ethernet and Infiniband networking (networking) 4. Specialized simulators, open source software, and proprietary modeling (application) 5. Huge user accounts across country boundaries (networking) 1. Multi-level detailed network 2. Visualization with interactions 1. Protection of PII of individuals used in modeling 2. Data protection and secure platform for computation 1. Data fusion from variety of data sources (i.e., Stata data files) 2. Data consistency and no corruption 3. Preprocessing of raw data 1. Efficient method of moving data 25 Biodiversity and 1. Special dedicated or overlay sensor 1. Web-based services, services, 1. Expandable storage resource 1. Access by mobile users 2. Advanced/ 1. Federated identity management 1. Data storage and archiving, data exchange and --
LifeWatch network 2. Storage: distributed, historical, and trends data archiving 3. Distributed data sources, including observation and monitoring facilities, sensor network, and satellites 4. Wide variety of data: satellite images/ information, climate and weather data, photos, video, sound recordings, etc. 5. Multi-type data combination and linkage, potentially unlimited data variety 6. Data streaming relational databases, NoSQL 2. Personalized virtual labs 3. Grid- and resources 4. Data analyzed incrementally and/or in real time at varying rates owing to variations in source processes 5. A variety of data and analytical and modeling tools to support analytics for diverse scientific communities 6. Parallel data streams and streaming analytics 7. Access and integration of multiple distributed databases for global users 2. Cloud community resource required visualization 3. 4D visualization computational models for mobile researchers and mobile sensors 2. Access control and accounting integration 2. Data life cycle management: data provenance, referral integrity and identification traceability back to initial observational data 3. Processed (secondary) data storage (in addition to original source data) for future uses 4. Provenance (and persistent identification [PID]) control of data, algorithms, and workflows 5. Curated (authorized) reference data (e.g. species name lists), algorithms, software code, workflows 26 Large-Scale Deep Learning -- -- 1. GPU 2. MPI and HPC Infiniband cluster -- -- -- --
Science study graph theoretic properties and behaviors of various algorithms 2. Asynchronous and real-time synchronous distributed computing analysis tools 2. Dynamic growth of the networks 3. Asynchronous and real-time synchronous distributed computing 4. Different parallel algorithms for different partitioning schemes for efficient operation network connectivity (networking) 3. Existing computing cluster 4. EC2 computing cluster 5. Various graph libraries, management tools, databases, semantic web tools 31 NIST Information Access Division 1. Large amounts of web pages, tweets, images, video 2. Scaling to larger data, intrinsic and annotation uncertainty measurement, performance measurement for incompletely annotated data, measuring analytic performance for heterogeneous data and analytic flows involving 1. Test analytic algorithms working with written language, speech, human imagery, etc. against real or realistic data; challenge: engineering artificial data that sufficiently captures the variability of real data involving humans 1. PERL, Python, C/C++, Matlab, R development tools; creation of ground-up test and measurement applications 1. Analytic flows involving users 1. Security requirements for protecting sensitive data while enabling meaningful developmental performance evaluation; shared evaluation testbeds that protect the intellectual property of analytic algorithm developers -- --
users 32 DataNet (iRODS) 1. Process key format types NetCDF, HDF5, Dicom 2. Real-time and batch data 1. Provision of general analytics workflows needed 1. iRODS data management software 2. interoperability across storage and network protocol types 1. General visualization workflows 1. Federate across existing authentication environments through Generic Security Service API and pluggable authentication modules (GSI, Kerberos, InCommon, Shibboleth) 2. Access controls on files independent of the storage location -- -- 33 The Discinnet Process 1. Integration of metadata approaches across disciplines -- 1. Software: Symfony-PHP, Linux, MySQL -- 1. Significant but optional security and privacy including secure servers and anonymization 1. Integration of metadata approaches across disciplines -- 34 Semantic 1. All data types, image to text, structures to protein sequence 1. Data graph processing 2. RDBMS 1. Cloud community resource required 1. Efficient visualization needed -- -- -- 35 Light source 1. Multiple streams of 1. Standard bioinformatics 1. High-volume data transfer to -- 1. Multiple security and -- --
beamlines data to be stored and analyzed later 2. Sample data to be analyzed in real time tools (BLAST, HMMER, multiple alignment and phylogenetic tools, gene callers, sequence feature predictors, etc.), Perl/Python wrapper scripts, Linux Cluster scheduling remote batch processing resource privacy requirements to be satisfied 36 Catalina Real-Time Transient Survey 1. ≈0.1 TB per day at present, will increase by factor of 100 1. A wide variety of the existing astronomical data analysis tools, plus a large number of custom developed tools and software programs, some research projects in and of themselves 2. Automated classification with machine learning tools given the very sparse and heterogeneous data, dynamically evolving in time as more data come in, with follow-up decision making reflecting limited follow-up -- 1. Visualization mechanisms for highly dimensional data parameter spaces -- -- --
resources 37 DOE Extreme Data from Sky Survey 1. ≈1 PB/year becoming 7 PB/year of observational data 1. Advanced analysis and visualization techniques and capabilities to support interpretation of results from detailed simulations 1. MPI, OpenMP, C, C++, F90, FFTW, viz packages, Python, FFTW, numpy, Boost, OpenMP, ScaLAPCK, PSQL and MySQL databases, Eigen, cfitsio, astrometry.net, and Minuit2 2. Methods/ tools to address supercomputer I/O subsystem limitations 1. Interpretation of results using advanced visualization techniques and capabilities -- -- -- 38 Large Survey Data for Cosmology 1. 20 TB of data/day 1. Analysis on both the simulation and observational data simultaneously 2. Techniques for handling Cholesky decomposition for thousands of simulations with matrices of order 1 million on a side 1. Standard astrophysics reduction software as well as Perl/Python wrapper scripts 2. Oracle RDBMS, Postgres psql, GPFS and Lustre file systems and tape archives 3. Parallel image storage -- -- 1. Links between remote telescopes and central analysis sites -- 39 Particle Physics at LHC 1. Real-time data from accelerator and analysis instruments 1. Experimental data from ALICE, ATLAS, CMS, LHB 2. Histograms, 1. Legacy computing infrastructure (computing 1. Histograms and model fits (visual) 1. Data protection 1. Data quality on complex apparatus --
2. Asynchronization data collection 3. Calibration of instruments scatter-plots with model fits 3. Monte-Carlo computations nodes) 2. Distributed cached files (storage) 3. Object databases (software package) 40 Belle II Physics Experiment 1. 120 PB of raw data -- 1. 120 PB raw data 2. International distributed computing model to augment that at accelerator (Japan) 3. Data transfer of ≈20 GB/ second at designed luminosity between Japan and United States 4. Software from Open Science Grid, Geant4, DIRAC, FTS, Belle II framework -- 1. Standard grid authentication -- -- 41 EISCAT 3D Incoherent Scatter Radar System 1. Remote sites generating 40 PB data/year by 2022 2. Hierarchical Data Format (HDF5) 3. Visualization of high-dimensional 1. Queen Bea architecture with mix of distributed on-sensor and central processing for 5 distributed sites 2. Real-time 1. Architecture compatible with ENVRI 1. Support needed for visualization of (≥5) data -- 1. Preservation of data and avoidance of lost data due to instrument malfunction 1. Support needed for real-time monitoring of equipment by partial streaming analysis
(≥5) data monitoring of equipment by partial streaming analysis 3. Hosting needed for rich set of radar image processing services using machine learning, statistical modelling, and graph algorithms 42 ENVRI Research 1. Huge volume of data from distributed data sources 2. Variety of instrumentation datasets and metadata 1. Diversified analytics tools 1. Variety of computing infrastructures and architectures (infrastructure) 2. Scattered repositories (storage) 1. Graph plotting tools 2. Time series interactive tools 3. flash playback 4. Earth map display 5. Visual tools for quality comparisons 1. Open data policy with minor restrictions 1. High data quality 2. Mirror archives 3. Various metadata frameworks 4. Scattered repositories and data curation 1. Various kinds of mobile sensor devices for data acquisition 43 CReSIS Remote Sensing 1. Provision of reliable data transmission from aircraft sensors/ instruments or removable disks from remote sites 2. Data gathering in real time 1. Legacy software (Matlab) and language (C/Java) binding for processing 2. Signal processing and advanced image processing to find 1. ≈0.5 PB/year of raw data 2. Transfer content from removable disk to computing cluster for parallel processing 3. Map/Reduce or 1. GIS user interface 2. Rich user interface for simulations 1. Security and privacy on sensitive political issues 2. Dynamic security and privacy policy mechanisms 1. Data quality assurance 1. Monitoring data collection instruments/ sensors
3. Varieties of datasets layers needed MPI plus language binding for C/Java 44 UAVSAR Data Processing 1. Angular and spatial data 2. Compatibility with other NASA radar systems and repositories (Alaska Satellite Facility) 1. Geolocated data that require GIS integration of data as custom overlays 2. Significant human intervention in data processing pipeline 3. Hosting of rich set of radar image processing services 4. ROI_PAC, GeoServer, GDAL, tools 1. Support for interoperable Cloud-HPC architecture 2. Hosting of rich set of radar image processing services 3. ROI_PAC, GeoServer, GDAL, tools 4. Compatibility with other NASA radar systems and repositories (Alaska Satellite Facility) 1. Support for field expedition users with phone/tablet interface and low-resolution downloads -- 1. Significant human intervention in data processing pipeline 2. Rich robust provenance defining complex machine/human processing 1. Support for field expedition users with phone/tablet interface and downloads 45 NASA LARC/ GSFC iRODS 1. Federate distributed heterogeneous datasets 1. CAaaS on clouds 1. Support virtual climate data server (vCDS) 2. GPFS parallel file system integrated with Hadoop 3. iRODS 1. Support needed to visualize distributed data -- -- -- 46 MERRA Analytic Services 1. Integrate simulation output and observational data, NetCDF files 2. Real-time and batch mode needed 1. CAaaS on clouds 1. NetCDF aware software 2. Map/Reduce 3. Interoperable use of AWS and local clusters 1. High-end distributed visualization -- -- 1. Smart phone and tablet access required 2. iRODS data management
3. Interoperable use of AWS and local clusters 4. iRODS data management 47 Atmospheric Turbulence 1. Real-time distributed datasets 2. Various formats, resolution, semantics, and metadata 1. Map/Reduce, SciDB, and other scientific databases 2. Continuous computing for updates 3. Event specification language for data mining and event searching 4. Semantics interpretation and optimal structuring for 4D data mining and predictive analysis 1. Other legacy computing systems (e.g. supercomputer) 2. high throughput data transmission over the network 1. Visualization to interpret results -- 1. Validation for output products (correlations) -- 48 Climate Studies 1. ≈100 PB data in 2017 streaming at high data rates from large supercomputers across the world 2. Integration of large-scale distributed data from simulations with diverse observations 3. Linking of 1. Data analytics close to data storage 1. Extension of architecture to several other fields 1. Worldwide climate data sharing 2. High-end distributed visualization -- -- 1. input and access
diverse data to novel HPC simulation 49 DOE-BER Subsurface 1. Heterogeneous diverse data with different domains and scales, translation across diverse datasets that cross domains and scales 2. Synthesis of diverse and disparate field, laboratory, omic, and simulation datasets across different semantic, spatial, and temporal scales 3. Linking of diverse data to novel HPC simulation -- 1. Postgres, HDF5 data technologies, and many custom software systems 1. input and access -- -- 1. input and access 50 DOE-BER AmeriFlux and FLUXNET Networks 1. Heterogeneous diverse data with different domains and scales, translation across diverse datasets that cross domains and scales 1. Custom software such as EddyPro, and custom analysis software, such as R, Python, neural networks, Matlab 1. Custom software, such as EddyPro, and custom analysis software, such as R, Python, neural networks, Matlab 2. Analytics including data 1. input and access -- -- 1. input and access
2. Link to many other environment and biology datasets 3. Link to HPC climate and other simulations 4. Link to European data sources and projects 5. Access to data from 500 distributed sources mining, data quality assessment, cross-correlation across datasets, data assimilation, data interpolation, statistics, quality assessment, data fusion, etc. 51 Forecasting in Smart Grids 1. Diverse data from smart grid sensors, city planning, weather, utilities 2. Data updated every 15 minutes 1. New machine learning analytics to predict consumption 1. SQL databases, CVS files, HDFS (platform) 2. R/Matlab, Weka, Hadoop (platform) -- 1. Privacy and anonymization by aggregation -- 1. Mobile access for clients
Truthy Twitter Data Data Consumer Requirements: Needs to support data retrieval and dynamic visualization. Needs to support data-driven interactive web interfaces. Needs to support API for data query. CINET for Network Science Data Consumer Requirements: Needs to support client-side visualization. NIST Information Access Division Data Consumer Requirements: Needs to support analytic flows involving users. DataNet (iRODS) Data Consumer Requirements: Needs to support general visualization workflows. Semantic Graph-Search Data Consumer Requirements: Needs to support efficient data-graph-based visualization. Catalina Real-Time Transient Survey Data Consumer Requirements: Needs to support visualization mechanisms for highly data parameter spaces. DOE Extreme Data from Cosmological Sky Survey Data Consumer Requirements: Needs to support interpretation of results using advanced visualization techniques and capabilities. Particle Physics at LHC Data Consumer Requirements: Needs to support histograms and model fits (visual). EISCAT 3D Incoherent Scatter Radar System Data Consumer Requirements: Needs to support visualization of high-dimensional (≥5) data. ENVRI Environmental Research Infrastructure Data Consumer Requirements: Needs to support graph-plotting tools. Needs to support time series interactive tools. Needs to support browser-based flash playback. Needs to support earth high-resolution map displays. Needs to support visual tools for quality comparisons. CReSIS Remote Sensing Data Consumer Requirements: Needs to support GIS user interface. Needs to support rich user interface for simulations. UAVSAR Data Processing Data Consumer Requirements: Needs to support field expedition users with phone/tablet interface and low-resolution downloads. NASA LARC/GSFC iRODS Data Consumer Requirements: Needs to support visualization of distributed heterogeneous data. MERRA Analytic Services Data Consumer Requirements: Needs to support high-end distributed visualization. Atmospheric Turbulence Data Consumer Requirements: Needs to support visualization to interpret results. Climate Studies Data Consumer Requirements: Needs to support worldwide climate data sharing. Needs to support high-end distributed visualization. DOE-BER Subsurface Biogeochemistry Data Consumer Requirements: Needs to support phone-based input and access. DOE-BER AmeriFlux and FLUXNET Networks Data Consumer Requirements: Needs to support phone-based input and access. Table D-5: Security and Privacy
4.8.4 What events are currently audited? * All data access must be audited Off Only selected / protected data must be audited Off Maintenance on user roles must be audited (new users, disabled user, updated roles or permissions) Off Purge and archive events Off Domain-dependent events (e.g., adding a new sensor) Off REST or SOAP events Off Changes in system configuration Off Organizational changes Off External project ownership / management changes Off Requirements are externally set, e.g., by PCI compliance Off Domain-specific events (patient death in a drug trial) Off Other: Yes Other text None 4.9 Application Provider Security 4.9.1 Describe Application Provider Security * There is a security mechanism implemented at the application level Off The app provider level is aware of PII or privacy data elements Off The app provider implements audit and logging Off The app provider security relies on framework-level security for its operation Off Does not apply to our application Yes Other: Off Other text 4.10 Framework Provider Security
4.10.1 Describe the framework provider security * Security is implemented at the framework level Off Roles can be defined at the framework level Off The framework level is aware of PII or related sensitive data Off Does not apply in our setting Yes Is provided by the Big Data tool Off Other: Off Other text 4.11 System Health 4.11.1 Measures to Ensure Availability * Deterrents to man-in-the-middle attacks Off Deterrents to denial of service attacks Off Replication, redundancy or other resilience measures Off Deterrents to data corruption, drops or other critical big data components Off Other: Yes Other text System resources are provided by the NASA Advanced Supercomputing Division (NAS) for the use case; NAS has responsibility for system availability 4.12 Permitted Use Cases 4.12.1 Describe Domain-specific Limitations on Use None 4.12.2 Paywall A paywall is in use at some stage in the workflow Off Not applicable Yes 5 Classify Use Cases with Tags Use Case 2-2