Use Case Requirements Summary Use Case Data Sources Transformation Capabilities Data Consumer Security & Privacy Lifecycle Management Others 1 Census 2010 and 2000 1. large document format from a centralized storage -- 1. large centralized storage (storage) -- 1. Title 13 data 1. long-term preservation of data as-is for 75 years 2. long-term preservation at the bit-level 3. curation process including format transformation 4. access and analytics processing after 75 years 5. needs to make sure no data loss -- 2 NARA: Search, Retrieve, 1. distributed data sources 2. large data storage 3. bursty data range from GB to hundreds of TB 4. wide variety of data formats including unstructured and structured data 5. distributed data sources in different clouds 1. crawl and index from distributed data sources 2. various analytics processing including ranking, data categorization, detect PII data 3. pre-processing of data 4. long-term preservation management fo large varied datasets. 5. hugh amount of data with high relevancy and recall. 1. large data storage 2. various storages such as NetApps, Hitachi, Magnetic tapes 1. high relevancy and high recall from search 2. high accuracy from categorization of records 3. various storages such as NetApps, Hitachi, Magnetic tapes 1. security policy 1. pre-process for virus scan 2. file format identification 3. indexing 4. categorize records 1. mobile search with similar interfaces/results from desktop 3 Statistical Survey Response 1. data size approximately one petabyte 1. analytics are required for recommendation systems, continued monitoring and general survey improvement. 1. software includes Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig 1. data visualization for data review, operational activity and general analysis. It continues to evolve. 1. improving recommendation systems that reduce costs and improve quality while providing confidentiality safeguards that are reliable and publically auditable. 2. both confidential and secure all data. 1. high veracity on data and systems must be very robust. The semantic integrity of conceptual metadata concerning what exactly is measured and the resulting limits of inference remain a 1. mobile access
consumer personalization 4. robust learning algorithms 5. continued analytic processing based on the monitoring and performance results 6. I/O intensive processing 8 Web Search 1. Needs to support distributed data sources 2. Needs to support streaming data 3. Needs to support multimedia content 1. dynamic fetching content over the network 2. linking user profiles and social network data 1. petabytes of text and rich media (storage) 1. search time in ~0.1 seconds 2. top 10 ranked results 3. page layout (visual) 1. access control 2. needs to protect sensitive content 1. purge data after certain time interval (few months) 2. data cleaning 1. mobile search and rendering 9 BC/DR) Within A Cloud -- 1. robust Backup algorithm 2. replicate recent changes 1. Hadoop 2. commercial cloud services -- 1. strong security for many applications -- -- 10 Cargo Shipping 1. centralized and real time distributed sites/sensors 1. tracking items based on the unique identification with its sensor information, GPS coordinates 2. real time updates on tracking items 1. Internet connectivity -- 1. security policy -- -- 11 Materials Data for 1. distributed data repositories for more than 500,000 commerical materials 2. may varieties of datasets 3. text, graphics, and images 1. (100s) of independent variables by collecting these variables to create robust datasets -- 1. visualization for materials discovery from many independent variables 2. visualization tools for materials 1. protection of proprietary of sensitive data 2. tools to mask proprietary information 1. how to handle data quality is poor or unknown -- 12 Simulation driven Materials 1. data streams from peta/exascale centralized simulation 1. computing data analysis for 1. massive (150K cores) of legacy infrastructure (infrastructure) 2. GPFS 1. browser-based to search growing material data 1. sandbox as independent working areas between different data stakeholders 1. validation and UQ of simulation with experimental data 1. mobile apps to access materials geonics information
Genomics systems 2. distributed web dataflows from central gateway to users responsiveness 2. mashup of simulation outputs across codes 3. search and crowd-driven with computation backend be flexibly for new targets 4. MapReduce and search to join simulation and experimental data (General Parallel File Sysem) (storage) 3. MonogDB systems (platform) 4. 10Gb networking 5. various of analytic tools such as PyMatGen, FireWorks, VASP, ABINIT, NWChem, BerkeleyGW, varied community codes 6. large storage (storage) 7. scalable and object store (platform) 8. data streams from peta/exascale centralized simulation systems 2. policy-driven federation of datasets 2. UQ in results from multiple datasets 13 Large Scale Geospatial Analysis and 1. geospatial data requires unique approaches to indexing and distributed analysis. 1. analytics include Closest point of approach, deviation from route, point density over time, PCA and ICA 2. geospatial data requires unique approaches to indexing and distributed analysis. 1. software includes Geospatially enabled RDBMS, Geospatial server/analysis software – ESRI ArcServer, Geoserver 1. visualization with GIS at high and low network bandwidths and on dedicated facilities and handhelds 1. sensitive data and must be completely secure in transit and at rest (particularly on handhelds) -- -- 14 Object and tracking 1. real-time data FMV – 30-60 frames per/sec at full color 1080P resolution and WALF – 1-10 frames per/sec at 10Kx10K full color resolution. 1. Rich analytics with object identification, pattern recognition, crowd behavior, economic activity and data fusion 1. wide range custom software and tools including traditional RDBM’s and display tools. 2. several network requirements 3. GPU Usage important 1. visualization of extracted outputs will typically be as overlays on a geospatial display. Overlay objects should be links back to the originating image/video segment. 1. significant security and privacy, sources and methods cannot be compromised the enemy should not be able to know what we see. 1. veracity of extracted objects --
nominal, discrete ordinal, discrete structured, binary large blobs (images and video). 5. data evolves over time in a highly variable fashion may be unique to a subset of the data. 5. robust security to prevent data breaches. 17 Pathology Imaging 1. high resolution spatial digitized pathology images 2. various image quality analysis algorithms 3. various image data formats especially BIGTIFF with structured data for analytical results 4. image analysis, spatial queries and analytics, feature clustering and classification 1. high performance image analysis to extract spatial information 2. spatial queries and analytics, and feature clustering and classification 3. analytic processing on huge large dataset and be able to correlate with other data types such as clinical data, -omic data. 1. legacy system and cloud (computing cluster) 2. huge legacy and new storage such as SAN or HDFS (storage) 3. high throughput network link (networking) 4. MPI image analysis, MapReduce, Hive with spatial extension (sw pkgs) 1. visualization for validation and training 1. security and privacy protection for protected health information 1. human annotations for validation 1. 3D visualization and rendering on mobile platforms 18 1. distributed multi-modal high resolution experimental sources of bioimages (instruments). 2. 50TB of data data formats include images. 1. computing with responsive analysis 2. segmentation of regions of interest, selection and extraction of features, and object classification, and organization, and search. 3. advance biosciences discovery through big data techniques / extreme scale computing…. processing and analytics. … Machine learning 1. ImageJ, OMERO, VolRover, advanced segmentation and feature detection methods from applied math researchers.... Scalable and object store databases needed. 2. NERSC.s Hopper infrastructure 3. database and image collections. 4. 10 GB and future 100 GB and advanced networking (SDN). 1. 3D structural modeling 1. significant but optional security & privacy including secure servers and anonymization 1. workflow components include data acquisition, storage, enhancement, minimizing noise -- dimensional
drop in cost of sequencing technology. 4. heterogeneous, complex, structural, and hierarchical biological data. 5. metagenomic samples can vary by several orders of magnitude, such as several hundred thousand genes to a billion genes interactive hosts 5. sequencing and comparative analysis techniques for highly complex data 6. descriptive statistics levels of resolution. Ability to view abstract representations of highly similar data. 21 Diabetes 1. distributed EHR data 2. over 5 million patients with thousands of properties each and many more that are derived from primary values. 3. each record, a range of 100 to 100,000 data property values average of 100 controlled vocabulary values and 1000 continuous values. 4. none real-time but data updated periodically. Data is timestamped with the time of observation [time that the value is recorded.] 5. structured data about a patient falls into two main categories: data with controlled vocabulary [CV] property values, and continuous property values [which are recorded / 1. data integration, using ontological annotation and taxonomies 2. parallel retrieval algorithms for both indexed and custom searches, identify data of interest. Patient cohorts, patients meeting certain criteria, patients sharing similar characteristics 3. distributed graph mining algorithms, pattern analysis and graph indexing, pattern searching on RDF triple graphs 4. robust statistical analysis tools to manage false discovery rate, determine true subgraph significance, validate results, eliminate false positive / false negative results 5. semantic graph mining algorithms to 1. data warehouse, open source indexed Hbase 2. supercomputers, cloud and parallel computing 3. I/O intensive processing 4. HDFS storage 5. custom code to develop new properties from stored data. 1. efficient based visualization is needed 1. protection of health data in accordance with legal requirements - eg, HIPAA - and privacy policies. 2. security policies for different user roles. 1. data annotated based on domain ontologies or taxonomies. 2. to ensure traceability of data, from origin [initial point of collection] through to use. 3. convert data from existing data warehouse into RDF triples 1. mobile access
captured more frequently]. 6. data consists of text, and Continuous Numerical values. identify graph patterns, index and search graph. 6. Semantic graph traversal. 22 Statistical Relational Artificial for Health Care 1. centralized data, with some data retrieved from internet sources 2. range from 100.s of GB sample size, to 1 petabyte for very large studies 3. both constant updates / additions [to subset of data] and scheduled batch inputs 4. large, longitudinal data 5. rich relational data comprised of multiple tables, different data types such as imaging, EHR, demographic, genetic and natural language data requiring rich representation 6. unpredictable arrival rates, in many cases data arrive in 1. relational probabalistic models / probability therory, software learns models from multiple data types and can possibly integrate the information and reason about complex queries. 2. robust and accurate learning methods to account for .data imbalance. [where large amount of data is available for a small number of subjects] 3. learning algorithms to identify skews in data, so as to not -- incorrectly -- model .noise. 4. learned models can be generalized and refined in order to be applied to diverse sets of data 5. challenging, must accept data in different modalities [and from disparate sources] 1. Java, some in house tools, [relational] database and NoSQL stores 2. cloud and parallel computing 3. high performance computer, 48 GB RAM [to performa analysis for a moderate sample size] 4. clusters for large datasets 5. 200 GB - 1 TB hard drive for test data 1. visualization of subsets of very large data 1. secure handling and processing of data is of crucial importance in medical domains 1. merging multiple tables before analysis 2. methods to validate data to minimize errors -- 23 World Population Scale 1. file-based synthetic population either centralized or distributed sites 2. large volume real time output 1. compute intensive and data intensive computation like supercomputer.s performance 2. unstructured 1. movement of very large amount of data for visualization (networking) 2. distributed MPI-based 1. visualization 1. protection of PII on individuals used in modeling 2. data protection and secure platform for 1. data quality and be able capture the traceability of quality from computation --
4. wide variety of data, including satellite climate and weather data, photos, video, sound recordings… 5. multi-type data combination and linkage potentially unlimited data variety 6. data streaming 5. a variety of data, analytical and modeling tools to support analytics for diverse scientific communities. 6. parallel data streams and streaming analytics 7. access and integration of multiple distributed databases may be stored for future uses 4. provenance (and persistent identification (PID)) control of data, algorithms, and workflows 5. curated (authorized) reference data (i.e. species names lists), algorithms, software code, workflows 26 Deep Learning -- -- 1. GPU 2. high performance MPI and HPC Infiniband cluster 3. libraries for single-machine or single-GPU computation are available (e.g., BLAS, CuBLAS, MAGMA, etc.), distributed computation of dense BLAS-like or LAPACK-like operations on GPUs remains poorly developed. Existing solutions (e.g., ScaLapack for CPUs) are not well-integrated with higher level languages and require low-level programming which lengthens experiment and development time. -- -- -- -- 27 1. over 500M images uploaded to social media sites each day 1. classifier (e.g. a Support Vector Machine), a process that is often hard to parallelize. 2. features seen 1. Hadoop or enhanced MapReduce 1. visualize large-scale 3-d reconstructions, and navigate large-scale collections of images that have 1. preserve privacy for users and digital rights for media. -- --
in many large scale image processing problems been aligned to maps. 28 Truthy 1. distributed data sources 2. large volume real time streaming 3. raw data in compressed formats 4. fully structured data in JSON, users metadata, data 5. multiple data schemas 1. various real time data analysis for anomaly detection, stream clustering, signal classification on time series and 1. Hadoop and HDFS (platform) 2. IndexedHBase, Hive, SciPy, NumPy (software) 3. in-memory database, MPI (platform) 4. high-speed Infiniband network (networking) 1. data retrieval and dynamic visualization 2. data driven interactive web interfaces 3. API for data query 1. security and privacy policy 1. standardized data with extremely high data quality 1. low-level data storage infrastructure for efficient mobile access to data 29 Crowd Sourcing -- 1. digitize existing photo and documents archives 2. analytics include pattern recognition of all kind (e.g., speech recognition, automatic A&V analysis, cultural patterns), identification of structures (lexical units, linguistic rules, etc.) -- -- 1. privacy issues in preserving anonymity of responses in spite of computer recording of access ID and reverse engineering of unusual user responses -- -- 30 CINET 1. a set of network topologies file to study graph theoretic properties and behaviors of various algorithms 2. asynchronous and real time synchronous distributed computing 1. environments to run various network and graph analysis tools 2. dynamic grow of the networks 3. asynchronous and real time synchronous distributed computing 4. different parallel algorithms for different partitioning schemes for efficient 1. large file system (storage) 2. various network connectivity (networking) 3. existing computing cluster 4. EC2 computing cluster 5. various graph libraries, management toos, databases, semantic web tools 1. client side visualization -- -- --
operation 31 NIST IAD 1. large amounts of web pages, tweets, images, video 2. scaling ground-truthing to larger data, intrinsic and annotation uncertainty measurement, performance measurement for incompletely annotated data, measuring analytic performance for heterogeneous data and analytic flows involving users 1. analytic algorithms working with written language, speech, human imagery, etc. must generally be tested against real or realistic data. It’s extremely challenging to engineer artificial data that sufficiently captures the variability of real data involving humans 1. PERL, Python, C/C++, Matlab, R development tools. Create ground-up test and measurement applications 1. analytic flows involving users 1. security requirements for protecting sensitive data while enabling meaningful developmental performance evaluation. Shared evaluation testbeds must protect the intellectual property of analytic algorithm developers -- -- 32 DataNet 1. process key format types NetCDF, HDF5, Dicom 2. real-time and batch data 1. needs to provide general analytics workflows 1. iRODS data management software 2. interoperability across Storage and Network Protocol Types 1. general visulaization workflows 1. Federate across existing authentication environments through Generic Security Service API and Pluggable Authentication Modules (GSI, Kerberos, InCommon, Shibboleth). 2. access controls on files independently of the storage location. -- -- 33 The Discinnet process 1. integration of metadata approaches across disciplines -- 1. software: Symfony-PHP, Linux, MySQL -- 1. significant but optional security & privacy including secure servers and anonymization 1. integration of metadata approaches across disciplines -- 34 Semantic 1. all data types, image to text, structures to protein sequence 1. data graph processing 2. RDMS 1. cloud community resource required 1. efficient based visualization is needed -- -- --
35 Light source beamlines 1. multiple streams of real time data to be stored and analyzed later 2. sample data to be analyzed in real time 1. standard bioinformatics tools (BLAST, HMMER, multiple alignment and phylogenetic tools, gene callers, sequence feature predictors…), Perl/Python wrapper scripts, Linux Cluster scheduling 1. high volume data transfer to remote batch processing resource -- 1. multiple security & privacy requirements to be satisfied -- -- 36 Catalina Real-Time Transient Survey 1. ~0.1TB per day at present will increase by factor 100 1. a wide variety of the existing astronomical data analysis tools, plus a large amount of custom developed tools and software, some of it a research project in itself 2. automated classification with machine learning tools given the very sparse and heterogeneous data, dynamically evolving in time as more data come in, with follow-up decision making reflecting limited follow up resources -- 1. visualization mechanisms for highly data parameter spaces -- -- -- 37 DOE Extreme Data 1. ~1 PB per year becoming 7PB a year observational datal 1. interpretation of results from detailed simulations requires advanced analysis and visualization techniques and capabilities 1. MPI, OpenMP, C, C++, F90, FFTW, viz packages, python, FFTW, numpy, Boost, OpenMP, ScaLAPCK, PSQL & MySQL databases, Eigen, cfitsio, 1. interpretation of results using advanced visualization techniques and capabilities -- -- --
astrometry.net, and Minuit2 2. supercomputer I/O subsystem limitations must be addressed 38 Large Survey Data for 1. 20TB data per day 1. analysis on both the simulation and observational data simultaneously 2. techniques for handling Cholesky decompostion for thousands of simulations with matricies of order 1M on a side 1. standard astrophysics reduction software as well as Perl/Python wrapper scripts 2. Oracle RDBMS, Postgres psql, as well as GPFS and Lustre file systems and tape archives 3. parallel image storage -- -- 1. links between remote telescopes and central analysis sites -- 39 Particle Physics 1. real time data from Accelerator and Analysis instruments 2. asynchronization data collection 3. calibration of instruments 1. experimental data from ALICE, ATLAS, CMS, LHb 2. histograms, scatter-plots with model fits 3. Monte-Carlo computations 1. legacy computing infrastructure (computing nodes) 2. distributed cached files (storage) 3. object databases (sw pkg) 1. histograms and model fits (visual) 1. data protection 1. data quality on complex apparatus -- 40 Belle II High Energy Physics 1. 120PB Raw data -- 1. 120PB Raw data 2. International distributed computing model to augment that at acceleartor (Japan) 3. data transfer of ~20Gbps at designed luminosity between Japan and US 4. software from Open Science Grid, Geant4, DIRAC, FTS, Belle II framework -- 1. standard Grid authentication -- -- 41 EISCAT 3D 1. remote sites generating 40PB data per year by 2022 1. Queen Bea architecture with mix of distributed 1. architecture compatible with ENVRI Environmental 1. needs to suupport visualization of high (>=5) -- 1. preservation of data and avoid lost data due to instrument 1. needs to suupport realtime monitoring of equipment by
processing services 4. ROI_PAC, GeoServer, GDAL, tools tools 4. compatibility with other NASA Radar systems and repositories (Alaska Satellite Facility) 45 NASA iRODS 1. Federate distributed heterogeneous datasets 1. Climate Analytics as a Service on Clouds 1. Support virtual Climate Data Server (vCDS) 2. GPFS Parallel File System integrated with Hadoop 3. iRODS 1. needs to suupport visualize distributed heterogeneous data -- -- -- 46 MERRA 1. Integrate simulation output and observational data, NetCDF files 2. real time and batch mode both needed 3. Interoperable Use of Amazon AWS and local clusters 4. iRODS data management 1. Climate Analytics as a Service on Clouds 1. NetCDF aware software 2. MapReduce 3. Interoperable Use of Amazon AWS and local clusters 1. high end distributed visualization -- -- 1. Smart phone and Tablet access required 2. iRODS data management 47 1. real time distributed datasets 2. various formats, resolution, semantics, and metadata 1. MapReduce, SciDB, and other scientific databases 2. continuously computing for updates 3. language for data mining and event searching 4. semantics interpretation and optimal structuring for data mining and predictive analysis 1. other legacy computing systems (e.g. supercomputer) 2. high throughput data transmission over the network 1. visualization to interpret results -- 1. validation for output products (correlations) -- 48 Climate Studies 1. ~100PB data in 2017 streaming at high data rates from large supercomputers 1. ata analytics close to data storage 1. extend architecture to several other fields 1. share data with worldwide climate 2. high end distributed visualization -- -- 1. phone based input and access
across world 2. Integrate large scale distributed data from simulations with diverse observations 3. link diverse data to novel HPC simulation 49 DOE-BER 1. heterogeneous Diverse data with different domains and scales, Translation across diverse datasets that cross domains and scales 2. synthesis of diverse and disparate field, laboratory, omic, and simulation datasets across different semantic, spatial, and temporal scales 3. link diverse data to novel HPC simulation -- 1. postgres, HDF5 data technologies and many custom software systems 1. phone based input and access -- -- 1. phone based input and access 50 DOE-BER AmeriFlux and Networks 1. heterogeneous Diverse data with different domains and scales, Translation across diverse datasets that cross domains and scales 2. link to many other environment and biology datasets 3. link to HPC Climate and other simulations 4. link to European data sources and projects 5. access data from 500 distributed sources 1. Custom Software: EddyPro, Custom analysis software, R, python, neural networks, Matlab 1. custom software like EddyPro and analysis software like R, python, neural netowrks, Matlab 2. analytics includes data mining, data quality assessment, cross-correlation across datasets, data assimilation, data interpolation, statistics, quality assessment, data fusion, etc. 1. phone based input and access -- -- 1. phone based input and access