seen program is Root from CERN that reads multiple event (AOD, NTUP) files from selected data sets and use physicist generated C++ code to calculate new quantities such as implied mass of an unstable (new) particle from workstations to small clusters. ROOT is the most common software stack used to analyze compact data formats generated on distributed computing resources. Data transfer is done using ATLAS and CMS DDM tools, which mostly rely on gridFTP middleware. XROOTD based direct data access is also gaining importance wherever high network bandwidth is available. See Figure 6: Particle Physics: Analysis of LHC Data: Discovery of Higgs Particle – CERN LHC location. See Figure 7: Particle Physics: Analysis of LHC Data: Discovery of Higgs Particle – The multi-tier LHC computing infrastructure. Astronomy and Physics> Use Case 40: Belle II Experiment Use Case Title Belle II Experiment Vertical (area) Scientific Research: High Energy Physics Author/Company/Email David Asner and Malachi Schram, PNNL, and Actors/Stakeholders and their roles and responsibilities David Asner is the Chief Scientist for the US Belle II Project Malachi Schram is Belle II network and data transfer coordinator and the PNNL Belle II computing center manager Goals Perform precision measurements to search for new phenomena beyond the Standard Model of Particle Physics Use Case Description Study numerous decay modes at the Upsilon(4S) resonance to search for new phenomena beyond the Standard Model of Particle Physics Current Solutions Compute(System) Distributed (Grid computing using DIRAC) Storage Distributed (various technologies) Networking Continuous RAW data transfer of ~20Gbps at designed luminosity between Japan and US Additional transfer rates are currently being investigated Software Open Science Grid, Geant4, DIRAC, FTS, Belle II framework Big Data Characteristics Data Source (distributed/centralized) Distributed data centers Primary data centers are in Japan (KEK) and US (PNNL) Volume (size) Total integrated RAW data ~120PB and physics data ~15PB and ~100PB MC samples Velocity (e.g. real time) Data will be re-calibrated and analyzed incrementally Data rates will increase based on the accelerator luminosity
Variety (multiple datasets, mashup) Measurements: different versions, formats, replicas, external sources ... System information: configuration, monitoring, logs/provenance ... Users’ metadata/data: experiments, analysis, sharing, communications … Variability (rate of change) In time, instantly, a few ms. Along the radar beams, 100ns. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Running 24/7, EISCAT_3D have very high demands on robustness. Data and performance assurance is vital for the and archive systems. These systems must be able to guarantee to meet minimum data rate acceptance at all times or scientific data will be lost. Similarly the systems must guarantee that data held is not volatile or corrupt. This latter requirement is particularly vital at the permanent archive where data is most likely to be accessed by scientific users and least easy to check; data corruption here has a significant possibility of being non-recoverable and of poisoning the scientific literature. Visualization Real-time visualisation of analysed data, e.g., with a figure of updating panels showing electron density, temperatures and ion velocity to those data for each beam. Non-real-time (post-experiment) visualisation of the physical parameters of interest, e.g., by standard plots, using three-dimensional block to show to spatial variation (in the user selected cuts), using animations to show the temporal variation, allow the visualisation of 5 or higher data, e.g., using the 'cut up and stack' technique to reduce the dimensionality, that is take one or more independent coordinates as discrete; or volume rendering technique to display a 2D projection of a 3D discretely sampled data set. (Interactive) Visualisation. E.g., to allow users to combine the information on several spectral features, e.g., by using colour coding, and to provide real-time visualisation facility to allow the users to link or plug in tailor-made data visualisation functions, and more importantly functions to signal for special observational conditions. Data Quality Monitoring software will be provided which allows The Operator to see incoming data via the
Appendix B: Summary of Key Properties Information related to five key properties was extracted from each use case. The five key properties were three Big Data characteristics (volume, velocity, and variety), software related information, and associated analytics. The extracted information is presented in Table B-1. Table B-1: Use Case Specific Information by Key Properties Use Case Volume Velocity Variety Software Analytics 1 Census 2000 and 2010 380 TB Static for 75 years Scanned documents Robust archival storage None for 75 years 2 NARA: Search, Retrieve, Preservation Hundreds of terabytes, and growing Data loaded in batches, so bursty Unstructured and structured data: textual documents, emails, photos, scanned documents, multimedia, social networks, web sites, databases, etc. Custom software, commercial search products, commercial databases Crawl/index, search, ranking, predictive search; data categorization (sensitive, confidential, etc.); personally identifiable information (PII) detection and flagging 3 Statistical Survey Response Improvement Approximately 1 PB Variable, field data streamed continuously, Census was ~150 million records transmitted Strings and numerical data Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig Recommendation systems, continued monitoring 4 Non-Traditional Data in Statistical Survey Response Improvement — — Survey data, other government administrative data, web-scraped data, wireless data, e-transaction data, (potentially) social media data and positioning data from various sources Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig New analytics to create reliable information from non-traditional disparate sources 5 Cloud Eco-System for Finance — Real time — Hadoop RDBMS XBRL Fraud detection 6 Mendeley 15 TB presently, growing about 1 TB per month Currently Hadoop batch jobs scheduled daily, PDF documents and log files of social network and client activities Hadoop, Scribe, Hive, Mahout, Python Standard libraries for machine learning and analytics, LDA,
real-time recommended in future reporting tools for aggregating readership and social activities per document 7 Netflix Movie Service Summer 2012 – 25 million subscribers, 4 million ratings per day, 3 million searches per day, 1 billion hours streamed in June 2012; Cloud storage – 2 petabytes in June 2013 Media (video and properties) and rankings continually updated Data vary from digital media to user rankings, user profiles, and media properties for content-based recommendations Hadoop and Pig; Cassandra; Teradata Personalized recommender systems using logistic/linear regression, elastic nets, matrix factorization, clustering, LDA, association rules, decision trees, and others; streaming video delivery 8 Web Search 45 billion web pages total, 500 million photos uploaded each day, 100 hours of video uploaded to YouTube each minute Real-time updating and real-time responses to queries Multiple media MapReduce + Bigtable; Dryad + Cosmos; PageRank; final step essentially a recommender engine Crawling; searching, including topic-based searches; ranking; recommending 9 Business Continuity and Disaster Recovery Within a Cloud Eco-System Terabytes up to petabytes Can be real time for recent changes Must work for all data Hadoop, MapReduce, open source, and/or vendor proprietary such as AWS, Google Cloud Services, and Microsoft Robust backup 10 Cargo Shipping — Needs to become real time, currently updated at events Event-based — Distributed event analysis identifying problems 11 Materials Data for Manufacturing 500,000 material types in 1980s, much growth since then Ongoing increase in new materials Many datasets with no standards National programs (Japan, Korea, and China), application areas (EU nuclear program), proprietary systems (Granta, etc.) No broadly applicable analytics
stored data 22 Statistical Relational Artificial Intelligence for Health Care Hundreds of gigabytes for a single cohort of a few hundred people; possibly on the order of 1 PB when dealing with millions of patients Constant updates to EHRs; in other controlled studies, data often in batches at regular intervals Critical feature – data typically in multiple tables, need to be merged to perform analysis Mainly Java-based, tools to process the data Relational probabilistic models (Statistical Relational AI) learned from multiple data types 23 World Population-Scale Epidemiological Study 100 TB Low number of data feeding into the simulation, massive amounts of real-time data generated by simulation Can be rich with various population activities, geographical, cultural variations Charm++, MPI Simulations on a synthetic population 24 Social Contagion Modeling for Planning Tens of terabytes per year During social unrest events, human interactions and mobility leads to rapid changes in data; e.g., who follows whom in Twitter Big issues – data fusion, combining data from different sources, dealing with missing or incomplete data Specialized simulators, open source software, proprietary modeling environments; databases Models of behavior of humans and hard infrastructures, models of their interactions, visualization of results 25 Biodiversity and LifeWatch N/A Real-time processing and analysis in case of natural or industrial disaster Rich variety and number of involved databases and observation data RDBMS Requires advanced and rich visualization 26 Large-Scale Deep Learning Current datasets typically 1 to 10 TB, possibly 100 million images to train a self-driving car Much faster than real-time processing; for autonomous driving, need to process thousands Neural net very heterogeneous as it learns many different features In-house GPU kernels and MPI-based communication developed by Stanford, C++/Python source Small degree of batch statistical pre-processing, all other data analysis performed by the learning algorithm itself
of high-resolution (six megapixels or more) images per second 27 Organizing Unstructured Collections of Consumer Photos 500+ billion photos on Facebook, 5+ billion photos on Flickr Over 500 million images uploaded to Facebook each day Images and metadata including EXIF (Exchangeable Image File) tags (focal distance, camera type, etc.) Hadoop MapReduce, simple hand-written multi-threaded tools (Secure Shell [SSH] and sockets for communication) Robust non-linear least squares optimization problem, SVM 28 Truthy Twitter Data 30 TB/year compressed data Near real-time data storage, querying and analysis Schema provided by social media data source; currently using Twitter only; plans to expand, incorporating Google+ and Facebook Hadoop IndexedHBase and HDFS; Hadoop, Hive, Redis for data management; Python: SciPy NumPy and MPI for data analysis Anomaly detection, stream clustering, signal classification, online learning; information diffusion, clustering, dynamic network visualization 29 Crowd Sourcing in Humanities Gigabytes (text, surveys, experiment values) to hundreds of terabytes (multimedia) Data continuously updated and analyzed incrementally So far mostly homogeneous small data sets; expected large distributed heterogeneous datasets XML technology, traditional relational databases Pattern recognition (e.g., speech recognition, automatic audio-visual analysis, cultural patterns), identification of structures (lexical units, linguistic rules, etc.) 30 CINET for Network Science Can be hundreds of gigabytes for a single network, networks and methods Dynamic networks, network collection growing Many types of networks Graph libraries (Galib, NetworkX); distributed workflow management (Simfrastructure, databases, semantic web tools) Network visualization 31 NIST Information Access Division >900 million web pages occupying 30 TB of storage, 100 million tweets, 100 million biometric Legacy evaluations mostly focused on retrospective analytics, newer evaluations focused on Wide variety of data types including textual search/extraction, machine translation, speech recognition, image and voice biometrics, object and PERL, Python, C/C++, Matlab, R development tools; create ground-up test and measurement applications Information extraction, filtering, search, and summarization; image and voice biometrics; speech recognition and understanding; machine
images, hundreds of thousands of partially video clips, terabytes of smaller fully ground-truthed test collections simulations of analytic challenges from multiple data streams person recognition and tracking, document analysis, human-computer dialogue, multimedia search/extraction translation; video person/object detection and tracking; event detection; imagery/document matching; novelty detection; structural semantic temporal analytics 32 DataNet (iRODS) Petabytes, hundreds of millions of files Real time and batch Rich iRODS Supports general analysis workflows 33 The Discinnet Process Small as metadata to Big Data Real time Can tackle arbitrary Big Data Symfony-PHP, Linux, MySQL -- 34 Semantic A few terabytes Evolving in time Rich Database Data graph processing 35 Light Source Beamlines 50–400 GB per day, total ~400 TB Continuous stream of data, but analysis need not be real time Images Octopus for Tomographic Reconstruction, Avizo (http://vsg3d.com) and FIJI (a distribution of ImageJ) Volume reconstruction, feature identification, etc. 36 Catalina Transient Survey ~100 TB total increasing by 0.1 TB a night accessing PBs of base astronomy data, 30 TB a night from successor LSST in 2020s Nightly update runs processes in real time Images, spectra, time series, catalogs Custom data processing pipeline and data analysis software Detection of rare events and relation to existing diverse data 37 DOE Extreme Data from Cosmological Sky Survey Several petabytes from Dark Energy Survey and Zwicky Transient Factory, simulations > 10 PB Analysis done in batch mode with data from observations and simulations updated daily Image and simulation data MPI, FFTW, viz packages, numpy, Boost, OpenMP, ScaLAPCK, PSQL and MySQL databases, Eigen, cfitsio, astrometry.net, and New analytics needed to analyze simulation results
workflow 43 CReSIS Remote Sensing Around 1 PB (current) increasing by 50–100 TB per mission, future expedition ~1 PB each Data taken in ~two-month missions including test analysis and then later batch processing Raw data, images with final layer data used for science Matlab for custom raw data processing, custom image processing software, GIS as user interface Custom signal processing to produce radar images that are analyzed by image processing to find layers 44 UAVSAR Data Processing 110 TB raw data and 40 TB processed, plus smaller samples Data come from aircraft and so incrementally added, data occasionally get reprocessed: new processing methods or parameters Image and annotation files ROI_PAC, GeoServer, GDAL, tools; moving to clouds Process raw data to get images that are run through image processing tools and accessed from GIS 45 NASA LARC/GSFC iRODS MERRA collection (below) represents most of total data, other smaller collections Periodic updates every six months Many applications to combine MERRA reanalysis data with other reanalyses and observational data such as CERES SGE Univa Grid Engine Version 8.1, iRODS Version 3.2 and/or 3.3, IBM GPFS Version 3.4, Cloudera Version 4.5.2-1 Federation software 46 MERRA Analytic Services 480 TB from MERRA Increases at ~1 TB/month Applications to combine MERRA reanalysis data with other re-analyses and observational data Cloudera, iRODS, Amazon AWS Climate (CAaaS) 47 Atmospheric Turbulence 200 TB (current), 500 TB within 5 years Data analyzed incrementally Re-analysis datasets are inconsistent in format, resolution, semantics, and metadata; interpretation/analysis of each of these input streams into a common product MapReduce or the like, SciDB or other scientific database Data mining customized for specific event types 48 Climate Studies Up to 30 PB/year from 15 end-to-end simulations at NERSC, more at 42 GB/second from simulations Variety across simulation groups and between observation and simulation National Center for Atmospheric Research (NCAR) PIO library and utilities NCL and NCO, Need analytics next to data storage
other HPC centers parallel NetCDF 49 DOE-BER Subsurface Biogeochemistry — — From omics of the microbes in the soil to watershed hydro-biogeochemistry, from observation to simulation PFLOWTran, postgres, HDF5, Akuna, NEWT, etc. Data mining, data quality assessment, across datasets, reduced model development, statistics, quality assessment, data fusion 50 DOE-BER AmeriFlux and FLUXNET Networks — Streaming data from ~150 towers in AmeriFlux and over 500 towers distributed globally collecting flux measurements Flux data merged with biological, disturbance, and other ancillary data EddyPro, custom analysis software, R, Python, neural networks, Matlab Data mining, data quality assessment, across datasets, data assimilation, data interpolation, statistics, quality assessment, data fusion 51 Consumption forecasting in Smart Grids 4 TB/year for a city with 1.4 million sensors, such as Los Angeles Streaming data from millions of sensors Tuple-based: timeseries, database rows; graph-based: network topology, customer connectivity; some semantic data for normalization R/Matlab, Weka, Hadoop; GIS-based visualization Forecasting models, machine learning models, time series analysis, clustering, motif detection, complex event processing, visual network analysis correlation
data sources in different clouds large varied datasets 5. Huge numbers of data with high relevancy and recall 3 Statistical Survey Response Improvement 1. Data size of approximately one petabyte 1. Analytics for recommendation systems, continued monitoring, and general survey improvement 1. Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig 1. Data visualization for data review, operational activity, and general analysis; continual evolution 1. Improved systems that reduce costs and improve quality while providing confidentiality safeguards that are reliable and publicly auditable 2. Confidential and secure data; processes that are auditable for security and confidentiality as required by various legal statutes 1. High veracity on data and very robust systems (challenges: semantic integrity of conceptual metadata concerning what exactly is measured and the resulting limits of inference) 1. Mobile access 4 Data in Statistical Survey Response -- 1. Analytics to create reliable estimates using data from traditional survey sources, government 1. Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, 1. Data visualization for data review, operational activity, and general 1. Confidential and secure data; processes that are auditable for security and confidentiality 1. High veracity on data and very robust systems (challenges: semantic integrity of conceptual metadata concerning what --
System Improvement administrative data sources, and sources from the digital economy Cassandra, Pig analysis; continual evolution as required by various legal statutes exactly is measured and the resulting limits of inference) 5 Cloud for Finance 1. Real-time ingestion of data 1. Real-time analytics -- -- 1. Strong security and privacy constraints -- 1. Mobile access 6 Mendeley 1. File-based documents with constant new uploads 2. Variety of file types such as PDFs, social network log files, client activities images, spreadsheet, presentation files 1. Standard machine learning and analytics libraries 2. Efficient scalable and parallelized way to match between documents 3. Third-party annotation tools or publisher watermarks and cover pages 1. Amazon Elastic Compute Cloud (EC2) with HDFS (infrastructure) 2. S3 (storage) 3. Hadoop (platform) 4. Scribe, Hive, Mahout, Python (language) 5. Moderate storage (15 TB with 1 TB/ month) 6. Batch and processing 1. reporting tools 2. Visualization tools such as networking graph, scatterplots, etc. 1. Access controls for who reads what content 1. Metadata management from PDF extraction 2. Identification of document duplication 3. Persistent identifier 4. Metadata between data repositories such as CrossRef, PubMed, and Arxiv 1. Windows Android and iOS mobile devices for content deliverables from Windows desktops 7 Netflix Movie Service 1. User profiles and ranking information 1. Streaming video contents to multiple clients 2. Analytic processing for matching client interest in movie selection 3. Various analytic processing techniques for 1. Hadoop (platform) 2. Pig (language) 3. Cassandra and Hive 4. Huge numbers of subscribers, ratings, and searches per day (DB) 5. Huge amounts 1. Streaming and rendering media 1. Preservation of users, privacy and digital rights for media 1. Continued ranking and updating based on user profile and analytic results 1. Smart interface accessing movie content on mobile platforms
consumer personalization 4. Robust learning algorithms 5. Continued analytic processing based on monitoring and performance results of storage (2 PB) 6. I/O intensive processing 8 Web Search 1. Distributed data sources 2. Streaming data 3. Multimedia content 1. Dynamic fetching content over the network 2. Linking of user profiles and social network data 1. Petabytes of text and rich media (storage) 1. Search time of ~0.1 seconds 2. Top 10 ranked results 3. Page layout (visual) 1. Access control 2. Protection of sensitive content 1. Data purge after certain time interval (a few months) 2. Data cleaning 1. Mobile search and rendering 9 Business Continuity and Disaster Recovery Within a Cloud Eco-System -- 1. Robust backup algorithm 2. Replication of recent changes 1. Hadoop 2. Commercial cloud services -- 1. Strong security for many applications -- -- Cargo Shipping 1. Centralized and real-time distributed sites/sensors 1. Tracking items based on the unique identification with its sensor information, GPS coordinates 2. Real-time updates on tracking items 1. Internet connectivity -- 1. Security policy -- -- 1. Distributed 1. Hundreds of -- 1. 1. Protection of 1. Handle data --
8. Data streams from peta/exascale centralized simulation systems Large-Scale Geospatial Analysis and Visualization 1. Unique approaches to indexing and distributed analysis required for geospatial data 1. Analytics: closest point of approach, deviation from route, point density over time, PCA and ICA 2. Unique approaches to indexing and distributed analysis required for geospatial data 1. Geospatially enabled RDBMS, geospatial server/analysis software, e.g., ESRI ArcServer, Geoserver 1. Visualization with GIS at high and low network bandwidths and on dedicated facilities and handhelds 1. Complete security of sensitive data in transit and at rest (particularly on handhelds) -- -- Object Identification and Tracking 1. Real-time data FMV (30–60 frames/ second at full-color 1080P resolution) and WALF (1–10 frames/ second at 10,000 x 10,000 full-color resolution) 1. Rich analytics with object identification, pattern recognition, crowd behavior, economic activity, and data fusion 1. Wide range of custom software and tools including traditional RDBMSs and display tools 2. Several network requirements 3. GPU usage important 1. Visualization of extracted outputs as overlays on a geospatial display; links back to the originating image/video segment as overlay objects 2. Output the form of Open Geospatial Consortium 1. Significant security and privacy issues; sources and methods never compromised 1. Veracity of extracted objects --
(patients), > 4 billion records or data points (discrete clinical observations), aggregate of > 20 TB raw data 3. Velocity: 500,000–1.5 million new transactions per day 4. Variety: formats include numeric, structured numeric, structured text, discrete nominal, discrete ordinal, discrete structured, binary large blobs (images and video) 5. Data evolve over time in a highly variable fashion techniques: information retrieval, NLP, machine learning decision models, maximum likelihood estimators, Bayesian networks 4. Various, with significant I/O intensive processing did not actually perform the analysis; specific visualization techniques informatics research scientists and health service researchers 2. Protection of all health data in compliance with governmental regulations 3. Protection of data in accordance with data providers, policies. 4. Security and privacy policies unique to a data subset 5. Robust security to prevent data breaches nomenclature and classification of content across disparate challenging in the health IT space, as the taxonomies continue to evolve— SNOMED, International Classification of Diseases (ICD) 9 and future ICD 10, etc. Pathology Imaging 1. High-resolution spatial digitized pathology images 2. Various image quality analyses algorithms 1. image analysis to extract spatial information 2. Spatial queries 1. Legacy system and cloud (computing cluster) 2. Huge legacy and new storage 1. Visualization for validation and training 1. Security and privacy protection for protected health information 1. Human annotations for validation 1. 3D visualization and rendering on mobile platforms
3. Various image data formats, especially BigTIFF with structured data for analytical results 4. Image analysis, spatial queries and analytics, feature clustering, and classification and analytics, feature clustering and classification 3. Analytic processing on huge multi-dimensional large dataset; correlation with other data types such as clinical data, omic data such as storage area network (SAN) or HDFS (storage) 3. network link (networking) 4. MPI image analysis, MapReduce, Hive with spatial extension (software packages) Computational Bioimaging 1. Distributed multi-modal experimental sources of bioimages (instruments) 2. 50 TB of data in formats that include images 1. High-throughput computing with responsive analysis 2. Segmentation of regions of interest; crowd-based selection and extraction of features; object classification, and organization; and search 3. Advanced biosciences discovery through Big Data techniques / extreme-scale computing; 1. ImageJ, OMERO, VolRover, advanced segmentation and feature detection methods from applied math researchers; scalable key-value and object store databases needed 2. NERSC’s Hopper infrastructure 3. database and image collections 4. 10 GB and future 100 GB and advanced 1. 3D structural modeling 1. Significant but optional security and privacy including secure servers and anonymization 1. Workflow components including data acquisition, storage, enhancement, minimizing noise --
Metagenomes and Genomes 2. Proteins and their structural features, core genomic data, new types of omics data such as transcriptomics, methylomics, and proteomics describing gene expression 3. Front real-time web UI interactive; backend data loading processing that keeps up with exponential growth of sequence data due to the rapid drop in cost of sequencing technology 4. Heterogeneous, complex, structural, and hierarchical biological data 5. Metagenomic samples that can vary by several 2. Real-time rapid and parallel bulk loading 3. Oracle RDBMS, SQLite files, flat text files, Lucy (a version of Lucene) for keyword searches, BLAST databases, USEARCH databases 4. Linux cluster, Oracle RDBMS server, large memory machines, standard Linux interactive hosts 5. Sequencing and comparative analysis techniques for highly complex data 6. Descriptive statistics loading capability 2. Interactive Web UI, backend batch job computation submission from the UI. 3. Download of assembled and annotated datasets for offline analysis 4. Ability to query and browse data via interactive web UI 5. Visualize data structure at different levels of resolution; ability to view abstract of highly similar data password 2. Creation of user account to submit and access dataset to system via web interface 3. Single capability (SSO) classification, reduction 3. Integration of new data/content into the system’s data store and data annotation
orders of magnitude, such as several hundred thousand genes to a billion genes Individualized Diabetes Management 1. Distributed EHR data 2. Over million patients with thousands of properties each and many more derived from primary values 3. Each record: a range of data property values, average of 100 controlled vocabulary values, and average of 1,000 continuous values 4. No real-time, but data updated periodically; data timestamped with the time of observation (time the value is recorded) 5. Two main categories of 1. Data integration using ontological annotation and taxonomies 2. Parallel retrieval algorithms for both indexed and custom searches; identification of data of interest; patient cohorts, patients’ meeting certain criteria, patients sharing similar characteristics 3. Distributed graph mining algorithms, pattern analysis and graph indexing, pattern searching on RDF triple graphs 4. Robust statistical analysis tools to manage false discovery rates, determine true sub-graph 1. data warehouse, open source indexed Hbase 2. supercomputers, cloud and parallel computing 3. I/O intensive processing 4. HDFS storage 5. custom code to develop new properties from stored data. 1. Efficient data visualization needed 1. Protection of health data in accordance with privacy policies and legal requirements, e.g., HIPAA. 2. Security policies for different user roles 1. Data annotated based on domain ontologies or taxonomies 2. Traceability of data from origin (initial point of collection) through use 3. Data conversion from existing data warehouse into RDF triples 1. Mobile access
structured data about a patient: data with controlled vocabulary (CV) property values and data with continuous property values (recorded/ captured more frequently) 6. Data consist of text and continuous numerical values significance, validate results, eliminate false positive/false negative results 5. Semantic graph mining algorithms to identify graph patterns, index and search graph 6. Semantic graph traversal Statistical Relational Artificial Intelligence for Health Care 1. Centralized data, with some data retrieved from Internet sources 2. Range from hundreds of gigabytes for a sample size to 1 PB for very large studies 3. Both constant updates/additions (to data subsets) and scheduled batch inputs 4. Large, longitudinal data 1. Relational probabilistic models/ probability theory; software that learns models from multiple data types and can possibly integrate the information and reason about complex queries 2. Robust and accurate learning methods to account for data imbalance (where large numbers of data are available 1. Java, some in house tools, [relational] database and NoSQL stores 2. Cloud and parallel computing 3. computer, 48 GB RAM (to perform analysis for a moderate sample size) 4. Dlusters for large datasets 5. 200 GB–1 TB hard drive for test 1. Visualization of very large data subsets 1. Secure handling and processing of data 1. Merging multiple tables before analysis 2. Methods to validate data to minimize errors --
trends data archiving 3. Distributed data sources, including observation and monitoring facilities, sensor network, and satellites 4. Wide variety of data: satellite images/ information, climate and weather data, photos, video, sound recordings, etc. 5. Multi-type data combination and linkage, potentially unlimited data variety 6. Data streaming 3. Grid- and resources 4. Data analyzed incrementally and/or in real time at varying rates owing to variations in source processes 5. A variety of data and analytical and modeling tools to support analytics for diverse scientific communities 6. Parallel data streams and streaming analytics 7. Access and integration of multiple distributed databases visualization computational models control and accounting integrity and identification traceability back to initial observational data 3. Processed (secondary) data storage (in addition to original source data) for future uses 4. Provenance (and persistent identification [PID]) control of data, algorithms, and workflows 5. Curated (authorized) reference data (e.g. species name lists), algorithms, software code, workflows Large-Scale Deep Learning -- -- 1. GPU 2. MPI and HPC Infiniband cluster 3. Libraries for single-machine or single-GPU computation – -- -- -- -- 6
available (e.g., BLAS, CuBLAS, MAGMA, etc.); distributed computation of dense BLAS-like or LAPACK-like operations on GPUs – poorly developed; existing solutions (e.g., ScaLapack for CPUs) – not well-integrated with higher-level languages and require low-level programming, lengthening experiment and development time Organizing Large-Scale Unstructured Collections of Consumer Photos 1. Over 500 million images uploaded to social media sites each day 1. Classifier (e.g. an SVM), a process that is often hard to parallelize 2. Features seen in many large-scale image processing problems 1. Hadoop or enhanced MapReduce 1. Visualize large-scale 3D navigate large-scale collections of images that have been aligned to maps 1. Preserve privacy for users and digital rights for media -- -- Truthy Twitter Data 1. Distributed data sources 2. Large volume 1. Various data analysis for anomaly 1. Hadoop and HDFS (platform) 2. IndexedHBase, 1. Data retrieval and dynamic 1. Security and privacy policy 1. Standardized data structures/ formats with extremely high 1. Low-level data storage
of real-time streaming data 3. Raw data in compressed formats 4. Fully structured data in JSON, user metadata, data 5. Multiple data schemas detection, stream clustering, signal classification on multi-dimensional time series, online learning Hive, SciPy, NumPy (software) 3. In-memory database, MPI (platform) 4. High-speed Infiniband network (networking) visualization 2. Data-driven interactive web interfaces 3. API for data query data quality for efficient mobile access to data Crowd Sourcing in Humanities -- 1. Digitize existing audio-video, photo, and documents archives 2. Analytics: pattern recognition of all kinds (e.g., speech recognition, automatic A&V analysis, cultural patterns), identification of structures (lexical units, linguistic rules, etc.) -- -- 1. Privacy issues in preserving anonymity of responses in spite of computer recording of access ID and reverse engineering of unusual user responses -- -- CINET for Network Science 1. A set of network topologies files to study graph theoretic properties and behaviors of 1. Environments to run various network and graph analysis tools 2. Dynamic growth of the networks 3. Asynchronous 1. Large file system (storage) 2. Various network connectivity (networking) 3. Existing 1. Client-side visualization -- -- --
be analyzed in real time gene callers, sequence feature predictors, etc.), Perl/Python wrapper scripts, Linux Cluster scheduling Catalina Transient Survey 1. ~0.1 TB per day at present, will increase by factor of 100 1. A wide variety of the existing astronomical data analysis tools, plus a large number of custom developed tools and software programs, some research projects in and of themselves 2. Automated classification with machine learning tools given the very sparse and heterogeneous data, dynamically evolving in time as more data come in, with follow-up decision making reflecting limited follow-up resources -- 1. Visualization mechanisms for highly dimensional data parameter spaces -- -- -- DOE Extreme Data from 1. ~1 PB/year becoming 7 PB/year of 1. Advanced analysis and visualization 1. MPI, OpenMP, C, C++, F90, FFTW, viz 1. Interpretation of results -- -- --
Cosmological Sky Survey observational data techniques and capabilities to support interpretation of results from detailed simulations packages, Python, FFTW, numpy, Boost, OpenMP, ScaLAPCK, PSQL and MySQL databases, Eigen, cfitsio, astrometry.net, and Minuit2 2. Methods/ tools to address supercomputer I/O subsystem limitations using advanced visualization techniques and capabilities Large Survey Data for Cosmology 1. 20 TB of data/day 1. Analysis on both the simulation and observational data simultaneously 2. Techniques for handling Cholesky decomposition for thousands of simulations with matrices of order 1 million on a side 1. Standard astrophysics reduction software as well as Perl/Python wrapper scripts 2. Oracle RDBMS, Postgres psql, GPFS and Lustre file systems and tape archives 3. Parallel image storage -- -- 1. Links between remote telescopes and central analysis sites -- Particle Physics at LHC 1. Real-time data from accelerator and analysis instruments 2. Asynchronization data collection 3. Calibration of 1. Experimental data from ALICE, ATLAS, CMS, LHB 2. Histograms, scatter-plots with model fits 3. Monte-Carlo computations 1. Legacy computing infrastructure (computing nodes) 2. Distributed cached files (storage) 1. Histograms and model fits (visual) 1. Data protection 1. Data quality on complex apparatus -- 9
instruments 3. Object databases (software package) Belle II Physics Experiment 1. 120 PB of raw data -- 1. 120 PB raw data 2. International distributed computing model to augment that at accelerator (Japan) 3. Data transfer of ~20 GB/ second at designed luminosity between Japan and United States 4. Software from Open Science Grid, Geant4, DIRAC, FTS, Belle II framework -- 1. Standard grid authentication -- -- EISCAT 3D Incoherent Scatter Radar System 1. Remote sites generating 40 PB data/year by 2022 2. Hierarchical Data Format (HDF5) 3. Visualization of high-dimensional (≥5) data 1. Queen Bea architecture with mix of distributed on-sensor and central processing for 5 distributed sites 2. Real-time monitoring of equipment by partial streaming analysis 1. Architecture compatible with ENVRI 1. Support needed for visualization of (≥5) data -- 1. Preservation of data and avoidance of lost data due to instrument malfunction 1. Support needed for real-time monitoring of equipment by partial streaming analysis
3. Hosting needed for rich set of radar image processing services using machine learning, statistical modelling, and graph algorithms ENVRI Environmental Research Infrastructure 1. Huge volume of data from distributed data sources 2. Variety of instrumentation datasets and metadata 1. Diversified analytics tools 1. Variety of computing infrastructures and architectures (infrastructure) 2. Scattered repositories (storage) 1. Graph plotting tools 2. Time series interactive tools 3. flash playback 4. Earth map display 5. Visual tools for quality comparisons 1. Open data policy with minor restrictions 1. High data quality 2. Mirror archives 3. Various metadata frameworks 4. Scattered repositories and data curation 1. Various kinds of mobile sensor devices for data acquisition CReSIS Remote Sensing 1. Provision of reliable data transmission from aircraft sensors/ instruments or removable disks from remote sites 2. Data gathering in real time 3. Varieties of datasets 1. Legacy software (Matlab) and language (C/Java) binding for processing 2. Signal processing and advanced image processing to find layers needed 1. ~0.5 PB/year of raw data 2. Transfer content from removable disk to computing cluster for parallel processing 3. MapReduce or MPI plus language binding for C/Java 1. GIS user interface 2. Rich user interface for simulations 1. Security and privacy on sensitive political issues 2. Dynamic security and privacy policy mechanisms 1. Data quality assurance 1. Monitoring data collection instruments/ sensors UAVSAR Data 1. Angular and spatial data 1. Geolocated data that require GIS 1. Support for interoperable 1. Support for field -- 1. Significant human intervention in data 1. Support for field
Processing 2. Compatibility with other NASA radar systems and repositories (Alaska Satellite Facility) integration of data as custom overlays 2. Significant human intervention in data processing pipeline 3. Hosting of rich set of radar image processing services 4. ROI_PAC, GeoServer, GDAL, tools Cloud-HPC architecture 2. Hosting of rich set of radar image processing services 3. ROI_PAC, GeoServer, GDAL, tools 4. Compatibility with other NASA radar systems and repositories (Alaska Satellite Facility) expedition users with phone/tablet interface and low-resolution downloads processing pipeline 2. Rich robust provenance defining complex machine/human processing expedition users with phone/tablet interface and downloads NASA LARC/ GSFC iRODS 1. Federate distributed heterogeneous datasets 1. CAaaS on clouds 1. Support virtual climate data server (vCDS) 2. GPFS parallel file system integrated with Hadoop 3. iRODS 1. Support needed to visualize distributed data -- -- -- MERRA Analytic Services 1. Integrate simulation output and observational data, NetCDF files 2. Real-time and batch mode needed 3. Interoperable use of AWS and local clusters 4. iRODS data 1. CAaaS on clouds 1. NetCDF aware software 2. MapReduce 3. Interoperable use of AWS and local clusters 1. High-end distributed visualization -- -- 1. Smart phone and tablet access required 2. iRODS data management
management Atmospheric Turbulence 1. Real-time distributed datasets 2. Various formats, resolution, semantics, and metadata 1. MapReduce, SciDB, and other scientific databases 2. Continuous computing for updates 3. Event specification language for data mining and event searching 4. Semantics interpretation and optimal structuring for 4D data mining and predictive analysis 1. Other legacy computing systems (e.g. supercomputer) 2. high throughput data transmission over the network 1. Visualization to interpret results -- 1. Validation for output products (correlations) -- Climate Studies 1. ~100 PB data in 2017 streaming at high data rates from large supercomputers across the world 2. Integration of large-scale distributed data from simulations with diverse observations 3. Linking of diverse data to novel HPC simulation 1. Data analytics close to data storage 1. Extension of architecture to several other fields 1. Worldwide climate data sharing 2. High-end distributed visualization -- -- 1. input and access 1. Heterogeneous -- 1. Postgres, HDF5 1. -- -- 1.
DOE-BER Subsurface diverse data with different domains and scales, translation across diverse datasets that cross domains and scales 2. Synthesis of diverse and disparate field, laboratory, omic, and simulation datasets across different semantic, spatial, and temporal scales 3. Linking of diverse data to novel HPC simulation data technologies, and many custom software systems input and access input and access DOE-BER AmeriFlux and FLUXNET Networks 1. Heterogeneous diverse data with different domains and scales, translation across diverse datasets that cross domains and scales 2. Link to many other environment and biology datasets 1. Custom software such as EddyPro, and custom analysis software, such as R, Python, neural networks, Matlab 1. Custom software, such as EddyPro, and custom analysis software, such as R, Python, neural networks, Matlab 2. Analytics including data mining, data quality assessment, cross-correlation 1. input and access -- -- 1. input and access
3. Link to HPC climate and other simulations 4. Link to European data sources and projects 5. Access to data from 500 distributed sources across datasets, data assimilation, data interpolation, statistics, quality assessment, data fusion, etc. Consumption Forecasting in Smart Grids 1. Diverse data from smart grid sensors, city planning, weather, utilities 2. Data updated every 15 minutes 1. New machine learning analytics to predict consumption 1. SQL databases, CVS files, HDFS (platform) 2. R/Matlab, Weka, Hadoop (platform) -- 1. Privacy and anonymization by aggregation -- 1. Mobile access for clients