Big Data Characteristics Data Source (distributed/centralized) Distributed – with replication/redundancy Volume (size) ~30TB/year compressed data Velocity (e.g. real time) Near real-time data storage, querying and analysis Variety (multiple datasets, mashup) Data schema provided by social media data source. Currently using Twitter only. We plan to expand incorporating Google+, Facebook Variability (rate of change) Continuous real-time data stream incoming from each source. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues, semantics) 99.99% uptime required for real-time data acquisition. Service outages might corrupt data integrity and significance. Visualization Information diffusion, clustering, and dynamic network visualization capabilities already exist. Data Quality (syntax) Data structured in standardized formats, the overall quality is extremely high. We generate aggregated statistics; expand the features set, etc., generating derived data. Data Types Fully-structured data (JSON format) enriched with users meta-data, geo-locations, etc. Data Analytics Stream clustering: data are aggregated according to topics, meta-data and additional features, using ad hoc online clustering algorithms. Classification: using time series to generate, network features, users, geographical, content features, etc., we classify information produced on the platform. Anomaly detection: real-time identification of anomalous events (e.g., induced by exogenous factors). Online learning: applying machine learning/deep learning methods to real-time information diffusion patterns analysis, users profiling, etc. Big Data Specific Challenges (Gaps) Dealing with real-time analysis of large volume of data. Providing a scalable infrastructure to allocate resources, storage space, etc. on-demand if required by increasing data volume over time. Big Data Specific Challenges in Mobility Implementing low-level data storage infrastructure features to guarantee efficient, mobile access to data. Security and Privacy Requirements Twitter publicly releases data collected by our platform. Although, data-sources incorporate user meta-data (in general, not sufficient to uniquely identify individuals) therefore some policy for data storage security and privacy protection must be implemented. Highlight issues for generalizing this use case (e.g. for ref. architecture) Definition of high-level data schema to incorporate multiple data-sources providing similarly structured data. More Information (URLs) dimensional
Appendix B: Summary of Key Properties Information related to five key properties was extracted from each use case. The five key properties were three Big Data characteristics (volume, velocity, and variety), software related information, and associated analytics. The extracted information is presented in Table B-1. Table B-1: Use Case Specific Information by Key Properties Use Case Volume Velocity Variety Software Analytics 1 Census 2000 and 2010 380 TB Static for 75 years Scanned documents Robust archival storage None for 75 years 2 NARA: Search, Retrieve, Preservation Hundreds of terabytes, and growing Data loaded in batches, so bursty Unstructured and structured data: textual documents, emails, photos, scanned documents, multimedia, social networks, web sites, databases, etc. Custom software, commercial search products, commercial databases Crawl/index, search, ranking, predictive search; data categorization (sensitive, confidential, etc.); personally identifiable information (PII) detection and flagging 3 Statistical Survey Response Improvement Approximately 1 PB Variable, field data streamed continuously, Census was ~150 million records transmitted Strings and numerical data Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig Recommendation systems, continued monitoring 4 Non-Traditional Data in Statistical Survey Response Improvement — — Survey data, other government administrative data, web-scraped data, wireless data, e-transaction data, (potentially) social media data and positioning data from various sources Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig New analytics to create reliable information from non-traditional disparate sources 5 Cloud Eco-System for Finance — Real time — Hadoop RDBMS XBRL Fraud detection 6 Mendeley 15 TB presently, growing about 1 TB per month Currently Hadoop batch jobs scheduled daily, PDF documents and log files of social network and client activities Hadoop, Scribe, Hive, Mahout, Python Standard libraries for machine learning and analytics, LDA,
real-time recommended in future reporting tools for aggregating readership and social activities per document 7 Netflix Movie Service Summer 2012 – 25 million subscribers, 4 million ratings per day, 3 million searches per day, 1 billion hours streamed in June 2012; Cloud storage – 2 petabytes in June 2013 Media (video and properties) and rankings continually updated Data vary from digital media to user rankings, user profiles, and media properties for content-based recommendations Hadoop and Pig; Cassandra; Teradata Personalized recommender systems using logistic/linear regression, elastic nets, matrix factorization, clustering, LDA, association rules, decision trees, and others; streaming video delivery 8 Web Search 45 billion web pages total, 500 million photos uploaded each day, 100 hours of video uploaded to YouTube each minute Real-time updating and real-time responses to queries Multiple media MapReduce + Bigtable; Dryad + Cosmos; PageRank; final step essentially a recommender engine Crawling; searching, including topic-based searches; ranking; recommending 9 Business Continuity and Disaster Recovery Within a Cloud Eco-System Terabytes up to petabytes Can be real time for recent changes Must work for all data Hadoop, MapReduce, open source, and/or vendor proprietary such as AWS, Google Cloud Services, and Microsoft Robust backup 10 Cargo Shipping — Needs to become real time, currently updated at events Event-based — Distributed event analysis identifying problems 11 Materials Data for Manufacturing 500,000 material types in 1980s, much growth since then Ongoing increase in new materials Many datasets with no standards National programs (Japan, Korea, and China), application areas (EU nuclear No broadly applicable analytics
17 Pathology Imaging 1 GB raw image data + 1.5 GB analytical results per 2D image, 1 TB raw image data + 1 TB analytical results per 3D image, 1 PB data per moderated hospital per year Once generated, data will not be changed Images MPI for image analysis, MapReduce + Hive with spatial extension Image analysis, spatial queries and analytics, feature clustering and classification 18 Computational Bioimaging Medical diagnostic imaging around 70 PB annually, 32 TB on emerging machines for a single scan Volume of data acquisition requires HPC back end Multi-modal imaging with disparate channels of data Scalable key-value and object store databases; ImageJ, OMERO, VolRover, advanced segmentation and feature detection methods Machine learning (support vector machine [SVM] and random forest [RF]) for classification and recommendation services 19 Genomic Measurements >100 TB in 1–2 years at NIST, many PBs in healthcare community ~300 GB of compressed data/day generated by DNA sequencers File formats not though some standards exist; generally structured data Open-source sequencing bioinformatics software from academic groups Processing of raw data to produce variant calls, clinical interpretation of variants 20 Comparative Analysis for Metagenomes and Genomes 50 TB New sequencers stream in data at growing rate Biological data that are inherently heterogeneous, complex, structural, and hierarchical; besides core genomic data, new types of omics data such as transcriptomics, methylomics, and proteomics Standard bioinformatics tools (BLAST, HMMER, multiple alignment and phylogenetic tools, gene callers, sequence feature predictors), Perl/Python wrapper scripts Descriptive statistics, statistical significance in hypothesis testing, data clustering and classification 21 Individualized Diabetes Management 5 million patients Not real time but updated periodically 100 controlled vocabulary values and 1,000 continuous values per patient, mostly time-stamped values HDFS supplementing Mayo internal data warehouse (EDT) Integration of data into semantic graphs, using graph traverse to replace SQL join; development of
semantic graph-mining algorithms to identify graph patterns, index graph, and search graph; indexed Hbase; custom code to develop new patient properties from stored data 22 Statistical Relational Artificial Intelligence for Health Care Hundreds of gigabytes for a single cohort of a few hundred people; possibly on the order of 1 PB when dealing with millions of patients Constant updates to EHRs; in other controlled studies, data often in batches at regular intervals Critical feature – data typically in multiple tables, need to be merged to perform analysis Mainly Java-based, tools to process the data Relational probabilistic models (Statistical Relational AI) learned from multiple data types 23 World Population-Scale Epidemiological Study 100 TB Low number of data feeding into the simulation, massive amounts of real-time data generated by simulation Can be rich with various population activities, geographical, cultural variations Charm++, MPI Simulations on a synthetic population 24 Social Contagion Modeling for Planning Tens of terabytes per year During social unrest events, human interactions and mobility leads to rapid changes in data; e.g., who follows whom in Twitter Big issues – data fusion, combining data from different sources, dealing with missing or incomplete data Specialized simulators, open source software, proprietary modeling environments; databases Models of behavior of humans and hard infrastructures, models of their interactions, visualization of results 25 Biodiversity and LifeWatch N/A Real-time processing and analysis in case of Rich variety and number of involved databases and observation data RDBMS Requires advanced and rich visualization
natural or industrial disaster 26 Large-Scale Deep Learning Current datasets typically 1 to 10 TB, possibly 100 million images to train a self-driving car Much faster than real-time processing; for autonomous driving, need to process thousands of high-resolution (six megapixels or more) images per second Neural net very heterogeneous as it learns many different features In-house GPU kernels and MPI-based communication developed by Stanford, C++/Python source Small degree of batch statistical pre-processing, all other data analysis performed by the learning algorithm itself 27 Organizing Unstructured Collections of Consumer Photos 500+ billion photos on Facebook, 5+ billion photos on Flickr Over 500 million images uploaded to Facebook each day Images and metadata including EXIF (Exchangeable Image File) tags (focal distance, camera type, etc.) Hadoop MapReduce, simple hand-written multi-threaded tools (Secure Shell [SSH] and sockets for communication) Robust non-linear least squares optimization problem, SVM 28 Truthy Twitter Data 30 TB/year compressed data Near real-time data storage, querying and analysis Schema provided by social media data source; currently using Twitter only; plans to expand, incorporating Google+ and Facebook Hadoop IndexedHBase and HDFS; Hadoop, Hive, Redis for data management; Python: SciPy NumPy and MPI for data analysis Anomaly detection, stream clustering, signal classification, online learning; information diffusion, clustering, dynamic network visualization 29 Crowd Sourcing in Humanities Gigabytes (text, surveys, experiment values) to hundreds of terabytes (multimedia) Data continuously updated and analyzed incrementally So far mostly homogeneous small data sets; expected large distributed heterogeneous datasets XML technology, traditional relational databases Pattern recognition (e.g., speech recognition, automatic audio-visual analysis, cultural patterns), identification of structures (lexical units, linguistic rules, etc.) 30 CINET for Network Science Can be hundreds of gigabytes for a single network, Dynamic networks, network collection growing Many types of networks Graph libraries (Galib, NetworkX); distributed workflow management Network visualization
networks and methods (Simfrastructure, databases, semantic web tools) 31 NIST Information Access Division >900 million web pages occupying 30 TB of storage, 100 million tweets, 100 million biometric images, hundreds of thousands of partially video clips, terabytes of smaller fully ground-truthed test collections Legacy evaluations mostly focused on retrospective analytics, newer evaluations focused on simulations of real-time analytic challenges from multiple data streams Wide variety of data types including textual search/extraction, machine translation, speech recognition, image and voice biometrics, object and person recognition and tracking, document analysis, human-computer dialogue, multimedia search/extraction PERL, Python, C/C++, Matlab, R development tools; create ground-up test and measurement applications Information extraction, filtering, search, and summarization; image and voice biometrics; speech recognition and understanding; machine translation; video person/object detection and tracking; event detection; imagery/document matching; novelty detection; structural semantic temporal analytics 32 DataNet (iRODS) Petabytes, hundreds of millions of files Real time and batch Rich iRODS Supports general analysis workflows 33 The Discinnet Process Small as metadata to Big Data Real time Can tackle arbitrary Big Data Symfony-PHP, Linux, MySQL -- 34 Semantic A few terabytes Evolving in time Rich Database Data graph processing 35 Light Source Beamlines 50–400 GB per day, total ~400 TB Continuous stream of data, but analysis need not be real time Images Octopus for Tomographic Reconstruction, Avizo () and FIJI (a distribution of ImageJ) Volume reconstruction, feature identification, etc. 36 Catalina Transient Survey ~100 TB total increasing by 0.1 TB a night accessing PBs of base astronomy data, 30 TB a night Nightly update runs processes in real time Images, spectra, time series, catalogs Custom data processing pipeline and data analysis software Detection of rare events and relation to existing diverse data
from successor LSST in 2020s 37 DOE Extreme Data from Cosmological Sky Survey Several petabytes from Dark Energy Survey and Zwicky Transient Factory, simulations > 10 PB Analysis done in batch mode with data from observations and simulations updated daily Image and simulation data MPI, FFTW, viz packages, numpy, Boost, OpenMP, ScaLAPCK, PSQL and MySQL databases, Eigen, cfitsio, astrometry.net, and Minuit2 New analytics needed to analyze simulation results 38 Large Survey Data for Cosmology Petabytes of data from Dark Energy Survey 400 images of 1 GB in size per night Images Linux cluster, Oracle RDBMS server, Postgres PSQL, large memory machines, standard Linux interactive hosts, GPFS; for simulations, HPC resources; standard astrophysics reduction software as well as Perl/Python wrapper scripts Machine learning to find optical transients, Cholesky decomposition for thousands of simulations with matrices of order 1 million on a side and parallel image storage 39 Particle Physics at LHC 15 PB of data (experiment and Monte Carlo combined) per year Data updated continuously with sophisticated selection and test analysis but all analyzed "properly" offline Different format for each stage in analysis but data uniform within each stage Grid-based environment with over 350,000 cores running simultaneously Sophisticated specialized data analysis code followed by basic exploratory statistics (histogram) with complex detector efficiency corrections 40 Belle II High Energy Physics Experiment Eventually 120 PB of Monte Carlo and observational data Data updated continuously with sophisticated selection and test analysis but all analyzed "properly" offline Different format for each stage in analysis but data uniform within each stage DIRAC Grid software Sophisticated specialized data analysis code followed by basic exploratory statistics (histogram) with complex detector efficiency corrections
41 EISCAT 3D incoherent scatter radar system Terabytes/year (current), 40 PB/year starting ~2022 Data updated continuously with real-time test analysis and batch full analysis Big data uniform Custom analysis based on flat file data storage Pattern recognition, demanding correlation routines, high-level parameter extraction 42 ENVRI Environmental Research Infrastructure Low volume (apart from EISCAT 3D given above), one system EPOS ~15 TB/year Mainly real-time data streams Six separate projects with common architecture for infrastructure, data very diverse across projects R and Python (Matplotlib) for visualization, custom software for processing Data assimilation, (statistical) analysis, data mining, data extraction, scientific modeling and simulation, scientific workflow 43 CReSIS Remote Sensing Around 1 PB (current) increasing by 50–100 TB per mission, future expedition ~1 PB each Data taken in ~two-month missions including test analysis and then later batch processing Raw data, images with final layer data used for science Matlab for custom raw data processing, custom image processing software, GIS as user interface Custom signal processing to produce radar images that are analyzed by image processing to find layers 44 UAVSAR Data Processing 110 TB raw data and 40 TB processed, plus smaller samples Data come from aircraft and so incrementally added, data occasionally get reprocessed: new processing methods or parameters Image and annotation files ROI_PAC, GeoServer, GDAL, tools; moving to clouds Process raw data to get images that are run through image processing tools and accessed from GIS 45 NASA LARC/GSFC iRODS MERRA collection (below) represents most of total data, other smaller collections Periodic updates every six months Many applications to combine MERRA reanalysis data with other reanalyses and observational data such as CERES SGE Univa Grid Engine Version 8.1, iRODS Version 3.2 and/or 3.3, IBM GPFS Version 3.4, Cloudera Version 4.5.2-1 Federation software 46 MERRA Analytic Services 480 TB from MERRA Increases at ~1 TB/month Applications to combine MERRA reanalysis data with other re-analyses and observational data Cloudera, iRODS, Amazon AWS Climate (CAaaS)
Appendix C: Use Case Requirements Summary Requirements were extracted from each use case within seven characteristic categories introduced in Section 3.1. The number of requirements within each category varied for each use case. Table C-1 contains the use case specific requirements. Table C-1: Use Case Specific Requirements Use Case Data Sources Data Transformation Capabilities Data Consumer Security and Privacy Lifecycle Management Other 1 Census 2010 and 2000 1. Large document format from centralized storage -- 1. Large centralized storage (storage) -- 1. Title 13 data 1. Long-term preservation of data as-is for 75 years 2. Long-term preservation at the bit level 3. Curation process including format transformation 4. Access and analytics processing after 75 years 5. No data loss -- 2 NARA: Search, Retrieve, Preservation 1. Distributed data sources 2. Large data storage 3. Bursty data ranging from gigabytes to hundreds of terabytes 4. Wide variety of data formats including unstructured and structured data 5. Distributed 1. Crawl and index from distributed data sources 2. Various analytics processing including ranking, data categorization, detection of PII data 3. Data 4. Long-term preservation management of 1. Large data storage 2. Various storage systems such as NetApps, Hitachi, magnetic tapes 1. High relevancy and high recall from search 2. High accuracy from categorization of records 3. Various storage systems such as NetApps, Hitachi, magnetic tapes 1. Security policy 1. Pre-process for virus scan 2. File format identification 3. Indexing 4. Records categorization 1. Mobile search with similar interfaces/ results from desktop
data sources in different clouds large varied datasets 5. Huge numbers of data with high relevancy and recall 3 Statistical Survey Response 1. Data size of approximately one petabyte 1. Analytics for recommendation systems, continued monitoring, and general survey improvement 1. Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig 1. Data visualization for data review, operational activity, and general analysis; continual evolution 1. Improved systems that reduce costs and improve quality while providing confidentiality safeguards that are reliable and publicly auditable 2. Confidential and secure data; processes that are auditable for security and confidentiality as required by various legal statutes 1. High veracity on data and very robust systems (challenges: semantic integrity of conceptual metadata concerning what exactly is measured and the resulting limits of inference) 1. Mobile access 4 Data in Statistical Survey Response -- 1. Analytics to create reliable estimates using data from traditional survey sources, government 1. Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, 1. Data visualization for data review, operational activity, and general 1. Confidential and secure data; processes that are auditable for security and confidentiality 1. High veracity on data and very robust systems (challenges: semantic integrity of conceptual metadata concerning what --
administrative data sources, and sources from the digital economy BigMemory, Cassandra, Pig analysis; continual evolution as required by various legal statutes exactly is measured and the resulting limits of inference) 5 Cloud for Finance 1. Real-time ingestion of data 1. Real-time analytics -- -- 1. Strong security and privacy constraints -- 1. Mobile access 6 Mendeley 1. File-based documents with constant new uploads 2. Variety of file types such as PDFs, social network log files, client activities images, spreadsheet, presentation files 1. Standard machine learning and analytics libraries 2. Efficient scalable and parallelized way to match between documents 3. Third-party annotation tools or publisher watermarks and cover pages 1. Amazon Elastic Compute Cloud (EC2) with HDFS (infrastructure) 2. S3 (storage) 3. Hadoop (platform) 4. Scribe, Hive, Mahout, Python (language) 5. Moderate storage (15 TB with 1 TB/ month) 6. Batch and processing 1. reporting tools 2. Visualization tools such as networking graph, scatterplots, etc. 1. Access controls for who reads what content 1. Metadata management from PDF extraction 2. Identification of document duplication 3. Persistent identifier 4. Metadata between data repositories such as CrossRef, PubMed, and Arxiv 1. Windows Android and iOS mobile devices for content deliverables from Windows desktops 7 Netflix Movie Service 1. User profiles and ranking information 1. Streaming video contents to multiple clients 2. Analytic processing for matching client interest in movie selection 3. Various analytic processing techniques for 1. Hadoop (platform) 2. Pig (language) 3. Cassandra and Hive 4. Huge numbers of subscribers, ratings, and searches per day (DB) 5. Huge amounts 1. Streaming and rendering media 1. Preservation of users, privacy and digital rights for media 1. Continued ranking and updating based on user profile and analytic results 1. Smart interface accessing movie content on mobile platforms
System consumer personalization 4. Robust learning algorithms 5. Continued analytic processing based on monitoring and performance results of storage (2 PB) 6. I/O intensive processing 8 Web Search 1. Distributed data sources 2. Streaming data 3. Multimedia content 1. Dynamic fetching content over the network 2. Linking of user profiles and social network data 1. Petabytes of text and rich media (storage) 1. Search time of ~0.1 seconds 2. Top 10 ranked results 3. Page layout (visual) 1. Access control 2. Protection of sensitive content 1. Data purge after certain time interval (a few months) 2. Data cleaning 1. Mobile search and rendering 9 Business Continuity and Disaster Recovery Within a Cloud -- 1. Robust backup algorithm 2. Replication of recent changes 1. Hadoop 2. Commercial cloud services -- 1. Strong security for many applications -- -- 10 Cargo Shipping 1. Centralized and real-time distributed sites/sensors 1. Tracking items based on the unique identification with its sensor information, GPS coordinates 2. Real-time updates on tracking items 1. Internet connectivity -- 1. Security policy -- --
11 Materials Data for 1. Distributed data repositories for more than 500,000 commercial materials 2. Many varieties of datasets 3. Text, graphics, and images 1. Hundreds of independent variables need to be collected to create robust datasets -- 1. Visualization for materials discovery from many independent variables 2. Visualization tools for materials 1. Protection of proprietary sensitive data 2. Tools to mask proprietary information 1. Handle data quality (currently poor or no process) -- 12 Materials Genomics 1. Data streams from peta/exascale centralized simulation systems 2. Distributed web dataflows from central gateway to users 1. High-throughput computing data analysis for web-like responsiveness 2. Mashup of simulation outputs across codes 3. Search and crowd-driven with computation backend, flexibility for new targets 4. MapReduce and search to join simulation and experimental data 1. Massive (150,000 cores) legacy infrastructure (infrastructure) 2. GPFS (storage) 3. MonogDB systems (platform) 4. 10 GB networking 5. Various analytic tools such as PyMatGen, FireWorks, VASP, ABINIT, NWChem, BerkeleyGW, varied community codes 6. Large storage (storage) 7. Scalable and object 1. search for growing materials data 1. Sandbox as independent working areas between different data stakeholders 2. Policy-driven federation of datasets 1. Validation and uncertainty quantification (UQ) of simulation with experimental data 2. UQ in results from multiple datasets 1. Mobile applications (apps) to access materials genomics information
store (platform) 8. Data streams from peta/exascale centralized simulation systems 13 Large-Scale Geospatial Analysis and Visualization 1. Unique approaches to indexing and distributed analysis required for geospatial data 1. Analytics: closest point of approach, deviation from route, point density over time, PCA and ICA 2. Unique approaches to indexing and distributed analysis required for geospatial data 1. Geospatially enabled RDBMS, geospatial server/analysis software, e.g., ESRI ArcServer, Geoserver 1. Visualization with GIS at high and low network bandwidths and on dedicated facilities and handhelds 1. Complete security of sensitive data in transit and at rest (particularly on handhelds) -- -- 14 Object and Tracking 1. Real-time data FMV (30–60 frames/ second at full-color 1080P resolution) and WALF (1–10 frames/ second at 10,000 x 10,000 full-color resolution) 1. Rich analytics with object identification, pattern recognition, crowd behavior, economic activity, and data fusion 1. Wide range of custom software and tools including traditional RDBMSs and display tools 2. Several network requirements 3. GPU usage important 1. Visualization of extracted outputs as overlays on a geospatial display; links back to the originating image/video segment as overlay objects 2. Output the form of Open Geospatial 1. Significant security and privacy issues; sources and methods never compromised 1. Veracity of extracted objects --
million entities (patients), > 4 billion records or data points (discrete clinical observations), aggregate of > 20 TB raw data 3. Velocity: 500,000–1.5 million new transactions per day 4. Variety: formats include numeric, structured numeric, structured text, discrete nominal, discrete ordinal, discrete structured, binary large blobs (images and video) 5. Data evolve over time in a highly variable fashion 2. Analytic techniques: information retrieval, NLP, machine learning decision models, maximum likelihood estimators, Bayesian networks MongoDB 4. Various, with significant I/O intensive processing i.e., those who did not actually perform the analysis; specific visualization techniques performed by informatics research scientists and health service researchers 2. Protection of all health data in compliance with governmental regulations 3. Protection of data in accordance with data providers, policies. 4. Security and privacy policies unique to a data subset 5. Robust security to prevent data breaches 3. Common nomenclature and classification of content across disparate challenging in the health IT space, as the taxonomies continue to evolve— SNOMED, International Classification of Diseases (ICD) 9 and future ICD 10, etc. 17 Pathology Imaging 1. High-resolution spatial digitized pathology images 2. Various image quality analyses 1. image analysis to extract spatial information 1. Legacy system and cloud (computing cluster) 2. Huge legacy 1. Visualization for validation and training 1. Security and privacy protection for protected 1. Human annotations for validation 1. 3D visualization and rendering on
algorithms 3. Various image data formats, especially BigTIFF with structured data for analytical results 4. Image analysis, spatial queries and analytics, feature clustering, and classification 2. Spatial queries and analytics, feature clustering and classification 3. Analytic processing on huge multi-dimensional large dataset; correlation with other data types such as clinical data, omic data and new storage such as storage area network (SAN) or HDFS (storage) 3. network link (networking) 4. MPI image analysis, MapReduce, Hive with spatial extension (software packages) health information mobile platforms 18 Bioimaging 1. Distributed multi-modal experimental sources of bioimages (instruments) 2. 50 TB of data in formats that include images 1. High-throughput computing with responsive analysis 2. Segmentation of regions of interest; crowd-based selection and extraction of features; object classification, and organization; and search 3. Advanced biosciences discovery through Big Data techniques / extreme-scale computing; 1. ImageJ, OMERO, VolRover, advanced segmentation and feature detection methods from applied math researchers; scalable key-value and object store databases needed 2. NERSC’s Hopper infrastructure 3. database and image collections 4. 10 GB and future 100 GB and 1. 3D structural modeling 1. Significant but optional security and privacy including secure servers and anonymization 1. Workflow components including data acquisition, storage, enhancement, minimizing noise --
processing and analytics; machine learning (SVM and RF) for classification and recommendation services; advanced algorithms for massive image analysis; computational solutions 4. Massive data analysis toward massive imaging datasets. advanced networking (software-defined networking [SDN]) 19 Genomic 1. compressed data (300 GB/day) from various DNA sequencers 2. Distributed data source (sequencers) 3. Various file formats with both structured and unstructured data 1. Processing raw data in variant calls 2. Challenge: characterizing machine learning for complex analysis on systematic errors from sequencing technologies 1. Legacy computing cluster and other PaaS and IaaS (computing cluster) 2. Huge data storage in PB range (storage) 3. Unix-based legacy sequencing bioinformatics software (software package) 1. Data format for genome browsers 1. Security and privacy protection of health records and clinical research databases -- 1. Mobile platforms for physicians accessing genomic data (mobile device) 20 Comparative 1. Multiple centralized data 2. Scalable RDBMS for heterogeneous 1. Huge data storage 1. Real-time interactive 1. Login security: 1. Methods to improve data quality --
categories of structured data about a patient: data with controlled vocabulary (CV) property values and data with continuous property values (recorded/ captured more frequently) 6. Data consist of text and continuous numerical values sub-graph significance, validate results, eliminate false positive/false negative results 5. Semantic graph mining algorithms to identify graph patterns, index and search graph 6. Semantic graph traversal 22 Statistical Relational Artificial Intelligence for Health Care 1. Centralized data, with some data retrieved from Internet sources 2. Range from hundreds of gigabytes for a sample size to 1 PB for very large studies 3. Both constant updates/additions (to data subsets) and scheduled batch inputs 4. Large, 1. Relational probabilistic models/ probability theory; software that learns models from multiple data types and can possibly integrate the information and reason about complex queries 2. Robust and accurate learning methods to account for data imbalance (where large numbers of 1. Java, some in house tools, [relational] database and NoSQL stores 2. Cloud and parallel computing 3. computer, 48 GB RAM (to perform analysis for a moderate sample size) 4. Dlusters for large datasets 5. 200 GB–1 TB 1. Visualization of very large data subsets 1. Secure handling and processing of data 1. Merging multiple tables before analysis 2. Methods to validate data to minimize errors --
network (networking) 24 Social Contagion Modeling for Planning 1. Traditional and new architecture for dynamic distributed processing on commodity clusters 2. Fine-resolution models and datasets to support Twitter network traffic 3. Huge data storage supporting annual data growth 1. Large-scale modeling for various events (disease, emotions, behaviors, etc.) 2. Scalable fusion between combined datasets 3. Multi-level analysis while generating sufficient results quickly 1. Computing infrastructure that can capture human-to-human interactions on various social events via the Internet (infrastructure) 2. File servers and databases (platform) 3. Ethernet and Infiniband networking (networking) 4. Specialized simulators, open source software, and proprietary modeling (application) 5. Huge user accounts across country boundaries (networking) 1. Multi-level detailed network 2. Visualization with interactions 1. Protection of PII of individuals used in modeling 2. Data protection and secure platform for computation 1. Data fusion from variety of data sources (i.e., Stata data files) 2. Data consistency and no corruption 3. Preprocessing of raw data 1. Efficient method of moving data 25 Biodiversity and LifeWatch 1. Special dedicated or overlay sensor network 2. Storage: distributed, 1. Web-based services, services, relational databases, NoSQL 2. Personalized 1. Expandable storage resource for global users 2. Cloud 1. Access by mobile users 2. Advanced/ visualization 1. Federated identity management for mobile researchers and mobile sensors 1. Data storage and archiving, data exchange and integration 2. Data lifecycle management: data --
historical, and trends data archiving 3. Distributed data sources, including observation and monitoring facilities, sensor network, and satellites 4. Wide variety of data: satellite images/ information, climate and weather data, photos, video, sound recordings, etc. 5. Multi-type data combination and linkage, potentially unlimited data variety 6. Data streaming virtual labs 3. Grid- and resources 4. Data analyzed incrementally and/or in real time at varying rates owing to variations in source processes 5. A variety of data and analytical and modeling tools to support analytics for diverse scientific communities 6. Parallel data streams and streaming analytics 7. Access and integration of multiple distributed databases community resource required 3. 4D visualization computational models 2. Access control and accounting provenance, referral integrity and identification traceability back to initial observational data 3. Processed (secondary) data storage (in addition to original source data) for future uses 4. Provenance (and persistent identification [PID]) control of data, algorithms, and workflows 5. Curated (authorized) reference data (e.g. species name lists), algorithms, software code, workflows 26 Large-Scale Deep Learning -- -- 1. GPU 2. MPI and HPC Infiniband cluster 3. Libraries for single-machine or single-GPU -- -- -- --
computation – available (e.g., BLAS, CuBLAS, MAGMA, etc.); distributed computation of dense BLAS-like or LAPACK-like operations on GPUs – poorly developed; existing solutions (e.g., ScaLapack for CPUs) – not well-integrated with higher-level languages and require low-level programming, lengthening experiment and development time 27 Organizing Large-Scale Collections of Consumer Photos 1. Over 500 million images uploaded to social media sites each day 1. Classifier (e.g. an SVM), a process that is often hard to parallelize 2. Features seen in many large-scale image processing problems 1. Hadoop or enhanced MapReduce 1. Visualize large-scale 3D navigate large-scale collections of images that have been aligned to maps 1. Preserve privacy for users and digital rights for media -- --
28 Truthy Twitter Data 1. Distributed data sources 2. Large volume of real-time streaming data 3. Raw data in compressed formats 4. Fully structured data in JSON, user metadata, data 5. Multiple data schemas 1. Various data analysis for anomaly detection, stream clustering, signal classification on multi-dimensional time series, online learning 1. Hadoop and HDFS (platform) 2. IndexedHBase, Hive, SciPy, NumPy (software) 3. In-memory database, MPI (platform) 4. High-speed Infiniband network (networking) 1. Data retrieval and dynamic visualization 2. Data-driven interactive web interfaces 3. API for data query 1. Security and privacy policy 1. Standardized data structures/ formats with extremely high data quality 1. Low-level data storage for efficient mobile access to data 29 Crowd Sourcing in Humanities -- 1. Digitize existing audio-video, photo, and documents archives 2. Analytics: pattern recognition of all kinds (e.g., speech recognition, automatic A&V analysis, cultural patterns), identification of structures (lexical units, linguistic rules, etc.) -- -- 1. Privacy issues in preserving anonymity of responses in spite of computer recording of access ID and reverse engineering of unusual user responses -- -- 30 CINET for Network Science 1. A set of network topologies files to study graph 1. Environments to run various network and graph analysis tools 1. Large file system (storage) 2. Various network 1. Client-side visualization -- -- --
theoretic properties and behaviors of various algorithms 2. Asynchronous and real-time synchronous distributed computing 2. Dynamic growth of the networks 3. Asynchronous and real-time synchronous distributed computing 4. Different parallel algorithms for different partitioning schemes for efficient operation connectivity (networking) 3. Existing computing cluster 4. EC2 computing cluster 5. Various graph libraries, management tools, databases, semantic web tools 31 NIST Information Access Division 1. Large amounts of web pages, tweets, images, video 2. Scaling to larger data, intrinsic and annotation uncertainty measurement, performance measurement for incompletely annotated data, measuring analytic performance for heterogeneous data and analytic flows involving users 1. Test analytic algorithms working with written language, speech, human imagery, etc. against real or realistic data; challenge: engineering artificial data that sufficiently captures the variability of real data involving humans 1. PERL, Python, C/C++, Matlab, R development tools; creation of ground-up test and measurement applications 1. Analytic flows involving users 1. Security requirements for protecting sensitive data while enabling meaningful developmental performance evaluation; shared evaluation testbeds that protect the intellectual property of analytic algorithm developers -- --
32 DataNet (iRODS) 1. Process key format types NetCDF, HDF5, Dicom 2. Real-time and batch data 1. Provision of general analytics workflows needed 1. iRODS data management software 2. interoperability across storage and network protocol types 1. General visualization workflows 1. Federate across existing authentication environments through Generic Security Service API and pluggable authentication modules (GSI, Kerberos, InCommon, Shibboleth) 2. Access controls on files independent of the storage location -- -- 33 The Discinnet Process 1. Integration of metadata approaches across disciplines -- 1. Software: Symfony-PHP, Linux, MySQL -- 1. Significant but optional security and privacy including secure servers and anonymization 1. Integration of metadata approaches across disciplines -- 34 Semantic 1. All data types, image to text, structures to protein sequence 1. Data graph processing 2. RDBMS 1. Cloud community resource required 1. Efficient visualization needed -- -- -- 35 Light source beamlines 1. Multiple streams of data to be 1. Standard bioinformatics tools (BLAST, 1. High-volume data transfer to remote batch -- 1. Multiple security and privacy -- --
37 DOE Extreme Data from Sky Survey 1. ~1 PB/year becoming 7 PB/year of observational data 1. Advanced analysis and visualization techniques and capabilities to support interpretation of results from detailed simulations 1. MPI, OpenMP, C, C++, F90, FFTW, viz packages, Python, FFTW, numpy, Boost, OpenMP, ScaLAPCK, PSQL and MySQL databases, Eigen, cfitsio, astrometry.net, and Minuit2 2. Methods/ tools to address supercomputer I/O subsystem limitations 1. Interpretation of results using advanced visualization techniques and capabilities -- -- -- 38 Large Survey Data for Cosmology 1. 20 TB of data/day 1. Analysis on both the simulation and observational data simultaneously 2. Techniques for handling Cholesky decomposition for thousands of simulations with matrices of order 1 million on a side 1. Standard astrophysics reduction software as well as Perl/Python wrapper scripts 2. Oracle RDBMS, Postgres psql, GPFS and Lustre file systems and tape archives 3. Parallel image storage -- -- 1. Links between remote telescopes and central analysis sites -- 39 Particle Physics at LHC 1. Real-time data from accelerator and analysis instruments 2. 1. Experimental data from ALICE, ATLAS, CMS, LHB 2. Histograms, scatter-plots with 1. Legacy computing infrastructure (computing nodes) 1. Histograms and model fits (visual) 1. Data protection 1. Data quality on complex apparatus --
Asynchronization data collection 3. Calibration of instruments model fits 3. Monte-Carlo computations 2. Distributed cached files (storage) 3. Object databases (software package) 40 Belle II Physics Experiment 1. 120 PB of raw data -- 1. 120 PB raw data 2. International distributed computing model to augment that at accelerator (Japan) 3. Data transfer of ~20 GB/ second at designed luminosity between Japan and United States 4. Software from Open Science Grid, Geant4, DIRAC, FTS, Belle II framework -- 1. Standard grid authentication -- -- 41 EISCAT 3D Incoherent Scatter Radar System 1. Remote sites generating 40 PB data/year by 2022 2. Hierarchical Data Format (HDF5) 3. Visualization of high-dimensional (≥5) data 1. Queen Bea architecture with mix of distributed on-sensor and central processing for 5 distributed sites 2. Real-time monitoring of 1. Architecture compatible with ENVRI 1. Support needed for visualization of (≥5) data -- 1. Preservation of data and avoidance of lost data due to instrument malfunction 1. Support needed for real-time monitoring of equipment by partial streaming analysis
equipment by partial streaming analysis 3. Hosting needed for rich set of radar image processing services using machine learning, statistical modelling, and graph algorithms 42 ENVRI Research 1. Huge volume of data from distributed data sources 2. Variety of instrumentation datasets and metadata 1. Diversified analytics tools 1. Variety of computing infrastructures and architectures (infrastructure) 2. Scattered repositories (storage) 1. Graph plotting tools 2. Time series interactive tools 3. flash playback 4. Earth map display 5. Visual tools for quality comparisons 1. Open data policy with minor restrictions 1. High data quality 2. Mirror archives 3. Various metadata frameworks 4. Scattered repositories and data curation 1. Various kinds of mobile sensor devices for data acquisition 43 CReSIS Remote Sensing 1. Provision of reliable data transmission from aircraft sensors/ instruments or removable disks from remote sites 2. Data gathering in real time 1. Legacy software (Matlab) and language (C/Java) binding for processing 2. Signal processing and advanced image processing to find layers needed 1. ~0.5 PB/year of raw data 2. Transfer content from removable disk to computing cluster for parallel processing 3. MapReduce or 1. GIS user interface 2. Rich user interface for simulations 1. Security and privacy on sensitive political issues 2. Dynamic security and privacy policy mechanisms 1. Data quality assurance 1. Monitoring data collection instruments/ sensors
3. Varieties of datasets MPI plus language binding for C/Java 44 UAVSAR Data Processing 1. Angular and spatial data 2. Compatibility with other NASA radar systems and repositories (Alaska Satellite Facility) 1. Geolocated data that require GIS integration of data as custom overlays 2. Significant human intervention in data processing pipeline 3. Hosting of rich set of radar image processing services 4. ROI_PAC, GeoServer, GDAL, tools 1. Support for interoperable Cloud-HPC architecture 2. Hosting of rich set of radar image processing services 3. ROI_PAC, GeoServer, GDAL, tools 4. Compatibility with other NASA radar systems and repositories (Alaska Satellite Facility) 1. Support for field expedition users with phone/tablet interface and low-resolution downloads -- 1. Significant human intervention in data processing pipeline 2. Rich robust provenance defining complex machine/human processing 1. Support for field expedition users with phone/tablet interface and downloads 45 NASA LARC/ GSFC iRODS 1. Federate distributed heterogeneous datasets 1. CAaaS on clouds 1. Support virtual climate data server (vCDS) 2. GPFS parallel file system integrated with Hadoop 3. iRODS 1. Support needed to visualize distributed data -- -- -- 46 MERRA Analytic Services 1. Integrate simulation output and observational data, NetCDF files 2. Real-time and batch mode needed 1. CAaaS on clouds 1. NetCDF aware software 2. MapReduce 3. Interoperable use of AWS and local clusters 1. High-end distributed visualization -- -- 1. Smart phone and tablet access required 2. iRODS data management
3. Interoperable use of AWS and local clusters 4. iRODS data management 47 Atmospheric Turbulence 1. Real-time distributed datasets 2. Various formats, resolution, semantics, and metadata 1. MapReduce, SciDB, and other scientific databases 2. Continuous computing for updates 3. Event specification language for data mining and event searching 4. Semantics interpretation and optimal structuring for 4D data mining and predictive analysis 1. Other legacy computing systems (e.g. supercomputer) 2. high throughput data transmission over the network 1. Visualization to interpret results -- 1. Validation for output products (correlations) -- 48 Climate Studies 1. ~100 PB data in 2017 streaming at high data rates from large supercomputers across the world 2. Integration of large-scale distributed data from simulations with diverse observations 3. Linking of 1. Data analytics close to data storage 1. Extension of architecture to several other fields 1. Worldwide climate data sharing 2. High-end distributed visualization -- -- 1. input and access
diverse data to novel HPC simulation 49 DOE-BER Subsurface 1. Heterogeneous diverse data with different domains and scales, translation across diverse datasets that cross domains and scales 2. Synthesis of diverse and disparate field, laboratory, omic, and simulation datasets across different semantic, spatial, and temporal scales 3. Linking of diverse data to novel HPC simulation -- 1. Postgres, HDF5 data technologies, and many custom software systems 1. input and access -- -- 1. input and access 50 DOE-BER AmeriFlux and FLUXNET Networks 1. Heterogeneous diverse data with different domains and scales, translation across diverse datasets that cross domains and scales 1. Custom software such as EddyPro, and custom analysis software, such as R, Python, neural networks, Matlab 1. Custom software, such as EddyPro, and custom analysis software, such as R, Python, neural networks, Matlab 2. Analytics including data 1. input and access -- -- 1. input and access