At this point, typical systems used are Extensible Markup Language (XML) technology and traditional relational databases. Other than pictures, not much multi-media is employed yet. Crowd sourcing is beginning to be used on a larger scale. However, the availability of sensors in mobile devices provides a huge potential for collecting large amount of data from numerous individuals. This possibility has not been explored on a large scale so far; existing crowd sourcing projects are usually of a limited scale and web-based. Privacy issues may be involved because of access to individuals’ audiovisual files; anonymization may be necessary but not always possible. Data management and curation are critical. With multimedia, the size could be hundreds of terabytes. 30: Submitted by Madhav Marathe and Keith Bisset, Virginia Tech CINET provides a common web-based platform that allows the end user seamless access to the following: Network and graph analysis tools such as SNAP, NetworkX, and Galib Real-world and synthetic networks Computing resources Data management systems. CINET uses an Infiniband-connected HPC cluster with 720 cores to provide HPC as a service. The platform is being used for research and education. CINET is used in classes and to support research by social science and social networking communities Rapid repository growth is expected to lead to at least 1,000 to 5,000 networks and methods in about a year. As more fields use graphs of increasing size, parallel algorithms will be important. Two critical challenges are data manipulation and bookkeeping of the derived data, as there are no well-defined and effective models and tools for unified management of various graph data. 31: NIST Submitted by John Garofolo, NIST Performance metrics, measurement methods, and community evaluations are needed to ground and accelerate development of advanced analytic technologies in the areas of speech and language processing, video and multimedia processing, biometric image processing, and heterogeneous data processing, as well as the interaction of analytics with users. Typically one of two processing models are employed: (1) push test data out to test participants, and analyze the output of participant systems, and (2) push algorithm test harness interfaces out to participants, bring in their algorithms, and test them on internal computing clusters.
case (e.g. for ref. architecture) More Information (URLs) Government Operation> Use Case 3: Statistical Survey Response Improvement Use Case Title Statistical Survey Response Improvement (Adaptive Design) Vertical (area) Government Statistical Logistics Author/Company/Email Cavan Capps: U.S. Census Actors/Stakeholders and their roles and responsibilities U.S. statistical agencies are charged to be the leading authoritative sources about the nation’s people and economy, while honoring privacy and rigorously protecting confidentiality. This is done by working with states, local governments and other government agencies. Goals To use advanced methods, that are open and scientifically objective, the statistical agencies endeavor to improve the quality, the specificity and the timeliness of statistics provided while reducing operational costs and maintaining the confidentiality of those measured. Use Case Description Survey costs are increasing as survey response declines. The goal of this work is to use advanced “recommendation system techniques” using data mashed up from several sources and historical survey para-data to drive operational processes in an effort to increase quality and reduce the cost of field surveys. Current Solutions Compute(System) Linux systems Storage SAN and Direct Storage Networking Fiber, 10 gigabit Ethernet, Infiniband 40 gigabit. Software Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig Big Data Characteristics Data Source (distributed/centralized) Survey data, other government administrative data, geographical positioning data from various sources. Volume (size) For this particular class of operational problem approximately one petabyte. Velocity (e.g. real time) Varies, paradata from field data streamed continuously, during the decennial census approximately 150 million records transmitted. Variety (multiple datasets, mashup) Data is typically defined strings and numerical fields. Data can be from multiple datasets mashed together for analytical use. Variability (rate of change) Varies depending on surveys in the field at a given time. High rate of velocity during a decennial census. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues, semantics) Data must have high veracity and systems must be very robust. The semantic integrity of conceptual metadata concerning what exactly is measured and the resulting limits of inference remain a challenge Visualization Data visualization is useful for data review, operational
Networking Infiniband. High bandwidth 3D Torus. Software Charm++, MPI Big Data Characteristics Data Source (distributed/centralized) Generated from synthetic population generator. Currently centralized. However, could be made distributed as part of post-processing. Volume (size) 100TB Velocity (e.g. real time) Interactions with experts and visualization routines generate large amount of real time data. Data feeding into the simulation is small but data generated by simulation is massive. Variety (multiple datasets, mashup) Variety depends upon the complexity of the model over which the simulation is being performed. Can be very complex if other aspects of the world population such as type of activity, geographical, socio-economic, cultural variations are taken into account. Variability (rate of change) Depends upon the evolution of the model and corresponding changes in the code. This is complex and time intensive. Hence low rate of change. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues, semantics) Robustness of the simulation is upon the quality of the model. However, robustness of the computation itself, although non-trivial, is tractable. Visualization Would require very large amount of movement of data to enable visualization. Data Quality (syntax) Consistent due to generation from a model Data Types Primarily network data. Data Analytics Summary of various runs and replicates of a simulation Big Data Specific Challenges (Gaps) Computation of the simulation is both compute intensive and data intensive. Moreover, due to unstructured and irregular nature of graph processing the problem is not easily decomposable. Therefore it is also bandwidth intensive. Hence, a supercomputer is applicable than cloud type clusters. Big Data Specific Challenges in Mobility None Security and Privacy Requirements Several issues at the synthetic population-modeling phase (see social contagion model). Highlight issues for generalizing this use case (e.g. for ref. architecture) In general contagion diffusion of various kinds: information, diseases, social unrest can be modeled and computed. All of them are agent-based model that utilize the underlying interaction network to study the evolution of the desired phenomena. More Information (URLs) Healthcare and Life Sciences> Use Case 24: Social Contagion Modeling Use Case Title Social Contagion Modeling
Networking Continuous RAW data transfer of ≈20Gbps at designed luminosity between Japan and US Additional transfer rates are currently being investigated Software Open Science Grid, Geant4, DIRAC, FTS, Belle II framework Big Data Characteristics Data Source (distributed/centralized) Distributed data centers Primary data centers are in Japan (KEK) and US (PNNL) Volume (size) Total integrated RAW data ≈120PB and physics data ≈15PB and ≈100PB MC samples Velocity (e.g. real time) Data will be re-calibrated and analyzed incrementally Data rates will increase based on the accelerator luminosity Variety (multiple datasets, mashup) Data will be re-calibrated and distributed incrementally. Variability (rate of change) Collisions will progressively increase until the designed luminosity is reached (3000 BB pairs per sec). Expected event size is ≈300kB per events. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Validation will be performed using known reference physics processes Visualization N/A Data Quality Output data will be re-calibrated and validated incrementally Data Types Tuple based output Data Analytics Data clustering and classification is an integral part of the computing model. Individual scientists define event level analytics. Big Data Specific Challenges (Gaps) Data movement and bookkeeping (file and event level meta-data). Big Data Specific Challenges in Mobility Network infrastructure required for continuous data transfer between Japan (KEK) and US (PNNL). Security and Privacy Requirements No special challenges. Data is accessed using grid authentication. Highlight issues for generalizing this use case (e.g. for ref. architecture) More Information (URLs) Earth, Environmental and Polar Science> Use Case 41: EISCAT 3D Incoherent Scatter Radar System Use Case Title EISCAT 3D incoherent scatter radar system Vertical (area) Environmental Science Author/Company/Email Yin Chen /Cardiff University/ Ingemar Häggström, Ingrid Mann, Craig Heinselman/ EISCAT Science Association/{
instruments. check data. Information: Offline Analysis L1B Transported Disks copied to (LUSTRE) File System Produce processed data as radar images Matlab Analysis code running in parallel and independently on each data sample ≈2500 cores running standard cluster tools N/A except results checked before release on CReSIS web site Information: L2/L3 Geolocation and Layer Finding Radar Images from L1B Input to Science as database with GIS frontend GIS and Metadata Tools Environment to support automatic and/or manual layer determination GIS (Geographical Information System). Cluster for Image Processing. As above Knowledge, Wisdom, Discovery: Science GIS interface to L2/L3 data Polar Science Research integrating multiple data sources e.g. for Climate change. Glacier bed data used in simulations of glacier flow Exploration on a cloud style GIS supporting access to data. Simulation is 3D partial differential equation solver on large cluster. Varies according to science use. Typically results open after research complete. See Figure 11: Radar Data Analysis for CReSIS Remote Sensing of Ice Sheets– Typical CReSIS radar data after analysis. See See Figure 13: Radar Data Analysis for CReSIS Remote Sensing of Ice Sheets – Typical echogram with detected boundaries. The upper (green) boundary is between air and ice layers, while the lower (red) boundary is between ice and terrain. Earth, Environmental and Polar Science> Use Case 44: UAVSAR Data Processing Use Case Title UAVSAR Data Processing, Data Product Delivery, and Data Services Vertical (area) Scientific Research: Earth Science Author/Company/Email Andrea Donnellan, NASA JPL, ; Jay Parker, NASA JPL, Actors/Stakeholders and their roles and responsibilities NASA UAVSAR team, NASA QuakeSim team, ASF (NASA SAR DAAC), USGS, CA Geological Survey Goals Use of Synthetic Aperture Radar (SAR) to identify landscape changes caused by seismic activity, landslides, deforestation, vegetation changes, flooding, etc.; increase its usability and accessibility by scientists. Use Case Description A scientist who wants to study the after effects of an earthquake examines multiple standard SAR products made available by NASA. The scientist may find it useful to interact with services provided by intermediate projects that add value to the official data product archive. Current Solutions Compute(System) Raw data processing at NASA AMES Pleiades, Endeavour. Commercial clouds for storage and service front ends have been explored. Storage File based.
Summary of Key Properties Information related to five key properties was extracted from each use case. The five key properties were three Big Data characteristics (volume, velocity, and variety), software related information, and associated analytics. The extracted information is presented in Table B-1. Table B-1: Use Case Specific Information by Key Properties Use Case Volume Velocity Variety Software Analytics 1 Census 2000 and 2010 380 TB Static for 75 years Scanned documents Robust archival storage None for 75 years 2 NARA: Search, Retrieve, Preservation Hundreds of terabytes, and growing Data loaded in batches, so bursty Unstructured and structured data: textual documents, emails, photos, scanned documents, multimedia, social networks, web sites, databases, etc. Custom software, commercial search products, commercial databases Crawl/index, search, ranking, predictive search; data categorization (sensitive, confidential, etc.); personally identifiable information (PII) detection and flagging 3 Statistical Survey Response Improvement Approximately 1 PB Variable, field data streamed continuously, Census was ≈150 million records transmitted Strings and numerical data Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig Recommendation systems, continued monitoring 4 Non-Traditional Data in Statistical Survey Response Improvement — — Survey data, other government administrative data, web-scraped data, wireless data, e-transaction data, (potentially) social media data and positioning data from various sources Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig New analytics to create reliable information from non-traditional disparate sources 5 Cloud Eco-System for Finance — Real time — Hadoop RDBMS XBRL Fraud detection
1 TB raw image data + 1 TB analytical results per 3D image, 1 PB data per moderated hospital per year classification 18 Computational Bioimaging Medical diagnostic imaging around 70 PB annually, 32 TB on emerging machines for a single scan Volume of data acquisition requires HPC back end Multi-modal imaging with disparate channels of data Scalable key-value and object store databases; ImageJ, OMERO, VolRover, advanced segmentation and feature detection methods Machine learning (support vector machine [SVM] and random forest [RF]) for classification and recommendation services 19 Genomic Measurements >100 TB in 1 to 2 years at NIST, many PBs in healthcare community ≈300 GB of compressed data/day generated by DNA sequencers File formats not though some standards exist; generally structured data Open-source sequencing bioinformatics software from academic groups Processing of raw data to produce variant calls, clinical interpretation of variants 20 Comparative Analysis for Metagenomes and Genomes 50 TB New sequencers stream in data at growing rate Biological data that are inherently heterogeneous, complex, structural, and hierarchical; besides core genomic data, new types of omics data such as transcriptomics, methylomics, and proteomics Standard bioinformatics tools (BLAST, HMMER, multiple alignment and phylogenetic tools, gene callers, sequence feature predictors), Perl/Python wrapper scripts Descriptive statistics, statistical significance in hypothesis testing, data clustering and classification 21 Individualized Diabetes Management 5 million patients Not real time but updated periodically 100 controlled vocabulary values and 1,000 continuous values per patient, mostly time-stamped values HDFS supplementing Mayo internal data warehouse (EDT) Integration of data into semantic graphs, using graph traverse to replace SQL join; development of semantic graph-mining algorithms to identify graph patterns, index graph, and search graph; indexed Hbase; custom
code to develop new patient properties from stored data 22 Statistical Relational Artificial Intelligence for Health Care Hundreds of GBs for a single cohort of a few hundred people; possibly on the order of 1 PB when dealing with millions of patients Constant updates to EHRs; in other controlled studies, data often in batches at regular intervals Critical feature – data typically in multiple tables, need to be merged to perform analysis Mainly Java-based, tools to process the data Relational probabilistic models (Statistical Relational AI) learned from multiple data types 23 World Population-Scale Epidemiological Study 100 TB Low number of data feeding into the simulation, massive amounts of real-time data generated by simulation Can be rich with various population activities, geographical, cultural variations Charm++, MPI Simulations on a synthetic population 24 Social Contagion Modeling for Planning Tens of terabytes per year During social unrest events, human interactions and mobility leads to rapid changes in data; e.g., who follows whom in Twitter Big issues – data fusion, combining data from different sources, dealing with missing or incomplete data Specialized simulators, open source software, proprietary modeling environments; databases Models of behavior of humans and hard infrastructures, models of their interactions, visualization of results 25 Biodiversity and LifeWatch N/A Real-time processing and analysis in case of natural or industrial disaster Rich variety and number of involved databases and observation data RDBMS Requires advanced and rich visualization 26 Large-Scale Deep Learning Current datasets typically 1 TB to 10 TB, possibly 100 million images to train a self-driving car Much faster than real-time processing; for autonomous driving, need to Neural net very heterogeneous as it learns many different features In-house GPU kernels and MPI-based communication developed by Stanford, C++/Python source Small degree of batch statistical pre-processing, all other data analysis performed by the learning algorithm itself
process thousands of high-resolution (six megapixels or more) images per second 27 Organizing Unstructured Collections of Consumer Photos 500+ billion photos on Facebook, 5+ billion photos on Flickr Over 500 million images uploaded to Facebook each day Images and metadata including EXIF (Exchangeable Image File) tags (focal distance, camera type, etc.) Hadoop Map/Reduce, simple hand-written multi-threaded tools (Secure Shell [SSH] and sockets for communication) Robust non-linear least squares optimization problem, SVM 28 Truthy Twitter Data 30 TB/year compressed data Near real-time data storage, querying and analysis Schema provided by social media data source; currently using Twitter only; plans to expand, incorporating Google+ and Facebook Hadoop IndexedHBase and HDFS; Hadoop, Hive, Redis for data management; Python: SciPy NumPy and MPI for data analysis Anomaly detection, stream clustering, signal classification, online learning; information diffusion, clustering, dynamic network visualization 29 Crowd Sourcing in Humanities GBs (text, surveys, experiment values) to hundreds of terabytes (multimedia) Data continuously updated and analyzed incrementally So far mostly homogeneous small data sets; expected large distributed heterogeneous datasets XML technology, traditional relational databases Pattern recognition (e.g., speech recognition, automatic audio-visual analysis, cultural patterns), identification of structures (lexical units, linguistic rules, etc.) 30 CINET for Network Science Can be hundreds of GBs for a single network, 1,000 to 5,000 networks and methods Dynamic networks, network collection growing Many types of networks Graph libraries (Galib, NetworkX); distributed workflow management (Simfrastructure, databases, semantic web tools) Network visualization 31 NIST Information Access Division >900 million web pages occupying 30 TB of storage, 100 million tweets, 100 million Legacy evaluations mostly focused on retrospective analytics, newer evaluations Wide variety of data types including textual search/extraction, machine translation, speech recognition, image and voice PERL, Python, C/C++, Matlab, R development tools; create ground-up test and measurement Information extraction, filtering, search, and summarization; image and voice biometrics; speech recognition and
biometric images, hundreds of thousands of partially ground-truthed video clips, terabytes of smaller fully test collections focused on simulations of analytic challenges from multiple data streams biometrics, object and person recognition and tracking, document analysis, dialogue, multimedia search/extraction applications understanding; machine translation; video person/object detection and tracking; event detection; imagery/document matching; novelty detection; structural semantic temporal analytics 32 DataNet (iRODS) Petabytes, hundreds of millions of files Real time and batch Rich iRODS Supports general analysis workflows 33 The Discinnet Process Small as metadata to Big Data Real time Can tackle arbitrary Big Data Symfony-PHP, Linux, MySQL -- 34 Semantic A few terabytes Evolving in time Rich Database Data graph processing 35 Light Source Beamlines 50 to 400 GB per day, total ≈400 TB Continuous stream of data, but analysis need not be real time Images Octopus for Tomographic Reconstruction, Avizo () and FIJI (a distribution of ImageJ) Volume reconstruction, feature identification, etc. 36 Catalina Transient Survey ≈100 TB total increasing by 0.1 TB a night accessing PBs of base astronomy data, 30 TB a night from successor LSST in 2020s Nightly update runs processes in real time Images, spectra, time series, catalogs Custom data processing pipeline and data analysis software Detection of rare events and relation to existing diverse data 37 DOE Extreme Data from Cosmological Sky Survey Several petabytes from Dark Energy Survey and Zwicky Transient Factory, simulations > 10 PB Analysis done in batch mode with data from observations and simulations updated daily Image and simulation data MPI, FFTW, viz packages, numpy, Boost, OpenMP, ScaLAPCK, PSQL and MySQL databases, Eigen, cfitsio, New analytics needed to analyze simulation results Time
astrometry.net, and Minuit2 38 Large Survey Data for Cosmology Petabytes of data from Dark Energy Survey 400 images of 1 GB in size per night Images Linux cluster, Oracle RDBMS server, Postgres PSQL, large memory machines, standard Linux interactive hosts, GPFS; for simulations, HPC resources; standard astrophysics reduction software as well as Perl/Python wrapper scripts Machine learning to find optical transients, Cholesky decomposition for thousands of simulations with matrices of order 1 million on a side and parallel image storage 39 Particle Physics at LHC 15 PB of data (experiment and Monte Carlo combined) per year Data updated continuously with sophisticated selection and test analysis but all analyzed "properly" offline Different format for each stage in analysis but data uniform within each stage Grid-based environment with over 350,000 cores running simultaneously Sophisticated specialized data analysis code followed by basic exploratory statistics (histogram) with complex detector efficiency corrections 40 Belle II High Energy Physics Experiment Eventually 120 PB of Monte Carlo and observational data Data updated continuously with sophisticated selection and test analysis but all analyzed "properly" offline Different format for each stage in analysis but data uniform within each stage DIRAC Grid software Sophisticated specialized data analysis code followed by basic exploratory statistics (histogram) with complex detector efficiency corrections 41 EISCAT 3D incoherent scatter radar system Terabytes/year (current), 40 PB/year starting ≈2022 Data updated continuously with real-time test analysis and batch full analysis Big data uniform Custom analysis based on flat file data storage Pattern recognition, demanding correlation routines, high-level parameter extraction 42 ENVRI Environmental Low volume (apart from EISCAT 3D given above), one system Mainly real-time data streams Six separate projects with common architecture for infrastructure, data very R and Python (Matplotlib) for visualization, custom Data assimilation, (statistical) analysis, data mining, data extraction,
Research Infrastructure EPOS ≈15 TB/year diverse across projects software for processing scientific modeling and simulation, scientific workflow 43 CReSIS Remote Sensing Around 1 PB (current) increasing by 50 to 100 TB per mission, future expedition ≈1 PB each Data taken in ≈two-month missions including test analysis and then later batch processing Raw data, images with final layer data used for science Matlab for custom raw data processing, custom image processing software, GIS as user interface Custom signal processing to produce radar images that are analyzed by image processing to find layers 44 UAVSAR Data Processing 110 TB raw data and 40 TB processed, plus smaller samples Data come from aircraft and so incrementally added, data occasionally get reprocessed: new processing methods or parameters Image and annotation files ROI_PAC, GeoServer, GDAL, tools; moving to clouds Process raw data to get images that are run through image processing tools and accessed from GIS 45 NASA LARC/GSFC iRODS MERRA collection (below) represents most of total data, other smaller collections Periodic updates every six months Many applications to combine MERRA reanalysis data with other reanalyses and observational data such as CERES SGE Univa Grid Engine Version 8.1, iRODS Version 3.2 and/or 3.3, IBM GPFS Version 3.4, Cloudera Version 4.5.2-1 Federation software 46 MERRA Analytic Services 480 TB from MERRA Increases at ≈1 TB/month Applications to combine MERRA reanalysis data with other re-analyses and observational data Cloudera, iRODS, Amazon AWS CAaaS 47 Atmospheric Turbulence 200 TB (current), 500 TB within 5 years Data analyzed incrementally Re-analysis datasets are inconsistent in format, resolution, semantics, and metadata; interpretation/analysis of each of these input streams into a common product Map/Reduce or the like, SciDB or other scientific database Data mining customized for specific event types 48 Up to 30 PB/year 42 GB/second from Variety across simulation National Center for Need analytics next to data
Climate Studies from 15 end-to-end simulations at NERSC, more at other HPC centers simulations groups and between observation and simulation Atmospheric Research (NCAR) PIO library and utilities NCL and NCO, parallel NetCDF storage 49 DOE-BER Subsurface Biogeochemistry — — From omics of the microbes in the soil to watershed hydro-biogeochemistry, from observation to simulation PFLOWTran, postgres, HDF5, Akuna, NEWT, etc. Data mining, data quality assessment, across datasets, reduced model development, statistics, quality assessment, data fusion 50 DOE-BER AmeriFlux and FLUXNET Networks — Streaming data from ≈150 towers in AmeriFlux and over 500 towers distributed globally collecting flux measurements Flux data merged with biological, disturbance, and other ancillary data EddyPro, custom analysis software, R, Python, neural networks, Matlab Data mining, data quality assessment, across datasets, data assimilation, data interpolation, statistics, quality assessment, data fusion 51 Consumption forecasting in Smart Grids 4 TB/year for a city with 1.4 million sensors, such as Los Angeles Streaming data from millions of sensors Tuple-based: timeseries, database rows; graph-based: network topology, customer connectivity; some semantic data for normalization R/Matlab, Weka, Hadoop; GIS-based visualization Forecasting models, machine learning models, time series analysis, clustering, motif detection, complex event processing, visual network analysis correlation
Use Case Requirements Summary Requirements were extracted from each use case within seven characteristic categories introduced in Section 3.1. The number of requirements within each category varied for each use case. Table C-1 contains the use case specific requirements. Table C-1: Use Case Specific Requirements Use Case Data Sources Data Transformation Capabilities Data Consumer Security and Privacy Life Cycle Management Other 1 Census 2010 and 2000 1. Large document format from centralized storage -- 1. Large centralized storage (storage) -- 1. Title 13 data 1. Long-term preservation of data as-is for 75 years 2. Long-term preservation at the bit level 3. Curation process including format transformation 4. Access and analytics processing after 75 years 5. No data loss -- 2 NARA: Search, Retrieve, Preservation 1. Distributed data sources 2. Large data storage 3. Bursty data ranging from GBs to hundreds of terabytes 4. Wide variety of data formats including unstructured and structured data 1. Crawl and index from distributed data sources 2. Various analytics processing including ranking, data categorization, detection of PII data 3. Data 4. Long-term 1. Large data storage 2. Various storage systems such as NetApps, Hitachi, magnetic tapes 1. High relevancy and high recall from search 2. High accuracy from categorization of records 3. Various storage systems such as NetApps, Hitachi, 1. Security policy 1. Pre-process for virus scan 2. File format identification 3. Indexing 4. Records categorization 1. Mobile search with similar interfaces/ results from desktop
5. Distributed data sources in different clouds preservation management of large varied datasets 5. Huge numbers of data with high relevancy and recall magnetic tapes 3 Statistical Survey Response 1. Data size of approximately one petabyte 1. Analytics for recommendation systems, continued monitoring, and general survey improvement 1. Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig 1. Data visualization for data review, operational activity, and general analysis; continual evolution 1. Improved systems that reduce costs and improve quality while providing confidentiality safeguards that are reliable and publicly auditable 2. Confidential and secure data; processes that are auditable for security and confidentiality as required by various legal statutes 1. High veracity on data and very robust systems (challenges: semantic integrity of conceptual metadata concerning what exactly is measured and the resulting limits of inference) 1. Mobile access 4 Data in Statistical -- 1. Analytics to create reliable estimates using data from traditional survey 1. Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, 1. Data visualization for data review, operational 1. Confidential and secure data; processes that are auditable for 1. High veracity on data and very robust systems (challenges: semantic integrity of conceptual --
Survey Response sources, government administrative data sources, and sources from the digital economy Storm, BigMemory, Cassandra, Pig activity, and general analysis; continual evolution security and confidentiality as required by various legal statutes metadata concerning what exactly is measured and the resulting limits of inference) 5 Cloud for Finance 1. Real-time ingestion of data 1. Real-time analytics -- -- 1. Strong security and privacy constraints -- 1. Mobile access 6 Mendeley 1. File-based documents with constant new uploads 2. Variety of file types such as PDFs, social network log files, client activities images, spreadsheet, presentation files 1. Standard machine learning and analytics libraries 2. Efficient scalable and parallelized way to match between documents 3. Third-party annotation tools or publisher watermarks and cover pages 1. Amazon Elastic Compute Cloud (EC2) with HDFS (infrastructure) 2. S3 (storage) 3. Hadoop (platform) 4. Scribe, Hive, Mahout, Python (language) 5. Moderate storage (15 TB with 1 TB/ month) 6. Batch and processing 1. reporting tools 2. Visualization tools such as networking graph, scatterplots, etc. 1. Access controls for who reads what content 1. Metadata management from PDF extraction 2. Identification of document duplication 3. Persistent identifier 4. Metadata between data repositories such as CrossRef, PubMed, and Arxiv 1. Windows Android and iOS mobile devices for content deliverables from Windows desktops 7 Netflix Movie Service 1. User profiles and ranking information 1. Streaming video contents to multiple clients 2. Analytic processing for matching client interest in movie selection 3. Various analytic 1. Hadoop (platform) 2. Pig (language) 3. Cassandra and Hive 4. Huge numbers of subscribers, ratings, and searches per day 1. Streaming and rendering media 1. Preservation of users, privacy and digital rights for media 1. Continued ranking and updating based on user profile and analytic results 1. Smart interface accessing movie content on mobile platforms
updates on tracking items 11 Materials Data for 1. Distributed data repositories for more than 500,000 commercial materials 2. Many varieties of datasets 3. Text, graphics, and images 1. Hundreds of independent variables need to be collected to create robust datasets -- 1. Visualization for materials discovery from many independent variables 2. Visualization tools for materials 1. Protection of proprietary sensitive data 2. Tools to mask proprietary information 1. Handle data quality (currently poor or no process) -- 12 Materials Genomics 1. Data streams from peta/exascale centralized simulation systems 2. Distributed web dataflows from central gateway to users 1. High-throughput computing data analysis for web-like responsiveness 2. Mashup of simulation outputs across codes 3. Search and crowd-driven with computation backend, flexibility for new targets 4. Map/Reduce and search to join simulation and experimental data 1. Massive (150,000 cores) legacy infrastructure (infrastructure) 2. GPFS (storage) 3. MonogDB systems (platform) 4. 10 GB networking 5. Various analytic tools such as PyMatGen, FireWorks, VASP, ABINIT, NWChem, BerkeleyGW, varied community codes 6. Large storage (storage) 1. search for growing materials data 1. Sandbox as independent working areas between different data stakeholders 2. Policy-driven federation of datasets 1. Validation and uncertainty quantification (UQ) of simulation with experimental data 2. UQ in results from multiple datasets 1. Mobile applications (apps) to access materials genomics information
7. Scalable and object store (platform) 8. Data streams from peta/exascale centralized simulation systems 13 Large-Scale Geospatial Analysis and Visualization 1. Unique approaches to indexing and distributed analysis required for geospatial data 1. Analytics: closest point of approach, deviation from route, point density over time, PCA and ICA 2. Unique approaches to indexing and distributed analysis required for geospatial data 1. Geospatially enabled RDBMS, geospatial server/analysis software, e.g., ESRI ArcServer, Geoserver 1. Visualization with GIS at high and low network bandwidths and on dedicated facilities and handhelds 1. Complete security of sensitive data in transit and at rest (particularly on handhelds) -- -- 14 Object and Tracking 1. Real-time data FMV (30 to 60 frames/ second at full-color 1080P resolution) and WALF (1 to 10 frames/ second at 10,000 x 10,000 full-color resolution) 1. Rich analytics with object identification, pattern recognition, crowd behavior, economic activity, and data fusion 1. Wide range of custom software and tools including traditional RDBMSs and display tools 2. Several network requirements 3. GPU usage important 1. Visualization of extracted outputs as overlays on a geospatial display; links back to the originating image/video segment as overlay objects 2. Output the 1. Significant security and privacy issues; sources and methods never compromised 1. Veracity of extracted objects --
form of Open Geospatial Consortium web features or standard geospatial files (shape files, KML) 15 Intelligence Data Processing and Analysis 1. Much real-time data with processing at near-real time (at worst) 2. Data in disparate silos, must be accessible through a semantically integrated data space 3. Diverse data: text files, raw media, imagery, video, audio, electronic data, human-generated data 1. Analytics: Near Real (NRT) alerts based on patterns and baseline changes 1. Tolerance of unreliable networks to warfighter and remote sensors 2. Up to hundreds of petabytes of data supported by modest to large clusters and clouds 3. Hadoop, Accumulo (Big Table), Solr, NLP (several variants), Puppet (for deployment and security), Storm, custom applications, visualization tools 1. Geospatial overlays (GIS) and network diagrams (primary visualizations) 1. Protection of data against unauthorized access or disclosure and tampering 1. Data provenance (e.g. tracking of all transfers and transformations) over the life of the data -- 16 EMR Data 1. Heterogeneous, high-volume, diverse data 1. A comprehensive and consistent view of data across 1. Hadoop, Hive, R. Unix-based 2. Cray supercomputer 1. Results of analytics provided for use by data 1. Data consumer direct access to data as well as 1. Standardize, aggregate, and normalize data from disparate sources 1. Security across mobile devices
sources 2. Volume: > 12 million entities (patients), > 4 billion records or data points (discrete clinical observations), aggregate of > 20 TB raw data 3. Velocity: 500,000 to 1.5 million new transactions per day 4. Variety: formats include numeric, structured numeric, structured text, discrete nominal, discrete ordinal, discrete structured, binary large blobs (images and video) 5. Data evolve over time in a highly variable fashion sources and over time 2. Analytic techniques: information retrieval, NLP, machine learning decision models, maximum likelihood estimators, Bayesian networks 3. Teradata, PostgreSQL, MongoDB 4. Various, with significant I/O intensive processing consumers/ stakeholders, i.e., those who did not actually perform the analysis; specific visualization techniques to the results of analytics performed by informatics research scientists and health service researchers 2. Protection of all health data in compliance with governmental regulations 3. Protection of data in accordance with data providers, policies. 4. Security and privacy policies unique to a data subset 5. Robust security to prevent data breaches 2. Reduce errors and bias 3. Common nomenclature and classification of content across disparate challenging in the health IT space, as the taxonomies continue to evolve— SNOMED, International Classification of Diseases (ICD) 9 and future ICD 10, etc. 17 Pathology Imaging 1. High-resolution spatial digitized pathology images 1. image analysis to 1. Legacy system and cloud (computing 1. Visualization for validation 1. Security and privacy protection for 1. Human annotations for validation 1. 3D visualization and
2. Various image quality analyses algorithms 3. Various image data formats, especially BigTIFF with structured data for analytical results 4. Image analysis, spatial queries and analytics, feature clustering, and classification extract spatial information 2. Spatial queries and analytics, feature clustering and classification 3. Analytic processing on huge multi-dimensional large dataset; correlation with other data types such as clinical data, omic data cluster) 2. Huge legacy and new storage such as storage area network (SAN) or HDFS (storage) 3. network link (networking) 4. MPI image analysis, Map/Reduce, Hive with spatial extension (software packages) and training protected health information rendering on mobile platforms 18 Bioimaging 1. Distributed multi-modal experimental sources of bioimages (instruments) 2. 50 TB of data in formats that include images 1. High-throughput computing with responsive analysis 2. Segmentation of regions of interest; crowd-based selection and extraction of features; object classification, and organization; and search 3. Advanced biosciences discovery through Big Data techniques / 1. ImageJ, OMERO, VolRover, advanced segmentation and feature detection methods from applied math researchers; scalable key-value and object store databases needed 2. NERSC’s Hopper infrastructure 3. database and image collections 1. 3D structural modeling 1. Significant but optional security and privacy including secure servers and anonymization 1. Workflow components including data acquisition, storage, enhancement, minimizing noise --
recorded) 5. Two main categories of structured data about a patient: data with controlled vocabulary (CV) property values and data with continuous property values (recorded/ captured more frequently) 6. Data consist of text and continuous numerical values discovery rates, determine true sub-graph significance, validate results, eliminate false positive/false negative results 5. Semantic graph mining algorithms to identify graph patterns, index and search graph 6. Semantic graph traversal 22 Statistical Relational Artificial Intelligence for Health Care 1. Centralized data, with some data retrieved from Internet sources 2. Range from hundreds of GBs for a sample size to 1 PB for very large studies 3. Both constant updates/additions (to data subsets) and scheduled batch inputs 4. Large, 1. Relational probabilistic models/ probability theory; software that learns models from multiple data types and can possibly integrate the information and reason about complex queries 2. Robust and accurate learning methods to account for data 1. Java, some in house tools, [relational] database and NoSQL stores 2. Cloud and parallel computing 3. computer, 48 GB RAM (to perform analysis for a moderate sample size) 4. Dlusters for 1. Visualization of very large data subsets 1. Secure handling and processing of data 1. Merging multiple tables before analysis 2. Methods to validate data to minimize errors --
longitudinal data 5. Rich relational data comprising multiple tables, different data types such as imaging, EHR, demographic, genetic, and natural language data requiring rich representation 6. Unpredictable arrival rates, often real time imbalance (where large numbers of data are available for a small number of subjects) 3. Learning algorithms to identify skews in data, so as to not to (incorrectly) model noise 4. Generalized and refined learned models for application to diverse sets of data 5. Challenge: acceptance of data in different modalities (and from disparate sources) large datasets 5. 200 GB–1 TB hard drive for test data 23 World Population Scale Study 1. File-based synthetic population, either centralized or distributed sites 2. Large volume of real-time output data 3. Variety of output datasets depending on the model’s complexity 1. and computation, like supercomputer performance 2. Unstructured and irregular nature of graph processing 3. Summary of various runs of simulation 1. Movement of very large volume of data for visualization (networking) 2. Distributed MPI-based simulation system (platform) 3. Charm++ on multi-nodes (software) 4. Network file 1. Visualization 1. Protection of PII on individuals used in modeling 2. Data protection and secure platform for computation 1. Data quality, ability to capture the traceability of quality from computation --
system (storage) 5. Infiniband network (networking) 24 Social Contagion Modeling for Planning 1. Traditional and new architecture for dynamic distributed processing on commodity clusters 2. Fine-resolution models and datasets to support Twitter network traffic 3. Huge data storage supporting annual data growth 1. Large-scale modeling for various events (disease, emotions, behaviors, etc.) 2. Scalable fusion between combined datasets 3. Multi-level analysis while generating sufficient results quickly 1. Computing infrastructure that can capture human-to-human interactions on various social events via the Internet (infrastructure) 2. File servers and databases (platform) 3. Ethernet and Infiniband networking (networking) 4. Specialized simulators, open source software, and proprietary modeling (application) 5. Huge user accounts across country boundaries (networking) 1. Multi-level detailed network 2. Visualization with interactions 1. Protection of PII of individuals used in modeling 2. Data protection and secure platform for computation 1. Data fusion from variety of data sources (i.e., Stata data files) 2. Data consistency and no corruption 3. Preprocessing of raw data 1. Efficient method of moving data 25 Biodiversity and LifeWatch 1. Special dedicated or overlay sensor network 1. Web-based services, services, relational 1. Expandable storage resource for global users 1. Access by mobile users 2. Advanced/ 1. Federated identity management for mobile 1. Data storage and archiving, data exchange and integration --
2. Storage: distributed, historical, and trends data archiving 3. Distributed data sources, including observation and monitoring facilities, sensor network, and satellites 4. Wide variety of data: satellite images/ information, climate and weather data, photos, video, sound recordings, etc. 5. Multi-type data combination and linkage, potentially unlimited data variety 6. Data streaming databases, NoSQL 2. Personalized virtual labs 3. Grid- and resources 4. Data analyzed incrementally and/or in real time at varying rates owing to variations in source processes 5. A variety of data and analytical and modeling tools to support analytics for diverse scientific communities 6. Parallel data streams and streaming analytics 7. Access and integration of multiple distributed databases 2. Cloud community resource required visualization 3. 4D visualization computational models researchers and mobile sensors 2. Access control and accounting 2. Data life cycle management: data provenance, referral integrity and identification traceability back to initial observational data 3. Processed (secondary) data storage (in addition to original source data) for future uses 4. Provenance (and persistent identification [PID]) control of data, algorithms, and workflows 5. Curated (authorized) reference data (e.g. species name lists), algorithms, software code, workflows 26 Large-Scale Deep Learning -- -- 1. GPU 2. MPI and HPC Infiniband cluster 3. Libraries for -- -- -- --
single-machine or single-GPU computation – available (e.g., BLAS, CuBLAS, MAGMA, etc.); distributed computation of dense BLAS-like or LAPACK-like operations on GPUs – poorly developed; existing solutions (e.g., ScaLapack for CPUs) – not well-integrated with higher-level languages and require low-level programming, lengthening experiment and development time 27 Organizing Large-Scale Collections of Consumer Photos 1. Over 500 million images uploaded to social media sites each day 1. Classifier (e.g. an SVM), a process that is often hard to parallelize 2. Features seen in many large-scale image processing problems 1. Hadoop or enhanced Map/Reduce 1. Visualize large-scale 3D navigate large-scale collections of images that have been aligned to maps 1. Preserve privacy for users and digital rights for media -- --
28 Truthy Twitter Data 1. Distributed data sources 2. Large volume of real-time streaming data 3. Raw data in compressed formats 4. Fully structured data in JSON, user metadata, data 5. Multiple data schemas 1. Various data analysis for anomaly detection, stream clustering, signal classification on multi-dimensional time series, online learning 1. Hadoop and HDFS (platform) 2. IndexedHBase, Hive, SciPy, NumPy (software) 3. In-memory database, MPI (platform) 4. High-speed Infiniband network (networking) 1. Data retrieval and dynamic visualization 2. Data-driven interactive web interfaces 3. API for data query 1. Security and privacy policy 1. Standardized data structures/ formats with extremely high data quality 1. Low-level data storage for efficient mobile access to data 29 Crowd Sourcing in Humanities -- 1. Digitize existing audio-video, photo, and documents archives 2. Analytics: pattern recognition of all kinds (e.g., speech recognition, automatic A&V analysis, cultural patterns), identification of structures (lexical units, linguistic rules, etc.) -- -- 1. Privacy issues in preserving anonymity of responses in spite of computer recording of access ID and reverse engineering of unusual user responses -- -- 30 CINET for Network Science 1. A set of network topologies files to study graph 1. Environments to run various network and graph analysis tools 1. Large file system (storage) 2. Various network 1. Client-side visualization -- -- --
theoretic properties and behaviors of various algorithms 2. Asynchronous and real-time synchronous distributed computing 2. Dynamic growth of the networks 3. Asynchronous and real-time synchronous distributed computing 4. Different parallel algorithms for different partitioning schemes for efficient operation connectivity (networking) 3. Existing computing cluster 4. EC2 computing cluster 5. Various graph libraries, management tools, databases, semantic web tools 31 NIST Information Access Division 1. Large amounts of web pages, tweets, images, video 2. Scaling to larger data, intrinsic and annotation uncertainty measurement, performance measurement for incompletely annotated data, measuring analytic performance for heterogeneous data and analytic flows involving users 1. Test analytic algorithms working with written language, speech, human imagery, etc. against real or realistic data; challenge: engineering artificial data that sufficiently captures the variability of real data involving humans 1. PERL, Python, C/C++, Matlab, R development tools; creation of ground-up test and measurement applications 1. Analytic flows involving users 1. Security requirements for protecting sensitive data while enabling meaningful developmental performance evaluation; shared evaluation testbeds that protect the intellectual property of analytic algorithm developers -- --
32 DataNet (iRODS) 1. Process key format types NetCDF, HDF5, Dicom 2. Real-time and batch data 1. Provision of general analytics workflows needed 1. iRODS data management software 2. interoperability across storage and network protocol types 1. General visualization workflows 1. Federate across existing authentication environments through Generic Security Service API and pluggable authentication modules (GSI, Kerberos, InCommon, Shibboleth) 2. Access controls on files independent of the storage location -- -- 33 The Discinnet Process 1. Integration of metadata approaches across disciplines -- 1. Software: Symfony-PHP, Linux, MySQL -- 1. Significant but optional security and privacy including secure servers and anonymization 1. Integration of metadata approaches across disciplines -- 34 Semantic 1. All data types, image to text, structures to protein sequence 1. Data graph processing 2. RDBMS 1. Cloud community resource required 1. Efficient visualization needed -- -- -- 35 Light source beamlines 1. Multiple streams of data to be 1. Standard bioinformatics tools (BLAST, 1. High-volume data transfer to remote batch -- 1. Multiple security and privacy -- --
stored and analyzed later 2. Sample data to be analyzed in real time HMMER, multiple alignment and phylogenetic tools, gene callers, sequence feature predictors, etc.), Perl/Python wrapper scripts, Linux Cluster scheduling processing resource requirements to be satisfied 36 Catalina Real-Time Transient Survey 1. ≈0.1 TB per day at present, will increase by factor of 100 1. A wide variety of the existing astronomical data analysis tools, plus a large number of custom developed tools and software programs, some research projects in and of themselves 2. Automated classification with machine learning tools given the very sparse and heterogeneous data, dynamically evolving in time as more data come in, with follow-up decision making reflecting limited follow-up resources -- 1. Visualization mechanisms for highly dimensional data parameter spaces -- -- --
Asynchronization data collection 3. Calibration of instruments model fits 3. Monte-Carlo computations 2. Distributed cached files (storage) 3. Object databases (software package) 40 Belle II Physics Experiment 1. 120 PB of raw data -- 1. 120 PB raw data 2. International distributed computing model to augment that at accelerator (Japan) 3. Data transfer of ≈20 GB/ second at designed luminosity between Japan and United States 4. Software from Open Science Grid, Geant4, DIRAC, FTS, Belle II framework -- 1. Standard grid authentication -- -- 41 EISCAT 3D Incoherent Scatter Radar System 1. Remote sites generating 40 PB data/year by 2022 2. Hierarchical Data Format (HDF5) 3. Visualization of high-dimensional (≥5) data 1. Queen Bea architecture with mix of distributed on-sensor and central processing for 5 distributed sites 2. Real-time monitoring of 1. Architecture compatible with ENVRI 1. Support needed for visualization of (≥5) data -- 1. Preservation of data and avoidance of lost data due to instrument malfunction 1. Support needed for real-time monitoring of equipment by partial streaming analysis
equipment by partial streaming analysis 3. Hosting needed for rich set of radar image processing services using machine learning, statistical modelling, and graph algorithms 42 ENVRI Research 1. Huge volume of data from distributed data sources 2. Variety of instrumentation datasets and metadata 1. Diversified analytics tools 1. Variety of computing infrastructures and architectures (infrastructure) 2. Scattered repositories (storage) 1. Graph plotting tools 2. Time series interactive tools 3. flash playback 4. Earth map display 5. Visual tools for quality comparisons 1. Open data policy with minor restrictions 1. High data quality 2. Mirror archives 3. Various metadata frameworks 4. Scattered repositories and data curation 1. Various kinds of mobile sensor devices for data acquisition 43 CReSIS Remote Sensing 1. Provision of reliable data transmission from aircraft sensors/ instruments or removable disks from remote sites 2. Data gathering in real time 3. Varieties of 1. Legacy software (Matlab) and language (C/Java) binding for processing 2. Signal processing and advanced image processing to find layers needed 1. ≈0.5 PB/year of raw data 2. Transfer content from removable disk to computing cluster for parallel processing 3. Map/Reduce or MPI plus language 1. GIS user interface 2. Rich user interface for simulations 1. Security and privacy on sensitive political issues 2. Dynamic security and privacy policy mechanisms 1. Data quality assurance 1. Monitoring data collection instruments/ sensors
datasets binding for C/Java 44 UAVSAR Data Processing 1. Angular and spatial data 2. Compatibility with other NASA radar systems and repositories (Alaska Satellite Facility) 1. Geolocated data that require GIS integration of data as custom overlays 2. Significant human intervention in data processing pipeline 3. Hosting of rich set of radar image processing services 4. ROI_PAC, GeoServer, GDAL, tools 1. Support for interoperable Cloud-HPC architecture 2. Hosting of rich set of radar image processing services 3. ROI_PAC, GeoServer, GDAL, tools 4. Compatibility with other NASA radar systems and repositories (Alaska Satellite Facility) 1. Support for field expedition users with phone/tablet interface and low-resolution downloads -- 1. Significant human intervention in data processing pipeline 2. Rich robust provenance defining complex machine/human processing 1. Support for field expedition users with phone/tablet interface and downloads 45 NASA LARC/ GSFC iRODS 1. Federate distributed heterogeneous datasets 1. CAaaS on clouds 1. Support virtual climate data server (vCDS) 2. GPFS parallel file system integrated with Hadoop 3. iRODS 1. Support needed to visualize distributed data -- -- -- 46 MERRA Analytic Services 1. Integrate simulation output and observational data, NetCDF files 2. Real-time and batch mode needed 3. Interoperable 1. CAaaS on clouds 1. NetCDF aware software 2. Map/Reduce 3. Interoperable use of AWS and local clusters 1. High-end distributed visualization -- -- 1. Smart phone and tablet access required 2. iRODS data management
novel HPC simulation 49 DOE-BER Subsurface 1. Heterogeneous diverse data with different domains and scales, translation across diverse datasets that cross domains and scales 2. Synthesis of diverse and disparate field, laboratory, omic, and simulation datasets across different semantic, spatial, and temporal scales 3. Linking of diverse data to novel HPC simulation -- 1. Postgres, HDF5 data technologies, and many custom software systems 1. input and access -- -- 1. input and access 50 DOE-BER AmeriFlux and FLUXNET Networks 1. Heterogeneous diverse data with different domains and scales, translation across diverse datasets that cross domains and scales 2. Link to many 1. Custom software such as EddyPro, and custom analysis software, such as R, Python, neural networks, Matlab 1. Custom software, such as EddyPro, and custom analysis software, such as R, Python, neural networks, Matlab 2. Analytics including data mining, data 1. input and access -- -- 1. input and access
other environment and biology datasets 3. Link to HPC climate and other simulations 4. Link to European data sources and projects 5. Access to data from 500 distributed sources quality assessment, cross-correlation across datasets, data assimilation, data interpolation, statistics, quality assessment, data fusion, etc. 51 Forecasting in Smart Grids 1. Diverse data from smart grid sensors, city planning, weather, utilities 2. Data updated every 15 minutes 1. New machine learning analytics to predict consumption 1. SQL databases, CVS files, HDFS (platform) 2. R/Matlab, Weka, Hadoop (platform) -- 1. Privacy and anonymization by aggregation -- 1. Mobile access for clients