on the monitoring and performance results 8 Web Search 1. Needs to support distributed data sources 2. Needs to support streaming data 3. Needs to support multimedia content 1. dynamic fetching content over the network 2. linking user profiles and social network data 1. petabytes of text and rich media (storage) 1. search time in ~0.1 seconds 2. top 10 ranked results 3. page layout (visual) 1. access control 2. needs to protect sensitive content 1. purge data after certain time interval (few months) 2. data cleaning 1. mobile search and rendering 9 Business Continuity & Disaster Recovery Within A Cloud -- 1. robust Backup algorithm 2. replicate recent changes 1. Hadoop 2. commercial cloud services -- 1. strong security for many applications -- -- 10 Cargo Shipping 1. centralized and real time distributed sites/sensors 1. tracking items based on the unique identification with its sensor information, GPS coordinates 2. real time updates on tracking items 1. Internet connectivity -- 1. security policy -- -- 11 Materials Data for 1. distributed data repositories for more than 500,000 commerical materials 2. may varieties of datasets 3. text, graphics, and images 1. (100s) of independent variables by collecting these variables to create robust datasets -- 1. visualization for materials discovery from many independent variables 2. visualization tools for materials 1. protection of proprietary of sensitive data 2. tools to mask proprietary information 1. how to handle data quality is poor or unknown -- 12 Simulation driven Materials Genomics 1. data streams from peta/exascale centralized simulation systems 2. distributed 1. computing data analysis for responsiveness 1. massive (150K cores) of legacy infrastructure (infrastructure) 2. GPFS (General Parallel File Sysem) 1. browser-based to search growing material data 1. sandbox as independent working areas between different data stakeholders 2. policy-driven 1. validation and UQ of simulation with experimental data 2. UQ in results from multiple datasets 1. mobile apps to access materials geonics information
web dataflows from central gateway to users 2. mashup of simulation outputs across codes 3. search and crowd-driven with computation backend be flexibly for new targets 4. MapReduce and search to join simulation and experimental data (storage) 3. MonogDB systems (platform) 4. 10Gb networking 5. various of analytic tools such as PyMatGen, FireWorks, VASP, ABINIT, NWChem, BerkeleyGW, varied community codes 6. large storage (storage) 7. scalable and object store (platform) 8. data streams from peta/exascale centralized simulation systems federation of datasets 13 Large Scale Geospatial Analysis and 1. geospatial data requires unique approaches to indexing and distributed analysis. 1. analytics include Closest point of approach, deviation from route, point density over time, PCA and ICA 2. geospatial data requires unique approaches to indexing and distributed analysis. 1. software includes Geospatially enabled RDBMS, Geospatial server/analysis software – ESRI ArcServer, Geoserver 1. visualization with GIS at high and low network bandwidths and on dedicated facilities and handhelds 1. sensitive data and must be completely secure in transit and at rest (particularly on handhelds) -- -- 14 Object and tracking 1. real-time data FMV – 30-60 frames per/sec at full color 1080P resolution and WALF – 1-10 frames per/sec at 10Kx10K full color resolution. 1. Rich analytics with object identification, pattern recognition, crowd behavior, economic activity and data fusion 1. wide range custom software and tools including traditional RDBM’s and display tools. 2. several network requirements 3. GPU Usage important 1. visualization of extracted outputs will typically be as overlays on a geospatial display. Overlay objects should be links back to the originating image/video segment. 2. output the form of OGC 1. significant security and privacy, sources and methods cannot be compromised the enemy should not be able to know what we see. 1. veracity of extracted objects -- value
numeric, structured text, discrete nominal, discrete ordinal, discrete structured, binary large blobs (images and video). 5. data evolves over time in a highly variable fashion a subset of the data. 5. robust security to prevent data breaches. 17 Pathology Imaging 1. high resolution spatial digitized pathology images 2. various image quality analysis algorithms 3. various image data formats especially BIGTIFF with structured data for analytical results 4. image analysis, spatial queries and analytics, feature clustering and classification 1. high performance image analysis to extract spatial information 2. spatial queries and analytics, and feature clustering and classification 3. analytic processing on huge large dataset and be able to correlate with other data types such as clinical data, -omic data. 1. legacy system and cloud (computing cluster) 2. huge legacy and new storage such as SAN or HDFS (storage) 3. high throughput network link (networking) 4. MPI image analysis, MapReduce, Hive with spatial extension (sw pkgs) 1. visualization for validation and training 1. security and privacy protection for protected health information 1. human annotations for validation 1. 3D visualization and rendering on mobile platforms 18 Bioimaging 1. distributed multi-modal high resolution experimental sources of bioimages (instruments). 2. 50TB of data data formats include images. 1. computing with responsive analysis 2. segmentation of regions of interest, selection and extraction of features, and object classification, and organization, and search. 3. advance 1. ImageJ, OMERO, VolRover, advanced segmentation and feature detection methods from applied math researchers.... Scalable and object store databases needed. 2. NERSC.s Hopper infrastructure 1. 3D structural modeling 1. significant but optional security & privacy including secure servers and anonymization 1. workflow components include data acquisition, storage, enhancement, minimizing noise -- dimensional value
biosciences discovery through big data techniques / extreme scale computing…. processing and analytics. … Machine learning (SVM and RF) for classification and recommendation services. … advanced algorithms for massive image analysis. computational solutions. 4. massive data analysis toward massive imaging data sets. 3. database and image collections. 4. 10 GB and future 100 GB and advanced networking (SDN). 19 Genomic 1. high throughput compressed data (300GB/day) from various DNA sequencers 2. distributed data source (sequencers) 3. various file formats either structured and unstructured data 1. for processing raw data in variant calls 2. machine learning for complex analysis on systematic errors from sequencing technologies are hard to characterize 1. legacy computing cluster and other PaaS and IaaS (computing cluster) 2. huge data storage in PB range (storage) 3. Unix-based legacy sequencing bioinformatics software (sw pkg) 1. data format for Genome browsers 1. security and privacy protection on health records and clinical research databases -- 1. mobile platforms for physicians accessing genomic data (mobile device) 20 analysis for and genomes 1. multiple centralized data sources 2. proteins and their structural features, core genomic data, new types of “Omics” data such as methylomics, and proteomics describing gene 2. scalable RDMS for heterogeneity biological data 2. real-time rapid and parallel bulk loading 3. Oracle RDBMS, SQLite files, flat text files, Lucy (a version of Lucene) for 1. huge data storage 1. real time interactive parallel bulk loading capability 2. interactive Web UI, backend precomputations, batch job computation submission from the UL 3. download assembled and 1. login security - username and password 2. creation of user account to submit and access dataset to system via web interface 3. single sign on capability (SSO) 1. methods to improve data quality required. 2. data clustering, classification, reduction. 3. Integrate new data / content into the system.s data store annotate data -- ,
expression 3. Front real time Web UI interactive. Back end data loading processing must keep up with exponential growth of sequence data due to the rapid drop in cost of sequencing technology. 4. heterogeneous, complex, structural, and hierarchical biological data. 5. metagenomic samples can vary by several orders of magnitude, such as several hundred thousand genes to a billion genes keyword searches, BLAST databases, USEARCH databases 4. Linux cluster, Oracle RDBMS server, large memory machines, standard Linux interactive hosts 5. sequencing and comparative analysis techniques for highly complex data 6. descriptive statistics annotated datasets for offline analysis. 4. ability to query and browse data via interactive Web UI. 5. visualize structure [of data] at different levels of resolution. Ability to view abstract representations of highly similar data. 21 Diabetes 1. distributed EHR data 2. over 5 million patients with thousands of properties each and many more that are derived from primary values. 3. each record, a range of 100 to 100,000 data property values average of 100 controlled vocabulary values and 1000 continuous values. 4. none but data updated 1. data integration, using ontological annotation and taxonomies 2. parallel retrieval algorithms for both indexed and custom searches, identify data of interest. Patient cohorts, patients meeting certain criteria, patients sharing similar characteristics 3. distributed graph mining algorithms, pattern analysis and graph indexing, pattern searching on 1. data warehouse, open source indexed Hbase 2. supercomputers, cloud and parallel computing 3. I/O intensive processing 4. HDFS storage 5. custom code to develop new properties from stored data. 1. efficient based visualization is needed 1. protection of health data in accordance with legal requirements - eg, HIPAA - and privacy policies. 2. security policies for different user roles. 1. data annotated based on domain ontologies or taxonomies. 2. to ensure traceability of data, from origin [initial point of collection] through to use. 3. convert data from existing data warehouse into RDF triples 1. mobile access
demographic, genetic and natural language data requiring rich representation 6. unpredictable arrival rates, in many cases data arrive in real-time 4. learned models can be generalized and refined in order to be applied to diverse sets of data 5. challenging, must accept data in different modalities [and from disparate sources] 23 World Population Scale Study 1. file-based synthetic population either centralized or distributed sites 2. large volume real time output data 3. variety of output datasets depends on the complexity of the model 1. compute intensive and data intensive computation like supercomputer.s performance 2. unstructured and irregular nature of graph processing 3. summary of various runs of simulation 1. movement of very large amount of data for visualization (networking) 2. distributed MPI-based simulation system (platform) 3. Charm++ on multi-nodes (software) 4. network file system (storage) 5. infiniband network (networking) 1. visualization 1. protection of PII on individuals used in modeling 2. data protection and secure platform for computation 1. data quality and be able capture the traceability of quality from computation -- 24 Social Contagion Modeling for Planning 1. traditional and new architecture for dynamic distributed processing on commondity clusters 2. models and datasets in to support Twitter network traffic 3. huge data storage per year 1. large scale modeling for various events (disease, emotions, behaviors, etc.) 2. scalable fusion between combined datasets 3. multi-levels analysis while generate sufficient results quickly 1. computing infrastructure which can capture interactions on various social events via the Internet (infrastructure) 2. file servers and databases (platform) 3. Ethernet and Infiniband networking (networking) 4. specialized simulators, open source software, and proprietary modeling (application) 5. huge user accounts across 1. multi-levels detail network representations 2. visualization with interactions 1. protection of PII on individuals used in modeling 2. data protection and secure platform for computation 1. data fusion from variety of dta sources 2. data consistency and no corruption 3. preprocessing of raw data 1. efficient method of moving data
operations on GPUs remains poorly developed. Existing solutions (e.g., ScaLapack for CPUs) are not well-integrated with higher level languages and require low-level programming which lengthens experiment and development time. 27 Organizing large-scale collections of consumer photos 1. over 500M images uploaded to social media sites each day 1. classifier (e.g. a Support Vector Machine), a process that is often hard to parallelize. 2. features seen in many large scale image processing problems 1. Hadoop or enhanced MapReduce 1. visualize large-scale 3-d reconstructions, and navigate large-scale collections of images that have been aligned to maps. 1. preserve privacy for users and digital rights for media. -- -- 28 Truthy Twitter Data 1. distributed data sources 2. large volume real time streaming 3. raw data in compressed formats 4. fully structured data in JSON, users metadata, data 5. multiple data schemas 1. various real time data analysis for anomaly detection, stream clustering, signal classification on time series and 1. Hadoop and HDFS (platform) 2. IndexedHBase, Hive, SciPy, NumPy (software) 3. in-memory database, MPI (platform) 4. high-speed Infiniband network (networking) 1. data retrieval and dynamic visualization 2. data driven interactive web interfaces 3. API for data query 1. security and privacy policy 1. standardized data structured/formats with extremely high data quality 1. low-level data storage infrastructure for efficient mobile access to data 29 Crowd Sourcing in Humanities -- 1. digitize existing photo and documents archives 2. analytics include pattern recognition of all kind (e.g., speech recognition, automatic A&V analysis, cultural patterns), -- -- 1. privacy issues in preserving anonymity of responses in spite of computer recording of access ID and reverse engineering of unusual user responses -- -- dimensional locations
2. real-time and batch data interoperability across Storage and Network Protocol Types through Generic Security Service API and Pluggable Authentication Modules (GSI, Kerberos, InCommon, Shibboleth). 2. access controls on files independently of the storage location. 33 The Discinnet process 1. integration of metadata approaches across disciplines -- 1. software: Symfony-PHP, Linux, MySQL -- 1. significant but optional security & privacy including secure servers and anonymization 1. integration of metadata approaches across disciplines -- 34 Semantic 1. all data types, image to text, structures to protein sequence 1. data graph processing 2. RDMS 1. cloud community resource required 1. efficient based visualization is needed -- -- -- 35 Light source beamlines 1. multiple streams of real time data to be stored and analyzed later 2. sample data to be analyzed in real time 1. standard bioinformatics tools (BLAST, HMMER, multiple alignment and phylogenetic tools, gene callers, sequence feature predictors…), Perl/Python wrapper scripts, Linux Cluster scheduling 1. high volume data transfer to remote batch processing resource -- 1. multiple security & privacy requirements to be satisfied -- -- 36 Catalina Real-Time Transient Survey 1. ~0.1TB per day at present will increase by factor 100 1. a wide variety of the existing astronomical data analysis tools, plus a large amount of custom developed tools and software, some of it a research project in itself 2. automated classification with machine -- 1. visualization mechanisms for highly data parameter spaces -- -- --
learning tools given the very sparse and heterogeneous data, dynamically evolving in time as more data come in, with follow-up decision making reflecting limited follow up resources 37 DOE Extreme Data from Sky Survey 1. ~1 PB per year becoming 7PB a year observational datal 1. interpretation of results from detailed simulations requires advanced analysis and visualization techniques and capabilities 1. MPI, OpenMP, C, C++, F90, FFTW, viz packages, python, FFTW, numpy, Boost, OpenMP, ScaLAPCK, PSQL & MySQL databases, Eigen, cfitsio, astrometry.net, and Minuit2 2. supercomputer I/O subsystem limitations must be addressed 1. interpretation of results using advanced visualization techniques and capabilities -- -- -- 38 Large Survey Data for Cosmology 1. 20TB data per day 1. analysis on both the simulation and observational data simultaneously 2. techniques for handling Cholesky decompostion for thousands of simulations with matricies of order 1M on a side 1. standard astrophysics reduction software as well as Perl/Python wrapper scripts 2. Oracle RDBMS, Postgres psql, as well as GPFS and Lustre file systems and tape archives 3. parallel image storage -- -- 1. links between remote telescopes and central analysis sites -- 39 Particle Physics at LHC 1. real time data from Accelerator and Analysis instruments 2. data collection 1. experimental data from ALICE, ATLAS, CMS, LHb 2. histograms, scatter-plots with model fits 3. Monte-Carlo computations 1. legacy computing infrastructure (computing nodes) 2. distributed cached files (storage) 3. object 1. histograms and model fits (visual) 1. data protection 1. data quality on complex apparatus --
3. calibration of instruments databases (sw pkg) 40 Belle II High Energy Physics Experiment 1. 120PB Raw data -- 1. 120PB Raw data 2. International distributed computing model to augment that at acceleartor (Japan) 3. data transfer of ~20Gbps at designed luminosity between Japan and US 4. software from Open Science Grid, Geant4, DIRAC, FTS, Belle II framework -- 1. standard Grid authentication -- -- 41 EISCAT 3D incoherent scatter radar system 1. remote sites generating 40PB data per year by 2022 2. HDF5 data format 3. visualization of high (>=5) dimension data 1. Queen Bea architecture with mix of distributed and central processing for 5 distributed sites 2. realtime monitoring of equipment by partial streaming analysis 3. needs to host rich set of Radar image processing services using machine learning, statistical modelling, and graph algorithms 1. architecture compatible with ENVRI Environmental Research Infrastructure collaboration 1. needs to suupport visualization of high (>=5) dimension data -- 1. preservation of data and avoid lost data due to instrument malfunction 1. needs to suupport realtime monitoring of equipment by partial streaming analysis 42 ENVRI Research 1. huge volume real time distributed data sources 2. variety of instrumentation datasets and metadata 1. diversified analytics tools 1. variety of computing infrastructures and architectures (infrastructure) 2. scattered repositories (storage) 1. graph plotting tools 2. time-series interactive tools 3. brower-based flash playback 4. earth map display 5. visual tools for quality comparisons 1. open data policy with minor restrictions 1. high quality on data 2. mirror archives 3. various metadata frameworks 4. scattered repositories and data curation 1. various kind of mobile sensor devices for data acquisition
clusters 4. iRODS data management 47 Turbulence 1. real time distributed datasets 2. various formats, resolution, semantics, and metadata 1. MapReduce, SciDB, and other scientific databases 2. continuously computing for updates 3. language for data mining and event searching 4. semantics interpretation and optimal structuring for data mining and predictive analysis 1. other legacy computing systems (e.g. supercomputer) 2. high throughput data transmission over the network 1. visualization to interpret results -- 1. validation for output products (correlations) -- 48 Climate Studies 1. ~100PB data in 2017 streaming at high data rates from large across world 2. Integrate large scale distributed data from simulations with diverse observations 3. link diverse data to novel HPC simulation 1. ata analytics close to data storage 1. extend architecture to several other fields 1. share data with worldwide climate 2. high end distributed visualization -- -- 1. phone based input and access 49 DOE-BER Subsurface 1. heterogeneous Diverse data with different domains and scales, Translation across diverse datasets that cross domains and scales 2. synthesis of diverse and disparate field, laboratory, -- 1. postgres, HDF5 data technologies and many custom software systems 1. phone based input and access -- -- 1. phone based input and access
Commercial: Mendeley NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Mendeley – An International Network of Research Vertical (area) Commercial Cloud Consumer Services Author/Company/Email William Gunn / Mendeley / william.gunn@mendeley.com Actors/Stakeholders and their roles and responsibilities Researchers, librarians, publishers, and funding organizations. Goals To promote more rapid advancement in scientific research by enabling researchers to efficiently collaborate, librarians to understand researcher needs, publishers to distribute research findings more quickly and broadly, and funding organizations to better understand the impact of the projects they fund. Use Case Description Mendeley has built a database of research documents and facilitates the creation of shared bibliographies. Mendeley uses the information collected about research reading patterns and other activities conducted via the software to build more efficient literature discovery and analysis tools. Text mining and classification systems enables automatic recommendation of relevant research, improving the cost and performance of research teams, particularly those engaged in curation of literature on a particular subject, such as the Mouse Genome Informatics group at Jackson Labs, which has a large team of manual curators who scan the literature. Other use cases include enabling publishers to more rapidly disseminate publications, facilitating research institutions and librarians with data management plan compliance, and enabling funders to better understand the impact of the work they fund via real-time data on the access and use of funded research. Current Solutions Compute(System) Amazon EC2 Storage HDFS Amazon S3 Networking Client-server connections between Mendeley and end user machines, connections between Mendeley offices and Amazon services. Software Hadoop, Scribe, Hive, Mahout, Python Big Data Characteristics Data Source (distributed/centralized) Distributed and centralized Volume (size) 15TB presently, growing about 1 TB/month Velocity (e.g. real time) Currently Hadoop batch jobs are scheduled daily, but work has begun on real-time recommendation Variety (multiple datasets, mashup) PDF documents and log files of social network and client activities
generalizing this use case (e.g. for ref. architecture) Amazon. Streaming video has features in common with other content providing services like iTunes, Google Play, Pandora and Last.fm More Information (URLs) by Xavier Amatriain Note: <additional comments> Commercial: Web Search NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Web Search (Bing, Google, Yahoo..) Vertical (area) Commercial Cloud Consumer Services Author/Company/Email Geoffrey Fox, Indiana University gcf@indiana.edu Actors/Stakeholders and their roles and responsibilities Owners of web information being searched; search engine companies; advertisers; users Goals Return in ~0.1 seconds, the results of a search based on average of 3 words; important to maximize “precision@10”; number of great responses in top 10 ranked results Use Case Description .1) Crawl the web; 2) Pre-process data to get searchable things (words, positions); 3) Form Inverted Index mapping words to documents; 4) Rank relevance of documents: PageRank; 5) Lots of technology for advertising, “reverse engineering ranking” “preventing reverse engineering”; 6) Clustering of documents into topics (as in Google News) 7) Update results efficiently Current Solutions Compute(System) Large Clouds Storage Inverted Index not huge; crawled documents are petabytes of text – rich media much more Networking Need excellent external network links; most operations pleasingly parallel and I/O sensitive. High performance internal network not needed Software MapReduce + Bigtable; Dryad + Cosmos. PageRank. Final step essentially a recommender engine Big Data Characteristics Data Source (distributed/centralized) Distributed web sites Volume (size) 45B web pages total, 500M photos uploaded each day, 100 hours of video uploaded to YouTube each minute Velocity (e.g. real time) Data continually updated Variety (multiple datasets, mashup) Rich set of functions. After processing, data similar for each page (except for media types) Variability (rate of change) Average page has life of a few months Big Data Science (collection, curation, Veracity (Robustness Issues) Exact results not essential but important to get main hubs and authorities for search query Visualization Not important although page layout critical
analysis, action) Data Quality A lot of duplication and spam Data Types Mainly text but more interest in rapidly growing image and video Data Analytics Crawling; searching including topic based search; ranking; recommending Big Data Specific Challenges (Gaps) Search of “deep web” (information behind query front ends) Ranking of responses sensitive to intrinsic (as in Pagerank) as well as advertising Link to user profiles and social network data Big Data Specific Challenges in Mobility Mobile search must have similar interfaces/results Security & Privacy Requirements Need to be sensitive to crawling restrictions. Avoid Spam results Highlight issues for generalizing this use case (e.g. for ref. architecture) Relation to Information retrieval such as search of scholarly works. More Information (URLs) http://www.slideshare.net/kleinerperkins/kpcb-internet-trends-2013 http://www.ifis.cs.tu-bs.de/teaching/ss-11/irws http://www.worldwidewebsize.com/ Note: <additional comments> Commercial: Cloud-based Continuity & Disaster Recovery NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title IaaS (Infrastructure as a Service) Big Data Business Continuity & Disaster Recovery (BC/DR) Within A Cloud Eco-System provided by Cloud Service Providers (CSPs) and Cloud Brokerage Service Providers (CBSPs) Vertical (area) Large Scale Reliable Data Storage Author/Company/Email Pw Carey, Compliance Partners, LLC, pwc.pwcarey@email.com Actors/Stakeholders and their roles and responsibilities Executive Management, Data Custodians, and Employees responsible for the integrity, protection, privacy, confidentiality, availability, safety, security and survivability of a business by ensuring the 3-As of data accessibility to an organizations services are satisfied; anytime, anyplace and on any device. Goals The following represents one approach to developing a workable BC/DR strategy. Prior to outsourcing an organizations BC/DR onto the backs/shoulders of a CSP or CBSP, the organization must perform the following Use Case, which will provide each organization with a baseline methodology for business continuity and disaster recovery (BC/DR) best practices, within a Cloud Eco-system for both Public and Private organizations.
Actors/Stakeholders and their roles and responsibilities End-users (Sender/Recipients) Transport Handlers (Truck/Ship/Plane) Telecom Providers (Cellular/SATCOM) Shippers (Shipping and Receiving) Goals Retention and analysis of items (Things) in transport Use Case Description The following use case defines the overview of a Big Data application related to the shipping industry (i.e. FedEx, UPS, DHL, etc.). The shipping industry represents possible the largest potential use case of Big Data that is in common use today. It relates to the identification, transport, and handling of item (Things) in the supply chain. The identification of an item begins with the sender to the recipients and for all those in between with a need to know the location and time of arrive of the items while in transport. A new aspect will be status condition of the items which will include sensor information, GPS coordinates, and a unique identification schema based upon a new ISO 29161 standards under development within ISO JTC1 SC31 WG2. The data is in near real-time being updated when a truck arrives at a depot or upon delivery of the item to the recipient. Intermediate conditions are not currently known, the location is not updated in real-time, items lost in a warehouse or while in shipment represent a problem potentially for homeland security. The records are retained in an archive and can be accessed for xx days. Current Solutions Compute(System) Unknown Storage Unknown Networking LAN/T1/Internet Web Pages Software Unknown Big Data Characteristics Data Source (distributed/centralized) Centralized today Volume (size) Large Velocity (e.g. real time) The system is not currently real-time. Variety (multiple datasets, mashup) Updated when the driver arrives at the depot and download the time and date the items were picked up. This is currently not real-time. Variability (rate of change) Today the information is updated only when the items that were checked with a bar code scanner are sent to the central server. The location is not currently displayed in real-time. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Visualization NONE Data Quality YES Data Types Not Available Data Analytics YES Big Data Specific Challenges (Gaps) Provide more rapid assessment of the identity, location, and conditions of the shipments, provide detailed analytics and location of problems in the system in real-time. Big Data Specific Challenges in Mobility
Pattern analysis of object (did the truck observed every weds afternoon take a different route today or is there a standard route this person takes every day). Crowd behavior/dynamics (is there a small group attempting to incite a riot. Is this person out of place in the crowd or behaving differently. Economic activity is the line at the bread store, the butcher, or the ice cream store, are more trucks traveling north with goods than trucks going south Has activity at or the size of stores in this market place increased or decreased over the past year. Fusion of data with other data to improve quality and confidence. Big Data Specific Challenges (Gaps) Processing the volume of data in NRT to support alerting and situational awareness. Big Data Specific Challenges in Mobility Getting data from mobile sensor to processing Security & Privacy Requirements Significant – sources and methods cannot be compromised the enemy should not be able to know what we see. Highlight issues for generalizing this use case (e.g. for ref. architecture) Typically this type of processing fits well into massively parallel computing such as provided by GPUs. Typical problem is integration of this processing into a larger cluster capable of processing data from several sensors in parallel and in NRT. Transmission of data from sensor to system is also a large challenge. More Information (URLs) Motion Imagery Standards - Some of many papers on object ident/tracking: General Articles on the need: Note: <additional comments>
Volume (size) The Mayo Clinic EHR dataset is a very large dataset containing over 5 million patients with thousands of properties each and many more that are derived from primary values. Velocity (e.g. real time) not real-time but updated periodically Variety (multiple datasets, mashup) Structured data, a patient has controlled vocabulary (CV) property values (demographics, diagnostic codes, medications, procedures, etc.) and continuous property values (lab tests, medication amounts, vitals, etc.). The number of property values could range from less than 100 (new patient) to more than 100,000 (long term patient) with typical patients composed of 100 CV values and 1000 continuous values. Most values are time based, i.e. a timestamp is recorded with the value at the time of observation. Variability (rate of change) Data will be updated or added during each patient visit. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Data are annotated based on domain ontologies or taxonomies. Semantics of data can vary from labs to labs. Visualization no visualization Data Quality Provenance is important to trace the origins of the data and data quality Data Types text, and Continuous Numerical values Data Analytics Integrating data into semantic graph, using graph traverse to replace SQL join. Developing semantic graph mining algorithms to identify graph patterns, index graph, and search graph. Indexed Hbase. Custom code to develop new patient properties from stored data. Big Data Specific Challenges (Gaps) For individualized cohort, we will effectively be building a datamart for each patient since the critical properties and indices will be specific to each patient. Due to the number of patients, this becomes an impractical approach. Fundamentally, the paradigm changes from relational lookup to semantic graph traversal. Big Data Specific Challenges in Mobility Physicians and patient may need access to this data on mobile platforms Security & Privacy Requirements Health records or clinical research databases must be kept secure/private. Highlight issues for generalizing this use case (e.g. for ref. architecture) Data integration: continuous values, ontological annotation, taxonomy Graph Search: indexing and searching graph Validation: Statistical validation More Information (URLs) Note: <additional comments>
Amsterdam Actors/Stakeholders and their roles and responsibilities End-users (biologists, ecologists, field researchers) Data analysts, data archive managers, e-Science Infrastructure managers, EU states national representatives Goals Research and monitor different ecosystems, biological species, their dynamics and migration. Use Case Description LifeWatch project and initiative intends to provide integrated access to a variety of data, analytical and modeling tools as served by a variety of collaborating initiatives. Another service is offered with data and tools in selected workflows for specific scientific communities. In addition, LifeWatch will provide opportunities to construct personalized ‘virtual labs', also allowing to enter new data and analytical tools. New data will be shared with the data facilities cooperating with LifeWatch. Particular case studies: Monitoring alien species, monitoring migrating birds, wetlands LifeWatch operates Global Biodiversity Information facility and Biodiversity Catalogue that is Biodiversity Science Web Services Catalogue Current Solutions Compute(System) Field facilities TBD Datacenter: General Grid and cloud based resources provided by national e-Science centers Storage Distributed, historical and trends data archiving Networking May require special dedicated or overlay sensor network. Software Web Services based, Grid based services, relational databases Big Data Characteristics Data Source (distributed/centralized) Ecological information from numerous observation and monitoring facilities and sensor network, satellite images/information, climate and weather, all recorded information. Information from field researchers Volume (size) Involves many existing data sets/sources Collected amount of data TBD Velocity (e.g. real time) Data analysed incrementally, processes dynamics corresponds to dynamics of biological and ecological processes. However may require real time processing and analysis in case of the natural or industrial disaster. May require data streaming processing. Variety (multiple datasets, mashup) Variety and number of involved databases and observation data is currently limited by available tools; in principle, unlimited with the growing ability to process data for identifying ecological changes, factors/reasons, species evolution and trends. See below in additional information. Variability (rate of change) Structure of the datasets and models may change depending on the data processing stage and tasks Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) In normal monitoring mode are data are statistically processed to achieve robustness. Some biodiversity research are critical to data veracity (reliability/trustworthiness). In case of natural and technogenic disasters data veracity is critical.
the science for supernovae. Variety (multiple datasets, mashup) While the imaging data is similar, the analysis for the 4 different types of cosmological measurements and comparisons to simulation data is quite different. Variability (rate of change) Weather and sky conditions can radically change both the quality and quantity of data. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Astrophysical data is a statistician’s nightmare as the both the uncertainties in a given measurement change from night-to-night in addition to the cadence being highly unpredictable. Also, most all of the cosmological measurements are systematically limited, and thus understanding these as best possible is the highest priority for a given survey. Visualization Interactive speed of web UI on very large data sets is an ongoing challenge. Basic querying and browsing of data to find new transients as well as monitoring the quality of the survey is a must. Ability to download large amounts of data for offline analysis is another requirement of the system. Ability to combine both simulation and observational data is also necessary. Data Quality Understanding the systematic uncertainties in the observational data is a prerequisite to a successful cosmological measurement. Beating down the uncertainties in the simulation data to under this level is a huge challenge for future surveys. Data Types Cf. above on “Variety” Data Analytics Big Data Specific Challenges (Gaps) New statistical techniques for understanding the limitations in simulation data would be beneficial. Often it is the case where there is not enough computing time to generate all the simulations one wants and thus there is a reliance on emulators to bridge the gaps. Techniques for handling Cholesky decompostion for thousands of simulations with matricies of order 1M on a side. Big Data Specific Challenges in Mobility Performing analysis on both the simulation and observational data simultaneously. Security & Privacy Requirements No special challenges. Data is either public or requires standard login with password. Highlight issues for generalizing this use case (e.g. for ref. architecture) Parallel databases which could handle imaging data would be an interesting avenue for future research. More Information (URLs) , http://desi.lbl.gov, & http://www.darkenergysurvey.org Note: <additional comments> Astronomy and Physics: Analysis of LHC (Large Hadron Collider) Data
Physics Analysis Information to Knowledge/Discovery Disk Files of Information including accelerator and Monte Carlo data. Include wisdom from lots of physicists (papers) in analysis choices Use simple statistical techniques (like histogramming, multi-variate analysis methods and other data analysis techniques and model fits to discover new effects (particles) and put limits on effects not seen Data reduction and processing steps with advanced physics algorithms to identify event properties, particle hypothesis etc. For interactive data analysis of those reduced and selected data sets the classic program is Root from CERN that reads multiple event (AOD, NTUP) files from selected data sets and use physicist generated C++ code to calculate new quantities such as implied mass of an unstable (new) particle While the bulk of data processing is done at Tier 1 and Tier 2 resources, the end stage analysis is usually done by users at a local Tier 3 facility. The scale of computing resources at Tier 3 sites range from workstations to small clusters. ROOT is the most common software stack used to analyze compact data formats generated on distributed computing resources. Data transfer is done using ATLAS and CMS DDM tools, which mostly rely on gridFTP middleware. XROOTD based direct data access is also gaining importance wherever high network bandwidth is available. Physics discoveries and results are confidential until certified by group and presented at meeting/journal. Data preserved so results reproducible Figure 1: The LHC Collider location at CERN Figure 2: The Multi-tier LHC computing infrastructure Astronomy and Physics: Belle II Experiment NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Belle II Experiment Vertical (area) Scientific Research: High Energy Physics Author/Company/Email David Asner & Malachi Schram, PNNL, & Actors/Stakeholders and their roles and responsibilities David Asner is the Chief Scientist for the US Belle II Project Malachi Schram is Belle II network and data transfer coordinator and the PNNL Belle II computing center manager
More Information (URLs) http://polargrid.org/polargrid https://www.cresis.ku.edu/ See movie at http://polargrid.org/polargrid/gallery Note: <additional comments> Use Case Stages Data Sources Data Usage Transformations (Data Analytics) Infrastructure Security & Privacy Radar Data Analysis for CReSIS (Scientific Research: Polar Science and Remote Sensing of Ice Sheets) Raw Data: Field Trip Raw Data from Radar instrument on Plane/Vehicle Capture Data on Disks for L1B. Check Data to monitor instruments. Robust Data Copying Utilities. Version of Full Analysis to check data. Rugged Laptops with small server (~2 CPU with ~40TB removable disk system) N/A Information: Offline Analysis L1B Transported Disks copied to (LUSTRE) File System Produce processed data as radar images Matlab Analysis code running in parallel and independently on each data sample ~2500 cores running standard cluster tools N/A except results checked before release on CReSIS web site Information: L2/L3 Geolocation & Layer Finding Radar Images from L1B Input to Science as database with GIS frontend GIS and Metadata Tools Environment to support automatic and/or manual layer determination GIS (Geographical Information System). Cluster for Image Processing. As above Knowledge, Wisdom, Discovery: Science GIS interface to L2/L3 data Polar Science Research integrating multiple data sources e.g. for Climate change. Glacier bed data used in simulations of glacier flow Exploration on a cloud style GIS supporting access to data. Simulation is 3D partial differential equation solver on large cluster. Varies according to science use. Typically results open after research complete. Figure 1: Typical Radar Data after analysis Figure 2: Typical flight paths of data gathering in survey region Figure 3. Typical echogram with Detected Boundaries. The upper (green) boundary is between air and ice layer while the lower (red) boundary is between ice and terrain