NERSC. More at other HPC centers between observation and simulation NCO, parallel NetCDF 49 DOE-BER Subsurface Biogeochemistry N/A N/A From 'omics of the microbes in the soil to watershed From observation to simulation PFLOWTran, postgres, HDF5, Akuna, NEWT, etc Data mining, data quality assessment, cross-correlation across datasets, reduced model development, statistics, quality assessment, data fusion 50 DOE-BER AmeriFlux and FLUXNET Networks N/A Streaming data from ~150 towers in AmeriFlux and over 500 towers distributed globally collecting flux measurements Flux data needs to be merged with biological, disturbance, and other ancillary data EddyPro, Custom analysis software, R, python, neural networks, Matlab Data mining, data quality assessment, cross-correlation across datasets, data assimilation, data interpolation, statistics, quality assessment, data fusion 51 Consumption forecasting in Smart Grids 4 TB a year for a city with 1.4M sensors like Los Angeles Streaming data from million(s) of sensors Tuple-based: Timeseries, database rows; Graph-based: Network topology, customer connectivity; Some semantic data for normalization R/Matlab, Weka, Hadoop. GIS based visualization Forecasting models, machine learning models, time series analysis, clustering, motif detection, complex event processing, visual network analysis Picture Book of Big Data Use Cases Figure 1: Cargo Shipping (Section 2.3.6) Scenario Figure 2: Pathology Imaging/digital pathology (Section 2.5.2) Examples of 2-D and 3-D pathology images. Figure 3: Pathology Imaging/digital pathology (Section 2.5.2) Architecture of Hadoop-GIS, a spatial data warehousing system over MapReduce to support spatial analytics for analytical pathology imaging. Figure 4: DataNet Federation Consortium DFC (Section 2.7.1) iRODS architecture Figure 5: Catalina Real-Time Transient Survey (CRTS): a digital, panoramic, synoptic sky survey (Section 2.8.1) One possible schematic architecture for a cyber-infrastructure for time domain
Use Case Requirements Summary Use Case Data Sources Transformation Capabilities Data Consumer Security & Privacy Lifecycle Management Others 1 Census 2010 and 2000 1. large document format from a centralized storage -- 1. large centralized storage (storage) -- 1. Title 13 data 1. long-term preservation of data as-is for 75 years 2. long-term preservation at the bit-level 3. curation process including format transformation 4. access and analytics processing after 75 years 5. needs to make sure no data loss -- 2 NARA: Search, Retrieve, Preservation 1. distributed data sources 2. large data storage 3. bursty data range from GB to hundreds of TB 4. wide variety of data formats including unstructured and structured data 5. distributed data sources in different clouds 1. crawl and index from distributed data sources 2. various analytics processing including ranking, data categorization, detect PII data 3. pre-processing of data 4. long-term preservation management fo large varied datasets. 5. hugh amount of data with high relevancy and recall. 1. large data storage 2. various storages such as NetApps, Hitachi, Magnetic tapes 1. high relevancy and high recall from search 2. high accuracy from categorization of records 3. various storages such as NetApps, Hitachi, Magnetic tapes 1. security policy 1. pre-process for virus scan 2. file format identification 3. indexing 4. categorize records 1. mobile search with similar from desktop 3 Statistical Survey Response 1. data size approximately one petabyte 1. analytics are required for recommendation systems, continued monitoring and general survey improvement. 1. software includes Hadoop, Spark, Hive, R, SAS, Mahout, Allegrograph, MySQL, Oracle, Storm, BigMemory, Cassandra, Pig 1. data visualization for data review, operational activity and general analysis. It continues to evolve. 1. improving recommendation systems that reduce costs and improve quality while providing confidentiality safeguards that are reliable and publically auditable. 2. both confidential and secure all data. All processes must be auditable for security and 1. high veracity on data and systems must be very robust. The semantic integrity of conceptual metadata concerning what exactly is measured and the resulting limits of inference remain a challenge 1. mobile access
on the monitoring and performance results 8 Web Search 1. Needs to support distributed data sources 2. Needs to support streaming data 3. Needs to support multimedia content 1. dynamic fetching content over the network 2. linking user profiles and social network data 1. petabytes of text and rich media (storage) 1. search time in ~0.1 seconds 2. top 10 ranked results 3. page layout (visual) 1. access control 2. needs to protect sensitive content 1. purge data after certain time interval (few months) 2. data cleaning 1. mobile search and rendering 9 Business Continuity & Disaster Recovery Within A Cloud -- 1. robust Backup algorithm 2. replicate recent changes 1. Hadoop 2. commercial cloud services -- 1. strong security for many applications -- -- 10 Cargo Shipping 1. centralized and real time distributed sites/sensors 1. tracking items based on the unique identification with its sensor information, GPS coordinates 2. real time updates on tracking items 1. Internet connectivity -- 1. security policy -- -- 11 Materials Data for 1. distributed data repositories for more than 500,000 commerical materials 2. may varieties of datasets 3. text, graphics, and images 1. (100s) of independent variables by collecting these variables to create robust datasets -- 1. visualization for materials discovery from many independent variables 2. visualization tools for materials 1. protection of proprietary of sensitive data 2. tools to mask proprietary information 1. how to handle data quality is poor or unknown -- 12 Simulation driven Materials Genomics 1. data streams from peta/exascale centralized simulation systems 2. distributed 1. computing data analysis for responsiveness 1. massive (150K cores) of legacy infrastructure (infrastructure) 2. GPFS (General Parallel File Sysem) 1. browser-based to search growing material data 1. sandbox as independent working areas between different data stakeholders 2. policy-driven 1. validation and UQ of simulation with experimental data 2. UQ in results from multiple datasets 1. mobile apps to access materials geonics information
web dataflows from central gateway to users 2. mashup of simulation outputs across codes 3. search and crowd-driven with computation backend be flexibly for new targets 4. MapReduce and search to join simulation and experimental data (storage) 3. MonogDB systems (platform) 4. 10Gb networking 5. various of analytic tools such as PyMatGen, FireWorks, VASP, ABINIT, NWChem, BerkeleyGW, varied community codes 6. large storage (storage) 7. scalable and object store (platform) 8. data streams from peta/exascale centralized simulation systems federation of datasets 13 Large Scale Geospatial Analysis and 1. geospatial data requires unique approaches to indexing and distributed analysis. 1. analytics include Closest point of approach, deviation from route, point density over time, PCA and ICA 2. geospatial data requires unique approaches to indexing and distributed analysis. 1. software includes Geospatially enabled RDBMS, Geospatial server/analysis software – ESRI ArcServer, Geoserver 1. visualization with GIS at high and low network bandwidths and on dedicated facilities and handhelds 1. sensitive data and must be completely secure in transit and at rest (particularly on handhelds) -- -- 14 Object and tracking 1. real-time data FMV – 30-60 frames per/sec at full color 1080P resolution and WALF – 1-10 frames per/sec at 10Kx10K full color resolution. 1. Rich analytics with object identification, pattern recognition, crowd behavior, economic activity and data fusion 1. wide range custom software and tools including traditional RDBM’s and display tools. 2. several network requirements 3. GPU Usage important 1. visualization of extracted outputs will typically be as overlays on a geospatial display. Overlay objects should be links back to the originating image/video segment. 2. output the form of OGC 1. significant security and privacy, sources and methods cannot be compromised the enemy should not be able to know what we see. 1. veracity of extracted objects -- value
numeric, structured text, discrete nominal, discrete ordinal, discrete structured, binary large blobs (images and video). 5. data evolves over time in a highly variable fashion a subset of the data. 5. robust security to prevent data breaches. 17 Pathology Imaging 1. high resolution spatial digitized pathology images 2. various image quality analysis algorithms 3. various image data formats especially BIGTIFF with structured data for analytical results 4. image analysis, spatial queries and analytics, feature clustering and classification 1. high performance image analysis to extract spatial information 2. spatial queries and analytics, and feature clustering and classification 3. analytic processing on huge large dataset and be able to correlate with other data types such as clinical data, -omic data. 1. legacy system and cloud (computing cluster) 2. huge legacy and new storage such as SAN or HDFS (storage) 3. high throughput network link (networking) 4. MPI image analysis, MapReduce, Hive with spatial extension (sw pkgs) 1. visualization for validation and training 1. security and privacy protection for protected health information 1. human annotations for validation 1. 3D visualization and rendering on mobile platforms 18 Bioimaging 1. distributed multi-modal high resolution experimental sources of bioimages (instruments). 2. 50TB of data data formats include images. 1. computing with responsive analysis 2. segmentation of regions of interest, selection and extraction of features, and object classification, and organization, and search. 3. advance 1. ImageJ, OMERO, VolRover, advanced segmentation and feature detection methods from applied math researchers.... Scalable and object store databases needed. 2. NERSC.s Hopper infrastructure 1. 3D structural modeling 1. significant but optional security & privacy including secure servers and anonymization 1. workflow components include data acquisition, storage, enhancement, minimizing noise -- dimensional value
biosciences discovery through big data techniques / extreme scale computing…. processing and analytics. … Machine learning (SVM and RF) for classification and recommendation services. … advanced algorithms for massive image analysis. computational solutions. 4. massive data analysis toward massive imaging data sets. 3. database and image collections. 4. 10 GB and future 100 GB and advanced networking (SDN). 19 Genomic 1. high throughput compressed data (300GB/day) from various DNA sequencers 2. distributed data source (sequencers) 3. various file formats either structured and unstructured data 1. for processing raw data in variant calls 2. machine learning for complex analysis on systematic errors from sequencing technologies are hard to characterize 1. legacy computing cluster and other PaaS and IaaS (computing cluster) 2. huge data storage in PB range (storage) 3. Unix-based legacy sequencing bioinformatics software (sw pkg) 1. data format for Genome browsers 1. security and privacy protection on health records and clinical research databases -- 1. mobile platforms for physicians accessing genomic data (mobile device) 20 analysis for and genomes 1. multiple centralized data sources 2. proteins and their structural features, core genomic data, new types of “Omics” data such as methylomics, and proteomics describing gene 2. scalable RDMS for heterogeneity biological data 2. real-time rapid and parallel bulk loading 3. Oracle RDBMS, SQLite files, flat text files, Lucy (a version of Lucene) for 1. huge data storage 1. real time interactive parallel bulk loading capability 2. interactive Web UI, backend precomputations, batch job computation submission from the UL 3. download assembled and 1. login security - username and password 2. creation of user account to submit and access dataset to system via web interface 3. single sign on capability (SSO) 1. methods to improve data quality required. 2. data clustering, classification, reduction. 3. Integrate new data / content into the system.s data store annotate data -- ,
expression 3. Front real time Web UI interactive. Back end data loading processing must keep up with exponential growth of sequence data due to the rapid drop in cost of sequencing technology. 4. heterogeneous, complex, structural, and hierarchical biological data. 5. metagenomic samples can vary by several orders of magnitude, such as several hundred thousand genes to a billion genes keyword searches, BLAST databases, USEARCH databases 4. Linux cluster, Oracle RDBMS server, large memory machines, standard Linux interactive hosts 5. sequencing and comparative analysis techniques for highly complex data 6. descriptive statistics annotated datasets for offline analysis. 4. ability to query and browse data via interactive Web UI. 5. visualize structure [of data] at different levels of resolution. Ability to view abstract representations of highly similar data. 21 Diabetes 1. distributed EHR data 2. over 5 million patients with thousands of properties each and many more that are derived from primary values. 3. each record, a range of 100 to 100,000 data property values average of 100 controlled vocabulary values and 1000 continuous values. 4. none but data updated 1. data integration, using ontological annotation and taxonomies 2. parallel retrieval algorithms for both indexed and custom searches, identify data of interest. Patient cohorts, patients meeting certain criteria, patients sharing similar characteristics 3. distributed graph mining algorithms, pattern analysis and graph indexing, pattern searching on 1. data warehouse, open source indexed Hbase 2. supercomputers, cloud and parallel computing 3. I/O intensive processing 4. HDFS storage 5. custom code to develop new properties from stored data. 1. efficient based visualization is needed 1. protection of health data in accordance with legal requirements - eg, HIPAA - and privacy policies. 2. security policies for different user roles. 1. data annotated based on domain ontologies or taxonomies. 2. to ensure traceability of data, from origin [initial point of collection] through to use. 3. convert data from existing data warehouse into RDF triples 1. mobile access
periodically. Data is timestamped with the time of observation [time that the value is recorded.] 5. structured data about a patient falls into two main categories: data with controlled vocabulary [CV] property values, and continuous property values [which are recorded / captured more frequently]. 6. data consists of text, and Continuous Numerical values. RDF triple graphs 4. robust statistical analysis tools to manage false discovery rate, determine true subgraph significance, validate results, eliminate false positive / false negative results 5. semantic graph mining algorithms to identify graph patterns, index and search graph. 6. Semantic graph traversal. 22 Statistical Relational Artificial Intelligence for Health Care 1. centralized data, with some data retrieved from internet sources 2. range from 100.s of GB sample size, to 1 petabyte for very large studies 3. both constant updates / additions [to subset of data] and scheduled batch inputs 4. large, longitudinal data 5. rich relational data comprised of multiple tables, different data types such as imaging, EHR, 1. relational probabalistic models / probability therory, software learns models from multiple data types and can possibly integrate the information and reason about complex queries. 2. robust and accurate learning methods to account for .data imbalance. [where large amount of data is available for a small number of subjects] 3. learning algorithms to identify skews in data, so as to not -- incorrectly -- model .noise. 1. Java, some in house tools, [relational] database and NoSQL stores 2. cloud and parallel computing 3. high performance computer, 48 GB RAM [to performa analysis for a moderate sample size] 4. clusters for large datasets 5. 200 GB - 1 TB hard drive for test data 1. visualization of subsets of very large data 1. secure handling and processing of data is of crucial importance in medical domains 1. merging multiple tables before analysis 2. methods to validate data to minimize errors --
demographic, genetic and natural language data requiring rich representation 6. unpredictable arrival rates, in many cases data arrive in real-time 4. learned models can be generalized and refined in order to be applied to diverse sets of data 5. challenging, must accept data in different modalities [and from disparate sources] 23 World Population Scale Study 1. file-based synthetic population either centralized or distributed sites 2. large volume real time output data 3. variety of output datasets depends on the complexity of the model 1. compute intensive and data intensive computation like supercomputer.s performance 2. unstructured and irregular nature of graph processing 3. summary of various runs of simulation 1. movement of very large amount of data for visualization (networking) 2. distributed MPI-based simulation system (platform) 3. Charm++ on multi-nodes (software) 4. network file system (storage) 5. infiniband network (networking) 1. visualization 1. protection of PII on individuals used in modeling 2. data protection and secure platform for computation 1. data quality and be able capture the traceability of quality from computation -- 24 Social Contagion Modeling for Planning 1. traditional and new architecture for dynamic distributed processing on commondity clusters 2. models and datasets in to support Twitter network traffic 3. huge data storage per year 1. large scale modeling for various events (disease, emotions, behaviors, etc.) 2. scalable fusion between combined datasets 3. multi-levels analysis while generate sufficient results quickly 1. computing infrastructure which can capture interactions on various social events via the Internet (infrastructure) 2. file servers and databases (platform) 3. Ethernet and Infiniband networking (networking) 4. specialized simulators, open source software, and proprietary modeling (application) 5. huge user accounts across 1. multi-levels detail network representations 2. visualization with interactions 1. protection of PII on individuals used in modeling 2. data protection and secure platform for computation 1. data fusion from variety of dta sources 2. data consistency and no corruption 3. preprocessing of raw data 1. efficient method of moving data
operations on GPUs remains poorly developed. Existing solutions (e.g., ScaLapack for CPUs) are not well-integrated with higher level languages and require low-level programming which lengthens experiment and development time. 27 Organizing large-scale collections of consumer photos 1. over 500M images uploaded to social media sites each day 1. classifier (e.g. a Support Vector Machine), a process that is often hard to parallelize. 2. features seen in many large scale image processing problems 1. Hadoop or enhanced MapReduce 1. visualize large-scale 3-d reconstructions, and navigate large-scale collections of images that have been aligned to maps. 1. preserve privacy for users and digital rights for media. -- -- 28 Truthy Twitter Data 1. distributed data sources 2. large volume real time streaming 3. raw data in compressed formats 4. fully structured data in JSON, users metadata, data 5. multiple data schemas 1. various real time data analysis for anomaly detection, stream clustering, signal classification on time series and 1. Hadoop and HDFS (platform) 2. IndexedHBase, Hive, SciPy, NumPy (software) 3. in-memory database, MPI (platform) 4. high-speed Infiniband network (networking) 1. data retrieval and dynamic visualization 2. data driven interactive web interfaces 3. API for data query 1. security and privacy policy 1. standardized data structured/formats with extremely high data quality 1. low-level data storage infrastructure for efficient mobile access to data 29 Crowd Sourcing in Humanities -- 1. digitize existing photo and documents archives 2. analytics include pattern recognition of all kind (e.g., speech recognition, automatic A&V analysis, cultural patterns), -- -- 1. privacy issues in preserving anonymity of responses in spite of computer recording of access ID and reverse engineering of unusual user responses -- -- dimensional locations
identification of structures (lexical units, linguistic rules, etc.) 30 CINET for Network Science 1. a set of network topologies file to study graph theoretic properties and behaviors of various algorithms 2. asynchronous and real time synchronous distributed computing 1. environments to run various network and graph analysis tools 2. dynamic grow of the networks 3. asynchronous and real time synchronous distributed computing 4. different parallel algorithms for different partitioning schemes for efficient operation 1. large file system (storage) 2. various network connectivity (networking) 3. existing computing cluster 4. EC2 computing cluster 5. various graph libraries, management toos, databases, semantic web tools 1. client side visualization -- -- -- 31 NIST Information Access Division 1. large amounts of semi-annotated web pages, tweets, images, video 2. scaling to larger data, intrinsic and annotation uncertainty measurement, performance measurement for incompletely annotated data, measuring analytic performance for heterogeneous data and analytic flows involving users 1. analytic algorithms working with written language, speech, human imagery, etc. must generally be tested against real or realistic data. It’s extremely challenging to engineer artificial data that sufficiently captures the variability of real data involving humans 1. PERL, Python, C/C++, Matlab, R development tools. Create ground-up test and measurement applications 1. analytic flows involving users 1. security requirements for protecting sensitive data while enabling meaningful developmental performance evaluation. Shared evaluation testbeds must protect the intellectual property of analytic algorithm developers -- -- 32 DataNet (iRODS) 1. process key format types NetCDF, HDF5, Dicom 1. needs to provide general analytics workflows 1. iRODS data management software 2. 1. general visulaization workflows 1. Federate across existing authentication environments -- -- truthing
2. real-time and batch data interoperability across Storage and Network Protocol Types through Generic Security Service API and Pluggable Authentication Modules (GSI, Kerberos, InCommon, Shibboleth). 2. access controls on files independently of the storage location. 33 The Discinnet process 1. integration of metadata approaches across disciplines -- 1. software: Symfony-PHP, Linux, MySQL -- 1. significant but optional security & privacy including secure servers and anonymization 1. integration of metadata approaches across disciplines -- 34 Semantic 1. all data types, image to text, structures to protein sequence 1. data graph processing 2. RDMS 1. cloud community resource required 1. efficient based visualization is needed -- -- -- 35 Light source beamlines 1. multiple streams of real time data to be stored and analyzed later 2. sample data to be analyzed in real time 1. standard bioinformatics tools (BLAST, HMMER, multiple alignment and phylogenetic tools, gene callers, sequence feature predictors…), Perl/Python wrapper scripts, Linux Cluster scheduling 1. high volume data transfer to remote batch processing resource -- 1. multiple security & privacy requirements to be satisfied -- -- 36 Catalina Real-Time Transient Survey 1. ~0.1TB per day at present will increase by factor 100 1. a wide variety of the existing astronomical data analysis tools, plus a large amount of custom developed tools and software, some of it a research project in itself 2. automated classification with machine -- 1. visualization mechanisms for highly data parameter spaces -- -- --
learning tools given the very sparse and heterogeneous data, dynamically evolving in time as more data come in, with follow-up decision making reflecting limited follow up resources 37 DOE Extreme Data from Sky Survey 1. ~1 PB per year becoming 7PB a year observational datal 1. interpretation of results from detailed simulations requires advanced analysis and visualization techniques and capabilities 1. MPI, OpenMP, C, C++, F90, FFTW, viz packages, python, FFTW, numpy, Boost, OpenMP, ScaLAPCK, PSQL & MySQL databases, Eigen, cfitsio, astrometry.net, and Minuit2 2. supercomputer I/O subsystem limitations must be addressed 1. interpretation of results using advanced visualization techniques and capabilities -- -- -- 38 Large Survey Data for Cosmology 1. 20TB data per day 1. analysis on both the simulation and observational data simultaneously 2. techniques for handling Cholesky decompostion for thousands of simulations with matricies of order 1M on a side 1. standard astrophysics reduction software as well as Perl/Python wrapper scripts 2. Oracle RDBMS, Postgres psql, as well as GPFS and Lustre file systems and tape archives 3. parallel image storage -- -- 1. links between remote telescopes and central analysis sites -- 39 Particle Physics at LHC 1. real time data from Accelerator and Analysis instruments 2. data collection 1. experimental data from ALICE, ATLAS, CMS, LHb 2. histograms, scatter-plots with model fits 3. Monte-Carlo computations 1. legacy computing infrastructure (computing nodes) 2. distributed cached files (storage) 3. object 1. histograms and model fits (visual) 1. data protection 1. data quality on complex apparatus --
3. calibration of instruments databases (sw pkg) 40 Belle II High Energy Physics Experiment 1. 120PB Raw data -- 1. 120PB Raw data 2. International distributed computing model to augment that at acceleartor (Japan) 3. data transfer of ~20Gbps at designed luminosity between Japan and US 4. software from Open Science Grid, Geant4, DIRAC, FTS, Belle II framework -- 1. standard Grid authentication -- -- 41 EISCAT 3D incoherent scatter radar system 1. remote sites generating 40PB data per year by 2022 2. HDF5 data format 3. visualization of high (>=5) dimension data 1. Queen Bea architecture with mix of distributed and central processing for 5 distributed sites 2. realtime monitoring of equipment by partial streaming analysis 3. needs to host rich set of Radar image processing services using machine learning, statistical modelling, and graph algorithms 1. architecture compatible with ENVRI Environmental Research Infrastructure collaboration 1. needs to suupport visualization of high (>=5) dimension data -- 1. preservation of data and avoid lost data due to instrument malfunction 1. needs to suupport realtime monitoring of equipment by partial streaming analysis 42 ENVRI Research 1. huge volume real time distributed data sources 2. variety of instrumentation datasets and metadata 1. diversified analytics tools 1. variety of computing infrastructures and architectures (infrastructure) 2. scattered repositories (storage) 1. graph plotting tools 2. time-series interactive tools 3. brower-based flash playback 4. earth map display 5. visual tools for quality comparisons 1. open data policy with minor restrictions 1. high quality on data 2. mirror archives 3. various metadata frameworks 4. scattered repositories and data curation 1. various kind of mobile sensor devices for data acquisition
omic, and simulation datasets across different semantic, spatial, and temporal scales 3. link diverse data to novel HPC simulation 50 DOE-BER AmeriFlux and FLUXNET Networks 1. heterogeneous Diverse data with different domains and scales, Translation across diverse datasets that cross domains and scales 2. link to many other environment and biology datasets 3. link to HPC Climate and other simulations 4. link to European data sources and projects 5. access data from 500 distributed sources 1. Custom Software: EddyPro, Custom analysis software, R, python, neural networks, Matlab 1. custom software like EddyPro and analysis software like R, python, neural netowrks, Matlab 2. analytics includes data mining, data quality assessment, cross-correlation across datasets, data assimilation, data interpolation, statistics, quality assessment, data fusion, etc. 1. phone based input and access -- -- 1. phone based input and access 51 forecasting in Smart Grids 1. diverse data from Smart Grid sensors, City planning, weather, utilities 2, data updated every 15 minutes 1. new machine learning analytics to predict consumption 1. SQL databases, CVS files, HDFS (platform) 2. R/Matlab, Weka, Hadoop (platform) -- 1. privacy and anonymization by aggregation -- 1. mobile access for clients Use Case Detail Requirements To assist in interpreting this section of the report: This section is first organized by category [Data Sources, Transformation, etc.] Within each category is a list of numbered requirement statements.
Actors/Stakeholders and their roles and responsibilities End-users (Sender/Recipients) Transport Handlers (Truck/Ship/Plane) Telecom Providers (Cellular/SATCOM) Shippers (Shipping and Receiving) Goals Retention and analysis of items (Things) in transport Use Case Description The following use case defines the overview of a Big Data application related to the shipping industry (i.e. FedEx, UPS, DHL, etc.). The shipping industry represents possible the largest potential use case of Big Data that is in common use today. It relates to the identification, transport, and handling of item (Things) in the supply chain. The identification of an item begins with the sender to the recipients and for all those in between with a need to know the location and time of arrive of the items while in transport. A new aspect will be status condition of the items which will include sensor information, GPS coordinates, and a unique identification schema based upon a new ISO 29161 standards under development within ISO JTC1 SC31 WG2. The data is in near real-time being updated when a truck arrives at a depot or upon delivery of the item to the recipient. Intermediate conditions are not currently known, the location is not updated in real-time, items lost in a warehouse or while in shipment represent a problem potentially for homeland security. The records are retained in an archive and can be accessed for xx days. Current Solutions Compute(System) Unknown Storage Unknown Networking LAN/T1/Internet Web Pages Software Unknown Big Data Characteristics Data Source (distributed/centralized) Centralized today Volume (size) Large Velocity (e.g. real time) The system is not currently real-time. Variety (multiple datasets, mashup) Updated when the driver arrives at the depot and download the time and date the items were picked up. This is currently not real-time. Variability (rate of change) Today the information is updated only when the items that were checked with a bar code scanner are sent to the central server. The location is not currently displayed in real-time. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Visualization NONE Data Quality YES Data Types Not Available Data Analytics YES Big Data Specific Challenges (Gaps) Provide more rapid assessment of the identity, location, and conditions of the shipments, provide detailed analytics and location of problems in the system in real-time. Big Data Specific Challenges in Mobility
Pattern analysis of object (did the truck observed every weds afternoon take a different route today or is there a standard route this person takes every day). Crowd behavior/dynamics (is there a small group attempting to incite a riot. Is this person out of place in the crowd or behaving differently. Economic activity is the line at the bread store, the butcher, or the ice cream store, are more trucks traveling north with goods than trucks going south Has activity at or the size of stores in this market place increased or decreased over the past year. Fusion of data with other data to improve quality and confidence. Big Data Specific Challenges (Gaps) Processing the volume of data in NRT to support alerting and situational awareness. Big Data Specific Challenges in Mobility Getting data from mobile sensor to processing Security & Privacy Requirements Significant – sources and methods cannot be compromised the enemy should not be able to know what we see. Highlight issues for generalizing this use case (e.g. for ref. architecture) Typically this type of processing fits well into massively parallel computing such as provided by GPUs. Typical problem is integration of this processing into a larger cluster capable of processing data from several sensors in parallel and in NRT. Transmission of data from sensor to system is also a large challenge. More Information (URLs) Motion Imagery Standards - Some of many papers on object ident/tracking: General Articles on the need: Note: <additional comments>
Big Data Specific Challenges (Gaps) Data volumes increasing. Shipping disks clumsy but no other obvious solution. Image processing algorithms still very active research Big Data Specific Challenges in Mobility Smart phone interfaces not essential but LOW power technology essential in field Security & Privacy Requirements Himalaya studies fraught with political issues and require UAV. Data itself open after initial study Highlight issues for generalizing this use case (e.g. for ref. architecture) Loosely coupled clusters for signal processing. Must support Matlab. More Information (URLs) http://polargrid.org/polargrid https://www.cresis.ku.edu/ See movie at http://polargrid.org/polargrid/gallery Note: <additional comments> Use Case Stages Data Sources Data Usage Transformations (Data Analytics) Infrastructure Security & Privacy Radar Data Analysis for CReSIS (Scientific Research: Polar Science and Remote Sensing of Ice Sheets) Raw Data: Field Trip Raw Data from Radar instrument on Plane/Vehicle Capture Data on Disks for L1B. Check Data to monitor instruments. Robust Data Copying Utilities. Version of Full Analysis to check data. Rugged Laptops with small server (~2 CPU with ~40TB removable disk system) N/A Information: Offline Analysis L1B Transported Disks copied to (LUSTRE) File System Produce processed data as radar images Matlab Analysis code running in parallel and independently on each data sample ~2500 cores running standard cluster tools N/A except results checked before release on CReSIS web site Information: L2/L3 Geolocation & Layer Finding Radar Images from L1B Input to Science as database with GIS frontend GIS and Metadata Tools Environment to support automatic and/or manual layer determination GIS (Geographical Information System). Cluster for Image Processing. As above Knowledge, Wisdom, Discovery: Science GIS interface to L2/L3 data Polar Science Research integrating multiple data sources e.g. for Climate change. Glacier bed data used in simulations of glacier Exploration on a cloud style GIS supporting access to data. Simulation is 3D partial differential equation solver on large cluster. Varies according to science use. Typically results open after research complete.
Big Data Characteristics Data Source (distributed/centralized) Terabase-scale sequencing data from JGI, subsurface and surface hydrological and biogeochemical data from a variety of sensors (including dense geophysical datasets) experimental data from field and lab analysis Volume (size) Velocity (e.g. real time) Variety (multiple datasets, mashup) Data crosses all scales from genomics of the microbes in the soil to watershed The SFA requires the synthesis of diverse and disparate field, laboratory, and simulation datasets across different semantic, spatial, and temporal scales through GEWaSC. Such datasets will be generated by the different research areas and include simulation data, field data (hydrological, geochemical, geophysical), ‘omics data, and data from laboratory experiments. Variability (rate of change) Simulations and experiments Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) and Quality Each of the sources samples different properties with different footprints – extremely heterogeneous. Each of the soruces has different levels of uncertainty and precision associated with it. In addition, the translation across scales and domains introduces uncertainty as does the data mining. Data quality is critical. Visualization Visualization is crucial to understanding the data. Data Types Described in “Variety” above. Data Analytics Data mining, data quality assessment, across datasets, reduced model development, statistics, quality assessment, data fusion, etc. Big Data Specific Challenges (Gaps) Translation across diverse and large datasets that cross domains and scales. Big Data Specific Challenges in Mobility Field experiment data taking would be improved by access to existing data and automated entry of new data via mobile devices. Security & Privacy Requirements Highlight issues for generalizing this use case (e.g. for ref. architecture) A wide array of programs in the earth sciences are working on challenges that cross the same domains as this project.