Already significant value but now needs to be disseminated and advertised although maximal value to come from interdisciplinary/projective next version. Value is to detect quickly a paper/project of interest for its results and next step is trajectory of the field under types of interactions from diverse levels of oracles (subjects/objects) + from interdisciplinary context. Current Solutions Compute(System) Currently on OVH servers (mix shared + dedicated) Storage OVH Networking To be implemented with desired integration with others Software Current version with Symfony-PHP, Linux, MySQL Big Data Characteristics Data Source (distributed/centralized) Currently centralized, soon distributed per country and even per hosting institution interested by own platform Volume (size) Not significant : this is a metadata base, not big data Velocity (e.g. real time) Real time Variety (multiple datasets, mashup) Link to Big data still to be established in a relationship not yet implemented (with experimental databases and already 1st level related metadata) Variability (rate of change) Currently Real time, for further multiple locations and distributed architectures, periodic (such as nightly) Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues, semantics) Methods to detect overall consistency, holes, errors, misstatements, known but mostly to be implemented Visualization Multidimensional (hypercube) Data Quality (syntax) A priori correct (directly human captured) with sets of checking + evaluation processes partly implemented Data Types ‘cluster displays’ (image), vectors, categories, PDFs Data Analytics Big Data Specific Challenges (Gaps) Our goal is to contribute to Big 2 Metadata challenge by systematic reconciling between metadata from many complexity levels with ongoing input from researchers from ongoing research process. Current relationship with Richeact is to reach the interdisciplinary model, using meta-grammar itself to be experimented and its extent fully proven to bridge efficiently the gap between as remote complexity levels as semantic and most elementary (big) signals. Example with cosmological models versus many levels of intermediary models (particles, gases, galactic, nuclear, geometries). Others with computational versus semantic levels. Big Data Specific Challenges in Mobility Appropriate graphic interface power Security & Privacy Requirements Several levels already available and others planned, up to physical access keys and isolated servers. Optional anonymity, usual protected exchanges
(cross-discipline and internationally), or effectively (virtually no materials data meetings, standards groups, or dedicated funded programs). One of the greatest challenges that Big Data approaches can address is predicting the performance of real materials (gram to ton quantities) starting at the atomistic, nanometer, and/or micrometer level of description. As a result of the above considerations, decisions about materials usage are unnecessarily conservative, often based on older rather than newer materials R&D data, and not taking advantage of advances in modeling and simulations. Materials informatics is an area in which the new tools of data science can have major impact. Current Solutions Compute(System) None Storage Widely dispersed with many barriers to access Networking Virtually none Software Narrow approaches based on national programs (Japan, Korea, and China), applications (EU Nuclear program), proprietary solutions (Granta, etc.) Big Data Characteristics Data Source (distributed/centralized) Extremely distributed with data repositories existing only for a very few fundamental properties Volume (size) It is has been estimated (in the 1980s) that there were over 500,000 commercial materials made in the last fifty years. The last three decades has seen large growth in that number. Velocity (e.g. real time) Computer-designed and theoretically design materials (e.g., nanomaterials) are growing over time Variety (multiple datasets, mashup) Many data sets and virtually no standards for mashups Variability (rate of change) Materials are changing all the time, and new materials data are constantly being generated to describe the new materials Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) More complex material properties can require many (100s?) of independent variables to describe accurately. Virtually no activity no exists that is trying to identify and systematize the collection of these variables to create robust data sets. Visualization Important for materials discovery. Potentially important to understand the dependency of properties on the many independent variables. Virtually unaddressed. Data Quality Except for fundamental data on the structural and thermal properties, data quality is poor or unknown. See Munro’s NIST Standard Practice Guide. Data Types Numbers, graphical, images Data Analytics Empirical and narrow in scope
Solutions Storage NetApps, Hitachi, Magnetic tapes. Networking Software Custom software, commercial search products, commercial databases. Big Data Characteristics Data Source (distributed/centralized) Distributed data sources from federal agencies. Current solution requires transfer of those data to a centralized storage. In the future, those data sources may reside in different Cloud environments. Volume (size) Hundred of Terabytes, and growing. Velocity (e.g. real time) Input rate is relatively low compared to other use cases, but the trend is bursty. That is the data can arrive in batches of size ranging from GB to hundreds of TB. Variety (multiple datasets, mashup) Variety data types, unstructured and structured data: textual documents, emails, photos, scanned documents, multimedia, social networks, web sites, databases, etc. Variety of application domains, since records come from different agencies. Data come from variety of repositories, some of which can be cloud-based in the future. Variability (rate of change) Rate can change especially if input sources are variable, some having audio, video more, some more text, and other images, etc. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Search results should have high relevancy and high recall. Categorization of records should be highly accurate. Visualization TBD Data Quality Unknown. Data Types Variety data types: textual documents, emails, photos, scanned documents, multimedia, databases, etc. Data Analytics Crawl/index; search; ranking; predictive search. Data categorization (sensitive, confidential, etc.) PII data detection and flagging. Big Data Specific Challenges (Gaps) Perform pre-processing and manage for long-term of large and varied data. Search huge amount of data. Ensure high relevancy and recall. Data sources may be distributed in different clouds in future. Big Data Specific Challenges in Mobility Mobile search must have similar interfaces/results Security & Privacy Need to be sensitive to data access restrictions.
Note: <additional comments> NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Individualized Diabetes Management Vertical (area) Healthcare Author/Company/Email Peter Li, Ying Ding, Philip Yu, Geoffrey Fox, David Wild at Mayo Clinic, Indiana University, UIC; dingying@indiana.edu Actors/Stakeholders and their roles and responsibilities Mayo Clinic + IU/semantic integration of EHR data UIC/semantic graph mining of EHR data IU cloud and parallel computing Goals Develop advanced graph-based data mining techniques applied to EHR to search for these cohorts and extract their EHR data for outcome evaluation. These methods will push the boundaries of scalability and data mining technologies and advance knowledge and practice in these areas as well as clinical management of complex diseases. Use Case Description Diabetes is a growing illness in world population, affecting both developing and developed countries. Current management strategies do not adequately take into account of individual patient profiles, such as co-morbidities and medications, which are common in patients with chronic illnesses. We propose to approach this shortcoming by identifying similar patients from a large Electronic Health Record (EHR) database, i.e. an individualized cohort, and their respective management outcomes to formulate one best solution suited for a given patient with diabetes. Project under development as below Stage 1: Use the Semantic Linking for Property Values method to convert an existing data warehouse at Mayo Clinic, called the Enterprise Data Trust (EDT), into RDF triples that enables us to find similar patients much more efficiently through linking of both vocabulary-based and continuous values, Stage 2: Needs efficient parallel retrieval algorithms, suitable for cloud or HPC, using open source Hbase with both indexed and custom search to identify patients of possible interest. Stage 3: The EHR, as an RDF graph, provides a very rich environment for graph pattern mining. Needs new distributed graph mining algorithms to perform pattern analysis and graph indexing technique for pattern searching on RDF triple graphs. Stage 4: Given the size and complexity of graphs, mining subgraph patterns could generate numerous false positives and miss numerous false negatives. Needs robust statistical analysis tools to manage false discovery rate and determine true subgraph significance and validate these through several clinical use cases. Current Solutions Compute(System) supercomputers; cloud Storage HDFS Networking Varies. Significant I/O intensive processing needed
Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Visualization NONE Data Quality YES Data Types Not Available Data Analytics YES Big Data Specific Challenges (Gaps) Provide more rapid assessment of the identity, location, and conditions of the shipments, provide detailed analytics and location of problems in the system in real-time. Big Data Specific Challenges in Mobility Currently conditions are not monitored on-board trucks, ships, and aircraft Security & Privacy Requirements Security need to be more robust Highlight issues for generalizing this use case (e.g. for ref. architecture) This use case includes local data bases as well as the requirement to synchronize with the central server. This operation would eventually extend to mobile device and on-board systems which can track the location of the items and provide real-time update of the information including the status of the conditions, logging, and alerts to individuals who have a need to know. More Information (URLs) Note: <additional comments> NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Radar Data Analysis for CReSIS Vertical (area) Scientific Research: Polar Science and Remote Sensing of Ice Sheets Author/Company/Email Geoffrey Fox, Indiana University gcf@indiana.edu Actors/Stakeholders and their roles and responsibilities Research funded by NSF and NASA with relevance to near and long term climate change. Engineers designing novel radar with “field expeditions” for 1-2 months to remote sites. Results used by scientists building models and theories involving Ice Sheets Goals Determine the depths of glaciers and snow layers to be fed into higher level scientific Use Case Description Build radar; build UAV or use piloted aircraft; overfly remote sites (Arctic, Antarctic, Himalayas). Check in field that experiments configured correctly with detailed analysis later. Transport data by air-shipping disk as poor Internet connection. Use image processing to find ice/snow sheet depths. Use depths in scientific discovery of melting ice caps etc. Current Solutions Compute(System) Field is a low power cluster of rugged laptops plus classic 2-4 CPU servers with ~40 TB removable disk array. Off line is about 2500 cores Storage Removable disk in field. (Disks suffer in field so 2 copies made) Lustre or equivalent for offline Networking Terrible Internet linking field sites to continental USA.