Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Exact results not essential but important to get main hubs and authorities for search query Visualization Not important although page lay out critical Data Quality A lot of duplication and spam Data Types Mainly text but more interest in rapidly growing image and video Data Analytics Crawling; searching including topic based search; ranking; recommending Big Data Specific Challenges (Gaps) Search of “deep web” (information behind query front ends) Ranking of responses sensitive to intrinsic value (as in Pagerank) as well as advertising value Link to user profiles and social network data Big Data Specific Challenges in Mobility Mobile search must have similar interfaces/results Security & Privacy Requirements Need to be sensitive to crawling restrictions. Avoid Spam results Highlight issues for generalizing this use case (e.g. for ref. architecture) Relation to Information retrieval such as search of scholarly works. More Information (URLs) http://www.slideshare.net/kleinerperkins/kpcb-internet-trends-2013 http://www.ifis.cs.tu-bs.de/teaching/ss-11/irws http://www.worldwidewebsize.com/ Note: <additional comments> Commercial NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title IaaS (Infrastructure as a Service) Big Data Business Continuity & Disaster Recovery (BC/DR) Within A Cloud Eco-System provided by Cloud Service Providers (CSPs) and Cloud Brokerage Service Providers (CBSPs) Vertical (area) Large Scale Reliable Data Storage Author/Company/Email Pw Carey, Compliance Partners, LLC, pwc.pwcarey@email.com Actors/Stakeholders and their roles and responsibilities Executive Management, Data Custodians, and Employees responsible for the integrity, protection, privacy, confidentiality, availability, safety, security and survivability of a business by ensuring the 3-As of data accessibility to an organizations services are satisfied; anytime, anyplace and on any device. Goals The following represents one approach to developing a workable BC/DR strategy. Prior to outsourcing an organizations BC/DR onto the backs/shoulders of a CSP or CBSP, the organization must perform the following Use Case, which will provide each organization with a baseline methodology for business continuity and disaster recovery (BC/DR) best
starting at the atomistic, nanometer, and/or micrometer level of description. As a result of the above considerations, decisions about materials usage are unnecessarily conservative, often based on older rather than newer materials R&D data, and not taking advantage of advances in modeling and simulations. Materials informatics is an area in which the new tools of data science can have major impact. Current Solutions Compute(System) None Storage Widely dispersed with many barriers to access Networking Virtually none Software Narrow approaches based on national programs (Japan, Korea, and China), applications (EU Nuclear program), proprietary solutions (Granta, etc.) Big Data Characteristics Data Source (distributed/centralized) Extremely distributed with data repositories existing only for a very few fundamental properties Volume (size) It is has been estimated (in the 1980s) that there were over 500,000 commercial materials made in the last fifty years. The last three decades has seen large growth in that number. Velocity (e.g. real time) Computer-designed and theoretically design materials (e.g., nanomaterials) are growing over time Variety (multiple datasets, mashup) Many data sets and virtually no standards for mashups Variability (rate of change) Materials are changing all the time, and new materials data are constantly being generated to describe the new materials Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) More complex material properties can require many (100s?) of independent variables to describe accurately. Virtually no activity no exists that is trying to identify and systematize the collection of these variables to create robust data sets. Visualization Important for materials discovery. Potentially important to understand the dependency of properties on the many independent variables. Virtually unaddressed. Data Quality Except for fundamental data on the structural and thermal properties, data quality is poor or unknown. See Munro’s NIST Standard Practice Guide. Data Types Numbers, graphical, images Data Analytics Empirical and narrow in scope Big Data Specific Challenges (Gaps) Establishing materials data repositories beyond the existing ones that focus on fundamental data Developing internationally-accepted data recording standards that can be used by a very diverse materials community, including developers materials test standards (such as ASTM and ISO), testing companies, materials producers, and R&D labs
Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Important to make as accurate as possible, subject to limitations of computer vision technology. Visualization Visualize large-scale 3-d reconstructions, and navigate large-scale collections of images that have been aligned to maps. Data Quality Features observed in images are quite noisy due both to imperfect feature extraction and to properties of specific images (lens distortions, sensor noise, image effects added by user, etc.) Data Types Images, metadata Data Analytics Big Data Specific Challenges (Gaps) Analytics needs continued monitoring and improvement. Big Data Specific Challenges in Mobility Many/most images are captured by mobile devices; eventual goal is to push reconstruction and organization to phone to allow real-time interaction with the user. Security & Privacy Requirements Need to preserve privacy for users and digital rights for media. Highlight issues for generalizing this use case (e.g. for ref. architecture) Components of this use case including feature extraction, feature matching, and large-scale probabilistic inference appear in many or most computer vision and image processing problems, including recognition, stereo resolution, image denoising, etc. More Information (URLs) http://vision.soic.indiana.edu/disco Note: <additional comments> Deep Learning and Social Media NBD(NIST Big Data) Requirements WG Use Case Template Aug 11 2013 Use Case Title Truthy: Information diffusion research from Twitter Data Vertical (area) Scientific Research: Complex Networks and Systems research Author/Company/Email Filippo Menczer, Indiana University, ; Alessandro Flammini, Indiana University, ; Emilio Ferrara, Indiana University, ; Actors/Stakeholders and their roles and responsibilities Research funded by NFS, DARPA, and McDonnel Foundation. Goals Understanding how communication spreads on socio-technical networks. Detecting potentially harmful information spread at the early stage (e.g., deceiving messages, orchestrated campaigns, untrustworthy information, etc.) Use Case Description (1) Acquisition and storage of a large volume of continuous streaming data from Twitter (~100 million messages per day, ~500GB data/day increasing over time); (2) near real-time analysis of such data, for anomaly detection, stream clustering, signal classification and
Software MPI, OpenMP, C, C++, F90, FFTW, viz packages, python, FFTW, numpy, Boost, OpenMP, ScaLAPCK, PSQL & MySQL databases, Eigen, cfitsio, astrometry.net, and Minuit2 Big Data Characteristics Data Source (distributed/centralized) Observational data will be generated by the Dark Energy Survey (DES) and the Zwicky Transient Factory in 2015 and by the Large Synoptic Sky Survey starting in 2019. Simulated data will generated at DOE supercomputing centers. Volume (size) DES: 4 PB, ZTF 1 PB/year, LSST 7 PB/year, Simulations > 10 PB in 2017 Velocity (e.g. real time) LSST: 20 TB/day Variety (multiple datasets, mashup) 1) Raw Data from sky surveys 2) Processed Image data 3) Simulation data Variability (rate of change) Observations are taken nightly; supporting simulations are run throughout the year, but data can be produced sporadically depending on access to resources Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Visualization and Analytics Interpretation of results from detailed simulations requires advanced analysis and visualization techniques and capabilities. Supercomputer I/O subsystem limitations are forcing researchers to explore “in-situ” analysis to replace post-processing methods. Data Quality Data Types Image data from observations must be reduced and compared with physical quantities derived from simulations. Simulated sky maps must be produced to match observational formats. Big Data Specific Challenges (Gaps) Storage, sharing, and analysis of 10s of PBs of observational and simulated data. Big Data Specific Challenges in Mobility LSST will produce 20 TB of data per day. This must be archived and made available to researchers world-wide. Security & Privacy Requirements Highlight issues for generalizing this use case (e.g. for ref. architecture) More Information (URLs) Note: <additional comments>