NIST Big Data Reference Architecture DRAFT Version 0.5 Reference Architecture Subgroup NIST Big Data Working Group (NBD-WG) September, 2013 Version Date Changes abstract References Editor 0.1 9/6/13 Outline and Diagram M0202v1 Orit 0.2 9/11/13 Text from input docs M0142v3, M0189v1, M0015v1, M0126v4, M0231v1 Orit 0.3 - Additions to the diagram based on a weekly conference call - Clarification to the RA model based on email conversation - Intro text is added M0230v2, M0239v1 Orit 0.4 - Security and Management are shown as the fabrics around all blocks - Transformation sub-blocks are shown with same color - Editorial changes to Section 3 (as an input to roadmap) Weekly conf call. M0230v3 Orit
Data Provider can also provide an abstraction of data being transformed earlier by another system that can be either a legacy system or another big data system. In this case, Data Provider would represent a Data Consumer of that other system. For example, a (big) streaming data source could be generated by another system operation on (big) data at rest. Data Provider activities include: Creating the metadata describing the data source(s), usage policies/access rights, and other relevant attributes Publishing the availability of the information and the means to access it Making the data accessible by other RA components using suitable programmable interface. Data Service Abstraction The Data Service Abstraction would typically include a registry so that the transformation functions can locate a Data provider, identify what comparable data it contains, understand what types of access allowed, understand what types of analysis are supported, where the data source is located, how to access the data, security requirements for the data, privacy requirements for the data, etc. As such, the interface would include the means for registering the data source, for querying the registry, and a standard set of data contained by the registry. Because the data can be too large to economically move across the network, the interface could also allow submi analysis request (as a software code implementing a certain algorithm for execution) the results returned to the requestor. Subject to data characteristics (such as, volume, velocity, and variety) and system design considerations, interfaces for exposing and accessing data would vary in their complexity and can include both push and pull software mechanisms. These mechanisms can include subscription to events, listening to data feeds, querying for specific data properties or content, and the ability to submit a code for execution to process the data in situ. Note that not all use of data service abstraction would always be automated, but rather might involve a human role logging into the system and providing directions where new data should be transferred (for example, via FTP). Transformation Provider Transformation Provider is the role of executing a generic “vertical system” data life cycle including data collection from various sources, multiple data transformations being implemented using both traditional and new technologies, and diverse data usage. The Transformation functions would typically be specific to the vertical application and therefore are not candidates for standardization. However, the metadata and the policies defined and exchanged between the transformation blocks could be standardized.
As the data propagates through the ecosystem, it is being processed and transformed in different ways in order to extract the value from the information. Transformation sub-components can be implemented by independent stakeholders and deployed as stand-alone services. Each transformation function can use different specialized data infrastructure or capabilities best for its requirements, and can have its own privacy and other policy considerations. In its role, Transformation Provider typically executes the manipulations of the data lifecycle of a specific vertical system to meet the requirements or instructions established by the System Orchestrator. [Editor’s Note: Listed activities need to be aligned with the sub-components shown on the diagram.] Transformation Provider activities include: Data Collecti (connect, transport, stage): obtains connection to Data Provider APIs to collect into local system, or to access dynamically when requested. At the initial collection stage, sets of data (e.g., data records) of similar structure are collected (and combined) resulting in uniform security considerations, policies, etc. Initial metadata is created (e.g., subjects with keys are identified) to facilitate subsequent aggregation or lookup method(s). Data Curati: provides cleansing, outlier removal, standardization for the ingestion and storage processes. Data Aggregati: Data from different data providers with easily correlated metadata (e.g., identical keys) is aggregated into a single data set. As a result, the information about each object is enriched or the number of objects in the collection grows. Security considerations and policies concerning the resultant data set are typically similar to the original data. Data Matching: Data from different data providers with disparate metadata (e.g., keys) are aggregated into a single data set. As a result, the information about each object is enriched. The security considerations and policies concerning the resultant data set may differ from the original policies. Data Optimizi (Pre-analytics): determines the appropriate data manipulations and indexes to optimize subsequent transformation processes. Data Analysis: implements the techniques to extract knowledge from the data based on the requirements of the data scientist, who has specified the algorithms to process the data to produce new insights that will address the technical goal. Data Transfer: facilitates secure transfer of data between different repositories and/or between the Transformation and the Capabilities RA blocks. System Service Abstraction TBD
[7] Gartner, “The Importance of 'Big Data': A Definition”, [8] Hilbert, Martin and Lopez, Priscilla, “The World’s Technological Capacity to Store, Communicate, and Compute Information”, Science, 01 April 2011 [9] Department of Defense, “Reference Architecture Description”, June 2010, [10] Rechtin, Eberhardt, “The Art of Systems Architecting”, CRC Press; 3rd edition, 06 January 2009 [11] ISO/IEC/IEEE 42010 Systems and software engineering — Architecture description, 24 November 2011,