Big Data Characteristics Data Source (distributed/centralized) Add movies institutionally. Collect user rankings and profiles in a distributed fashion Volume (size) Summer 2012. 25 million subscribers; 4 million ratings per day; 3 million searches per day; 1 billion hours streamed in June 2012. Cloud storage 2 petabytes (June 2013) Velocity (e.g. real time) Media (video and properties) and Rankings continually updated Variety (multiple datasets, mashup) Data varies from digital media to user rankings, user profiles and media properties for recommendations Variability (rate of change) Very competitive business. Need to aware of other companies and trends in both content (which Movies are hot) and technology. Need to investigate new business initiatives such as Netflix sponsored content Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Success of business requires excellent quality of service Visualization Streaming media and quality user-experience to allow choice of content Data Quality Rankings are intrinsically “rough” data and need robust learning algorithms Data Types Media content, user profiles, “bag” of user rankings Data Analytics Recommender systems and streaming video delivery. Recommender systems are always personalized and use logistic/linear regression, elastic nets, matrix factorization, clustering, latent Dirichlet allocation, association rules, gradient boosted decision trees and others. Winner of Netflix competition (to improve ratings by 10%) combined over 100 different algorithms. Big Data Specific Challenges (Gaps) Analytics needs continued monitoring and improvement. Big Data Specific Challenges in Mobility Mobile access important Security & Privacy Requirements Need to preserve privacy for users and digital rights for media. Highlight issues for generalizing this use case (e.g. for ref. architecture) Recommender systems have features in common to e-commerce like Amazon. Streaming video has features in common with other content providing services like iTunes, Google Play, Pandora and Last.fm More Information (URLs) by Xavier Amatriain Note: <additional comments>
know, the location is not updated in real-time, items lost in a warehouse or while in shipment represent a problem potentially for homeland security. The records are retained in an archive and can be accessed for xx days. Current Solutions Compute(System) Unknown Storage Unknown Networking LAN/T1/Internet Web Pages Software Unknown Big Data Characteristics Data Source (distributed/centralized) Centralized today Volume (size) Large Velocity (e.g. real time) The system is not currently real-time. Variety (multiple datasets, mashup) Updated when the driver arrives at the depot and download the time and date the items were picked up. This is currently not real-time. Variability (rate of change) Today the information is updated only when the items that were checked with a bar code scanner are sent to the central server. The location is not currently displayed in real-time. Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) Visualization NONE Data Quality YES Data Types Not Available Data Analytics YES Big Data Specific Challenges (Gaps) Provide more rapid assessment of the identity, location, and conditions of the shipments, provide detailed analytics and location of problems in the system in real-time. Big Data Specific Challenges in Mobility Currently conditions are not monitored on-board trucks, ships, and aircraft Security & Privacy Requirements Security need to be more robust Highlight issues for generalizing this use case (e.g. for ref. architecture) This use case includes local data bases as well as the requirement to synchronize with the central server. This operation would eventually extend to mobile device and on-board systems which can track the location of the items and provide real-time update of the information including the status of the conditions, logging, and alerts to individuals who have a need to know. More Information (URLs) Note: <additional comments> Commercial
Variety (multiple datasets, mashup) Variety and number of involved databases and observation data is currently limited by available tools; in principle, unlimited with the growing ability to process data for identifying ecological changes, factors/reasons, species evolution and trends. See below in additional information. Variability (rate of change) Structure of the datasets and models may change depending on the data processing stage and tasks Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues) In normal monitoring mode are data are statistically processed to achieve robustness. Some biodiversity research are critical to data veracity (reliability/trustworthiness). In case of natural and technogenic disasters data veracity is critical. Visualization Requires advanced and rich visualization, high definition visualisation facilities, visualisation data 4D visualization Visualizing effects of parameter change in (computational) models Comparing model outcomes with actual observations (multi dimensional) Data Quality Depends on and ensued by initial observation data. Quality of analytical data depends on used mode and algorithms that are constantly improved. Repeating data analytics should be possible to initial observation data. Actionable data are human aided. Data Types Multi-type. Relational data, key-value, complex semantically rich data Data Analytics Parallel data streams and streaming analytics Big Data Specific Challenges (Gaps) Variety, multi-type data: SQL and no-SQL, distributed multi-source data. Visualisation, distributed sensor networks. Data storage and archiving, data exchange and integration; data linkage: from the initial observation data to processed data and reported/visualised data. Historical unique data Curated (authorized) reference data (i.e. species names lists), algorithms, software code, workflows Processed (secondary) data serving as input for other researchers Provenance (and persistent identification (PID)) control of data, algorithms, and workflows Big Data Specific Challenges in Mobility Require supporting mobile sensors (e.g. birds migration) and mobile researchers (both for information feed and catalogue search) Instrumented field vehicles, Ships, Planes, Submarines, floating buoys, sensor tagging on organisms Photos, video, sound recording Security & Privacy Requirements Data integrity, referral integrity of the datasets.
Software Custom data processing pipeline and data analysis software, operating under Linux. Some archives on Windows machines, running a MS SQL server databases. Big Data Characteristics Data Source (distributed/centralized) Distributed: Survey data from 3 (soon more?) telescopes Archival data from a variety of resources connected through the VO framework Follow-up observations from separate telescopes Volume (size) The survey generates up to ~ 0.1 TB per clear night; ~ 100 TB in current data holdings. observational data amount to no more than a few % of that. Archival data in external (VO-connected) archives are in PBs, but only a minor fraction is used. Velocity (e.g. real time) Up to ~ 0.1 TB / night of the raw survey data. Variety (multiple datasets, mashup) The primary survey data in the form of images, processed to catalogs of sources (db tables), and time series for individual objects (light curves). Follow-up observations consist of images and spectra. Archival data from the VO data grid include all of the above, from a wide variety of sources and different wavelengths. Variability (rate of change) Daily data traffic fluctuates from ~ 0.01 to ~ 0.1 TB / day, not including major data transfers between the principal archives (Caltech, UA, and IUCAA). Big Data Science (collection, curation, analysis, action) Veracity (Robustness Issues, semantics) A variety of automated and human inspection quality control mechanisms is implemented at all stages of the process. Visualization Standard image display and data plotting packages are used. We are exploring visualization mechanisms for highly dimensional data parameter spaces. Data Quality (syntax) It varies, depending on the observing conditions, and it is evaluated automatically: error bars are estimated for all relevant quantities. Data Types Images, spectra, time series, catalogs. Data Analytics A wide variety of the existing astronomical data analysis tools, plus a large amount of custom developed tools and software, some of it a research project in itself. Big Data Specific Challenges (Gaps) Development of machine learning tools for data exploration, and in particular for an automated, real-time classification of transient events, given the data sparsity and heterogeneity. Effective visualization of hyper-dimensional parameter spaces is a major challenge for all of us. Big Data Specific Challenges in Mobility Not a significant limitation at this time. Security & Privacy None.