raw/raw_bsc__introduction_to_big_data.md

BSc: Introduction To Big Data
=============================






Contents
--------


* [1 Introduction to Big Data](#Introduction_to_Big_Data)
	+ [1.1 Short Description](#Short_Description)
	+ [1.2 Prerequisites](#Prerequisites)
		- [1.2.1 Prerequisite subjects](#Prerequisite_subjects)
		- [1.2.2 Prerequisite topics](#Prerequisite_topics)
	+ [1.3 Course Topics](#Course_Topics)
	+ [1.4 Intended Learning Outcomes (ILOs)](#Intended_Learning_Outcomes_.28ILOs.29)
		- [1.4.1 What is the main purpose of this course?](#What_is_the_main_purpose_of_this_course.3F)
		- [1.4.2 ILOs defined at three levels](#ILOs_defined_at_three_levels)
			* [1.4.2.1 Level 1: What concepts should a student know/remember/explain?](#Level_1:_What_concepts_should_a_student_know.2Fremember.2Fexplain.3F)
			* [1.4.2.2 Level 2: What basic practical skills should a student be able to perform?](#Level_2:_What_basic_practical_skills_should_a_student_be_able_to_perform.3F)
			* [1.4.2.3 Level 3: What complex comprehensive skills should a student be able to apply in real-life scenarios?](#Level_3:_What_complex_comprehensive_skills_should_a_student_be_able_to_apply_in_real-life_scenarios.3F)
	+ [1.5 Grading](#Grading)
		- [1.5.1 Course grading range](#Course_grading_range)
		- [1.5.2 Course activities and grading breakdown](#Course_activities_and_grading_breakdown)
		- [1.5.3 Recommendations for students on how to succeed in the course](#Recommendations_for_students_on_how_to_succeed_in_the_course)
	+ [1.6 Resources, literature and reference materials](#Resources.2C_literature_and_reference_materials)
		- [1.6.1 Open access resources](#Open_access_resources)
		- [1.6.2 Closed access resources](#Closed_access_resources)
		- [1.6.3 Software and tools used within the course](#Software_and_tools_used_within_the_course)
* [2 Teaching Methodology: Methods, techniques, & activities](#Teaching_Methodology:_Methods.2C_techniques.2C_.26_activities)
	+ [2.1 Activities and Teaching Methods](#Activities_and_Teaching_Methods)
	+ [2.2 Formative Assessment and Course Activities](#Formative_Assessment_and_Course_Activities)
		- [2.2.1 Ongoing performance assessment](#Ongoing_performance_assessment)
			* [2.2.1.1 Section 1](#Section_1)
			* [2.2.1.2 Section 2](#Section_2)
			* [2.2.1.3 Section 3](#Section_3)
			* [2.2.1.4 Section 4](#Section_4)
			* [2.2.1.5 Section 5](#Section_5)
			* [2.2.1.6 Section 6](#Section_6)
			* [2.2.1.7 Section 7](#Section_7)
		- [2.2.2 Final assessment](#Final_assessment)
		- [2.2.3 The retake exam](#The_retake_exam)



Introduction to Big Data
========================


* **Course name**: Introduction to Big Data
* **Code discipline**: N/A
* **Subject area**:


Short Description
-----------------


This course covers the following concepts: Distributed data organization; Distributed data processing.



Prerequisites
-------------


### Prerequisite subjects


### Prerequisite topics


Course Topics
-------------




Course Sections and Topics
| Section | Topics within the section
 |
| --- | --- |
| Introduction | 1. What is Big Data
2. Characteristics of Big Data
3. Data Structures
4. Types of Analytics
 |
| Hadoop | 1. Data storage
2. Clustering
3. Design decisions
4. Scaling
5. Distributed systems
6. The ecosystem
 |
| HDFS | 1. Distributed storage
2. Types of nodes
3. Files and blocks
4. Replication
5. Memory usage
 |
| MapReduce | 1. Distributed processing
2. MapReduce model
3. Applications
4. Tasks management
5. Patterns
 |
| YARN | 1. Resource manager
2. Components
3. Run an application
4. Schedules
 |
| Optimizing Data Processing | 1. CAP theorem
2. Distributed storage and computation
3. Batch Processing
4. Stream Processing
5. Usage patterns
6. NoSQL databases
 |
| Spark | 1. Architecture
2. Use cases
3. Job scheduling
4. Data types
5. SparkML
6. GraphX
 |


Intended Learning Outcomes (ILOs)
---------------------------------


### What is the main purpose of this course?


Software systems are increasingly based on large amount of data that come from a wide range of sources (e.g., logs, sensors, user-generated content, etc.). However, data are useful only if it can be analyzed properly to extract meaningful information can be used (e.g., to take decisions, to make predictions, etc.). This course provides an overview of the state-of-the-art technologies, tools, architectures, and systems constituting the big data computing solutions landscape. Particular attention will be given to the Hadoop ecosystem that is widely adopted in the industry.



### ILOs defined at three levels


#### Level 1: What concepts should a student know/remember/explain?


By the end of the course, the students should be able to ...



* The most common structures of distributed storage.
* Batch processing techniques
* Stream processing techniques
* Basic distributed data processing algorithms
* Basic tools to address specific processing needs


#### Level 2: What basic practical skills should a student be able to perform?


By the end of the course, the students should be able to ...



* The basis of the CAP theorem
* The structure of the MapReduce
* How to process batch data
* How to process stream data
* The characteristics of a NoSQL database


#### Level 3: What complex comprehensive skills should a student be able to apply in real-life scenarios?


By the end of the course, the students should be able to ...



* Use a NoSQL database
* Write a program for batch processing
* Write a program for stream processing


Grading
-------


### Course grading range





| Grade | Range | Description of performance
 |
| --- | --- | --- |
| A. Excellent | 90-100 | -
 |
| B. Good | 75-89 | -
 |
| C. Satisfactory | 60-74 | -
 |
| D. Poor | 0-59 | -
 |


### Course activities and grading breakdown





| Activity Type | Percentage of the overall course grade
 |
| --- | --- |
| Labs/seminar classes | 30
 |
| Interim performance assessment | 30
 |
| Exams | 40
 |


### Recommendations for students on how to succeed in the course


Resources, literature and reference materials
---------------------------------------------


### Open access resources


* Slides and material provided during the course.
* Vignesh Prajapati. Big Data Analytics with R and Hadoop. Packt Publishing, 2013
* Jules J. Berman. Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2013


### Closed access resources


### Software and tools used within the course


Teaching Methodology: Methods, techniques, & activities
=======================================================


Activities and Teaching Methods
-------------------------------




Activities within each section
| Learning Activities | Section 1 | Section 2 | Section 3 | Section 4 | Section 5 | Section 6 | Section 7
 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Testing (written or computer based) | 1 | 1 | 1 | 1 | 1 | 1 | 1
 |
| Discussions | 1 | 1 | 1 | 1 | 1 | 1 | 1
 |
| Development of individual parts of software product code | 0 | 0 | 1 | 1 | 1 | 1 | 1
 |
| Homework and group projects | 0 | 0 | 1 | 1 | 1 | 1 | 1
 |
| Midterm evaluation | 0 | 0 | 1 | 1 | 1 | 1 | 1
 |


Formative Assessment and Course Activities
------------------------------------------


### Ongoing performance assessment


#### Section 1





| Activity Type | Content | Is Graded?
 |
| --- | --- | --- |
| Question | Describe the 6 Vs | 1
 |
| Question | Describe the types of analytics | 1
 |
| Question | Design the structure of a DB to address a specific analytics type | 0
 |
| Question | Give examples of the 6 Vs in real systems | 0
 |


#### Section 2





| Activity Type | Content | Is Graded?
 |
| --- | --- | --- |
| Question | Describe the Hadoop ecosystem | 1
 |
| Question | Structure of an Hadoop cluster | 1
 |
| Question | Describe the scaling techniques | 1
 |
| Question | Configure a basic Hadoop node | 0
 |
| Question | Configure a basic Hadoop cluster | 0
 |


#### Section 3





| Activity Type | Content | Is Graded?
 |
| --- | --- | --- |
| Question | Describe the characteristics of the different nodes | 1
 |
| Question | How files and blocks are managed | 1
 |
| Question | How memory is managed | 1
 |
| Question | How replication works | 1
 |
| Question | Configure a HDFS cluster | 0
 |
| Question | Configure different replication approaches | 0
 |
| Question | Build a HDFS client | 0
 |
| Question | Use a HDFS command line | 0
 |


#### Section 4





| Activity Type | Content | Is Graded?
 |
| --- | --- | --- |
| Question | Describe the MapReduce model | 1
 |
| Question | Describe tasks management | 1
 |
| Question | Describe patterns of usage | 1
 |
| Question | Solve with MapReduce a specific problem | 0
 |
| Question | Implement a usage pattern | 0
 |


#### Section 5





| Activity Type | Content | Is Graded?
 |
| --- | --- | --- |
| Question | Describe the resource manager | 1
 |
| Question | Describe the lifecycle of an application | 1
 |
| Question | Describe and compare the scheduling approaches | 1
 |
| Question | Compare the performance of the different schedules in different load conditions | 0
 |
| Question | Configure YARN | 0
 |
| Question | Evaluate the overall performance of YARN | 0
 |


#### Section 6





| Activity Type | Content | Is Graded?
 |
| --- | --- | --- |
| Question | Analyze the CAP theorem | 1
 |
| Question | Define the kinds of data storage available | 1
 |
| Question | Characteristics of batch processing | 1
 |
| Question | Characteristics of stream processing | 1
 |
| Question | Describe the usage patterns | 1
 |
| Question | Compare NoSQL databases | 1
 |
| Question | Build a program to solve a problem with batch processing | 0
 |
| Question | Build a program to solve a problem with stream processing | 0
 |
| Question | Interact with a NoSQL database | 0
 |


#### Section 7





| Activity Type | Content | Is Graded?
 |
| --- | --- | --- |
| Question | Describe the architecture of Spark | 1
 |
| Question | Describe the types of schedulers | 1
 |
| Question | Different characteristics of the data types | 1
 |
| Question | Features of SparkML | 1
 |
| Question | Features of GraphX | 1
 |
| Question | Analyze the performance of different schedulers | 0
 |
| Question | Write a program exploiting the features of each data type | 0
 |
| Question | Write a program using SparkML | 0
 |
| Question | Write a program using GraphX | 0
 |


### Final assessment


**Section 1**



1. Design the structure of a DB to address a specific analytics type
2. Give examples of the 6 Vs in real systems


**Section 2**



1. Identify the Hadoop components useful to address a specific problem.
2. Configure an multi-node Hadoop system.


**Section 3**



1. Configure a HDFS cluster with some specific replication approaches
2. Build a HDFS client


**Section 4**



1. Describe the advantages and disadvantages of the MapReduce model
2. Solve a task designing the solution using MapReduce
3. Solve a task designing the solution using a composition of usage patterns


**Section 5**



1. Evaluate the performance of a specific configuration
2. Compare the different schedules


**Section 6**



1. Identify problems and solutions related to the CAP theorem
2. Compare solutions with batch and stream processing approaches
3. Design a system using a NoSQL database


**Section 7**



1. Compare the performance of different schedules with different loads
2. Extend the SparkML library with a custom algorithm
3. Extend the GraphX library with a custom algorithm


### The retake exam


**Section 1**


**Section 2**


**Section 3**


**Section 4**


**Section 5**


**Section 6**


**Section 7**