How working/install Pig with Notebooks?
Tags: bigdata, datascience, opensource, analytics
Published: 2023-01-27 00:55:32

# 🐷📝 Basic commands to work with Pig in Notebooks 
---
## 🔗Related content 
### You can find post related in: 
📀[Google Colab](https://colab.research.google.com/github/xlmriosx/data-projects/blob/main/how-working-install-Pig-with-Notebooks/how_working_install_Pig_with_Notebooks.ipynb) 

### You can find repo related in:
🐱‍🏍[GitHub](https://github.com/xlmriosx/data-projects/tree/main/how-working-install-Pig-with-Notebooks) 

### You can connect with me in:
🧬[LinkedIn](https://www.linkedin.com/in/xlmriosx/) 

--- 

### Resume 🧾

I will install Hadoop with Pig program and will use a library of Python to write a job that answer the question, how many row exists by each rating?

First I install Hadoop using same commands that I have used before but without put a number of step.

---

### Install Hadoop 🐘

I use following command but you can change to get current last version:

`!wget https://downloads.apache.org/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz`

You would can get other version if you need in: https://downloads.apache.org/hadoop/common/ and later replace it in the before command.

---

### Unzip and copy 🔓

I use following command:

`!tar -xzvf hadoop-3.3.4.tar.gz && cp -r hadoop-3.3.4/ /usr/local/`

---

### Set up Hadoop's Java ☕

I use following command:
```
#To find the default Java path and add export in hadoop-env.sh
JAVA_HOME = !readlink -f /usr/bin/java | sed "s:bin/java::"
java_home_text = JAVA_HOME[0]
java_home_text_command = f"$ {JAVA_HOME[0]} "
!echo export JAVA_HOME=$java_home_text >>/usr/local/hadoop-3.3.4/etc/hadoop/hadoop-env.sh
```

---

### Set Hadoop home variables 🏡

I use following command:
```
# Set environment variables
import os
os.environ['HADOOP_HOME']="/usr/local/hadoop-3.3.4"
os.environ['JAVA_HOME']=java_home_text
```

---

### 1st - Install Pig 🐷

I use following command but you can change to get current last version:

`!wget https://downloads.apache.org/pig/pig-0.17.0/pig-0.17.0.tar.gz`

You would can get other version if you need in: https://downloads.apache.org/pig/ and later replace it in the before command.

---

### 2nd - Unzip and copy 🔓

I use following command:

`!tar -xzvf pig-0.17.0.tar.gz`

---

### 3rd - Set Pig home variables 🏡

I use following command:
```
# Set environment variables
import os
os.environ['PIG_HOME']="/content/pig-0.17.0"
os.environ['PIG_CLASSPATH']="/usr/local/hadoop-3.3.1/conf"
os.environ["PATH"] += os.pathsep + "/content/pig-0.17.0/bin"
```

We can validate installation with command:

`!pig -version`

---

### 4th - Create a folder with HDFS 🌎📂

I use following command:

`!/usr/local/hadoop-3.3.4/bin/hadoop fs -mkdir file:///content/data_pig`

---

#### 4.1 - Remove folder with HDFS ♻

Maybe, later you need remove it. To do that you must apply following command:

`!/usr/local/hadoop-3.3.4/bin/hadoop fs -rm -r file:///content/data_pig`

---

### 5th - Getting a dataset to anlyze with Pig 💾

I use a dataset from grouplens. You can get other in:
http://files.grouplens.org/datasets/

This time I use movieslens and you can download it using:

`!wget http://files.grouplens.org/datasets/movielens/ml-100k.zip`


To use data extract files. I extract files in path later of -d in command:

`!unzip "/content/ml-100k.zip" -d "file:///content/data_pig"`

For list them:

`!/usr/local/hadoop-3.3.4/bin/hadoop fs -ls /content/data_pig/ml-100k`

---

### 6th - Creating process to use Pig with Pig Syntax 🐖

To create job in Pig, you must see structure of dataset to configure jobs.
In this case we print dataset with following command:

`!head /content/data_pig/ml-100k/u.data`

I can get following information of dataset:

- First column reference to userID.
- Second column reference to movieID.
- Third column reference to rating.
- Fourth column reference to timestamp.

```
# Create pig script
%%writefile id.pig
/* id.pig */

student = LOAD 'file:///content/data_pig/ml-100k/u.data' USING PigStorage('	')
   as (userId:int, movieId:int, rating:int, timestamp:int);
	
student_order = ORDER student BY rating DESC;
  
Dump student_order;
```

---

### 7th - Running the process 🙈

Here we run the process specifing some parameters:
- Pig file program is `id.pig `
- Dataset is in `file:///content/data_pig/ml-100k/u.data`

When run process, maybe take a few minutes...

You can run script with:

`!pig -x local id.pig`

But we run script and save results in a file .txt:

`!pig -x local id.pig > results.txt`

---

### 8th - Advancing in the logic of the scripts 😎

Now we will advance in logic of the script to get answer to next questions:
- What are the oldest 5 star movies?
- What are the worst movies?

---

#### 8.1 - Find oldest 5 star movies start ⭐

```
%%writefile fiveStarMovies.pig

ratings = LOAD 'file:///content/data_pig/ml-100k/u.data'
	AS (userID:int, movieID:int, rating:int, ratingTime:int);
metadata = LOAD 'file:///content/data_pig/ml-100k/u.item' USING PigStorage('|')
	AS (movieID:int, movieTitle:chararray, releaseDate:chararray, videoRealese:chararray, imdblink:chararray);
   
nameLookup = FOREACH metadata GENERATE movieID, movieTitle,	ToUnixTime(ToDate(releaseDate, 'dd-MMM-yyyy')) AS releaseTime;
   
ratingsByMovie = GROUP ratings BY movieID;

avgRatings = FOREACH ratingsByMovie GENERATE group as movieID, AVG(ratings.rating) as avgRating;

fiveStarMovies = FILTER avgRatings BY avgRating > 4.0;

fiveStarsWithData = JOIN fiveStarMovies BY movieID, nameLookup BY movieID;

oldestFiveStarMovies = ORDER fiveStarsWithData BY nameLookup::releaseTime;

DUMP oldestFiveStarMovies;
```

Run script and save results in a file .txt:

`!pig -x local fiveStarMovies.pig > fiveStarMovies.txt`

---

#### 8.2 - Find most rated bad movies ⭐

```
%%writefile BadPopularMovies.pig

ratings = LOAD 'file:///content/data_pig/ml-100k/u.data'
  AS (userID:int, movieID:int, rating:int, ratingTime:int);
metadata = LOAD 'file:///content/data_pig/ml-100k/u.item' USING PigStorage('|')
	AS (movieID:int, movieTitle:chararray, releaseDate:chararray, videoRealese:chararray, imdblink:chararray);
   
nameLookup = FOREACH metadata GENERATE movieID, movieTitle;

groupedRating = GROUP ratings by movieID;

avgRatings = FOREACH groupedRating GENERATE group as movieID, AVG(ratings.rating) as avgRating, COUNT(ratings.rating) AS numRatings;  

badMovies = FILTER avgRatings BY avgRating < 2.0;

namedBadMovies = JOIN badMovies BY movieID, nameLookup BY movieID;

results = FOREACH namedBadMovies GENERATE nameLookup::movieTitle as movieName,
          badMovies::avgRating as avgRating, badMovies::numRatings as numRatings;

finalResults = ORDER results BY numRatings DESC;

DUMP finalResults;
```

Run script and save results in a file .txt:

`!pig -x local BadPopularMovies.pig > BadPopularMovies.txt`

---

### 9th - Say thanks, give like and share if this has been of help/interest 😁🖖

---