Spark programming basics (Python version)
Tags: awscommunity, spark, python, hadoop
Published: 2022-03-29 08:20:15

# [1.Master the installation and environment configuration of spark](#1)
# [2.Master the installation and environment configuration of spark](#2)
# [3.Master Python version management and third-party installation under Ubuntu](#3)
# [4.Master the synchronous connection between pychar and Ubuntu under Windows](#4)
# [5.Master the data read by spark from the file system](#5)
### Reference website:
[https://spark.apache.org/docs/1.1.1/quick-start.html](https://spark.apache.org/docs/1.1.1/quick-start.html)
<h2 id="1">1.Write first: experimental environment</h2>
#### You Must be have Hadoop environment,You can read my other blog:[Click Here](https://dev.to/leifengflying/build-hadoop-2x-fully-distributed-environment-by-ubuntu-1604-173c)

```bash

Operating system: Ubuntu 16 04；
Spark version: 2.4.6;
Hadoop version: 2.7.2.
Python version: 3.5.
```
Click Here Download:[spark-2.4.6-bin-without-hadoop.tgz](https://archive.apache.org/dist/spark/spark-2.4.6/spark-2.4.6-bin-without-hadoop.tgz)
![在这里插入图片描述](https://img-blog.csdnimg.cn/614428ffedd94c15b8815a11f0c76155.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

<h2 id="2">2.Master the installation and environment configuration of spark</h2>
#### 1. Unzip the spark package and move it

```powershell
ubuntu@adserver:~$ tar zxf spark-2.4.6-bin-without-hadoop.tgz 
ubuntu@adserver:~$ ls -lh
total 536M
-rw-rw-r--  1 ubuntu ubuntu 203M Mar 28 09:59 hadoop-2.7.2.tar.gz
-rw-rw-r--  1 ubuntu ubuntu 173M Mar 28 09:11 jdk-8u77-linux-x64.tar.gz
-rw-rw-r--  1 ubuntu ubuntu  458 Mar 28 13:31 jps
drwxr-xr-x 13 ubuntu ubuntu 4.0K May 30  2020 spark-2.4.6-bin-without-hadoop
-rw-rw-r--  1 ubuntu ubuntu 161M Mar 26 19:34 spark-2.4.6-bin-without-hadoop.tgz
ubuntu@adserver:~$ sudo mv spark-2.4.6-bin-without-hadoop /usr/local/
ubuntu@adserver:~$ cd /usr/local/
ubuntu@adserver:/usr/local$ sudo mv spark-2.4.6-bin-without-hadoop/ spark
ubuntu@adserver:/usr/local$ ls -lh spark/
```

![在这里插入图片描述](https://img-blog.csdnimg.cn/a5201436beb246f8a25fe6cded6c7af5.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

#### 2.Modify spark environment variable file

```powershell
ubuntu@adserver:~$ cd /usr/local/spark/conf/
ubuntu@adserver:/usr/local/spark/conf$ pwd
/usr/local/spark/conf
ubuntu@adserver:/usr/local/spark/conf$ cp spark-env.sh.template spark-env.sh
ubuntu@adserver:/usr/local/spark/conf$ vi spark-env.sh

```
Edit spark env SH file (vi ./conf/spark-env.sh), add the following configuration information on the first line:

```powershell
export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop-2.7.2/bin/hadoop classpath)
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/a8a0a38d8ba6455ea1a0e9f0ffb76cef.png)
With the above configuration information, spark can store data in Hadoop distributed file system HDFS or read data from HDFS. If the above information is not configured, spark can only read and write local data, but cannot read and write HDFS data. After configuration, it can be used directly without running the startup command like Hadoop. Verify whether spark is successfully installed by running the example provided by spark.
During execution, a lot of running information will be output, and the output result is not easy to find. It can be filtered through the grep command (2 > &amp; 1 in the command can output all information to stdout, otherwise it will be output to the screen due to the nature of the output log):

```powershell
ubuntu@adserver:/usr/local/spark$ ./bin/run-example SparkPi 2>&1 | grep "Pi is"
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/d062eeb42e374a128bae0b92ec105d59.png)
Modify the contents of /usr/local/spark/bin/pyspark file
Change line 45 Python to python3
![在这里插入图片描述](https://img-blog.csdnimg.cn/28ec23d4410047b88f0a62fc8677c36a.png)
Execute command`sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 150`

```powershell
ubuntu@adserver:/usr/local/spark/bin$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 150
```

![在这里插入图片描述](https://img-blog.csdnimg.cn/c38d98bbf6be4be583b722510cf528ea.png)




<h2 id="3">3.Master Python version management and third-party installation under Ubuntu</h2>

```powershell
whereis python3  #  Confirm Python 3 directory
cd  /usr/lib/python3.5   # Switch directory 
sudo apt-get install python3-pip   #  install pip 
sudo pip3 install -i https://pypi.doubanio.com/simple matplotlib  #  install matplotlib 
```

```powershell
ubuntu@adserver:~$ whereis python3
python3: /usr/bin/python3 /usr/bin/python3.5 /usr/bin/python3.5m /usr/lib/python3 /usr/lib/python3.5 /etc/python3 /etc/python3.5 /usr/local/lib/python3.5 /usr/share/python3 /usr/share/man/man1/python3.1.gz
ubuntu@adserver:~$ cd /usr/lib/python3.5/
ubuntu@adserver:/usr/lib/python3.5$ sudo apt-get install python3-pip
ubuntu@adserver:/usr/lib/python3.5$ sudo pip3 install -i https://pypi.doubanio.com/simple matplotlib
```

![在这里插入图片描述](https://img-blog.csdnimg.cn/22abd7f6f86c4f988a576f60cc0a3fd4.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)
![在这里插入图片描述](https://img-blog.csdnimg.cn/977f1089aaa94ded8ed776752a958fac.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)





<h2 id="4">4.Master the synchronous connection between pychar and Ubuntu under Windows</h2>
#### 1) Open pychart and open file -- > settings -- >
Click the + sign, and then select SSH interpreter to set the server; Enter the IP address, user name and password of the virtual machine Ubuntu
![在这里插入图片描述](https://img-blog.csdnimg.cn/e425647624284797be7c0820f7e74a48.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)
![在这里插入图片描述](https://img-blog.csdnimg.cn/bfabc1ec96e445eaa2d7dd789e447d9d.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)



<h2 id="5">5.Master the data read by spark from the file system</h2>

#### 1）Read the local file "/home/Hadoop/test.txt" of Linux system in pyspark, and then count the number of lines of the file;
First create the test file

```powershell
$ vi /home/hadoop/test.txt
this is line 1
this is line 2
this is line 3
this is line 4
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/4ee8312e9ac148c3b9dfce8795cd5e7d.png)

```powershell
ubuntu@adserver:/usr/local/spark/bin$ pwd
/usr/local/spark/bin
ubuntu@adserver:/usr/local/spark/bin$ ./pyspark 
Python 3.5.2 (default, Jan 26 2021, 13:30:48) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
22/03/28 15:57:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.4.6
      /_/

Using Python version 3.5.2 (default, Jan 26 2021 13:30:48)
SparkSession available as 'spark'.
>>> lines = sc.textFile("file:/home/hadoop/test.txt")
>>> lines.count()
4
>>> 
```

![在这里插入图片描述](https://img-blog.csdnimg.cn/079959dc452a4f3f8d05b397790c0ba2.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)






#### 2）Read the HDFS system file "/user/Hadoop/test.txt" (if the file does not exist, please create it first), and then count the number of lines of the file;

```powershell
ubuntu@adserver:~$ cd /home/hadoop/
ubuntu@adserver:/home/hadoop$ hadoop fs -ls /
Found 1 items
drwxrwx---   - ubuntu supergroup          0 2022-03-28 17:15 /tmp
ubuntu@adserver:/home/hadoop$ hadoop fs -mkdir -p /user/hadoop
ubuntu@adserver:/home/hadoop$ hadoop fs -put test.txt /user/hadoop/
ubuntu@adserver:/home/hadoop$ hadoop fs -ls /user/hadoop/
Found 1 items
-rw-r--r--   3 ubuntu supergroup         60 2022-03-28 17:17 /user/hadoop/test.txt
ubuntu@adserver:/home/hadoop$ /usr/local/spark/bin/pyspark 
>>> lines = sc.textFile("/user/hadoop/test.txt")     
>>> lines.count()
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/90186245eaa4467cba37344156a5e2c3.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)






#### 3）Write an independent application, read the HDFS system file "/user/Hadoop/test.txt" (if the file does not exist, please create it first), and then count the number of lines of the file;

```python
ubuntu@adserver:/home/hadoop$ sudo vi st-app.py
  
from pyspark import SparkContext

logFile = "/user/hadoop/test.txt"  # Should be some file on your system
sc = SparkContext("local", "Simple App")
logData = sc.textFile(logFile).cache()
print("The HDFS file lines : ",logData.count())
```

![在这里插入图片描述](https://img-blog.csdnimg.cn/048b244380e5497181b086f13023da88.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

```powershell
ubuntu@adserver:/home/hadoop$ /usr/local/spark/bin/spark-submit --master local[4] st-app.py 2>&1 | grep "The HDFS"
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/3da6ffed374649a7ac05ca35cc19dd37.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

# Hey,If you think this blog is useful, please like it and Follow Me. Thank you.Have a nice day!!!!!!