Scrape Github User details with Python. 
Tags: python, github, scrape, watercooler
Published: 2020-12-13 09:33:41

When i was learning web scraping , one of the ideas that came to my mind is a Github Scraper.
Here i will try my best to describe each process.
## __Lets start..__
We have to install a couple of packages first.
* Beautifulsoup
* requests
* htmlparser
```python
pip install requests
pip install html5lib
pip install beautifulsoup4
```

***

* Then open [https://github.com/yourusername]()
* Open Devtools.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/jzo4rgh3z7n34ub8yb2c.png)
* This is what i see when i open my dashboard and devtools.
* While we scrape web , we need the element's id ,classname or xpath to scrape it.

* We will be scraping Name, Username , No of Repos, Followers , Following and profile image.

```python
import requests
from bs4 import BeautifulSoup
import html5lib
```
* Import the modules.

***

```python
r=requests.get("https://github.com/fredysomy")
soup=BeautifulSoup(r.content,'html5lib')
```
* Make a request into the website.
* Parse the html recieved as response in ``r.content`` using beautifulsoup and html5lib.

* From here we are starting scraping.

***

```python
namediv=soup.find("h1" ,class_="vcard-names pl-2 pl-md-0")
name=namediv.find_all('span')[0].getText()
u_name=namediv.find_all('span')[1].getText()
```
* Here we are getting all element in the element of class name ``vcard-names pl-2 pl-md-0"``
* Name and Username are in the span element in the above div.
* We have assigned the content into namediv variable.
* We are finding all span elements and selecting (0:name,1:Username) and getting the text using getText() function.

***


```python
statstab=soup.find(class_="flex-order-1 flex-md-order-none mt-2 mt-md-0")
elements=statstab.find(class_="mb-3")
followers=elements.find_all('a')[0].find('span').getText().strip(' ')
following=elements.find_all('a')[1].find('span').getText().strip(' ')
totstars=elements.find_all('a')[2].find('span').getText().strip(' ')
```

* Here the same thing happens.
* Followers,Following,Stargazers are inside element of classname ``flex-order-1 flex-md-order-none mt-2 mt-md-0`` and in  ``mb-3`` which is inside that.

* Lets get that and store it in elements variable.
* Getting the span inside <a></a> inside the elements returns a list.
  * Followers is having index=0
  * Following is having index=1
  * Stargazer is having index=2
```python
elements.find_all('a')[2].find('span').getText().strip(' ')
```
* Here we are getting the second index item in a element  and then ``getText()`` from the span inside it. We are using ``strip('')`` to remove unneccesary blank spaces in the result.
```python
u_img=soup.find(class_="avatar avatar-user width-full border bg-white")['src']
```
* The above code gives the image tag and we are getting the src attribute.

```python
repo_num=soup.find(class_="UnderlineNav-body").find('span',class_="Counter").getText()
```
* Here we are getting the no of repos user haves.

* That is all you need to scrape user details with python.
### __Source Code__
```python
import requests
from bs4 import BeautifulSoup
import html5lib
r=requests.get("https://github.com/fredysomy")
soup=BeautifulSoup(r.content,'html5lib')
namediv=soup.find("h1" ,class_="vcard-names pl-2 pl-md-0")
name=namediv.find_all('span')[0].getText()
u_name=namediv.find_all('span')[1].getText()
statstab=soup.find(class_="flex-order-1 flex-md-order-none mt-2 mt-md-0")
elements=statstab.find(class_="mb-3")
followers=elements.find_all('a')[0].find('span').getText().strip(' ')
following=elements.find_all('a')[1].find('span').getText().strip(' ')
totstars=elements.find_all('a')[2].find('span').getText().strip(' ')
u_img=soup.find(class_="avatar avatar-user width-full border bg-white")['src']
repo_num=soup.find(class_="UnderlineNav-body").find('span',class_="Counter").getText()
```
* The idea is that, we should make the program to navigate to the element we want and select the required element.
* Refer some beautifulsoup methods [here](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)

* I have also made a pypi module to scrape Github.See it [here](https://github.com/fredysomy/gitscrpy) and give a star if you like it.

__If you have any doubts or need clarification, comment down below.__

__Stay tuned for part 2 where we will scrape the user repo details.__