Extracting Data from Transfermarkt: An Introduction to WebScraping
Tags: python, begginers, webscraping, beautifulsoup
Published: 2020-07-28 18:04:41

This a translated version of [my tutorial](https://dev.to/lisandramelo/recebendo-informacoes-do-transfermarkt-uma-introducao-ao-web-scraping-188o) originaly published in Brazilian Portuguese. The repository with the code from this tutorial is in my [gitlab profile](https://gitlab.com/veronicamars73/introducao-ao-web-scraping).

Getting data and transforming it into information is the foundation of fields such as Data Science. Sometimes obtaining it is very simple, for example, you can, right now, visit the Brazilian government website [data.gov.br] (http://www.dados.gov.br/) and get access to several raw data files from the government and then perform the analysis of a .csv file (a file format that transmits data) in an easy, simple and fast way.

However, in some situations the data is somewhat difficult to obtain, for example, you may need to receive data that is only available on a web page to perform an analysis. In this situation you can use **Beautiful Soup**, a **Python** library, to perform *web scraping*.

Beautiful Soup is the most popular Python library for receiving web data, it is capable of extracting data from HTML and XML files, it has several methods that make the search for specific data on web pages rather simple an fast.

For this tutorial, we will extract data from the website [Transfermarkt] (https://www.transfermarkt.co.uk/) which is a web plataform that contains news and data about games, transfers, clubs and players from the football/soccer world.

![Transfermarkt Homepage] (https://dev-to-uploads.s3.amazonaws.com/i/4hmpvi3ianfhjw84osrk.png)
*Transfermarkt Homepage*

We will receive the name, country of the previous league and the price of the 25 most expensive players in the history of the AFC Ajax club, this information can be found on the [Transfermarkt page](https://www.transfermarkt.co.uk/ajax-amsterdam/transferrekorde/verein/610/saison_id//pos//detailpos/0/w_s//altersklasse//plus/1/).

![Page which contains the informations about the 25 biggest AFC Ajax signs](https://dev-to-uploads.s3.amazonaws.com/i/xpqwdoj72u781b55hxv4.png)
*Page which contains the informations about the 25 biggest AFC Ajax signs*

## Extracting Data
Before obtaining the data itself, we will import the libraries required for the execution of the program, these will be: *Beautiful Soup*, *Pandas* and *Requests*.

```python
import requests
from bs4 import BeautifulSoup
import pandas as pd
```
After that, we will download the webpage in our program using the `requests` library, which requests the information from the page, and the BeautifulSoup library, which transforms the data received in requests (a `Response` object) into a` BeautifulSoup` object that will be used in data extraction.

```python
"""
To make the request to the page we have to inform the
website that we are a browser and that is why we
use the headers variable
"""
headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

# endereco_da_pagina stands for the data page address
endereco_da_pagina = "https://www.transfermarkt.co.uk/ajax-amsterdam/transferrekorde/verein/610/saison_id//pos//detailpos/0/w_s//altersklasse//plus/1"

# In the objeto_response variable we will the download of the web page
objeto_response = requests.get(endereco_da_pagina, headers=headers)

"""
Now we will create a BeautifulSoup object from our object_response.
The 'html.parser' parameter represents which parser we will use when creating our object,
a parser is a software responsible for converting an entry to a data structure.
"""
pagina_bs = BeautifulSoup(objeto_response.content, 'html.parser')
```
`pagina_bs` is now a variable that contains all the HTML content inside our data page.

Now let's extract the data that is in our variable, note that the information we need is in a table. Each row in this table represents a player, with his name, represented in HTML by an anchor (`<a>`) with the class "spielprofil_tooltip", country of origin league, represented as a flag image (`<img>`) with a class "flaggenrahmen" in the seventh column (`<td>`) of each row, and cost represented by a table cell (`<td>`) of the class "rechts hauptlink"

We will then get this data using the BeautifulSoup library.

First we will get the names of the players.
```python
nomes_jogadores = [] # List that will receive all the players names

# The find_all () method is able to return all tags that meet restrictions within parentheses
tags_jogadores = pagina_bs.find_all("a", {"class": "spielprofil_tooltip"})
# In our case, we are finding all anchors with the class "spielprofil_tooltip"

# Now we will get only the names of all players
for tag_jogador in tags_jogadores:
    nomes_jogadores.append(tag_jogador.text)
```
Now we will get the countries of the players’s previous leagues.

```python
pais_jogadores = [] # List that will receive all the names of the countries of the players’s previous leagues.

tags_ligas = pagina_bs.find_all("td",{"class": None})
# Now we will receive all the cells in the table that have no class atribute set

for tag_liga in tags_ligas:
    # The find() function will find the first image whose class is "flaggenrahmen" and has a title
    imagem_pais = tag_liga.find("img", {"class": "flaggenrahmen"}, {"title":True})
    # The country_image variable will be a structure with all the image information,
    # one of them is the title that contains the name of the country of the flag image
    if(imagem_pais != None): # We will test if we have found any matches than add them
        pais_jogadores.append(imagem_pais['title'])
```
Finally, we will get the players' prices.
```python
custos_jogadores = []

tags_custos = pagina_bs.find_all("td", {"class": "rechts hauptlink"})

for tag_custo in tags_custos:
    texto_preco = tag_custo.text
    # The price text contains characters that we don’t need like £ (euros) and m (million) so we’ll remove them
    texto_preco = texto_preco.replace("£", "").replace("m","")
    # We will now convert the value to a numeric variable (float)
    preco_numerico = float(texto_preco)
    custos_jogadores.append(preco_numerico)
```
Now that we have got all the data we wanted, let's make it understandable to improve any analysis we want to do. For this, we will use the pandas library and its `DataFrame` class, which is a class that represents a tabular data structure, that is, it is similar to a common table.

```python
# Creating a DataFrame with our data
df = pd.DataFrame({"Jogador":nomes_jogadores,"Preço (milhão de euro)":custos_jogadores,"País de Origem":pais_jogadores})

# Printing our gathered data
print(df)
```
Now we can see all our data obtained with `web scraping` organized in the DataFrame!

I hope I have helped in any way and if you have any problems or questions, feel free to leave a comment on this post or send an [email to me](mailto:mendie73@gmail.com) ;).