How to Scrape Data from Websites in C#

Published: 2022-02-11 06:26:05

---
canonical_url: https://ironsoftware.com/csharp/webscraper/tutorials/webscraping-in-c-sharp/
---

IronWebscraper is a .Net Library for web scraping, web data extraction, and web content parsing. It is an easy to use library that can be added to Microsoft Visual Studio projects for use in development and production.

IronWebscraper has lots of unique features and capabilities such as controlling allowed and prohibited pages, objects, media, etc. It also allows management of multiple identities, web cache and lots of other features we will discuss in this tutorial document.

## Target Audience

This tutorial targets software developers with basic or advanced programming skills, who wish to build and implement solutions for advanced scraping capabilities (web sites scraping, website data gathering and extraction, websites contents parsing, web harvesting).

![Image 1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ilz6fm79du8hi0js3i8a.png)

### Skills required

1. Basic fundamentals of programming with skills using one of Microsoft Programming languages such as C# or VB.NET
2. Basic understand of Web Technologies (HTML, JavaScript, JQuery, CSS, etc.) and how they work
3. Basic knowledge of DOM, XPath, HTML and CSS Selectors

### Tools

1. Microsoft Visual Studio 2010 or above
2. Web developer extensions for browsers such as web inspector for Chrome or Firebug for Firefox

## Why Scrape? (Reasons and Concepts)

If you want to build a product or solution that has the capabilities to:

1. Extract web site data
2. Compare contents, prices, features, etc. from multiple web sites
3. Scanning and caching websites content

If you have one or more reasons from the above, then IronWebscraper is a great library to fit your needs

## How to Install IronWebScraper?

After you Create a New Project (See Appendix A) you can add IronWebScraper library to your project by automatically inserting the library using NuGet or by Manually installing the DLL.

### Install using NuGet

To add IronWebScraper library to our project using NuGet we can do it using the visual interface (NuGet Package Manager) or by command using the Package Manager Console.

**Using NuGet Package Manager**

1. Using mouse -> right click on project name -> Select manage NuGet Package

![Image 2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w6xdspatnvnklpse1p8b.png)

2. From browse tab -> search for IronWebScraper -> Install

![Image 3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cyvtxfmld66y25g2ja6v.png)
 
3. Click Ok

![Image 4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7bz9a7m9yf68ms94mbxk.png)
 
4. And we are Done

![Image 5](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/60bq33e5h70y2xpkyzx9.png)

**Using NuGet Package Console**

1. From tools -> NuGet Package Manager -> Package Manager Console

![Image 6](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/isynlurmhvf52dnqygd5.png)

2. Choose Class Library Project as Default Project

3. Run command -> Install-Package IronWebScraper

![Image 7](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9d4ifu8iyj1se9f73o85.png)
 
### Install Manually

1. Go to [ironsoftware.com](https://ironsoftware.com/)
2. Click IronWebScraper or visit its Page Directly using URL [https://ironsoftware.com/csharp/webscraper](https://ironsoftware.com/csharp/webscraper)
3. Click Download DLL.
4. Extract downloaded compressed file
5. In visual studio right click on project -> add -> reference -> browse

![Image 8](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5qjy00hb4tgsh7x1jtuv.png)
 
6. Go to extracted folder -> bin -> select “IronWebScraper .dll”

![Image 9](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i3nk6nrgnd2zeqder88b.png)

7. And it’s done!

## HelloScraper - Our First IronWebScraper Sample

As usual, we will start implementing the Hello Scraper App to make our first step using IronWebScraper.

- We have Created a New Console Application with the name “IronWebScraperSample”

### Steps to Create IronWebScraper Sample

1. Create a Folder and name it “HelloScraperSample”

2. Then a new class and name it “HelloScraper”

![Image 10](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7engxngjrek6yupswb1u.png)
 
3. Add this Code snippet to HelloScraper

**C#:**
```
public class HelloScraper : WebScraper
{
    /// <summary>
    /// Override this method initialize your web-scraper.
    /// Important tasks will be to Request at least one start url... and set allowed/banned domain or url patterns.
    /// </summary>
    public override void Init()
    {
        License.LicenseKey = "LicenseKey"; // Write License Key
        this.LoggingLevel = WebScraper.LogLevel.All; // All Events Are Logged
        this.Request("https://blog.scrapinghub.com", Parse);
    }
    /// <summary>
    /// Override this method to create the default Response handler for your web scraper.
    /// If you have multiple page types, you can add additional similar methods.
    /// </summary>
    /// <param name="response">The http Response object to parse</param>
    public override void Parse(Response response)
    {
        // set working directory for the project
        this.WorkingDirectory = AppSetting.GetAppRoot()+ @"\HelloScraperSample\Output\";
        // Loop on all Links
        foreach (var title_link in response.Css("h2.entry-title a"))
        {
            // Read Link Text
            string strTitle = title_link.TextContentClean;
            // Save Result to File
            Scrape(new ScrapedData() { { "Title", strTitle } }, "HelloScraper.Jsonl");
        }
        // Loop On All Links
        if (response.CssExists("div.prev-post > a[href]"))
        {
            // Get Link URL
            var next_page = response.Css("div.prev-post > a[href]")[0].Attributes["href"];
            // Scrape Next URL
            this.Request(next_page, Parse);
        }
    }
}
```

**VB:**
```
Public Class HelloScraper
	Inherits WebScraper
	''' <summary>
	''' Override this method initialize your web-scraper.
	''' Important tasks will be to Request at least one start url... and set allowed/banned domain or url patterns.
	''' </summary>
	Public Overrides Sub Init()
		License.LicenseKey = "LicenseKey" ' Write License Key
		Me.LoggingLevel = WebScraper.LogLevel.All ' All Events Are Logged
		Me.Request("https://blog.scrapinghub.com", AddressOf Parse)
	End Sub
	''' <summary>
	''' Override this method to create the default Response handler for your web scraper.
	''' If you have multiple page types, you can add additional similar methods.
	''' </summary>
	''' <param name="response">The http Response object to parse</param>
	Public Overrides Sub Parse(ByVal response As Response)
		' set working directory for the project
		Me.WorkingDirectory = AppSetting.GetAppRoot() & "\HelloScraperSample\Output\"
		' Loop on all Links
		For Each title_link In response.Css("h2.entry-title a")
			' Read Link Text
			Dim strTitle As String = title_link.TextContentClean
			' Save Result to File
			Scrape(New ScrapedData() From {
				{ "Title", strTitle }
			},
			"HelloScraper.Jsonl")
		Next title_link
		' Loop On All Links
		If response.CssExists("div.prev-post > a[href]") Then
			' Get Link URL
			Dim next_page = response.Css("div.prev-post > a[href]")(0).Attributes("href")
			' Scrape Next URL
			Me.Request(next_page, AddressOf Parse)
		End If
	End Sub
End Class
```

4. Now to start Scrape Add This Code Snippet To Main

**C#:**
```
static void Main(string[] args)
{
    // Create Object From Hello Scrape class
    HelloScraperSample.HelloScraper scrape = new HelloScraperSample.HelloScraper();
        // Start Scraping
        scrape.Start();
}
```

**VB:**
```
Shared Sub Main(ByVal args() As String)
	' Create Object From Hello Scrape class
	Dim scrape As New HelloScraperSample.HelloScraper()
		' Start Scraping
		scrape.Start()
End Sub
```

The result will be saved in a file with the format WebSraper.WorkingDirecty/classname.Json

![Image 11](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/616j32d1v628xjjrw9gr.png)
 
**Code Overview**

Scrape.Start() => fires scrape logic as follow:

1. Call the Init() Method first for initiating the variables, scrape properties and behavior attributes.,

2. As we can see it sets the starting page to Request("https://blog.scrapinghub.com", Parse) and Parse (Response response) defined to be the process that used to parse response.

3. Webscraper manages in parallel: http and threads… keeping all your code easy to debug and synchronous.

4. Parse method start after Init() to parse the page.
     a. You can find elements using (Css selectors, Js DOM, XPath)
     b. Selected elements are casted to the type ScrapedData Class, you can cast them to any custom Class Like (Product, Employee, News, etc.)
     c. The objects saved in a file with Json Format in the (“bin/Scrape/”) Directory. Or you can set the path of the file as a parameter as we will see later in other examples.

## IronWebScraper Library Functions and Options

You can find updated documentation inside the zip file that was downloaded with the manual installation method (IronWebScraper Documentation.chm File)

Or you can check online Documentation for library last update on [https://ironsoftware.com/csharp/webscraper/object-reference/](https://ironsoftware.com/csharp/webscraper/object-reference/)

To start using IronWebscraper in your project you must inherit from (IronWebScraper.WebScraper) class that extends your class library and adds scraping functionality to it.

Also you must implement {Init(), Parse(Response response)} methods.

**C#:**
```
namespace IronWebScraperEngine
{
    public class NewsScraper : IronWebScraper.WebScraper
    {
        public override void Init()
        {
            throw new NotImplementedException();
        }
        public override void Parse(Response response)
        {
            throw new NotImplementedException();
        }
    }
}
```

**VB:**
```
Namespace IronWebScraperEngine
	Public Class NewsScraper
		Inherits IronWebScraper.WebScraper
		Public Overrides Sub Init()
			Throw New NotImplementedException()
		End Sub
		Public Overrides Sub Parse(ByVal response As Response)
			Throw New NotImplementedException()
		End Sub
	End Class
End Namespace
```

![Image 12](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7fybimrrustlkj8fuqjm.png)
![Image 13](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/488rot691h2kg9q0jrgp.png)
 
## Real World Samples and Practice

### Scraping an Online Movie Website

Let’s start another example from a real world website. We'll choose to scrape a movie website.

Let’s add a new class and name it “MovieScraper”:

![Image 14](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/44dg3o3xtihdufxgr0px.png)

Now let’s take a look on the site that we will scrape:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/38uj7i78hnk7z3iec8q7.png)
 
This is part of the homepage HTML we see on the website:

```html
<div id="movie-featured" class="movies-list movies-list-full tab-pane in fade active">
    <div data-movie-id="20746" class="ml-item">
        <a href="https://website.com/film/king-arthur-legend-of-the-sword-20746/">
            <span class="mli-quality">CAM</span>
            <img data-original="https://img.gocdn.online/2017/05/16/poster/2116d6719c710eabe83b377463230fbe-king-arthur-legend-of-the-sword.jpg" 
                 class="lazy thumb mli-thumb" alt="King Arthur: Legend of the Sword"
                  src="https://img.gocdn.online/2017/05/16/poster/2116d6719c710eabe83b377463230fbe-king-arthur-legend-of-the-sword.jpg" 
                 style="display: inline-block;">
            <span class="mli-info"><h2>King Arthur: Legend of the Sword</h2></span>
        </a>
    </div>
    <div data-movie-id="20724" class="ml-item">
        <a href="https://website.com/film/snatched-20724/" >
            <span class="mli-quality">CAM</span>
            <img data-original="https://img.gocdn.online/2017/05/16/poster/5ef66403dc331009bdb5aa37cfe819ba-snatched.jpg" 
                 class="lazy thumb mli-thumb" alt="Snatched" 
                 src="https://img.gocdn.online/2017/05/16/poster/5ef66403dc331009bdb5aa37cfe819ba-snatched.jpg" 
                 style="display: inline-block;">
            <span class="mli-info"><h2>Snatched</h2></span>
        </a>
    </div>
</div>
```

As we can see, we have a movie ID, Title, and Link to Detailed Page.

Let’s start to scrape this set of data:

**C#:**
```
public class MovieScraper : WebScraper
{
    public override void Init()
    {
        License.LicenseKey = "LicenseKey";
        this.LoggingLevel = WebScraper.LogLevel.All;
        this.WorkingDirectory = AppSetting.GetAppRoot() + @"\MovieSample\Output\";
        this.Request("www.website.com", Parse);
    }
    public override void Parse(Response response)
    {
        foreach (var Divs in response.Css("#movie-featured > div"))
        {
            if (Divs.Attributes["class"] != "clearfix")
            {
                var MovieId = Divs.GetAttribute("data-movie-id");
                var link = Divs.Css("a")[0];
                var MovieTitle = link.TextContentClean;
                Scrape(new ScrapedData() { { "MovieId", MovieId }, { "MovieTitle", MovieTitle } }, "Movie.Jsonl");
            }
        }           
    }
}
```

**VB:**
```
Public Class MovieScraper
	Inherits WebScraper
	Public Overrides Sub Init()
		License.LicenseKey = "LicenseKey"
		Me.LoggingLevel = WebScraper.LogLevel.All
		Me.WorkingDirectory = AppSetting.GetAppRoot() & "\MovieSample\Output\"
		Me.Request("www.website.com", AddressOf Parse)
	End Sub
	Public Overrides Sub Parse(ByVal response As Response)
		For Each Divs In response.Css("#movie-featured > div")
			If Divs.Attributes("class") <> "clearfix" Then
				Dim MovieId = Divs.GetAttribute("data-movie-id")
				Dim link = Divs.Css("a")(0)
				Dim MovieTitle = link.TextContentClean
				Scrape(New ScrapedData() From {
					{ "MovieId", MovieId },
					{ "MovieTitle", MovieTitle }
				},
				"Movie.Jsonl")
			End If
		Next Divs
	End Sub
End Class
```

_What’s new in this code?_

The Working Directory property is used to set the main working directory for all scraped data and its related files.

_Let's do more._

What if we need to build typed objects that will hold scraped data in formatted objects?

Let’s implement a movie class that will hold our formatted data:

**C#:**
```
public class Movie
{
    public int Id { get; set; }
    public string Title { get; set; }
    public string URL { get; set; }
}
```

**VB:**
```
Public Class Movie
	Public Property Id() As Integer
	Public Property Title() As String
	Public Property URL() As String
End Class
```

Now we will update our code:

**C#:**
```
public class MovieScraper : WebScraper
{
    public override void Init()
    {
        License.LicenseKey = "LicenseKey";
        this.LoggingLevel = WebScraper.LogLevel.All;
        this.WorkingDirectory = AppSetting.GetAppRoot() + @"\MovieSample\Output\";
        this.Request("https://website.com/", Parse);
    }
    public override void Parse(Response response)
    {
        foreach (var Divs in response.Css("#movie-featured > div"))
        {
            if (Divs.Attributes["class"] != "clearfix")
            {
                var movie = new Movie();
                movie.Id = Convert.ToInt32( Divs.GetAttribute("data-movie-id"));
                var link = Divs.Css("a")[0];
                movie.Title = link.TextContentClean;
                movie.URL = link.Attributes["href"];
                Scrape(movie, "Movie.Jsonl");
            }
        }
    }
}
```

**VB:**
```
Public Class MovieScraper
	Inherits WebScraper
	Public Overrides Sub Init()
		License.LicenseKey = "LicenseKey"
		Me.LoggingLevel = WebScraper.LogLevel.All
		Me.WorkingDirectory = AppSetting.GetAppRoot() & "\MovieSample\Output\"
		Me.Request("https://website.com/", AddressOf Parse)
	End Sub
	Public Overrides Sub Parse(ByVal response As Response)
		For Each Divs In response.Css("#movie-featured > div")
			If Divs.Attributes("class") <> "clearfix" Then
				Dim movie As New Movie()
				movie.Id = Convert.ToInt32(Divs.GetAttribute("data-movie-id"))
				Dim link = Divs.Css("a")(0)
				movie.Title = link.TextContentClean
				movie.URL = link.Attributes("href")
				Scrape(movie, "Movie.Jsonl")
			End If
		Next Divs
	End Sub
End Class
```

_What’s new?_

1. We implement Movie Class to hold our scraped data
2. We pass movie objects to the Scrape Method and it understands our format and saves in a defined format as we can see here:

![Image 16](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gt1cyfug1vdupzqoi3j0.png)
 
Let’s start scraping a more detailed page.

The Movie Page looks like this:

![Image 17](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8m41tbrip3zzbphkbpv4.png)
 
```html
<div class="mvi-content">
    <div class="thumb mvic-thumb"
         style="background-image: url(https://img.gocdn.online/2017/04/28/poster/5a08e94ba02118f22dc30f298c603210-guardians-of-the-galaxy-vol-2.jpg);"></div>
    <div class="mvic-desc">
        <h3>Guardians of the Galaxy Vol. 2</h3>        
        <div class="desc">
            Set to the backdrop of Awesome Mixtape #2, Marvel's Guardians of the Galaxy Vol. 2 continues the team's adventures as they travel throughout the cosmos to help Peter Quill learn more about his true parentage.
        </div>
        <div class="mvic-info">
            <div class="mvici-left">
                <p>
                    <strong>Genre: </strong>
                    <a href="https://Domain/genre/action/" title="Action">Action</a>,
                    <a href="https://Domain/genre/adventure/" title="Adventure">Adventure</a>,
                    <a href="https://Domain/genre/sci-fi/" title="Sci-Fi">Sci-Fi</a>
                </p>
                <p>
                    <strong>Actor: </strong>
                    <a target="_blank" href="https://Domain/actor/chris-pratt" title="Chris Pratt">Chris Pratt</a>,
                    <a target="_blank" href="https://Domain/actor/-zoe-saldana" title="Zoe Saldana">Zoe Saldana</a>,
                    <a target="_blank" href="https://Domain/actor/-dave-bautista-" title="Dave Bautista">Dave Bautista</a>
                </p>
                <p>
                    <strong>Director: </strong>
                    <a href="#" title="James Gunn">James Gunn</a>
                </p>
                <p>
                    <strong>Country: </strong>
                    <a href="https://Domain/country/us" title="United States">United States</a>
                </p>
            </div>
            <div class="mvici-right">
                <p><strong>Duration:</strong> 136 min</p>
                <p><strong>Quality:</strong> <span class="quality">CAM</span></p>
                <p><strong>Release:</strong> 2017</p>
                <p><strong>IMDb:</strong> 8.3</p>
            </div>
            <div class="clearfix"></div>
        </div>
        <div class="clearfix"></div>
    </div>
    <div class="clearfix"></div>
</div>
```

We can extend our movie class with new properties (Description, Genre, Actor, Director, Country, Duration, IMDB Score) but we’ll use (Description, Genre, Actor) only for our sample.

**C#:**
```
public class Movie
{
    public int Id { get; set; }
    public string Title { get; set; }
    public string URL { get; set; }
    public string Description { get; set; }
    public List<string> Genre { get; set; }
    public List<string> Actor { get; set; }
}
```

**VB:**
```
Public Class Movie
	Public Property Id() As Integer
	Public Property Title() As String
	Public Property URL() As String
	Public Property Description() As String
	Public Property Genre() As List(Of String)
	Public Property Actor() As List(Of String)
End Class
```

Now we’ll navigate to the Detailed page to scrape it.

IronWebScraper enables you to add more to the scrape function to scrape different types of page formats

As we can see here:

```html
Public Class MovieScraper
	Inherits WebScraper
	Public Overrides Sub Init()
		License.LicenseKey = "LicenseKey"
		Me.LoggingLevel = WebScraper.LogLevel.All
		Me.WorkingDirectory = AppSetting.GetAppRoot() & "\MovieSample\Output\"
		Me.Request("https://domain/", AddressOf Parse)
	End Sub
	Public Overrides Sub Parse(ByVal response As Response)
		For Each Divs In response.Css("#movie-featured > div")
			If Divs.Attributes("class") <> "clearfix" Then
				Dim movie As New Movie()
				movie.Id = Convert.ToInt32(Divs.GetAttribute("data-movie-id"))
				Dim link = Divs.Css("a")(0)
				movie.Title = link.TextContentClean
				movie.URL = link.Attributes("href")
				Me.Request(movie.URL, AddressOf ParseDetails, New MetaData() From {
					{ "movie", movie }
				}) ' to scrap Detailed Page
			End If
		Next Divs
	End Sub
	Public Sub ParseDetails(ByVal response As Response)
		Dim movie = response.MetaData.Get(Of Movie)("movie")
		Dim Div = response.Css("div.mvic-desc")(0)
		movie.Description = Div.Css("div.desc")(0).TextContentClean
		For Each Genre In Div.Css("div > p > a")
			movie.Genre.Add(Genre.TextContentClean)
		Next Genre
		For Each Actor In Div.Css("div > p:nth-child(2) > a")
			movie.Actor.Add(Actor.TextContentClean)
		Next Actor
		Scrape(movie, "Movie.Jsonl")
	End Sub
End Class
```

_What’s new?_

1. We can add scrape functions (ParseDetails) to scrape detailed pages
2. We moved the Scrape function that generates our file to the new function
3. We used the IronWebScraper feature (MetaData) to pass our movie object to the new scrape function
4. We scraped the page and saved our movie object data to a file

![Image 18](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q7zxkigwcbne79s69n6i.png)
 
### Scrape Content from a Shopping Website

We select a shopping site to scrape the content from it

![Image 19](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xn7ngwib0i8q18x4jaqs.png)
 
As you can see from the image, we have a left bar that contains links for the site's product categories

So our first step is to investigate the site HTML, and plan how we wish to scrape it.

![Image 20](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lzop84em57psk0hm9i3d.png)
 
The fashion site categories have sub categories (Men, Women, Kids)

```html
<li class="menu-item" data-id="">
    <a href="https://domain.com/fashion-by-/" class="main-category">
        <i class="cat-icon osh-font-fashion"></i> <span class="nav-subTxt">FASHION </span> <i class="osh-font-light-arrow-left"></i><i class="osh-font-light-arrow-right"></i>
    </a> <div class="navLayerWrapper" style="width: 633px; display: none;"><div class="submenu"><div class="column"><div class="categories"><a class="category" href="https://domain.com/fashion-by-/?sort=newest&amp;dir=desc&amp;viewType=gridView3">New Arrivals !</a>  </div><div class="categories"><a class="category" href="https://domain.com/men-fashion/">Men</a>   <a class="subcategory" href="https://domain.com/mens-shoes/">Shoes</a>   <a class="subcategory" href="https://domain.com/mens-clothing/">Clothing</a>   <a class="subcategory" href="https://domain.com/mens-accessories/">Accessories</a>  </div><div class="categories"><a class="category" href="https://domain.com/women-fashion/">Women</a>   <a class="subcategory" href="https://domain.com/womens-shoes/">Shoes</a>   <a class="subcategory" href="https://domain.com/womens-clothing/">Clothing</a>   <a class="subcategory" href="https://domain.com/womens-accessories/">Accessories</a>  </div><div class="categories"><a class="category" href="https://domain.com/girls-boys-fashion/">Kids</a>   <a class="subcategory" href="https://domain.com/boys-fashion/">Boys</a>   <a class="subcategory" href="https://domain.com/girls/">Girls</a>  </div><div class="categories"><a class="category" href="https://domain.com/maternity-clothes/">Maternity Clothes</a>  </div></div><div class="column"><div class="categories"> <span class="category defaultCursor">Men Best Sellers</span>  <a class="subcategory" href="https://domain.com/mens-casual-shoes/">Casual Shoes</a>   <a class="subcategory" href="https://domain.com/mens-sneakers/">Sneakers</a>   <a class="subcategory" href="https://domain.com/mens-t-shirts/">T-shirts</a>   <a class="subcategory" href="https://domain.com/mens-polos/">Polos</a>  </div><div class="categories"> <span class="category defaultCursor">Women Best Sellers</span>  <a class="subcategory" href="https://domain.com/womens-sandals/">Sandals</a>   <a class="subcategory" href="https://domain.com/womens-sneakers/">Sneakers</a>   <a class="subcategory" href="https://domain.com/women-dresses/">Dresses</a>   <a class="subcategory" href="https://domain.com/women-tops/">Tops</a>  </div><div class="categories"><a class="category" href="https://domain.com/womens-curvy-clothing/">Women's Curvy Clothing</a>  </div><div class="categories"><a class="category" href="https://domain.com/fashion-bundles/v/">Fashion Bundles</a>  </div><div class="categories"><a class="category" href="https://domain.com/hijab-fashion/">Hijab Fashion</a>  </div></div><div class="column"><div class="categories"><a class="category" href="https://domain.com/brands/fashion-by-/">SEE ALL BRANDS</a>   <a class="subcategory" href="https://domain.com/adidas/">Adidas</a>   <a class="subcategory" href="https://domain.com/converse/">Converse</a>   <a class="subcategory" href="https://domain.com/ravin/">Ravin</a>   <a class="subcategory" href="https://domain.com/dejavu/">Dejavu</a>   <a class="subcategory" href="https://domain.com/agu/">Agu</a>   <a class="subcategory" href="https://domain.com/activ/">Activ</a>   <a class="subcategory" href="https://domain.com/oxford--bellini--tie-house--milano/">Tie House</a>   <a class="subcategory" href="https://domain.com/shoe-room/">Shoe Room</a>   <a class="subcategory" href="https://domain.com/town-team/">Town Team</a>  </div></div></div></div>
</li>
```

Let's set up a project

1. Create a new Console App or Add a new folder for our new sample with the name “ShoppingSiteSample”
2. Add new class with the name “ShoppingScraper”
3. The first step will be to scrape the site categories and its sub categories

Let’s create a Categories Model:

**C#:**
```
public class Category
{
    /// <summary>
    /// Gets or sets the name.
    /// </summary>
    /// <value>
    /// The name.
    /// </value>
    public string Name { get; set; }
    /// <summary>
    /// Gets or sets the URL.
    /// </summary>
    /// <value>
    /// The URL.
    /// </value>
    public string URL { get; set; }
    /// <summary>
    /// Gets or sets the sub categories.
    /// </summary>
    /// <value>
    /// The sub categories.
    /// </value>
    public List<Category> SubCategories { get; set; }
}
```

**VB:**
```
Public Class Category
	''' <summary>
	''' Gets or sets the name.
	''' </summary>
	''' <value>
	''' The name.
	''' </value>
	Public Property Name() As String
	''' <summary>
	''' Gets or sets the URL.
	''' </summary>
	''' <value>
	''' The URL.
	''' </value>
	Public Property URL() As String
	''' <summary>
	''' Gets or sets the sub categories.
	''' </summary>
	''' <value>
	''' The sub categories.
	''' </value>
	Public Property SubCategories() As List(Of Category)
End Class
```

4. Now let’s build our scrape logic

**C#:**
```
public class ShoppingScraper : WebScraper
{
/// <summary>
/// Override this method initialize your web-scraper.
/// Important tasks will be to Request at least one start url... and set allowed/banned domain or url patterns.
/// </summary>
public override void Init()
{
    License.LicenseKey = "LicenseKey";
    this.LoggingLevel = WebScraper.LogLevel.All;
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";
    this.Request("www.webSite.com", Parse);
}
/// <summary>
/// Override this method to create the default Response handler for your web scraper.
/// If you have multiple page types, you can add additional similar methods.
/// </summary>
/// <param name="response">The http Response object to parse</param>
public override void Parse(Response response)
{
    var categoryList = new List<Category>();
    foreach (var Links in response.Css("#menuFixed > ul > li > a "))
    {
        var cat = new Category();
        cat.URL = Links.Attributes["href"];
        cat.Name = Links.InnerText;
        categoryList.Add(cat);
    }
    Scrape(categoryList, "Shopping.Jsonl");
}
}
```

**VB:**
```
Public Class ShoppingScraper
	Inherits WebScraper
''' <summary>
''' Override this method initialize your web-scraper.
''' Important tasks will be to Request at least one start url... and set allowed/banned domain or url patterns.
''' </summary>
Public Overrides Sub Init()
	License.LicenseKey = "LicenseKey"
	Me.LoggingLevel = WebScraper.LogLevel.All
	Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"
	Me.Request("www.webSite.com", AddressOf Parse)
End Sub
''' <summary>
''' Override this method to create the default Response handler for your web scraper.
''' If you have multiple page types, you can add additional similar methods.
''' </summary>
''' <param name="response">The http Response object to parse</param>
Public Overrides Sub Parse(ByVal response As Response)
	Dim categoryList = New List(Of Category)()
	For Each Links In response.Css("#menuFixed > ul > li > a ")
		Dim cat = New Category()
		cat.URL = Links.Attributes("href")
		cat.Name = Links.InnerText
		categoryList.Add(cat)
	Next Links
	Scrape(categoryList, "Shopping.Jsonl")
End Sub
End Class
```

Scraping links from the menu

![Image 21](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l3s7245c73osyd1ln2az.png)
 
Let’s update our code to scrape the Main Categories and all its sub links

**C#:**
```
public override void Parse(Response response)
{
    // List of Categories Links (Root)
    var categoryList = new List<Category>();
    foreach (var li in response.Css("#menuFixed > ul > li"))
    {
        // List Of Main Links
        foreach (var Links in li.Css("a"))
        {
            var cat = new Category();
            cat.URL = Links.Attributes["href"];
            cat.Name = Links.InnerText;
            cat.SubCategories = new List<Category>();
            // List of Sub Catgories Links
            foreach (var subCategory in li.Css("a[class=subcategory]"))
            {
                var subcat = new Category();
                subcat.URL = Links.Attributes["href"];
                subcat.Name = Links.InnerText;
                // Check If Link Exist Before 
                if (cat.SubCategories.Find(c=>c.Name== subcat.Name && c.URL == subcat.URL) == null)
                {
                    // Add Sublinks
                    cat.SubCategories.Add(subcat);
                }
            }
            // Add Categories
            categoryList.Add(cat);
        }
    }
    Scrape(categoryList, "Shopping.Jsonl");
}
```

**VB:**
```
Public Overrides Sub Parse(ByVal response As Response)
	' List of Categories Links (Root)
	Dim categoryList = New List(Of Category)()
	For Each li In response.Css("#menuFixed > ul > li")
		' List Of Main Links
		For Each Links In li.Css("a")
			Dim cat = New Category()
			cat.URL = Links.Attributes("href")
			cat.Name = Links.InnerText
			cat.SubCategories = New List(Of Category)()
			' List of Sub Catgories Links
			For Each subCategory In li.Css("a[class=subcategory]")
				Dim subcat = New Category()
				subcat.URL = Links.Attributes("href")
				subcat.Name = Links.InnerText
				' Check If Link Exist Before 
				If cat.SubCategories.Find(Function(c) c.Name= subcat.Name AndAlso c.URL = subcat.URL) Is Nothing Then
					' Add Sublinks
					cat.SubCategories.Add(subcat)
				End If
			Next subCategory
			' Add Categories
			categoryList.Add(cat)
		Next Links
	Next li
	Scrape(categoryList, "Shopping.Jsonl")
End Sub
```

Now we have links to all site categories, let’s start scraping the products within each category

Let’s navigate to any category and check the content.

![Image 22](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/km6niauez1798uw3s1xe.png)
 
Let’s see its code

```html
<section class="products">
    <div class="sku -gallery -validate-size " data-sku="AG249FA0T2PSGNAFAMZ" ft-product-sizes="41,42,43,44,45" ft-product-color="Multicolour">
        <a class="link" href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html">
            <div class="image-wrapper default-state">
                <img class="lazy image -loaded" alt="Bundle Of 2 Sneakers - Black &amp;amp; Navy Blue" data-image-vertical="1" width="210" height="262" src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" data-sku="AG249FA0T2PSGNAFAMZ" data-src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" data-placeholder="placeholder_m_1.jpg"><noscript>&lt;img src="https://static.WebSite.com/p/agu-6208-488356-1-catalog_grid_3.jpg" width="210" height="262" class="image" /&gt;</noscript>
            </div> <h2 class="title">
                <span class="brand ">Agu&nbsp;</span>
                <span class="name" dir="ltr">Bundle Of 2 Sneakers - Black &amp; Navy Blue</span>
            </h2><div class="price-container clearfix">
                <span class="price-box">
                    <span class="price">
                        <span data-currency-iso="EGP">EGP</span>
                        <span dir="ltr" data-price="299">299</span>
                    </span>   <span class="price -old  -no-special"></span>
                </span>
            </div><div class="rating-stars"><div class="stars-container"><div class="stars" style="width: 62%"></div></div> <div class="total-ratings">(30)</div> </div>    <span class="shop-first-logo-container"><img src="http://www.WebSite.com/images/local/logos/shop_first/ShoppingSite/logo_normal.png" data-src="http://www.WebSite.com/images/local/logos/shop_first/ShoppingSite/logo_normal.png" class="lazy shop-first-logo-img -mbxs -loaded"> </span>
            <span class="osh-icon -ShoppingSite-local shop_local--logo -block -mbs -mts"></span>
            <div class="list -sizes" data-selected-sku="">
                <span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=41">41</span>     <span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=42">42</span>
                <span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=43">43</span>     <span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=44">44</span>
                <span class="js-link sku-size" data-href="http://www.WebSite.com/agu-bundle-of-2-sneakers-black-navy-blue-653884.html?size=45">45</span>
            </div>
        </a>
    </div>
    <div class="sku -gallery -validate-size " data-sku="LE047FA01SRK4NAFAMZ" ft-product-sizes="110,115,120,125,130,135" ft-product-color="Black">
        <a class="link" href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html">
            <div class="image-wrapper default-state"><img class="lazy image -loaded" alt="Genuine Leather Belt - Black" data-image-vertical="1" width="210" height="262" src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" data-sku="LE047FA01SRK4NAFAMZ" data-src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" data-placeholder="placeholder_m_1.jpg"><noscript>&lt;img src="https://static.WebSite.com/p/leather-shop-1831-030217-1-catalog_grid_3.jpg" width="210" height="262" class="image" /&gt;</noscript></div>
            <h2 class="title"><span class="brand ">Leather Shop&nbsp;</span> <span class="name" dir="ltr">Genuine Leather Belt - Black</span></h2><div class="price-container clearfix">
                <span class="sale-flag-percent">-29%</span>  <span class="price-box"> <span class="price"><span data-currency-iso="EGP">EGP</span> <span dir="ltr" data-price="96">96</span> </span>   <span class="price -old "><span data-currency-iso="EGP">EGP</span> <span dir="ltr" data-price="135">135</span> </span> </span>
            </div><div class="rating-stars"><div class="stars-container"><div class="stars" style="width: 100%"></div></div> <div class="total-ratings">(1)</div> </div>
            <span class="osh-icon -ShoppingSite-local shop_local--logo -block -mbs -mts"></span>    <div class="list -sizes" data-selected-sku="">
                <span class="js-link sku-size" data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=110">110</span>     <span class="js-link sku-size" data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=115">115</span>
                <span class="js-link sku-size" data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=120">120</span>     <span class="js-link sku-size" data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=125">125</span>     <span class="js-link sku-size" data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=130">130</span>
                <span class="js-link sku-size" data-href="http://www.WebSite.com/leather-shop-genuine-leather-belt-black-712030.html?size=135">135</span>
            </div>
        </a>
    </div>
</section>
```

Let’s build our product model for this content.

**C#:**
```
public class Product
{
    /// <summary>
    /// Gets or sets the name.
    /// </summary>
    /// <value>
    /// The name.
    /// </value>
    public string Name { get; set; }
    /// <summary>
    /// Gets or sets the price.
    /// </summary>
    /// <value>
    /// The price.
    /// </value>
    public string Price { get; set; }
    /// <summary>
    /// Gets or sets the image.
    /// </summary>
    /// <value>
    /// The image.
    /// </value>
    public string Image { get; set; }
}
```

**VB:**
```
Public Class Product
	''' <summary>
	''' Gets or sets the name.
	''' </summary>
	''' <value>
	''' The name.
	''' </value>
	Public Property Name() As String
	''' <summary>
	''' Gets or sets the price.
	''' </summary>
	''' <value>
	''' The price.
	''' </value>
	Public Property Price() As String
	''' <summary>
	''' Gets or sets the image.
	''' </summary>
	''' <value>
	''' The image.
	''' </value>
	Public Property Image() As String
End Class
```

To scrape category pages, we add a new scrape method:

**C#:**
```
public void ParseCatgory(Response response)
{          
    // List of Products Links (Root)
    var productList = new List<Product>();
    foreach (var Links in response.Css("body > main > section.osh-content > section.products > div > a"))
    {
        var product = new Product();
        product.Name = Links.InnerText;
        product.Image = Links.Css("div.image-wrapper.default-state > img")[0].Attributes["src"];                
        productList.Add(product);
    }
    Scrape(productList, "Products.Jsonl");
}
```

**VB:**
```
Public Sub ParseCatgory(ByVal response As Response)
	' List of Products Links (Root)
	Dim productList = New List(Of Product)()
	For Each Links In response.Css("body > main > section.osh-content > section.products > div > a")
		Dim product As New Product()
		product.Name = Links.InnerText
		product.Image = Links.Css("div.image-wrapper.default-state > img")(0).Attributes("src")
		productList.Add(product)
	Next Links
	Scrape(productList, "Products.Jsonl")
End Sub
```

### Advanced Webscraping Features

**HttpIdentity Feature:**

Some website systems require the user to be logged in to view the content; in this case we can use a HttpIdentity: -

**C#:**
```
HttpIdentity id = new HttpIdentity();
id.NetworkUsername = "username";
id.NetworkPassword = "pwd";
Identities.Add(id); 
```

**VB:**
```
Dim id As New HttpIdentity()
id.NetworkUsername = "username"
id.NetworkPassword = "pwd"
Identities.Add(id)
```

One of the most impressive and powerful features in IronWebScraper, is the ability to use thousands of unique (user’s credentials and/or browser engines) to spoof or scrape websites using multi login sessions.

**C#:**
```
public override void Init()
{
    License.LicenseKey = " LicenseKey ";
    this.LoggingLevel = WebScraper.LogLevel.All;
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";
    var proxies = "IP-Proxy1: 8080,IP-Proxy2: 8081".Split(',');
    foreach (var UA in IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents)
    {
        foreach (var proxy in proxies)
        {
            Identities.Add(new HttpIdentity()
            {
                UserAgent = UA,
                UseCookies = true,
                Proxy = proxy
            });
        }
    }
    this.Request("http://www.Website.com", Parse);
}
```

**VB:**
```
Public Overrides Sub Init()
	License.LicenseKey = " LicenseKey "
	Me.LoggingLevel = WebScraper.LogLevel.All
	Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"
	Dim proxies = "IP-Proxy1: 8080,IP-Proxy2: 8081".Split(","c)
	For Each UA In IronWebScraper.CommonUserAgents.ChromeDesktopUserAgents
		For Each proxy In proxies
			Identities.Add(New HttpIdentity() With {
				.UserAgent = UA,
				.UseCookies = True,
				.Proxy = proxy
			})
		Next proxy
	Next UA
	Me.Request("http://www.Website.com", Parse)
End Sub
```

You have multiple properties to give you different behaviors, so preventing websites from blocking you.

Some of these properties: -

- **NetworkDomain**: The network domain to be used for user authentication. Supports Windows, NTLM, Keroberos, Linux, BSD, and Mac OS X networks. Must be used with (NetworkUsername and NetworkPassword)
- **NetworkUsername**: The network/http username to be used for user authentication. Supports HTTP, Windows networks, NTLM, Kerberos, Linux networks, BSD networks and Mac OS.
- **NetworkPassword**: The network/HTTP password to be used for user authentication. Supports HTTP, Windows networks, NTLM, Keroberos, Linux networks, BSD networks, and Mac OS.
- **Proxy** : to set proxy settings
- **UserAgent**: to set browser engine (chrome desktop, chrome mobile, chrome tablet, IE and Firefox, etc.)
- **HttpRequestHeaders**: for custom header values that will be used with this identity, and it accepts dictionary object (Dictionary <string, string>)
- **UseCookies**: enable/disable using cookies

IronWebScraper runs the scraper using random identities. If we need to specify the use of a specific identity to parse a page we can do so.

**C#:**
```
public override void Init()
{
    License.LicenseKey = " LicenseKey ";
    this.LoggingLevel = WebScraper.LogLevel.All;
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";
    HttpIdentity identity = new HttpIdentity();
    identity.NetworkUsername = "username";
    identity.NetworkPassword = "pwd";
    Identities.Add(id);
    this.Request("http://www.Website.com", Parse, identity);
}
```

**VB:**
```
Public Overrides Sub Init()
	License.LicenseKey = " LicenseKey "
	Me.LoggingLevel = WebScraper.LogLevel.All
	Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"
	Dim identity As New HttpIdentity()
	identity.NetworkUsername = "username"
	identity.NetworkPassword = "pwd"
	Identities.Add(id)
	Me.Request("http://www.Website.com", Parse, identity)
End Sub
```

**Enable the Web Cache Feature:**

This feature is used to cache Requested Pages. It is often used in development and testing phases; enabling developers to cache required pages for reuse after updating code. This enables you to execute your code on cached pages after restarting your Web scraper and not needing to connect to the live website every time (action-replay).

_You can use it in Init() Method_

EnableWebCache();

OR

EnableWebCache(Timespan Expiry);

It will save your cached data to the WebCache folder under the working directory folder

**C#:**
```
public override void Init()
{
    License.LicenseKey = " LicenseKey ";
    this.LoggingLevel = WebScraper.LogLevel.All;
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";
    EnableWebCache(new TimeSpan(1,30,30));
    this.Request("http://www.WebSite.com", Parse);
}
```

**VB:**
```
Public Overrides Sub Init()
	License.LicenseKey = " LicenseKey "
	Me.LoggingLevel = WebScraper.LogLevel.All
	Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"
	EnableWebCache(New TimeSpan(1,30,30))
	Me.Request("http://www.WebSite.com", Parse)
End Sub
```

IronWebScraper also has features to enable your engine to continue scraping after restarting code by setting the engine start process name using Start(CrawlID)

**C#:**
```
static void Main(string[] args)
{
    // Create Object From Scraper class
    EngineScraper scrape = new EngineScraper();
    // Start Scraping
    scrape.Start("enginestate");
}
```

**VB:**
```
Shared Sub Main(ByVal args() As String)
	' Create Object From Scraper class
	Dim scrape As New EngineScraper()
	' Start Scraping
	scrape.Start("enginestate")
End Sub
```

The execution request and response will be saved in the SavedState folder inside the working directory.

**Throttling**

We can control the minimum and maximum connection numbers and connection speed per domain.

**C#:**
```
public override void Init()
{
    License.LicenseKey = "LicenseKey";
    this.LoggingLevel = WebScraper.LogLevel.All;
    this.WorkingDirectory = AppSetting.GetAppRoot() + @"\ShoppingSiteSample\Output\";
    // Gets or sets the total number of allowed open HTTP requests (threads)
    this.MaxHttpConnectionLimit = 80;
    // Gets or sets minimum polite delay (pause)between request to a given domain or IP address.
    this.RateLimitPerHost = TimeSpan.FromMilliseconds(50);            
    //     Gets or sets the allowed number of concurrent HTTP requests (threads) per hostname
    //     or IP address. This helps protect hosts against too many requests.
    this.OpenConnectionLimitPerHost = 25;
    this.ObeyRobotsDotTxt = false;
    //     Makes the WebSraper intelligently throttle requests not only by hostname, but
    //     also by host servers' IP addresses. This is polite in-case multiple scraped domains
    //     are hosted on the same machine.
    this.ThrottleMode = Throttle.ByDomainHostName;
    this.Request("https://www.Website.com", Parse);
}
```

**VB:**
```
Public Overrides Sub Init()
	License.LicenseKey = "LicenseKey"
	Me.LoggingLevel = WebScraper.LogLevel.All
	Me.WorkingDirectory = AppSetting.GetAppRoot() & "\ShoppingSiteSample\Output\"
	' Gets or sets the total number of allowed open HTTP requests (threads)
	Me.MaxHttpConnectionLimit = 80
	' Gets or sets minimum polite delay (pause)between request to a given domain or IP address.
	Me.RateLimitPerHost = TimeSpan.FromMilliseconds(50)
	'     Gets or sets the allowed number of concurrent HTTP requests (threads) per hostname
	'     or IP address. This helps protect hosts against too many requests.
	Me.OpenConnectionLimitPerHost = 25
	Me.ObeyRobotsDotTxt = False
	'     Makes the WebSraper intelligently throttle requests not only by hostname, but
	'     also by host servers' IP addresses. This is polite in-case multiple scraped domains
	'     are hosted on the same machine.
	Me.ThrottleMode = Throttle.ByDomainHostName
	Me.Request("https://www.Website.com", Parse)
End Sub
```

**Throttling properties**

- **MaxHttpConnectionLimit**
total number of allowed open HTTP requests (threads)

- **RateLimitPerHost**
minimum polite delay or pause (in milliseconds) between request to a given domain or IP address

- **OpenConnectionLimitPerHost**
allowed number of concurrent HTTP requests (threads)

- **ThrottleMode**
Makes the WebSraper intelligently throttle requests not only by hostname, but also by host servers' IP addresses. This is polite in-case multiple scraped domains are hosted on the same machine.

## Appendix

### How to Create a Windows Form Application?

We should use Visual Studio 2013 or higher for this.

Follow these steps to create new a Windows Forms Project:

1. Open visual studio

![Image 23](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hxy0l2ga3xlzskz75f4i.png)
 
2. File -> New -> Project

![Image 24](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8dx0vsxu7scqjtj2e458.png)

3. From Template, Choose the programming language (Visual C# or VB) -> Windows -> Windows Forms Application

![Image 25](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lgvkyg51le9eky5has0n.png)
 
**Name of project**: IronScraperSample
**Location**: Choose a location on your hard disk

![Image 26](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yyh2hu4257d4gvaghyrw.png)
 
### How to Create a Web Form Application?

You should use Visual Studio 2013 or higher for this.

Follow the steps to Create a New Asp.NET Web forms Project

1. Open Visual Studio

![Image 27](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0s4yjd26nq5suaqmhuqz.png)
 
2. File -> New -> Project

![Image 28](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qgo60dqpr623zsmcsdjm.png)
 
From Template Choose programming language (Visual C# or VB) -> Web -> ASP.NET Web Application (.NET Framework).

![Image 29](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v0bovavorh896pbxg353.png)
 
Name of project: IronScraperSample
Location: choose a location from your Hard disk

4. From your ASP.NET Templates

a. Select Empty Template
b. Check Web Forms

![Image 30](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yyhu9q9a6kakrcztg11n.png)
 
5. Now your basic ASP.NET web form Project is Created

![Image 31](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4fp608eujx7tc8nah149.png)

[Click here](https://ironsoftware.com/downloads/assets/tutorials/webscraping-in-c-sharp/IronWebScraperSample.zip) to download the full tutorial sample project code project.