Image downloader with puppeteer and the fetch API
Tags: webdev, webmonetization, node, javascript
Published: 2020-11-24 11:26:07

In this tutorial, we are going to build a webpage image downloader. Assuming you visit a webpage and saw that the images in that page are cool and you want to have your own copy without downloading them one by one, this simple tool we will build is going to be a life saver for you. This little project is also a good way to practice and hone your webscraping skills.

We will create a new directory called `image-downloader` and navigate into it. Pop open your terminal window and type in the following commands.

```bash
mkdir image-downloader && cd image-downloader
```


I will assume that you have node js and npm installed on your machine. We will then initialize this directory with the standard `package.json` file by running `npm init -y` and then install two dependencies namely `puppeteer` and `node-fetch`. Run the following commands to get them installed.

```bash
npm install --save puppeteer node-fetch --verbose
```

You probably just saw a new npm flag `--verbose`. When installing puppeteer, what happens behind the scenes is that `npm` also installs the chromium browser because it is a dependency of `puppeteer`. This file is usually large and we are using the `--verbose` flag to see the progress of the installation, nothing fancy, but let's just use it because we can. 

One more thing to do before getting our hands dirty with code is to create a directory where we want all our images to be downloaded. Let's name that directory `images`. We will also create `index.js` file where all the app's logic will go.

```bash
mkdir images && touch index.js
```

Actually, it's great to clearly outline our thought process before writing a single line of code.
1. Get all image tags from the page and extract the `href` property from each of these image tags
2. Make request to those `href` links and store them into the `images` directory (Saving images to disk)

## Step one 1: Getting all image tags and `href` property
```js
'use strict';

const puppeteer = require('puppeteer');
const fetch = require('node-fetch');
const fs = require('fs')

// Extract all imageLinks from the page
async function extractImageLinks(){
    const browser = await puppeteer.launch({
		headless: false
	})

	const page = await browser.newPage()

	// Get the page url from the user
	let baseURL = process.argv[2] ? process.argv[2] : "https://stocksnap.io"
    
    try {
		await page.goto(baseURL, {waitUntil: 'networkidle0'})
		await page.waitForSelector('body')

		let imageBank = await page.evaluate(() => {
			let imgTags = Array.from(document.querySelectorAll('img'))
             
            let imageArray = []
			
			imgTags.map((image) => {
				let src = image.src

				let srcArray = src.split('/')
				let pos = srcArray.length - 1
				let filename = srcArray[pos]
			
				imageArray.push({
					src,
					filename
				})
			})

			return imageArray
		})

		await browser.close()
		return imageBank

	} catch (err) {
		console.log(err)
	}
}
```


Now let me explain what is happening here. First, we created an `async` function called `extractImageLinks`. In that function, we created an instance of a browser page using puppeteer and stored it in the `page` constant. Think of this `page` as the new page you get after launching your chrome browser. We can now heedlessly control this page from our code. We then get the url of the page we want to download the image from the user and stored it in a variable named `baseURL`. We then navigate to that URL using the `page.goto()` function. The `{waitUntil: 'networkidle0'}` object passed as the second argument to this function is to ensure that we wait for the for the network request to complete before we proceed with parsing the page. `page.waitForSelector('body')` is telling puppeteer to wait for the html `body` tag to render before we start extracting anything from the page.

The `page.evaluate()` function allows us to run JavaScript code in that page instance as if we were in our Google Chrome Devtools. To get all image tags from the page, we call the `document.querySelectorAll("img")` function. However, this function returns an `NodeList` and not an array. So to convert this to an array, we wrapped the first function with the `Array.from()` method. Now we have an array to work with.

We then store all the image tags in the `imgTags` variable and initialized `imageArray` variable as a placeholder for all the `href` values. Since `imgTags` has been converted into an array, we then map through every tag in that array and extract the `src` property from each image tag.

Now time for some little hack, we want to download the image from the webpage maintianing the original filename as it appears in the webpage. For instance, we have this image src `https://cdn.stocksnap.io/img-thumbs/960w/green-leaf_BVKZ4QW8LS.jpg`. We wnat to get the `green-leaf_BVKZ4QW8LS.jpg` from that URL. One way to do this is to split the string using the `"/"` delimeter. We then end up with something like this: 

```js
let src = `https://cdn.stocksnap.io/img-thumbs/960w/green-leaf_BVKZ4QW8LS.jpg`.split("/")

// Output
["https:", "", "cdn.stocksnap.io", "img-thumbs", "960w", "green-leaf_BVKZ4QW8LS.jpg"]
```

Now the last index of the array after running the `split` array method on the image source contains the image's name and the extension as well, awesome!!! 

#### Note: to get the last item from any array, we subtract `1` from the  `length`m of that array like so: 

```js
let arr = [40,61,12] 
let lastItemIndex = arr.length - 1 // This is the index of the last item

console.log(lastItemIndex)
// Output
2

console.log(arr[lastItemIndex])
// Output
12
``` 

So we store the index of the last item in the `pos` variable and then store the name of the file in the `filename` variable as well. Now we have the source of the file and the file name of the current image in the loop, we then push these values as an object in the `imageArray` variable. After the mapping is done, we return the `imageArray` because by now it has been populated. We also return the `imageBank` variable which now contains the image links (sources) and the filenames.

## Saving images to disk
```js
function saveImageToDisk(url, filename){
	fetch(url)
	.then(res => {
		const dest = fs.createWriteStream(filename);
		res.body.pipe(dest)
	})
	.catch((err) => {
		console.log(err)
	})
}


// Run the script on auto-pilot
(async function(){
	let imageLinks = await extractImageLinks()
	console.log(imageLinks)

	imageLinks.map((image) => {
		let filename = `./images/${image.filename}`
		saveImageToDisk(image.src, filename)
	})
})()
```

Now let's decipher this little piece. In the anonymous IIFE, we are running the `extractImageLinks` to get the array containing the `src` and `filename`. Since the function is returns an array, we run the `map` function on that array and then pass the required parameters (`url` and `filename`) to `saveImageToDisk`. We then use the `fetch` API to make a `GET` request to that `url` and as the response is coming down the wire, we are concurrently piping it into the `filename` destination, in this case, a writable stream on our filesystem. This is very efficient because we are not waiting for the image to be fully loaded in memory before saving to disk but instead saving every chunk we get from the response directly.

Lets's run the code, cross our fingers and checkout our `images` directory

```bash
node index.js  https://stocksnap.io
```

We should see some cool images there in. Wooo! You can add this to your portfolio. There are so many improvements that can be done to this little software, such as allowing the user to specify the directory they want to download the image, handling `Data URI` images, proper error handling, code refactoring, creating a standalone CLI utility for it. Hint: use the `commander` npm package for that, etc. You can go ahead and extend this app and I'll be glad to see what improvements you will make it.

## Full code
```js
'use strict';

const puppeteer = require('puppeteer');
const fetch = require('node-fetch');
const fs = require('fs')

// Browser and page instance
async function instance(){
	const browser = await puppeteer.launch({
		headless: false
	})

	const page = await browser.newPage()
	return {page, browser}
}

// Extract all imageLinks from the page
async function extractImageLinks(){
	const {page, browser} = await instance()

	// Get the page url from the user
	let baseURL = process.argv[2] ? process.argv[2] : "https://stocksnap.io"

	try {
		await page.goto(baseURL, {waitUntil: 'networkidle0'})
		await page.waitForSelector('body')

		let imageLinks = await page.evaluate(() => {
			let imgTags = Array.from(document.querySelectorAll('img'))
			
			let imageArray = []
			
			imgTags.map((image) => {
				let src = image.src

				let srcArray = src.split('/')
				let pos = srcArray.length - 1
				let filename = srcArray[pos]
			
				imageArray.push({
					src,
					filename
				})
			})

			return imageArray
		})

		await browser.close()
		return imageLinks

	} catch (err) {
		console.log(err)
	}
}

(async function(){
	console.log("Downloading images...")

	let imageLinks = await extractImageLinks()

	imageLinks.map((image) => {
		let filename = `./images/${image.filename}`
		saveImageToDisk(image.src, filename)
	})

	console.log("Download complete, check the images folder")
})()

function saveImageToDisk(url, filename){
	fetch(url)
	.then(res => {
		const dest = fs.createWriteStream(filename);
		res.body.pipe(dest)
	})
	.catch((err) => {
		console.log(err)
	})
}
```

Cheers! :tada: