Netlify Dynamic Site Challenge: Web Voyager, a website screenshot service using Netlify Blob storage.
Tags: netlifychallenge, devchallenge, webdev, javascript
Published: 2024-05-12 18:18:38

*This is a submission for the [Netlify Dynamic Site Challenge](https://dev.to/challenges/netlify): Build with Blobs.*

[https://github.com/shikhar13012001/netlify-dyn-site-challenge](https://github.com/shikhar13012001/netlify-dyn-site-challenge)
## What I Built
<!-- Share an overview about your project. -->
Introducing a cutting-edge service that revolutionizes web monitoring by allowing users to capture high-quality screenshots directly from input URLs or CSV files containing multiple URLs. Each URL generates a screenshot, and CSV files have persistent URLs for easy sharing and future reference. This service leverages Netlify's blob storage, utilizing a deployment blob store with a `key: blob|JSON|..` storage model, ensuring efficient, reliable, and scalable screenshot management. Ideal for developers and marketers, this tool simplifies web monitoring and enhances collaboration. Thank you, Netlify, for powering this innovative solution!
## Demo
<!-- Share a link to your Netlify app and include screenshots here. -->
The deployment is in a digital ocean droplet [http://64.227.150.39/](http://64.227.150.39/) because of the timeouts in Netlify lambda functions.
###  Single URL webshot 
The single web shot URL is persistent at `/screenshot/image/<key_id>`
![Single Url web shot of netlify.com from my service](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/67q148vt53i0g585pd61.png)

![Obtained screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jeu61jpr3qabmjyc3oll.png)
As you can observe, the screenshot has both an instant download feature and a shareable link.

### Bulk screenshot feature
Note: I have made a URL csv (`urls.csv`) file for convenience.
For simplicity I have taken 4 URLs in the `urls.csv`

![urls.csv content](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/plq0l5pvd2yjp8sfcyuz.png)


![urls.csv is attached to the service from the local file system](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w565rjjxy27utej87i5k.png)
I just wanted to show the cool loading screen while it works on it.

![Magic UI loading screen(animated beams)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0iphxjysqltjrrstx4ie.png)

**Bulk screenshot network tab view**
The persistent URL of the bulk shot exists at `/screenshot/bucket/<generated_bucket_key>`

![Bulk screenshot network tab view](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nvgyvdvu0qjw7l4e26qw.png)







## Platform Primitives

Leveraging Netlify Blobs for Screenshot Service

In developing the website screenshot service, Netlify Blobs was crucial in managing and storing screenshots efficiently. Here’s a detailed breakdown of how we harnessed this platform primitive to enhance our service:

### **Blob Storage Integration**

- **Initialization and Configuration**:
   The integration begins with setting up the Netlify Blobs storage, a straightforward process thanks to the `@netlify/blobs` package. We initialize the blob store with our site's unique identifier and an API token for authentication. This setup ensures that our screenshots are securely stored and easily accessible.
   
   ```javascript
   const { getStore } = require("@netlify/blobs");
   const NETLIFY_API_TOKEN = "your_netlify_api_token";
   const NETLIFY_SITE_ID = "your_site_id";
   const store = getStore({
     name: "screenshots",
     siteID: NETLIFY_SITE_ID,
     token: NETLIFY_API_TOKEN,
   });
   ```

- **Screenshot Capture**:
   Using Puppeteer, we capture screenshots of the specified URLs. Puppeteer is configured to run in a headless mode with several options to optimize performance and compatibility with different web pages.

   ```javascript
   const browserOptions = {
     headless: true,
     args: [
       "--disable-setuid-sandbox",
       "--no-sandbox",
       "--single-process",
       "--no-zygote",
       "--autoplay-policy=user-gesture-required",
       "--disable-background-networking",
     ],
   };

   const browser = await puppeteer.launch({ ...browserOptions });
   const page = await browser.newPage();
   await page.setViewport({ width: 1920, height: 1080 });
   await page.goto(url, { waitUntil: "domcontentloaded" });
   const screenshot = await page.screenshot({ fullPage: true });
   await browser.close();
   ```

-  **Storing Screenshots**:
   After capturing the screenshot, we use the Netlify Blob store to save the image. Each screenshot is stored with a unique key generated by `uuidv4()`, ensuring easy retrieval and avoiding filename conflicts.

   ```javascript
   const key = uuidv4();
   await store.set(key, screenshot, {
     metadata: {
       url,
       type: "image/png",
       size: { width: 1920, height: 1080 },
     },
   });
   ```

- **Persistent URLs**:
   Netlify Blobs provide persistent URLs for each stored screenshot, making it easy to share and reference them. This feature is particularly useful for bulk screenshot functionality, where multiple URLs are processed, and each screenshot is stored and accessed via its unique URL.

   ```javascript
   return res.json({ key });
   ```

- **Bulk screenshot storage**:
For bulk screenshot persistence, I have used a function called `saveKeys`, a member function of `Screenshot.api.ts'`  `Webshot` class.
The code below gets all keys from the screenshots, which runs in parallel to lower the compute time to the max of all screenshot capture times( around a minute right now!!!).
 ```javascript
 const keys = await Promise.all(
        urls.map((url) => this.takeScreenshot(url, OPTIONS))
      );
```
Another route, `/api/bucketList`, has a conditional statement for saving or retrieving keys using the `setJson` function available in Netlify blob storage.
```javascript
let bucketKeys;

    // Handle operations
    if (opCode === "save") {
      bucketKeys = await store.setJSON(bucketKey, { keys });
    } else if (opCode === "retrieve") {
      bucketKeys = await store.get(bucketKey, { type: "json" });
    } 
```
```javascript
    return NextResponse.json(
      { bucketKeys },
      {
        status: 200,
        headers: {
          "Netlify-CDN-Cache-Control":
            "public, s-maxage=31536000, stale-while-revalidate=59",
        },
      }
    );
```
Network call for `bucketList` call
![Network call for bucketList call](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hp6vyx5jzvk0p9nzzv37.png)


By integrating Netlify Blobs, we have built a robust screenshot service that caters to single and bulk URL processing needs. This service not only simplifies web monitoring for developers and marketers but also leverages the powerful features of Netlify Blobs to provide a scalable, reliable, and secure solution.