How to generate a Laravel sitemaps on the fly?
Tags: laravel, laravelseo, sitemap, crawler
Published: 2020-03-26 09:44:54

---
title: How to generate a Laravel sitemaps on the fly?
published: true
description: Simple crawler and sitemap generator for Laravel. No headless browser - just a crawler.
tags:
  - Laravel
  - Laravel SEO
  - Sitemap
  - Crawler
canonical_url: https://spekulatius.hashnode.dev/how-to-generate-a-laravel-sitemaps-dynamically-ck88kil3g00207rs1r9fv0239
---

First things first: What is a sitemap?
------------------

Sitemaps are xml-files containing structured data about the pages of the website. Each page has an entry similar to this one:

```xml
  <url>
    <loc>https://startupnamecheck.com</loc>
    <lastmod>2020-03-06T20:31:03+00:00</lastmod>
    <priority>0.9</priority>
    <changefreq>monthly</changefreq>
  </url>

```

What are sitemaps good for?
---------------------------

Sitemaps are helpers for search-engines to discover all relevant pages and content on a website. While there are also sitemaps for images, the focus here is on web-pages only.

How can I generate a sitemap?
-----------------------------

A sitemap can be created in various ways. If you are using a framework such as [Laravel](https://laravel.com) you can create these on the fly or whenever you publish or update your content.

After some experiments and checking several solutions on GitHub I've not found the solution I was looking for:

-   A simple, permanent crawler of the actual website.
-   It considers `noindex` robots tags as well as canonicals and of course the `article:modified_time` tag.
-   Ignores JavaScript as Google does mostly. This allows it to run much faster than executing a headless browser only to access a pure HTML5/CSS3 page.

My solution for sitemaps
------------------------

As mentioned, after some research I haven't found what I had in mind. So, being a developer at heart, I've opted to build my own solution. It's heavily reliant on [PHP Spider](https://mvdbos.github.io/php-spider/), a crawler package for PHP. Besides this, the package is using some regex to identify the most interesting parts of the website. Other values, such as `priority` are guessed by the depth within the website (nesting level). More detail can also be found on the [GitHub repo for Laravel-Sitemaps](https://github.com/bringyourownideas/laravel-sitemap).

How can I get this?
-------------------

The package is distributed using [composer](https://getcomposer.org/) and can be installed using:

```bash
composer require bringyourownideas/laravel-sitemap
```

This will automatically configure the required Laravel ServiceProvider. If you opted out of package discovery you can install it manually using:

```bash
php artisan vendor:publish --provider="BringYourOwnIdeas\LaravelSitemap\SitemapServiceProvider"
```

How to use the package
----------------------

The package registers an `artisan`-command called `generate:sitemap`. This triggers a crawl of your site and writing out of the sitemap. For convenience, you can add this to your deployment steps.

Regular updates of the sitemap
------------------------------

If you'd like to run updates of the `sitemap.xml` regularly, you can add a new line in `app/Console/Kernel.php` in the `schedule` function:

```php
/**
 * Define the application's command schedule.
 *
 * @param  \Illuminate\Console\Scheduling\Schedule  $schedule
 * @return void
 */
protected function schedule(Schedule $schedule)
{
    $schedule->command('generate:sitemap')->daily();

    // ...or with a defined time...

    $schedule->command('generate:sitemap')->daily()->at('02:50');
}
```

Summary & Feedback
-------------------

If you've got issues please raise an [issue on GitHub](https://github.com/bringyourownideas/laravel-sitemap/issues). To stay updated please subscribe to my newsletter (below). More information can also be found in the BYOI article around the [Laravel Sitemap Generator](https://bringyourownideas.com/blog/simple-laravel-sitemap-generator/).