Mirroring a Website with Wget
Tags: archiving
Published: 2022-12-24 23:59:30

---
title: Mirroring a Website with Wget
published: true
date: 2022-12-21 19:05:00 UTC
tags: Archiving
canonical_url: https://kevincox.ca/2022/12/21/wget-mirror/
---

<main style="--width:min(36em,100%);display:grid;grid-template-columns:1fr var(--width) 1fr;overflow:auto;padding:0;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><p style="box-sizing:border-box;grid-column:2;margin:0 1rem .8em">Occasionally I want to archive a site. Maybe it is something important or maybe the site is going offline. I often struggle to concoct the right Wget command so I’ve recorded it here for future reference.</p>
<div style="background:#222;border-bottom:1px solid #444;border-top:1px solid #444;box-sizing:border-box;display:grid;grid-column:2;grid-column:1/4;grid-template-columns:1fr var(--width) 1fr;margin:0 0 .8em;overflow-x:auto;padding:0"><pre style="box-sizing:border-box;grid-column:2;margin:.4em 1rem"><code style="background:none!important;box-sizing:border-box;color:#fff;margin:0;white-space:pre-wrap;white-space:pre">wget
	<span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">--adjust-extension</span> <span style="box-sizing:border-box;color:#b3d733;margin:0;padding:0">\</span>
	<span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">--convert-links</span> <span style="box-sizing:border-box;color:#b3d733;margin:0;padding:0">\</span>
	<span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">--mirror</span> <span style="box-sizing:border-box;color:#b3d733;margin:0;padding:0">\</span>
	<span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">--no-parent</span> <span style="box-sizing:border-box;color:#b3d733;margin:0;padding:0">\</span>
	<span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">--page-requisites</span> <span style="box-sizing:border-box;color:#b3d733;margin:0;padding:0">\</span>
	<span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">--restrict-file-names</span><span style="box-sizing:border-box;color:#f5a3a3;font-weight:700;margin:0;padding:0">=</span>windows <span style="box-sizing:border-box;color:#b3d733;margin:0;padding:0">\</span>
	<span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">--trust-server-names</span> <span style="box-sizing:border-box;color:#b3d733;margin:0;padding:0">\</span>
	<span style="box-sizing:border-box;color:#b3d733;margin:0;padding:0">"https://example.com/tutorials"</span>
</code></pre></div>
<p style="box-sizing:border-box;grid-column:2;margin:0 1rem .8em">That is the base command. You may also want <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap"><span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">--span-hosts</span></code> if you want to include all third-party dependencies or <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap"><span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">--domains</span><span style="box-sizing:border-box;color:#f5a3a3;font-weight:700;margin:0;padding:0">=</span>cdn.example.com</code> if you want to include assets from some domains.</p>
<h2 id="flag-descriptions" style="box-sizing:border-box;grid-column:2;margin:.4em 1rem .8em"><a href="https://kevincox.ca/2022/12/21/wget-mirror/#flag-descriptions" style="box-sizing:border-box;color:inherit!important;margin:0;text-decoration:inherit;text-decoration:underline;-webkit-text-decoration-color:gray;text-decoration-color:gray;text-shadow:inherit!important">Flag Descriptions</a></h2>
<h3 id="--adjust-extension" style="box-sizing:border-box;grid-column:2;margin:.4em 1rem .8em"><a href="https://kevincox.ca/2022/12/21/wget-mirror/#--adjust-extension" style="box-sizing:border-box;color:inherit!important;margin:0;text-decoration:inherit;text-decoration:underline;-webkit-text-decoration-color:gray;text-decoration-color:gray;text-shadow:inherit!important"><code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">--adjust-extension</code></a></h3>
<p style="box-sizing:border-box;grid-column:2;margin:0 1rem .8em">This ensures that some file types end with a common extension. For example HTML with <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">.html</code> CSS with <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">.css</code> and similar. This unfortunately doesn’t adjust all downloaded files but provides some improvement. Using common extensions helps when serving content from general-purpose webservers that will set the <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">Content-Type</code> header based on the extension.</p>
<h3 id="--convert-links" style="box-sizing:border-box;grid-column:2;margin:.4em 1rem .8em"><a href="https://kevincox.ca/2022/12/21/wget-mirror/#--convert-links" style="box-sizing:border-box;color:inherit!important;margin:0;text-decoration:inherit;text-decoration:underline;-webkit-text-decoration-color:gray;text-decoration-color:gray;text-shadow:inherit!important"><code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">--convert-links</code></a></h3>
<p style="box-sizing:border-box;grid-column:2;margin:0 1rem .8em">Updates links in the content to relative links. For example if the page contains <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap"><span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">&lt;a</span> <span style="box-sizing:border-box;color:#ffb433;margin:0;padding:0">href=</span><span style="box-sizing:border-box;color:#b3d733;margin:0;padding:0">https://example.com/foo</span><span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">&gt;</span></code> will be converted to <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap"><span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">&lt;a</span> <span style="box-sizing:border-box;color:#ffb433;margin:0;padding:0">href=</span><span style="box-sizing:border-box;color:#b3d733;margin:0;padding:0">foo</span><span style="box-sizing:border-box;color:#aaf;margin:0;padding:0">&gt;</span></code> so that it works as expected when browsing the archive. It will also convert links that haven’t been archived to be absolute so that they continue working.</p>
<h3 id="--mirror" style="box-sizing:border-box;grid-column:2;margin:.4em 1rem .8em"><a href="https://kevincox.ca/2022/12/21/wget-mirror/#--mirror" style="box-sizing:border-box;color:inherit!important;margin:0;text-decoration:inherit;text-decoration:underline;-webkit-text-decoration-color:gray;text-decoration-color:gray;text-shadow:inherit!important"><code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">--mirror</code></a></h3>
<p style="box-sizing:border-box;grid-column:2;margin:0 1rem .8em">This turns on a handful of options that are recommended for mirroring. See the man page for details.</p>
<h3 id="--no-parent" style="box-sizing:border-box;grid-column:2;margin:.4em 1rem .8em"><a href="https://kevincox.ca/2022/12/21/wget-mirror/#--no-parent" style="box-sizing:border-box;color:inherit!important;margin:0;text-decoration:inherit;text-decoration:underline;-webkit-text-decoration-color:gray;text-decoration-color:gray;text-shadow:inherit!important"><code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">--no-parent</code></a></h3>
<p style="box-sizing:border-box;grid-column:2;margin:0 1rem .8em">Prevent archiving of top-level pages that are “above” the root URL. For example if archiving <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">https://example.com/music</code> links to <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">https://other.example</code> will not be followed and links to <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">/work</code> will not be followed. Note that this doesn’t prevent archiving of page requisites. So if <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">https://example.com/music/player</code> loads <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">/js/music-player.js</code> that file will still be downloaded even though it is above the root.</p>
<h3 id="--page-requisites" style="box-sizing:border-box;grid-column:2;margin:.4em 1rem .8em"><a href="https://kevincox.ca/2022/12/21/wget-mirror/#--page-requisites" style="box-sizing:border-box;color:inherit!important;margin:0;text-decoration:inherit;text-decoration:underline;-webkit-text-decoration-color:gray;text-decoration-color:gray;text-shadow:inherit!important"><code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">--page-requisites</code></a></h3>
<p style="box-sizing:border-box;grid-column:2;margin:0 1rem .8em">Download files needed to display a page. For example images, JavaScript and CSS. Note that Wget doesn’t execute JavaScript so dynamically loaded resources won’t be found.</p>
<h3 id="--restrict-file-names" style="box-sizing:border-box;grid-column:2;margin:.4em 1rem .8em"><a href="https://kevincox.ca/2022/12/21/wget-mirror/#--restrict-file-names" style="box-sizing:border-box;color:inherit!important;margin:0;text-decoration:inherit;text-decoration:underline;-webkit-text-decoration-color:gray;text-decoration-color:gray;text-shadow:inherit!important"><code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">--restrict-file-names</code></a></h3>
<p style="box-sizing:border-box;grid-column:2;margin:0 1rem .8em">By default Wget will save files to whatever the URL path is. This doesn’t play well with all web servers. Besides filename restrictions many strip a query string such as <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">?v=123</code> before requesting the file. This means that if Wget saves <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">example.css?v=123</code> it will be effectively unreachable.</p>
<p style="box-sizing:border-box;grid-column:2;margin:0 1rem .8em">I use <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">windows</code> because it is built in and seems to cover the most important cases.</p>
<h3 id="--trust-server-names" style="box-sizing:border-box;grid-column:2;margin:.4em 1rem .8em"><a href="https://kevincox.ca/2022/12/21/wget-mirror/#--trust-server-names" style="box-sizing:border-box;color:inherit!important;margin:0;text-decoration:inherit;text-decoration:underline;-webkit-text-decoration-color:gray;text-decoration-color:gray;text-shadow:inherit!important"><code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">--trust-server-names</code></a></h3>
<p style="box-sizing:border-box;grid-column:2;margin:0 1rem .8em">This flag changes how Wget names files when it encounters a redirect. By default it saves based on the initial URL, this makes it save based on the final URL. This is useful because of how Wget deals with mutliple URLs that end up at the same URL after resolving any redirects. By default it will pick a random initial URL and rewrite all links to that, this can result in weird URLs appearing. For example Wordpress sites have links to <code style="background:#000;border:1px solid #555;border-radius:.3em;box-sizing:border-box;color:#fff;margin:0;padding:.1em;white-space:pre-wrap">/index.html?p=985</code> which redirect to the regular post URL. Often Wget will end up selecting one of these and rewriting all of the links to point at it which is unsightly.</p>
<h2 id="other-options" style="box-sizing:border-box;grid-column:2;margin:.4em 1rem .8em"><a href="https://kevincox.ca/2022/12/21/wget-mirror/#other-options" style="box-sizing:border-box;color:inherit!important;margin:0;text-decoration:inherit;text-decoration:underline;-webkit-text-decoration-color:gray;text-decoration-color:gray;text-shadow:inherit!important">Other Options</a></h2>
<p style="box-sizing:border-box;grid-column:2;margin:0 1rem .8em">I’m not aware of any other easy options for full-site archiving. One interesting approach would be using something like <a href="https://github.com/assetgraph/assetgraph" style="box-sizing:border-box;margin:0;text-decoration:underline;-webkit-text-decoration-color:gray;text-decoration-color:gray">AssetGraph</a> to fetch the site. It has a much more powerful analysis engine (including parsing scripts) that would catch more dynamic URL references as well as a scriptable AST-like reference that would allow for better cleanup of the site (for example merging URLs with different parameters but identical content).</p></main>