Unnecessary upload evasion with lftp mirrors
Tags: linux, devops
Published: 2024-06-12 07:42:00

---
title: Unnecessary upload evasion with lftp mirrors
published: true
date: 2024-06-10 22:00:00 UTC
tags: Linux,DevOps
canonical_url: https://peateasea.de/unnecessary-upload-evasion-with-lftp-mirrors/
cover_image: https://peateasea.de/assets/images/avoid-old-file-upload.png
---

*Originally published on [peateasea.de](https://peateasea.de/unnecessary-upload-evasion-with-lftp-mirrors/).*

I’ve been using `lftp`’s reverse mirror feature for years to upload files to my blog. I’d never worked out how to avoid repeated file uploads. Until now.

## A bit of background

[Jekyll](https://jekyllrb.com/) creates and populates a directory called `_site` when building the production version of a static site. To upload the files in the `_site` directory to my hosting provider, I’ve been using an `lftp` script like this:<sup id="fnref:cannot-use-rsync" role="doc-noteref"><a href="#fn:cannot-use-rsync" rel="footnote">1</a></sup>

```
open sftp://<username>:not-a-password@ftp.<domain-name>
mirror -v --delete --reverse _site/ /public_html/
```

The `open` command opens a connection<sup id="fnref:captain-obvious" role="doc-noteref"><a href="#fn:captain-obvious" rel="footnote">2</a></sup> to the FTP server that my hosting service provides for my domain and logs me in. Since the protocol is SFTP, the connection uses SSH and I can use an SSH key for authentication. This is great because then I don’t need to use a password.

Note that the `not-a-password` component is an important placeholder: it’s there because a password is still expected before the `@` symbol even though authentication uses public key encryption. The authentication mechanism thus ignores it. I chose this placeholder value to remind me that it isn’t a password and that I shouldn’t ever put one here.

The `mirror` command normally _downloads_ files from an upstream (usually remote) source to a local system. Thus the usual mirror process _pulls_ remote files from upstream. Using the `--reverse` option swaps the sense of the mirror mechanism and files are instead _uploaded_ (i.e. pushed) to the upstream system.

In the case I describe here, I push all files from within the `_site/` directory to the `/public_html/` directory on my hosting service.

When [reading the documentation](https://lftp.yar.ru/lftp-man.html) you will see terms like “source” and “target”. When mirroring to a local system then the “source” is the remote system and the “target” is the local system. With the `--reverse` option, these are swapped and the local system is now the “source” and the upstream system is the “target”.

The `--delete` option removes any files from the target system which are not present in the source. In our case, this is anything within the `/public_html/` directory tree. This ensures that if I delete or rename a file, it isn’t still floating around on the production system, which might confuse someone in the future.

The `-v` option turns on the first level of verbosity so that I can get feedback about what’s happening when mirroring the files to the upstream system.

## The problem

So what’s the issue? Well, each time I mirror the site to production, the `lftp` script re-transfers _all_ my files. In particular, `lftp` _removes_ each file from upstream before uploading it again. This happens even if the files haven’t changed. Here’s what I mean:

```shell
$ lftp -f deploy_site.lftp
Removing old file `feed.xml'
Transferring file `feed.xml'
Removing old file `index.html'
Transferring file `index.html'
Removing old file `sitemap.xml'
Transferring file `sitemap.xml'
Removing old file `about/index.html'
Transferring file `about/index.html'
Removing old file `add-favicon-to-mm-jekyll-site/index.html'
Transferring file `add-favicon-to-mm-jekyll-site/index.html'

<snip>
```

Not only is this annoying, but it’s a waste of network resources and time. I’d tried to get `lftp` to only upload changed files in the past, but never seemed to have found the right incantation. Until today. Today, I finally found the information I needed to make this work.

## The solution

If you read the [`lftp` man page](https://lftp.yar.ru/lftp-man.html), you’ll find in the `mirror` section the `--only-newer` option. Adding this option to the `mirror` command mentioned earlier, we get

```
mirror -v --only-newer --delete --reverse _site/ /public_html/
```

Using this command you’ll find that _it still transfers all files upstream_. Gah! Why doesn’t this work?

Today I managed to stumble upon why this is so. [An answer](https://stackoverflow.com/a/15320869) to the StackOverflow question [Why lftp mirror –only-newer does not transfer “only newer” file?](https://stackoverflow.com/q/11490145) mentions a subtlety noted on [Matthieu Bouthours’ blog](https://www.bouthors.fr/wiki/doku.php?id=en:linux:synchro_lftp) and seemingly not mentioned anywhere else:

> When uploading, it is not possible to set the date/time on the files uploaded, that’s why `--ignore-time` is needed.

Therefore, as mentioned in the [StackOverflow answer](https://stackoverflow.com/q/11490145):

> [I]f you use the flag combination `--only-newer` and `--ignore-time` you can achieve decent backup properties, in such a way that all files that differ in size are replaced. Of course it doesn’t help if you really need to rely on time-synchronization but if it is just to perform a regular backup of data, it’ll do the job.

Updating the `mirror` command like so:

```
mirror -v --only-newer --ignore-time --delete --reverse _site/ /public_html/
```

fixes the issue and only uploads new or newly changed files, which is the desired behaviour. Yay! :partying_face:

Implementing this change in my build scripts reduced build and deployment times from 5.5 minutes to 2 minutes. That’s more than halved the time! Brilliant!

## A word of caution

There’s a caveat here, though. If a file is changed and just so happens to be of the same size as its counterpart upstream, it won’t be transferred. One needs to bear this in mind.

To be honest, I’d prefer to use `rsync` because it generates checksums of the files to detect file changes. Then I could be more certain that my scripts upload only newer files and don’t transfer older ones unnecessarily. However, until I have that option, this will do the job nicely.

1. Unfortunately my hosting service doesn’t allow `rsync` (at least not at my service level) and hence I can’t use a more sophisticated synchronisation mechanism. [↩](#fnref:cannot-use-rsync)

2. Thank you, [Captain Obvious!](https://peateasea.de/assets/images/captain-obvious-imgur-com.gif) [↩](#fnref:captain-obvious)