romt 0.9.0
Romt (Rust Offline Mirror Tool) enables mirroring of Rust programming language tools and crates for use in an offline context.
License: MIT
Requires Python: <3.14,>=3.9
Keywords: Rust, mirror, toolchain, crates

*******************************
ROMT - Rust Offline Mirror Tool
*******************************

Romt (Rust Offline Mirror Tool) aids in using the Rust programming language in
an offline context.  Instructions and tooling are provided for:

- Mirroring of Rust ecosystem artifacts:

  - Toolchains (Rustc, Cargo, libraries, etc.)
  - Rustup (toolchain multiplexer)
  - Crates.io (community-supplied Crates) with "sparse" index support.

- Incremental artifact downloading (with a configurable number of simultaneous
  download jobs).

- Incremental artifact transfer to offline network.

- Artifact serving in offline context (offline computer, disconnected network).

Scenarios
=========

Romt support two main mirroring scenarios:

- Development laptop scenario:  Download Rust artifacts to the laptop when
  connected to the Internet, then serve the artifacts from the laptop when
  offline.

- Disconnected network scenario:  Download Rust artifacts on an
  Internet-connected "Export" machine, transfer them to an offline network, then
  serve the artifacts from an offline "Import" machine.

Instructions are provided for serving the artifacts using Romt itself via
unencrypted HTTP or via the nginx web server.

Alternative Tooling
===================

- Panamax is an alternative to Romt, written in Rust:
  https://github.com/panamax-rs/panamax

Requirements
============

- Python 3.8+ for running ``romt`` (requires some packages from pypi.org).
- Git is required for manipulating the crates.io-index repository.
- Internet-connected computer for initial downloading (Linux, Windows, Mac
  [#]_).
- Offline computer for serving artifacts (Linux, Windows, Mac).
- [Optional] Gnu Privacy Guard (GPG), if installed, is used used for signature
  checking.

.. [#] Note: the author does not have access to a Mac, so support for Romt on
   Mac is untested but hopefully close to working.  Pull requests for
   Mac-specific fixes are welcome.

Romt installation
=================

.. note::

  Take note of the instructions for upgrading from Romt versions before 0.4.0
  if you have existing crate mirrors created from older Romt versions.

Install prerequisites
---------------------

First install prerequisites for Romt:

- Ensure Git is installed; it is required for proper manipulation of the
  crates.io-index repository.

- For signature checking, GNU Privacy Guard (gpg) should be installed as well.
  If gpg is not available, signature files (``*.asc``) will still be transferred
  but signature checking will be skipped.

Next, choose an option for installation of Romt itself.

Option 1: Install a pre-built executable
----------------------------------------

The simplest method of installation is to use a pre-built self-contained
executable from the Github release area:
https://github.com/drmikehenry/romt/releases

Option 2: Install from Python Package Index
-------------------------------------------

Romt is also available in the Python Package Index (PyPI).

- Download ``romt`` source and all dependencies on a host with direct Internet
  access:

  - Prepare ``romt`` download area:

    .. code-block:: sh

      mkdir romt
      cd romt

  - Download poetry and dependencies:

    .. code-block:: sh

      pip download poetry

  - Download the ``romt`` source:

    .. code-block:: sh

      pip download --no-binary :all: --no-deps romt

  - Unpack the ``romt`` source tarball ``romt-*.tar.gz``:

    .. code-block:: sh

      # Example for Linux:
      tar -zxf romt-*.tar.gz

  - Download the dependencies from ``requirements.txt``:

    .. code-block:: sh

      pip download -r romt-*/requirements.txt

- If installing to an offline host, transfer the entire ``romt/`` download area
  to that host.

- Ensure that the ``PATH`` contains the directory that holds installed Python
  packages::

      # For Linux:
      ~/.local/bin

      # For Windows with Python version X.Y:
      %APPDATA%\Python\PythonXY\Scripts

- Install ``romt`` from the current directory of sources (ensuring the current
  working directory is the ``romt/`` download area):

  .. code-block:: sh

    pip install --user --no-index --find-links . romt

Option 3: Work with source
--------------------------

If desired, the source may be cloned from Github and installed into a virtual
environment.

- Install Poetry globally as described in the documentation:
  https://python-poetry.org/docs/#installation

  Include the ``poetry-plugin-export`` plugin as well.  Assuming ``pipx`` was
  used for installation of poetry itself, this is done via::

    pipx inject poetry poetry-plugin-export

  This plugin is needed for generating a ``requirements.txt`` file.

  Optionally include the ``poetry-plugin-shell`` plugin as well.  Assuming
  ``pipx`` was used for installation of poetry itself, this is done via::

    pipx inject poetry poetry-plugin-shell

  This plugin is needed for running ``poetry shell``.  This is optional, as
  ``poetry run some_command`` can be used to run ``some_command`` instead of
  doing ``poetry shell`` followed by ``some_command``.

- Clone source:

  .. code-block:: sh

    git clone https://github.com/drmikehenry/romt
    cd romt

- Run a Poetry install (which creates a virtual environment installed with Romt
  and all dependencies)::

    poetry install

- Optionally build an executable for your platform; this requires ``poetry run``
  to run ``nox`` within the Poetry virtual environment::

    poetry run nox -s build

  Find executables in ``dist/`` tree based on your platform, e.g.::

    dist/x86_64-linux/romt
    dist/x86_64-windows/romt.exe
    dist/aarch64-darwin/romt

Romt usage overview
===================

Romt is a Python-based command-line tool with several commands:

- ``romt toolchain``: mirror and manage Rust toolchains.
- ``romt rustup``: mirror and manage Rustup.
- ``romt crate``: mirror and manage crate files from crates.io.
- ``romt serve``: simple HTTP server for toolchains, rustup, and crates.

See ``romt --help`` for overall usage help.

In particular, note that ``romt --readme`` will display the contents of this
README file for reference.

Quick-start development-laptop server
=====================================

For the development-laptop scenario, follow these steps to get a working server
configuration with mirrored Rust content.

- Ensure the laptop has Internet access.

- Install Romt (as above).

- Create area for mirrored artifacts:

  .. code-block:: sh

    mkdir mirror
    cd mirror

- Download latest stable toolchain:

  .. code-block:: sh

    # Change ``linux`` to ``windows`` or ``darwin`` as appropriate:
    romt toolchain -v -s stable -t linux download

- Download latest stable rustup version:

  .. code-block:: sh

    # Change ``linux`` to ``windows`` or ``darwin`` as appropriate:
    romt rustup -v -s stable -t linux download

- Setup crate mirror (one-time only):

  .. code-block:: sh

    romt crate init

- Download full crates.io mirror:

  .. code-block:: sh

    romt crate -v --keep-going update

  .. note::

    A few crates have been removed from crates.io and are therefore not
    available, so a few download failures (``403 Client Error: Forbidden``)
    should be expected.  The ``--keep-going`` option allows romt to continue
    in the face of these missing crates.

    Currently (April 2022), versions of the following crates are missing:

    - bork
    - css-modules
    - css-modules-macros
    - deploy
    - doccy
    - etch
    - glib-2-0-sys
    - glue
    - gobject-2-0-sys
    - peek
    - pose

- Configure crate mirror to be served from localhost (one-time only):

  .. code-block:: sh

    romt crate config

- Start Romt as a server on http://localhost:8000:

  .. code-block:: sh

    romt serve

  .. note::

    Leave the server running in this dedicated terminal.

Quick-start disconnected-network server
=======================================

Setting up a server for the disconnected-network scenario is similar to that for
the development-laptop scenario above; explanations that overlap that scenario
are omitted below.

- On Internet-connected Export machine:

  - Install Romt (as above).

  - Create area for mirrored artifacts:

    .. code-block:: sh

      mkdir mirror
      cd mirror

  - Download latest stable toolchain and create ``toolchain.tar.gz``:

    .. code-block:: sh

      # Change ``linux`` to ``windows`` or ``darwin`` as appropriate:
      romt toolchain -v -s stable -t linux download pack

  - Download latest stable rustup version and create ``rustup.tar.gz``:

    .. code-block:: sh

      # Change ``linux`` to ``windows`` or ``darwin`` as appropriate:
      romt rustup -v -s stable -t linux download pack

  - Setup crate mirror (one-time only):

    .. code-block:: sh

      romt crate init

  - Download and create ``crates.tar.gz``:

    .. code-block:: sh

      romt crate -v --keep-going export

  - Transfer ``toolchain.tar.gz, ``rustup.tar.gz``, and ``crates.tar.gz`` to
    Import machine.

- On Disconnected network Import machine:

  - Install Romt (as above).

  - Create area for mirrored artifacts (one-time only):

    .. code-block:: sh

      mkdir mirror

  - Place exported ``toolchain.tar.gz, ``rustup.tar.gz``, and ``crates.tar.gz``
    files into this ``mirror/`` directory, and enter the directory at a prompt:

    .. code-block:: sh

      cd mirror

  - Import toolchain and rustup:

    .. code-block:: sh

      romt toolchain -v unpack
      romt rustup -v unpack

  - Setup crate mirror (one-time only):

    .. code-block:: sh

      romt crate init-import

  - Import ``crates.tar.gz``:

    .. code-block:: sh

      romt crate -v --keep-going import

  - Configure crate mirror to be served from localhost (one-time only):

    .. code-block:: sh

      romt crate config

  - Start Romt as a server on http://localhost:8000:

    .. code-block:: sh

      romt serve

    .. note::

      Leave the server running in this dedicated terminal.

Quick-start client setup
========================

Follow these steps to configure Rust tooling for use with a mirror server on
localhost using either Quick-start server configuration above.

- Setup environment variables to point to the server.  By default, this will be
  at http://localhost:8000; adjust all uses of ``localhost:8000`` below for
  different server address:port combinations:

  .. code-block:: sh

    # For Linux/Mac:
    export RUSTUP_DIST_SERVER=http://localhost:8000
    export RUSTUP_UPDATE_ROOT=http://localhost:8000/rustup

    # For Windows:
    set RUSTUP_DIST_SERVER=http://localhost:8000
    set RUSTUP_UPDATE_ROOT=http://localhost:8000/rustup

  .. note::

    These variables must be set in each terminal window before using the mirror
    server.

- Download the ``rustup-init`` installer for your platform from the Romt server
  using the appropriate URL below, saving it into the current directory:

  - Linux:
    http://localhost:8000/rustup/dist/x86_64-unknown-linux-gnu/rustup-init

  - Windows:
    http://localhost:8000/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe

  - Mac:
    http://localhost:8000/rustup/dist/aarch64-apple-darwin/rustup-init

- Run the installer, accepting the defaults:

  .. code-block:: sh

    # Linux/Mac:
    chmod +x rustup-init
    ./rustup-init

    # Windows
    rustup-init

- Ensure environment changes take place in current shell:

  .. code-block:: sh

    # For Linux/Mac:
    source $HOME/.cargo/env

    # For Windows:
    PATH %USERPROFILE%\.cargo\bin;%PATH%

- Try out some rustup commands::

    rustup self update
    rustup component add rust-src

- Create the text file ``~/.cargo/config.toml``
  (``%USERPROFILE%\.cargo\config.toml`` on Windows) to use ``romt serve``. With
  a Rust toolchain from 2022-06-20 or later, the "sparse" protocol may be used.
  This is significantly faster than the older Git-based method.

  - For the sparse index method, use the following contents for the
    ``config.toml`` file::

      [source.crates-io]
      registry = 'sparse+http://localhost:8000/crates-index/'

      # Disable cert revocation checking (necessary only on Windows):
      [http]
      check-revoke = false

  - For the older Git-based index method, use the following contents for the
    ``config.toml`` file::

      [source.crates-io]
      registry = 'http://localhost:8000/git/crates.io-index'

      # Disable cert revocation checking (necessary only on Windows):
      [http]
      check-revoke = false

      # For greatly improved performance, have Cargo use the Git command-line
      # client to acquire `crates.io-index` repository. See
      # https://github.com/rust-lang/cargo/issues/9167 for details.
      [net]
      git-fetch-with-cli = true

- Create a sample project to demonstrate crate usage:

  .. code-block:: sh

    cargo new rand_test
    cd rand_test

- Add the ``rand`` crate to the build:

  .. code-block:: sh

    cargo add rand

- Fetch ``rand`` and its dependencies::

    cargo fetch

Upgrading Romt
==============

When upgrading Romt, it's recommended to use the same version of Romt on both
the Internet-connected and offline hosts.

Romt aims for a high degree of backward compatibility.  Mostly you should be
able to use a newer version of Romt without issue.

However, if your crate mirror was created by Romt prior to version 0.4.0
(released in April, 2022), you may be using mixed-case prefixes without a Crate
configuration file; if that's your case, you'll need to see the section
"Mixed-case crate prefixes" for more information how prefix handling has changed
in Romt 0.8.0 and 0.4.0.

Commonalities
=============

Romt has some features that are shared across two or more commands.

TARGET
------

The TARGET specifies the platform for executables using standard tuple values
(e.g., ``x86_64-unknown-linux-gnu``).  Any tuples supported by Rust are valid.
Typical values are shown below; in parentheses are aliases Romt provides for
ease of typing these common targets:

- ``x86_64-unknown-linux-gnu`` (alias ``linux``)
- ``x86_64-pc-windows-msvc`` (alias ``windows``)
- ``aarch64-apple-darwin`` (alias ``darwin``)

TARGET values are given by the option ``--target TARGET``.  Multiple TARGET
options may be given, and each TARGET will be split at commas and whitespace to
produce a list of desired TARGET values, e.g.::

  --target linux,windows --target 'darwin i686-pc-windows-msvc'

A TARGET may be a literal ``all`` that expands to all known targets.  For ``romt
toolchain``, this list comes from the manifest file.  For ``romt rustup``, it
comes from a hard-code list within Romt; this is an ever-changing list that may
be out-of-date in an old release of Romt.

A TARGET may be a literal ``*`` (asterisk) that expands to all targets with at
least one on-disk file for the given SPEC.

SHA256 hashes
-------------

- Each file named ``{file}.sha256`` contains the SHA256 hash of the
  corresponding file named ``{file}``.  Romt verifies all hashes to ensure file
  integrity.  Where possible, Romt will create ``{file}.sha256`` files for
  toolchain components based on the hash provided in the toolchain manifest.
  This allows Romt to work with mirrors that do not contain these ``.sha256``
  files.

Command-line option details
---------------------------

- The option ``--num-jobs`` controls how many simultaneous download jobs Romt
  may use at a time.  By default, ``--num-jobs=4``, which should be a
  conservative value that won't stress the servers heavily.

- The option ``--timeout`` controls the timeout in seconds for downloading.
  A value of zero disables the timeout functionality altogether.

- The option ``--assume-ok`` instructs Romt that all files already on-disk are
  to be assumed OK; no hashes or signatures are checked for such files.

SSL verification
----------------

Romt provides control over SSL verification of servers using HTTPS.  Romt uses
the ``httpx`` library for downloading Rust artifacts.  By default, ``httpx``
always verifies server certificates.  Verification is based on a bundled set of
certificate authorities; in particular, ``httpx`` will not use certificate
authorities installed in the host system.  Romt uses several environment
variables to control server certificate verification:

- ``SSL_VERIFY`` allows verification to be disabled.  If this variable is set to
  the string ``false``, verification will be disabled.  The string is compared
  case-insensitively.

- ``SSL_CERT_FILE`` may point to a certificate file.  For example, to use the
  system certificates on an Ubuntu system::

    SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

- ``SSL_CERT_DIR`` may point to directory of certificate files.  To use the
  certificates found in the directory ``/etc/ssl/certs``::

    SSL_CERT_DIR=/etc/ssl/certs

``toolchain`` operation
=======================

The ``toolchain`` operation deals with Rust toolchains.

SPEC
----

Each toolchain is identified by a SPEC value which takes on one of the below
forms::

  {channel}
  {channel}-{date}
  {date}

In the above SPEC forms:

- ``{channel}`` is typically one of the channel names ``nightly``, ``beta``,
  ``stable``.  It may also be a version number of the form ``X.Y.Z`` or a
  literal ``*`` (asterisk) as a wildcard that expands to the set
  ``nightly,beta,stable``.

- ``{date}`` is typically of the form ``YYYY-MM-DD`` (e.g., ``2020-04-30``).  It
  may also be a literal ``*`` (asterisk) as a wildcard that expands to all
  toolchain dates on-disk, or a literal ``latest`` that expands to the most
  recent toolchain date on-disk.

- Note that a SPEC value consisting of a single ``*`` represents a wildcarded
  ``{date}`` value, not a ``{channel}`` value.  It is equivalent to ``*-*``
  (making both ``{channel}`` and ``{date}`` wild).

- Wildcards (``*`` and ``latest``) may not be used when downloading, and the
  ``{channel}`` is always required.  The ``{date}`` field may be omitted to
  download the most recent toolchain for the given channel.

- SPEC values are given by the option ``--select SPEC``.  Multiple SPEC options
  may be given, and each SPEC will be split at commas and whitespace to produce
  a list of desired SPEC values.  E.g.::

    --select nightly,stable --select beta-2020-01-23

TARGET
------

See the TARGET section of Commonalities above for details.

Manifest file
-------------

A manifest file provides details about a toolchain for a given SPEC, enumerating
valid combinations of toolchain components and targets.

The manifest filename is of the form ``channel-rust-{channel}.toml``, where
``{channel}`` is one of ``nightly``, ``beta``, or ``stable``.  For ``stable``
manifests, the manifest is duplicated into a file of the form
``channel-rust-{version}.toml``, where ``{version}`` is a version number of the
form ``X.Y.Z``.

Downloading
-----------

Downloading is requested via the ``romt toolchain download`` command.

A toolchain is specified by a SPEC/TARGET pair.  Both must be given.
Wildcarding (via ``*`` or ``latest``) is not permitted, though the ``{date}``
may be omitted from the SPEC value, and TARGET may be the literal ``all`` to
download all known targets for the SPEC.

By default, all toolchain components will be downloaded; but when the switch
``--cross`` is supplied, only the Rust standard library component ``rust-std``
will be downloaded.  This is to support cross-compilation to a given target
without the need to download all toolchain components for that target.

Rust provides toolchain files in the ``dist/`` subdirectory of the Rust dist
server.  The dist server defaults to ``https://static.rust-lang.org``, but this
default may be overridden in Rust tooling and in Romt by setting the environment
variable ``RUST_DIST_SERVER``.  If ``RUST_DIST_SERVER`` is unset, files are by
default downloaded from ``https://static.rust-lang.org/dist``; otherwise, file
are by default downloaded from ``$RUSTUP_DIST_SERVER/dist``.  This default
download URL may be changed via the option ``--url <URL>`` (e.g., ``romt
toolchain --url http://my.server/rust/dist``).

Files are downloaded to the destination directory ``dist/`` by default; this
may be changed via the option ``--dest DEST``.

When downloaded, the toolchain will be stored on-disk in the following layout::

  dist/
    YYYY-MM-DD/
      channel-rust-{channel}.toml
      channel-rust-{channel}.toml.asc
      channel-rust-{channel}.toml.sha256
      {component}-{channel}.tar.xz
      {component}-{channel}.tar.xz.asc
      {component}-{channel}.tar.xz.sha256
      {component}-{channel}-{target}.tar.xz
      {component}-{channel}-{target}.tar.xz.asc
      {component}-{channel}-{target}.tar.xz.sha256

Where:

- ``YYYY-MM-DD`` is the toolchain date.
- ``{channel}`` is one of ``nightly``, ``beta``, or ``stable``.
- ``{component}`` represents a toolchain component (e.g., ``rust``, ``cargo``,
  ``rust-src``).
- ``{target}`` represents a target tuple (e.g., ``x86_64-unknown-linux-gnu``).
  Components lacking a ``{target}`` are common across all targets; currently
  this is limited to the ``rust-src`` component.

- Each file named ``{file}.asc`` contains the Gnu Privacy Guard (GPG) digital
  signature of the corresponding file named ``{file}``.  Checking signature
  requires GPG; if it is not installed, signature files won't be checked but
  they will still be transferred.  The verification key is available at
  https://static.rust-lang.org/rust-key.gpg.ascii; this key is built into Romt
  itself for offline use.

For example, after downloading with this command:

.. code-block:: sh

  romt toolchain download --select nightly-2020-04-30 --target linux

The tree would contain (among other files)::

  dist/
    2020-04-30/
      channel-rust-nightly.toml
      channel-rust-nightly.toml.asc
      channel-rust-nightly.toml.sha256
      rust-src-nightly.tar.xz
      rust-src-nightly.tar.xz.asc
      rust-src-nightly.tar.xz.sha256
      rust-nightly-x86_64-unknown-linux-gnu.tar.xz
      rust-nightly-x86_64-unknown-linux-gnu.tar.xz.asc
      rust-nightly-x86_64-unknown-linux-gnu.tar.xz.sha256

For convenience, the most recently released toolchain for each channel
(``nightly``, ``beta``, or ``stable``) will be copied directly into the
``dist/`` directory.  This is especially helpful for ``stable`` and ``beta``
builds so that the date of the most recent release need not be known in advance.
For ``stable`` manifests, the version-specific copy of the manifest is placed
into ``dist/`` as well.

For example, as of 2020-05-06, the most recent manifests were for SPEC values
of:

- ``nightly-2020-05-06``
- ``beta-2020-04-26``
- ``stable-2020-04-23`` (version ``1.43.0``)

On that date, performing a download with ``--target linux`` and ``--select
nightly,beta,stable`` would yield the following downloaded manifests::

  dist/
    channel-rust-beta.toml
    channel-rust-nightly.toml
    channel-rust-stable.toml
    channel-rust-1.43.0.toml
    2020-04-23/
      channel-rust-stable.toml
      channel-rust-1.43.0.toml
    2020-04-26/
      channel-rust-beta.toml
    2020-05-06/
      channel-rust-nightly.toml

Where the dateless manifests housed directly in ``dist/`` are copies of those
from the dated directories.

Because the contents of dateless manifests are subject to change, cached copies
of these files are re-downloaded during a ``download`` command.

When Romt downloads a top-level manifest, it will copy it into the corresponding
dated directory with its canonical name. This allows Romt to work with mirrors
that lack manifests in the dated directories.  For example, Rust 1.76.0 lives in
dated directory 2024-02-08.  A top-level manifest of
``channel-rust-1.76.0.toml`` will be copied to the canonical location and name
``2024-02-08/channel-rust-stable.toml``.

Packing/unpacking
-----------------

Downloaded toolchains may be packed into an ``ARCHIVE`` file using the ``romt
toolchain pack`` command.

The archive file may be moved to another machine and unpacked using the ``romt
toolchain unpack`` command.

Romt will detect when a toolchain has been downloaded via the ``--cross``
switch, in which case only the ``rust-std`` component (along with whatever other
toolchain components are present, if any) will be processed.

For both ``pack`` and ``unpack``, the ``ARCHIVE`` file is named
``toolchain.tar.gz`` by default; this may be changed via the option ``--archive
ARCHIVE``.

An ``unpack`` command automatically performs a ``verify`` (described below).  In
addition, dateless manifests are reconstructed automatically during ``unpack``
as part of a fixup operation (described below).

An archive file contains files from dated subdirectories only.  Given the
example above for the ``download`` command, the ``ARCHIVE`` would contain only
these manifests::

  dist/
    2020-04-23/
      channel-rust-stable.toml
    2020-04-26/
      channel-rust-beta.toml
    2020-05-06/
      channel-rust-nightly.toml

Fixup
-----

Each toolchain identified by a SPEC has a canonical manifest file stored in the
toolchain's dated directory.  This file has a path of the form
``YYYY-MM-DD/channel-rust-{channel}.toml``, where ``{channel}`` is one of the
channel names ``nightly``, ``beta``, or ``stable``.

The "fixup" operation is responsible for making any necessary copies of each
canonical manifest in the ``dist/`` tree.  If the given on-disk manifest is
found in the latest dated directory, it will be copied into the top-level
``dist/`` directory.  In addition, for each SPEC on the ``stable`` channel a
version-specific manifest file of the form ``channel-rust-X.Y.Z.toml`` will be
copied into the dated directory and the top-level ``dist/`` directory.

A fixup operation may be explicitly requested via the ``romt toolchain fixup``
command, though that should rarely be required because it is automatically
performed after any ``download`` or ``unpack`` command.

Consider the example above for the ``download`` command; it would generate an
archive containing only these canonical manifests::

  dist/
    2020-04-23/
      channel-rust-stable.toml
    2020-04-26/
      channel-rust-beta.toml
    2020-05-06/
      channel-rust-nightly.toml

The ``fixup`` command would copy these manifests to create::

  dist/
    channel-rust-beta.toml
    channel-rust-nightly.toml
    channel-rust-stable.toml
    channel-rust-1.43.0.toml
    2020-04-23/
      channel-rust-stable.toml
      channel-rust-1.43.0.toml
    2020-04-26/
      channel-rust-beta.toml
    2020-05-06/
      channel-rust-nightly.toml

Listing downloaded toolchains
-----------------------------

The ``romt toolchain list`` command prints information about on-disk toolchains
for the provided SPEC values.  Wildcards are permitted.

For example, the most recent on-disk ``stable`` release can be shown via:

.. code-block:: sh

  romt toolchain list --select 'stable-latest'

With example output::

  stable-2024-02-08(1.76.0)    targets[10/94]   packages[34/425]
    x86_64-unknown-linux-gnu                      native-target
    x86_64-unknown-linux-musl                     cross-target

Next to each target name is that target's "type", one of:

- ``native-target`` (a full toolchain)
- ``cross-target`` (a toolchain for cross-compilation)
- ``minimal`` (minimal toolchain components with no compilation support)

A ``native-target`` contains a full toolchain capable of running on the target
natively (and compiling Rust code to that target as well).

A ``cross-target`` does not contain a compiler that runs on the target; but it
does contain the Rust standard library component ``rust-std`` for the target,
enabling cross-compilation to the target (using a ``native-target`` toolchain on
another host).

A ``minimal`` target lacks ``rust-std`` but has some minimal components
available.  Typically a minimal target shows up by coincidence because it shares
one or more components with another target.  For example, at the time of this
writing the minimal target ``mips-unknown-linux-gnu`` has no components of its
own in toolchain 1.76.0, but it shares the component ``rust-docs`` with the more
common target ``x86_64-unknown-linux-gnu``; therefore, downloading the full
toolchain for ``x86_64-unknown-linux-gnu`` will cause ``mips-unknown-linux-gnu``
to be present as a minimal toolchain.

Because most ``minimal`` targets are present only by coincidence and not useful,
listing them is suppressed by default.  Use ``--verbose`` to include them,
e.g.::

.. code-block:: sh

  romt toolchain list --select 'stable-latest' --verbose

With example output::

  List: stable-2024-02-08
  [verify] dist/2024-02-08/channel-rust-stable.toml
  stable-2024-02-08(1.76.0)    targets[10/94]   packages[34/425]
    mips-unknown-linux-gnu                        minimal
    mips64-unknown-linux-gnuabi64                 minimal
    mips64el-unknown-linux-gnuabi64               minimal
    mipsel-unknown-linux-gnu                      minimal
    mipsisa32r6-unknown-linux-gnu                 minimal
    mipsisa32r6el-unknown-linux-gnu               minimal
    mipsisa64r6-unknown-linux-gnuabi64            minimal
    mipsisa64r6el-unknown-linux-gnuabi64          minimal
    x86_64-unknown-linux-gnu                      native-target
    x86_64-unknown-linux-musl                     cross-target

To suppress information about targets, use ``--quiet``:

.. code-block:: sh

  romt toolchain list --select 'stable-latest' --quiet

With example output::

  stable-2024-02-08(1.76.0)

With wildcards, Romt can provide a listing of all available toolchains for a
given channel:

.. code-block:: sh

  romt toolchain list -s 'nightly-*'

With example output::

  nightly-2024-02-14(1.78.0)   targets[9/94]    packages[54/486]
    x86_64-unknown-linux-gnu                      native-target
  nightly-2023-10-31(1.75.0)   targets[9/95]    packages[54/488]
    x86_64-unknown-linux-gnu                      native-target
  nightly-2023-07-04(1.72.0)   targets[5/96]    packages[53/529]
    x86_64-unknown-linux-gnu                      native-target

After toolchain importation, it may be useful to list toolchains for each
channel for reference:

.. code-block:: sh

  romt toolchain list -s 'nightly-*' > nightly.txt
  romt toolchain list -s 'beta-*' > beta.txt
  romt toolchain list -s 'stable-*' > stable.txt

``toolchain`` scenarios
-----------------------

For the laptop scenario, only the ``download`` command is needed.  After
downloading a toolchain, it will be available for serving via ``romt serve``
(or other means).  For example, to download the latest stable toolchain for
Linux:

.. code-block:: sh

    romt toolchain download --select stable --target linux

For the disconnected network scenario, toolchains are downloaded and packed on
an Internet-connected Export machine, then unpacked on an Import machine, e.g.:

- On the Export machine:

  - First, download the latest stable toolchain for Linux into a local ``dist/``
    directory and pack it into an archive for transfer:

    .. code-block:: sh

      romt toolchain download pack --select stable --target linux

  - Transfer the resulting ``toolchain.tar.gz`` file onto the Import machine.

- On the Import machine:

  - Unpack the archive into a local ``dist/`` directory:

    .. code-block:: sh

      romt toolchain unpack

Miscellaneous commands
----------------------

A few additional commands are provided for ``romt toolchain``.

``romt toolchain fetch-manifest`` is the same as ``download``, but only the
manifest is downloaded.

``romt toolchain verify`` validates the SHA256 hashes and GPG signatures of
on-disk toolchains.  It is implicitly done as part of ``download`` and
``unpack``.

``romt toolchain all-targets`` prints a list of all known targets mentioned in
the given SPEC.

``romt toolchain all-components`` prints a list of all known toolchain
components mentioned in the given SPEC.  This is useful to determine the
component names for use with the ``--components`` option.  Sample component
names include::

    cargo
    clippy-preview
    llvm-bitcode-linker-preview
    llvm-tools-preview
    reproducible-artifacts
    rls-preview
    rust
    rust-analysis
    rust-analyzer-preview
    rust-docs
    rust-mingw
    rust-src
    rust-std
    rustc
    rustc-dev
    rustc-docs
    rustfmt-preview

Command-line option details
---------------------------

The option ``--warn-signature`` instructs Romt to treat signature failures as
warnings instead of as failures.  Signature files will still be downloaded and
transferred.  This might be helpful in case the signing key changes.

The option ``--no-signature`` prevents both downloading and checking of GPG
signature files (``*.asc``).  This is mainly for testing.

The option ``--components`` allows for processing a subset of toolchain
components.  This may be useful to ignore unnecessary components (such as
``rustc-dev`` and to work with mirrors that do not contain the complete set of
components.  By default, all components are included.  To include only a
specific subset of components, provide the list of component names to
``--components``.  Names may be separated by whitespace or commas, and
``--components`` may be provided multiple times.  For example, to include the
four components ``cargo``, ``rust``, ``rust-std``, and ``rustc``, you could use
``--components 'cargo rust' --components rust-std,rustc``.  To exclude a
component, prefix the name with ``!``; e.g., to exclude ``rust-src``, you could
use ``--components '!rust-src'``.

``rustup`` operation
====================

The ``rustup`` operation deals with the Rustup toolchain multiplexer.

SPEC
----

Each rustup version is identified by a SPEC value which takes on one of the
below forms::

  {version}
  stable
  latest
  *

In the above SPEC forms:

- ``{version}`` is a version number of the form ``X.Y.Z``.

- A literal ``stable`` refers to the current stable version given in the
  ``release-stable.toml`` file (described later).

- A literal ``*`` (asterisk) is a wildcard that expands to all on-disk versions.

- A literal ``latest`` is a wildcard that expands to the latest on-disk version.

- Wildcards (``*`` and ``latest``) may not be used when downloading, but
  ``stable`` is permitted.

- SPEC values are given by the option ``--select SPEC``.  Multiple SPEC options
  may be given, and each SPEC will be split at commas and whitespace to produce
  a list of desired SPEC values.  E.g.::

    --select stable,1.20.0 --select '1.19.0 1.20.1'

TARGET
------

See the TARGET section of Commonalities above for details.

Downloading
-----------

Downloading is requested via the ``romt rustup download`` command.

A rustup executable is specified by a SPEC/TARGET pair.  Both must be given.
Wildcarding (via ``*`` or ``latest``) is not permitted, though SPEC may be the
literal ``stable`` to download the latest stable release, and TARGET may be the
literal ``all`` to download all known targets for the SPEC.

Files are downloaded from ``https://static.rust-lang.org/rustup`` by default;
this may be changed via the option ``--url <URL>``.

Files are downloaded to the destination directory ``rustup/`` by default;
this may be changed via the option ``--dest DEST``.

When downloaded, files will be stored on-disk in the following layout::

  rustup/
    release-stable.toml
    archive/
      {version}/
        {target}/
          {rustup}
          {rustup}.sha256
    dist/
      {target}/

Where:

- ``release-stable.toml`` is a configuration file that indicates the most recent
  stable version of rustup.
- ``{version}`` is a rustup version of the form ``X.Y.Z``.
- ``{target}`` represents a target tuple (e.g., ``x86_64-unknown-linux-gnu``).
- ``{rustup}`` is the name of the rustup executable.  On most platforms, this is
  ``rustup-init``; on Windows, it's ``rustup-init.exe``.

For example, if version 1.21.1 were the most recent stable version, after
downloading with this command:

.. code-block:: sh

  romt rustup download --select stable --target linux

The tree would contain::

  rustup/
    release-stable.toml
    dist/
      x86_64-unknown-linux-gnu/
        rustup-init
        rustup-init.sha256
    archive/
      1.21.1/
        x86_64-unknown-linux-gnu/
          rustup-init
          rustup-init.sha256

For convenience, all targets found in the most recently released rustup version
will be copied directly into the ``rustup/dist/`` directory.

Because the ``release-stable.toml`` file is subject to change, this file will be
re-downloaded during a ``download`` command when SPEC is ``stable``.

Packing/unpacking
-----------------

Downloaded rustup executables may be packed into an ``ARCHIVE`` file using the
``romt rustup pack`` command.

The archive file may be moved to another machine and unpacked using the
``romt rustup unpack`` command.

For both ``pack`` and ``unpack``, the ``ARCHIVE`` file is named
``rustup.tar.gz`` by default; this may be changed via the option ``--archive
ARCHIVE``.

An ``unpack`` command automatically performs a ``verify`` (described below).  In
addition, the ``rustup/dist/`` tree is created automatically during ``unpack``
as part of a fixup operation (described below).

An archive file contains files from ``rustup/archive/{version}`` subdirectories
only.  Given the example above for the ``download`` command, the ``ARCHIVE``
would contain only these files::

  rustup/
    archive/
      1.21.1/
        x86_64-unknown-linux-gnu/
          rustup-init
          rustup-init.sha256

Fixup
-----

Each rustup version is stored in a directory of the form
``rustup/archive/{version}``.

The "fixup" operation is responsible for copying the most recent on-disk
rustup version to ``rustup/dist/``, and for updating
``rustup/release-stable.toml`` to contain the most recent version number.

A fixup operation may be explicitly requested via the ``romt rustup fixup``
command, though that should rarely be required because it is automatically
performed after any ``download`` or ``unpack`` command.

Consider the example above for the ``download`` command that generated the
following archive contents::

  rustup/
    archive/
      1.21.1/
        x86_64-unknown-linux-gnu/
          rustup-init
          rustup-init.sha256

Assuming this is the latest on-disk version, the ``fixup`` command would copy
``rustup/archive/1.21.1`` to ``rustup/archive`` as shown below, and it would
create ``release-stable.toml`` to point to version ``1.21.1``::

  rustup/
    release-stable.toml
    archive/
      1.21.1/
        x86_64-unknown-linux-gnu/
          rustup-init
          rustup-init.sha256
    dist/
      x86_64-unknown-linux-gnu/
        rustup-init
        rustup-init.sha256

Listing downloaded rustup versions
----------------------------------

The ``romt rustup list`` command prints information about on-disk rustup
versions for the provided SPEC values.  Wildcards are permitted.

For example, the most recent on-disk version can be shown via:

.. code-block:: sh

  romt rustup list --select 'latest'

With example output::

  List: 1.21.1
  1.21.1   targets[1]
    x86_64-unknown-linux-gnu

To suppress information about targets, use ``--quiet``:

.. code-block:: sh

  romt rustup list --select 'latest' --quiet

With example output::

  1.21.1

With wildcards, Romt can provide a listing of all available rustup versions:

.. code-block:: sh

  romt rustup list -s '*'

With example output::

  List: 1.21.1
  1.21.1   targets[1]
    x86_64-unknown-linux-gnu
  List: 1.21.0
  1.21.0   targets[1]
    x86_64-unknown-linux-gnu
  List: 1.20.0
  1.20.0   targets[1]
    x86_64-unknown-linux-gnu

``rustup`` scenarios
--------------------

For the laptop scenario, only the ``download`` command is needed.  After
downloading a rustup executable, it will be available for serving via ``romt
serve`` (or other means).  For example, to download the latest stable rustup for
Linux:

.. code-block:: sh

    romt rustup download --select stable --target linux

For the disconnected network scenario, rustup versions a