Patching Salt Modules
Tags: devops, saltstack, salt, iac
Published: 2020-01-27 13:26:59

---
title: Patching Salt Modules
published: true
date: 2020-01-27 13:26:36 UTC
tags: devops, saltstack, salt, iac
cover_image: https://salt.tips/patching-salt-modules/salt-modules-cover.png
canonical_url: https://salt.tips/patching-salt-modules/
---

Let’s say you were bitten by a critical bug in Salt (2019.2.0 & 2019.2.1 were [painful for many of us](https://github.com/saltstack/salt/issues/54755)) that had an existing but unreleased fix. Or a cool feature that you badly need has landed in the old develop branch and is waiting forever to be [migrated to the new master](https://github.com/saltstack/salt-enhancement-proposals/blob/master/accepted/0014-dev-overhaul.md#pr-migration-plan). Even if it was released, maybe you can’t yet upgrade Salt across the whole minion fleet. Or maybe you were wrongly expecting that your PR against the master branch would be included in the nearest release but were told that the upcoming release is feature frozen, and you have to wait another 4 months for the next one.

As a DevOps engineer, you need to get things done quickly. Fortunately, SaltStack has a built-in way to patch itself! After reading this guide, you’ll be able to patch almost any part of Salt code, even if you run multiple versions simultaneously. Some of the ways are hacky, but bugs are bugs, and you need to fix them in the most efficient way.

The methods described are mostly suitable for applying a couple of simple fixes. If you have a large or constantly evolving patchset, you’ll keep yourself sane by making your own fork and installing it via pip or custom packages.

## Contents

- [Determining affected minions](#determining-affected-minions)
- [How to sync Salt modules](#how-to-sync-salt-modules)
- [Extracting upstream patches](#extracting-upstream-patches)
- [Overriding a whole module](#overriding-a-whole-module)
- [Adding a new function](#adding-a-new-function)
- [Version-aware guard](#versionaware-guard)
- [Version-aware override](#versionaware-override)
- [Non-syncable modules](#nonsyncable-modules)
- [The dreaded utils namespace](#the-dreaded-utils-namespace)
- [Self-patching](#selfpatching)
- [Bootstrapping patched minions](#bootstrapping-patched-minions)
- [Useful commands](#useful-commands)
- [Future developments](#future-developments)
- [If you need more help](#if-you-need-more-help)
- [Unexplored areas](#unexplored-areas)

## Determining affected minions

The first thing to do is understand what minion versions you have (you can discover that there are older minions you forgot to upgrade):

```
% sudo salt '*' test.version

minion3:
    2019.2.1
minion2:
    2019.2.1
minion1:
    2019.2.3
```

Another way to do it is the `manage.versions` runner (it includes the master version too):

```
% sudo salt-run manage.versions

Master:
    2019.2.3
Minion requires update:
    ----------
    minion2:
        2019.2.1
    minion3:
        2019.2.1
Up to date:
    ----------
    minion1:
        2019.2.3
```

If that is too verbose, use the `survey` runner:

```
% sudo salt-run survey.hash '*' test.version
|_
  ----------
  pool:
      - minion2
      - minion3
  result:
      2019.2.1
|_
  ----------
  pool:
      - minion1
  result:
      2019.2.3
```

And if all you want is just version counters, use the following command:

```
% sudo salt '*' test.version --out json --static | jq '.[]' | sort | uniq -c
      2 "2019.2.1"
      1 "2019.2.3"
```

Now you need to understand which of these versions you need to target:

- If you found a bug, chances are you already know that
- Otherwise, search the [list of issues](https://github.com/saltstack/salt/issues) and the [list of pull requests](https://github.com/saltstack/salt/pulls) (including closed ones)

The general advice is to look at the PR contents and see if the changes are applicable to your installed Salt versions.

## How to sync Salt modules

In the official docs, you can find the [up-to-date list of syncable modules](https://docs.saltstack.com/en/master/topics/development/modules/#module-types) (contributed by [Jamie Bliss](https://github.com/astronouth7303) in [#50633](https://github.com/saltstack/salt/pull/50633/)). The majority of module types could be loaded (synced) from a fileserver, except the ones with a footnote.

The simplest way to sync them all across a minion fleet is to use the following command:

```
% sudo salt '*' saltutil.sync_all
```

Alternatively, you can use the individual [`saltutil.sync_*`](https://docs.saltstack.com/en/master/ref/modules/all/salt.modules.saltutil.html#salt.modules.saltutil.sync_beacons) commands to sync different kinds of modules.

In Salt Neon, you can also use the new [`saltutil.sync_*`](https://github.com/saltstack/salt/pull/50197) states to do the same.

Some module types could be synced to a master (unfortunately, [excluding rosters](https://github.com/saltstack/salt/issues/53914)):

```
% sudo salt-run saltutil.sync_all
```

Or individually via the [`saltutil.sync_*`](https://docs.saltstack.com/en/master/ref/runners/all/salt.runners.saltutil.html) runner commands.

Also, custom modules are automatically synced whenever a highstate is performed.

## Extracting upstream patches

The general advice is to avoid grabbing the latest modules from the upstream Git repo and putting them into your `salt://` file tree. The only scenario when it’ll likely end up well is when a module doesn’t exist yet in your Salt version and doesn’t depend on any other modules. In all other cases, it is much safer to extract a specific patch and adapt it to apply cleanly to your specific Salt version.

Let’s say you want to backport the ability to pass arbitrary `pip` arguments through `pip.installed` from Neon to 2019.2.3 (PR [#52327](https://github.com/saltstack/salt/pull/52327)).

A simple way to extract the feature is to open the [https://github.com/saltstack/salt/pull/52327.diff](https://github.com/saltstack/salt/pull/52327.diff) link, download the patch and remove unnecessary hunks/files that are related to tests and docs. You can see the resulting patch here: [pip-extra-args-orig.diff](https://salt.tips/patching-salt-modules/files/pip-extra-args-orig.diff)

The patch won’t apply cleanly; however, if you inspect the `salt/modules/pip.py.rej` and `salt/states/pip_state.py.rej`, you’ll see that they contain non-functional changes and could be ignored. For more complex conflicts, you’ll need to spend some time to port the patch.

By comparing the original `salt-2019.2.3` tree with the patched one (using the `diff -Naur -x '*.orig' -x '*.rej' salt-2019.2.3{.orig,}` command), you can get the patch that applies cleanly to 2019.2.3: [pip-extra-args-2019-2-3.diff](https://salt.tips/patching-salt-modules/files/pip-extra-args-2019-2-3.diff)

## Overriding a whole module

1. Grab the patched modules. For the example above, that would be `salt/modules/pip.py` and `salt/states/pip_state.py`.
2. Put them into the `salt://_modules/` and `salt://_states/` folders respectively.
3. Sync the modules:

```
% sudo salt minion1 saltutil.sync_modules,saltutil.sync_states ,

minion1:
    ----------
    saltutil.sync_modules:
        - modules.pip
    saltutil.sync_states:
        - states.pip_state
```

Viola! Now you can use the `extra_args` option:

```
$ sudo salt minion1 sys.state_doc pip.installed | grep -A 8 extra_args

            extra_args
                pip keyword and positional arguments not yet implemented in salt

                    pandas:
                      pip.installed:
                        - name: pandas
                        - extra_args:
                          - --latest-pip-kwarg: param
                          - --latest-pip-arg

                Warning:

                    If unsupported options are passed here that are not supported in a
                    minion's version of pip, a `No such option error` will be thrown.
```

Below is an example of how to use this new option to install a package into the `/root/.local` directory (instead of `/usr/local`):

```yaml
install_poetry:
  pip.installed:
    - name: poetry
    - extra_args:
      - --user
```

## Adding a new function

Sometimes you want to extend a module with just one new function, but do not want to override the whole module because you are afraid of possible future version conflicts. Let’s say you want to add the `delete` function to `sdb.sqlite3` module (PR [#51479](https://github.com/saltstack/salt/pull/51479)).

The original module is named `sqlite3.py`. Custom one should have a unique file name that contains the original one (see [#50812](https://github.com/saltstack/salt/pull/50812#issue-237715602) and [#52521](https://github.com/saltstack/salt/issues/52521) to understand why). Let’s name it as `sqlite3_custom.py`:

```python
# The imports below are copied from the original module
from __future__ import absolute_import, print_function, unicode_literals

try:
    import sqlite3
    HAS_SQLITE3 = True
except ImportError:
    HAS_SQLITE3 = False

# The original module doesn't define a virtualname. However, it is absolutely
# critical to define it in a custom module, otherwise the loader
# won't be able to find our delete function in the same sqlite3 namespace
# (this will happen when the __virtual__ () return value is not a string)
__virtualname__ = 'sqlite3'

# This function is also copied from the original module
def __virtual__ ():
    '''
    Only load if sqlite3 is available.
    '''
    if not HAS_SQLITE3:
        return False
    return True

# This is the new function that we want to add
# See https://github.com/saltstack/salt/pull/51479
def delete(key, profile=None):
    '''
    Delete a key/value pair from sqlite3
    '''
    if not profile:
        return None
    conn, cur, table = _connect(profile)
    q = profile.get('delete_query', ('DELETE FROM {0} WHERE '
                                     'key=:key'.format(table)))
    res = cur.execute(q, {'key': key})
    conn.commit()
    return cur.rowcount
```

Put the following configuration snippet to `/etc/salt/minion.d/mysqlite.conf` on a minion and restart it:

```yaml
mysqlite:
  driver: sqlite3
  database: /tmp/sdb.sqlite
  table: sdb
  create_table: True
```

Then put the above `sqlite3_custom.py` module into the `salt://_sdb/` directory and sync it:

```
% sudo salt minion1 saltutil.sync_sdb

minion1:
    - sdb.sqlite3_custom
```

Let’s try it:

```
% sudo salt minion1 sdb.set sdb://mysqlite/password 12345678

minion1:
    True

% sudo salt minion1 sdb.get sdb://mysqlite/password

minion1:
    12345678

% sudo salt minion1 sdb.delete sdb://mysqlite/password

minion1:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/salt/minion.py", line 1664, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/modules/sdb.py", line 58, in delete
        return salt.utils.sdb.sdb_delete(uri, __opts__ , __utils__ )
      File "/usr/lib/python3/dist-packages/salt/utils/sdb.py", line 108, in sdb_delete
        return loaded_db[fun](query, profile=profile)
      File "/var/cache/salt/minion/extmods/sdb/sqlite3_custom.py", line 28, in delete
        conn, cur, table = _connect(profile)
    NameError: name '_connect' is not defined
ERROR: Minions returned with non-zero exit code
```

Oops! Our custom function needs to have access to `_connect`, any related imports, private functions, and module-level variables. Salt loader only exposes public functions (without leading underscores), so any public function calls from the same module should be changed to ` __salt__ ['sqlite3.function']()` syntax.

Let’s fix this specific example to make it work:

```python
# The import below is copied from the original module
from __future__ import absolute_import, print_function, unicode_literals

# This private function is imported from the original module
from salt.sdb.sqlite3 import _connect

# The import below is copied from the original module
try:
    import sqlite3
    HAS_SQLITE3 = True
except ImportError:
    HAS_SQLITE3 = False

# The original module doesn't define a virtualname. However, it is absolutely
# critical to define it in a custom module, otherwise the loader
# won't be able to find our delete function in the same sqlite3 namespace
# (this will happen when the __virtual__ () return value is not a string)
__virtualname__ = 'sqlite3'

# This function is also copied from the original module
def __virtual__ ():
    '''
    Only load if sqlite3 is available.
    '''
    if not HAS_SQLITE3:
        return False
    return True

# This is the new function that we want to add
# See https://github.com/saltstack/salt/pull/51479
def delete(key, profile=None):
    '''
    Delete a key/value pair from sqlite3
    '''
    if not profile:
        return None
    conn, cur, table = _connect(profile)
    q = profile.get('delete_query', ('DELETE FROM {0} WHERE '
                                     'key=:key'.format(table)))
    res = cur.execute(q, {'key': key})
    conn.commit()
    return cur.rowcount
```

Let’s try it again:

```
% sudo salt minion1 saltutil.sync_sdb

minion1:
    - sdb.sqlite3_custom

% sudo salt minion1 sdb.delete sdb://mysqlite/password

minion1:
    1

% sudo salt minion1 sdb.get sdb://mysqlite/password

minion1:
    None
```

It works! Please note, that because the overlay module is named differently than the original one, all that initialization code in the upstream module is still triggered. Also, direct imports could potentially have unintended side effects.

CAVEAT: your custom `delete` function will be silently ignored (even with a version-aware guard) if the upstream `sqlite3.py` module gets its own `delete` function (i.e., after a Salt upgrade).

## Version-aware guard

It would be nice if a patched module just stopped working with any Salt version that doesn’t match the target one. Then any upgrades would lead to a loud failure, and you won’t forget to remove or update the patched module. Let’s implement exactly that!

For example, there is an issue [#54755](https://github.com/saltstack/salt/issues/54755) (`pip failures even when not using pip`) that badly affected Salt 2019.2.1. The fix is located in [#54826](https://github.com/saltstack/salt/pull/54826); however, it can’t be applied cleanly on top of 2019.2.1 because the module was changed multiple times between the releases. Fortunately, all these changes are relevant, so we can just grab the [`state/pip_state.py`](https://raw.githubusercontent.com/saltstack/salt/7780a3ed289770a1c02e1ac21a6809563d70aa75/salt/states/pip_state.py) module from 2019.2.2 or 2019.2.3. You can inspect the changes using the following command on two extracted release tarballs:

```
% diff -u salt-2019.2.{1,3}/salt/states/pip_state.py

...
DIFF HERE
...
```

Put the `pip_state.py` module from 2019.2.3 into the `salt://_states/` directory and sync it:

```
% sudo salt minion1 saltutil.sync_states

minion1:
    - states.pip_state
```

After that, the state that triggered the error will run just fine:

```
# pip_test_state.sls
/tmp/file.txt:
  file.managed:
    - contents: |
        Content
    - unless:
      - 'grep "Content" /tmp/file.txt'

% sudo salt minion2 state.apply pip_test_state

minion1:
----------
          ID: /tmp/file.txt
    Function: file.managed
      Result: True
     Comment: unless condition is true
     Started: 06:47:19.338294
    Duration: 4058.666 ms
     Changes:

Summary for minion1
------------
Succeeded: 1
Failed: 0
------------
Total states run: 1
Total run time: 4.059 s
```

Now comes the most important part. We need to add a version-aware guard to the module, so it will only work on 2019.2.1. There are multiple ways to check Salt version:

1. Compare ` __grains__ ['saltversion']` with a string like `'2019.2.1'`
2. Compare ` __grains__ ['saltversioninfo']` with a list like `[2019, 2, 1, 0]`
3. Compare the `salt.utils.versions.LooseVersion` instances
4. Use the `salt.utils.versions.warn_until` helper
5. Compare `salt.version. __saltstack_version__ ` with an instance of `salt.version.SaltStackVersion`
6. Compare release codenames using the new Compare release codenames using the new [`salt_version`](https://salt.tips/whats-new-in-salt-neon/#salt-version-execution-module) execution module in Salt Neon
7. Use the new [`salt.utils.versions.warn_until_date`](https://salt.tips/whats-new-in-salt-neon/#date-based-deprecation-warning) helper in Salt Neon

Option (5) seems to be the most robust (it should work with release candidates and git snapshots too, and also support the [new versioning scheme](https://github.com/saltstack/salt/issues/55896)).

We need to place the version comparison in a module function, which is triggered as early as possible. For the majority of modules, that would be ` __virtual__ ()` or ` __init__ (opts)`. The added lines are marked with `--- VERSION CHECK ---`:

```python
def __virtual__ ():
    '''
    Only load if the pip module is available in __salt__
    '''
    # --- VERSION CHECK ---
    from salt.version import SaltStackVersion, __saltstack_version__ as sv
    ver = SaltStackVersion(2019, 2, 1, 0, u'', 0, 0, None)
    if sv != ver:
        msg = 'The %s module is not compatible with %s' % ( __name__ , ver)
        log.critical(msg)
        raise Exception(msg)
    # --- VERSION CHECK END ---

    if HAS_PKG_RESOURCES is False:
        return False, 'The pkg_resources python library is not installed'
    if 'pip.list' in __salt__ :
        return __virtualname__
    return False
```

Now, if you apply the `pip.installed` state to 2019.2.3 minion, it will fail because the module is not loaded:

```
% sudo salt minion1 state.single pip.installed ipython

minion1:
    Data failed to compile:
----------
    Specified state 'pip.installed' was not found
```

Another option is to place this check into the `installed(...)` function (right after the docstring and before the function body). This is more risky, because the module will be loaded and some parts could be executed before the check. On the positive side, the failure is more visible:

```
% sudo salt minion1 state.single pip.installed ipython

minion1:
----------
          ID: ipython
    Function: pip.installed
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3/dist-packages/salt/state.py", line 1933, in call
                  **cdata['kwargs'])
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 1951, in wrapper
                  return f(*args, **kwargs)
                File "/var/cache/salt/minion/extmods/states/pip_state.py", line 675, in installed
                  raise Exception(msg)
              Exception: The salt.loaded.ext.states.pip_state module is not compatible with 2019.2.1
     Started: 07:49:24.007851
    Duration: 4.435 ms
     Changes:

Summary for minion1
------------
Succeeded: 0
Failed: 1
------------
Total states run: 1
Total run time: 4.435 ms
```

A couple of essential details to summarize:

- An overlay module needs to be named exactly as the original one (see the [Overriding a whole module](#overriding-a-whole-module) section)
- If an overlay module fails with an exception, the original one [won’t be loaded](https://github.com/saltstack/salt/issues/16439)
- A module that extends the original one needs to contain its name in the filename (see the [Adding a new function](#adding-a-new-function) section)
- If an extension module fails with an exception, the original one will be still loaded
- Because the loader is lazy, the warning would be only triggered when the function is used
- If a module is used implicitly (i.e., not by your states), avoid raising exceptions and instead find other ways to inject the warning

## Version-aware override

So, we got a little safety guard that helps us to detect when our custom modules are outdated. But the rabbit hole is much deeper because we can run multiple Salt versions at the same time and may need to have different versions of custom modules.

The solution is to use Salt [environments](https://docs.saltstack.com/en/latest/ref/states/top.html#multiple-environments). First, set up the [`file_roots`](https://docs.saltstack.com/en/latest/ref/file_server/file_roots.html#directory-overlay) master setting:

```yaml
file_roots:
  base:
    - /srv/salt
  v20190201:
    - /srv/salt_20190201
```

Second, make sure that any custom modules for 2019.2.1 are placed into the corresponding `/srv/salt_20190201/` subdirs (`_modules`, `_states`, `_grains`, etc.) and not into the base environment `/srv/salt` dir:

```
% tree /srv
├── pillar
│   └── top.sls
├── salt
│   ├── top.sls
├── salt_20190201
│   └── _modules
│   └── test.py
```

There are two ways to pin minions to specific saltenvs:

1. Using the [top file](https://docs.saltstack.com/en/latest/ref/states/top.html#multiple-environments)
2. Using the [`saltenv`](https://docs.saltstack.com/en/latest/ref/configuration/minion.html#saltenv) minion setting

Unfortunately, the second approach doesn’t affect how the `saltutil.sync_*` functions work (no modules are synced). And the explicit `saltutil.sync_all saltenv=v20190201` will sync the modules to all minions, ignoring the `saltenv` setting. So, the only option left is to specify saltenvs in the top file:

```yaml
base:
  '*': []
v20190201:
  'saltversion:2019.2.1':
    - match: grain
```

Now the `saltutil.sync_modules` function will work as intended and sync our custom `test.py` only to `minion2`:

```
% salt '*' test.version
minion1:
    2019.2.3
minion2:
    2019.2.1

% sudo salt '*' saltutil.sync_modules
minion1:
minion2:
    - modules.test
```

By adding more environments, you can distribute different custom modules to different minions using any targeting criteria (including the [compound matchers](https://docs.saltstack.com/en/latest/topics/targeting/compound.html)).

CAVEAT 1: custom modules are cached and won’t be automatically deleted when you update Salt (at least for Ubuntu packages; other distros may contain uninstaller scripts that clear these caches). To mitigate that, run `saltutil.clear_cache` before the update (and maybe `saltutil.refresh_*` if necessary).

CAVEAT 2: Fileserver environments won’t be usable for proxymodules until the [#56222](https://github.com/saltstack/salt/pull/56222) is merged. However, you can fix this problem by using the [self-patching](#selfpatching) method and then proceed as usual.

If you are paranoid, you can use environments in combination with [version-aware guards](#versionaware-guard) to prevent modules from being run on updated minions or if you accidentally mess up Salt environments.

## Non-syncable modules

Some module types do not have a corresponding `saltutil.sync_*` command. For example, before Salt Neon, it was impossible to [sync custom executors](https://github.com/saltstack/salt/pull/55190).

Below is a customized `direct_call.py` executor that logs the function name:

```python
# -*- coding: utf-8 -*-
# direct_call.py
from __future__ import absolute_import, print_function, unicode_literals
import logging

log = logging.getLogger( __name__ )

def execute(opts, data, func, args, kwargs):
    log.info("Executing the %s function", func)
    return func(*args, **kwargs)
```

There are three ways to distribute this custom executor module.

### 1. Place the module into a cache dir

The cache dir is defined by the `extension_modules` config option (`/var/cache/salt/minion/extmods`, there is no need to change it).

```yaml
# override_executor.sls
/var/cache/salt/minion/extmods/executors:
  file.directory:
    - makedirs: True

/var/cache/salt/minion/extmods/executors/direct_call.py:
  file.managed:
    - source: salt://_executors/direct_call.py
    - require:
        - file: /var/cache/salt/minion/extmods/executors
```

Run `sudo salt minion1 state.apply override_executor` to apply the state.

The biggest downside is that the cache directory is volatile and may be cleaned (e.g., if you run `saltutil.clear_cache`, `saltutil.sync_all`, `state.highstate`, or `state.apply` commands).

### 2. Module search path

Salt has multiple settings to control the search path for each module type ([`extension_modules`](https://docs.saltstack.com/en/latest/ref/configuration/master.html#extension-modules) and various [`*_dirs`](https://docs.saltstack.com/en/latest/ref/configuration/minion.html#module-dirs) options). First, you need to configure the minion. Place the following snippet into `/etc/salt/minion.d/extmods.conf` and restart the minion (unfortunately, [pillar-based minion configuration](https://docs.saltstack.com/en/latest/topics/tutorials/pillar.html#minion-config-in-pillar) doesn’t work in this case):

```yaml
# /etc/salt/minion.d/extmods.conf
executor_dirs:
  - /srv/extmods/executors
```

Then run `sudo salt minion1 state.apply override_executor` to apply the following state:

```yaml
# override_executor.sls
/srv/extmods/executors:
  file.directory:
    - makedirs: True

/srv/extmods/executors/direct_call.py:
  file.managed:
    - source: salt://_executors/direct_call.py
    - require:
        - file: /srv/extmods/executors
```

### 3. Setuptools entry points

This is the most complex method; however, it allows installing arbitrary sets of custom modules along with their dependencies. I started writing some examples for this blog post but then abandoned the idea for the sake of brevity. If you are interested in how to do that, visit the [Packaging Modules](https://docs.saltstack.com/en/latest/topics/tutorials/packaging_modules.html) documentation page.

Do not forget to add a [version-aware guard](#versionaware-guard) whenever you override a built-in module.

## The dreaded utils namespace

Utils are a special breed of modules (despite being syncable through `saltutil.sync_utils`). Sometimes they are used through the loader (the [` __utils__ `](https://docs.saltstack.com/en/latest/topics/development/modules/developing.html#utils) dunder), but most of the time they are imported directly:

```
% find salt/ -name '*.py' -exec grep -l ' __utils__ \[' \{\} \; | sort -u | wc -l
     159

% find salt/ -name '*.py' -exec grep -l ' salt\.utils' \{\} \; | sort -u | wc -l
    1101
```

There are a couple of problems with `utils`:

- They are nested, and Salt loader can’t override nested modules (e.g., `salt.utils.docker.translate`) because it only supports the `module.function` notation
- Although custom utils are added to `sys.path` if the `utils` cache directory is present, they aren’t importable as `salt.utils.*`
- Even if they were importable, the minion process wouldn’t reload already imported modules unless restarted
- When utils aren’t called via the ` __utils__ ` dunder, ` __opts__ ` and other dunders won’t be injected by the loader
- Custom Jinja filters do not work because they are added at import time via the `@jinja_filter` decorator, but the loader is lazy and won’t import anything unless it is used somewhere

This explains the long list of issues that people have when they try to override utils: [#32500](https://github.com/saltstack/salt/issues/32500), [#46841](https://github.com/saltstack/salt/pull/46841), [#46911](https://github.com/saltstack/salt/issues/46911), [#51719](https://github.com/saltstack/salt/issues/51719), [#51958](https://github.com/saltstack/salt/issues/51958), [#52136](https://github.com/saltstack/salt/issues/52136), [#52222](https://github.com/saltstack/salt/issues/52222), [#53666](https://github.com/saltstack/salt/issues/53666), [#55232](https://github.com/saltstack/salt/issues/55232)…

There are two attempts to partially solve this use-case ([#52001](https://github.com/saltstack/salt/pull/52001) and [#53167](https://github.com/saltstack/salt/pull/53167) by [Alberto Planas](https://github.com/aplanas)), but so far none of them has been released.

I was able to find two ways to deal with this mess when a patch touches both regular modules and utils:

1. Copy all the changed functions (and functions that use these functions!) from the utils patch into the modules that use it, and change all the calls accordingly
2. Alternatively, put the patched utils module into the `_utils` dir, then change all the calls to use the ` __utils__ ` notation

With both methods, you’ll end up running two versions of functions from the utils module (the old one is imported as usual, and the newer one is inlined or called from the overlay module).

## Self-patching

The trick is described by [DatuX](https://github.com/psy0rz) in [#51932](https://github.com/saltstack/salt/issues/51932#issuecomment-493394790). For example, let’s fix the horrible typo that prevented Salt 2019.2.1 from starting when the `ping_interval` option was set (see [#54777](https://github.com/saltstack/salt/pull/54777)). The patch is simple:

```diff
diff --git a/salt/minion.py b/salt/minion.py
index d1c0c00751..28b4956a49 100644
--- a/salt/minion.py
+++ b/salt/minion.py
@@ -2730,7 +2730,7 @@ class Minion(MinionBase):
                     self._fire_master('ping', 'minion_ping', sync=False, timeout_handler=ping_timeout_handler)
                 except Exception:
                     log.warning('Attempt to ping master failed.', exc_on_loglevel=logging.DEBUG)
- self.remove_periodic_callbback('ping', ping_master)
+ self.remove_periodic_callback('ping')
             self.add_periodic_callback('ping', ping_master, ping_interval)

         # add handler to subscriber

```

Put the patch into the `salt://patches/files/2019.2.1-periodic-callback.diff` file, then create the following state and put it into `salt://patches/periodic_callback.sls`:

```yaml
{% if grains.saltversion == "2019.2.1" %}

# Just in case there is no patch command installed
patch:
  pkg.installed

# We use directory name here (saltpath) to apply patches that touch multiple files
periodic_callback_patch:
  file.patch:
    - name: '{{ grains.saltpath }}'
    - source: salt://patches/files/2019.2.1-periodic-callback.diff
    - strip: 2
    - require:
        - pkg: patch

# Set the ping_interval option after the patch is applied
set_ping_interval:
  file.managed:
    - name: '{{ salt["file.join"](salt["config.get"]("config_dir"), "minion.d", "ping_interval.conf") }}'
    - contents: |
        ping_interval: 60
    - onchanges:
        - periodic_callback_patch

# Restart salt minion after the patch is applied and the config option is set
# A naive module.run with service.restart stopped working in 2019.2.1
# (https://github.com/saltstack/salt/issues/55201), so we have to use salt-call
# https://docs.saltstack.com/en/latest/faq.html#restart-using-states
restart_salt_minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: true
    - onchanges:
      - file: set_ping_interval

{%- endif %}
```

The state will apply the patch, set the required config option, and restart the minion. However, there are a couple of things that could be improved:

- The state is idempotent, but to improve the overall performance, we can prevent it from running the second time. You can use [grain](https://github.com/saltstack/salt/issues/51932#issuecomment-493394790) to mark the patch as applied.
- The `file.patch` state is not implemented for Windows (see [#44783](https://github.com/saltstack/salt/issues/44783))
- There are other Windows-specific tweaks that are missing

## Bootstrapping patched minions

All these patching methods are super useful, but it is also essential to know how to apply them to new minions. Earlier I described [a few commands](#how-to-sync-salt-modules) that you can use to sync the modules manually. Now it is time to automate that.

There are many ways to bootstrap Salt minions: salt-bootstrap, salt-ssh, different cloud drivers (including the saltify one), `salt-run manage.bootstrap`, and countless custom scripts. The most important thing is to apply any overrides/patches to minions as early as possible (before any other states are run). There are multiple ways to do so:

1. You do not need to do anything when you apply a highstate. With the [`autoload_dynamic_modules`](https://docs.saltstack.com/en/latest/ref/configuration/minion.html#autoload-dynamic-modules) minion config option (`True` by default), Salt runs `saltutil.sync_all` automatically before applying the highstate.
2. If for some reason, you disabled the above option, you can place a state that syncs modules at the beginning of the `top.sls` file, so it will be applied first
3. Configure the [`startup_states`](https://docs.saltstack.com/en/latest/ref/states/startup.html) minion setting
4. Add a master [reactor config](https://docs.saltstack.com/en/latest/topics/reactor/#mapping-events-to-reactor-sls-files) for the `salt/minion/*/start` event
5. Hook into the salt-cloud bootstrap process
6. Write a custom bootstrap script

If you need to restart a minion once for the patch to take effect, you can combine methods (3) or (4) with the [restart technique](#selfpatching). Methods (5) and (6) allow you to patch minions before they are started.

## Useful commands

A couple of commands to troubleshoot custom modules:

1. `salt-call` together with print or logging statements (or with any interactive Python debugger like `ipdb`)
2. [`saltutil.clear_cache`](https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.saltutil.html#salt.modules.saltutil.clear_cache) to clear the `extmods` directory
3. [`cp.list_master prefix=_modules/`](https://docs.saltstack.com/en/master/ref/modules/all/salt.modules.cp.html#salt.modules.cp.list_master) (or any other directory) to see which custom modules are available on the fileserver
4. [`saltutil.list_extmods`](https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.saltutil.html#salt.modules.saltutil.list_extmods) to see the list of synced modules
5. [`sys.list_*`](https://docs.saltstack.com/en/master/ref/modules/all/salt.modules.sysmod.html#salt.modules.sysmod.list_functions) to list different kinds of modules

## Future developments

- We can expect [Salt loader refactoring](https://youtu.be/q-DoWnkte5Y?t=378) to happen in 2020
- In 2019 SaltStack introduced [POP](https://www.brighttalk.com/webcast/16869/382336/saltstack-open-source-innovation-a-first-look-at-pop-heist-and-umbra) (warn: gated content) as a standalone plugin system that is better than Salt loader. So far, no integration plans have been announced.

## If you need more help

Hopefully, this guide has enough details to help you apply any patches you want. However, if you need more help to do that, I’m available for part-time consulting:

- I can port any existing upstream patch (that is not overly invasive) to an older Salt version. Examples: [1](https://github.com/saltstack/salt/issues/56118#issuecomment-585081144), [2](https://github.com/saltstack/salt/issues/56119#issuecomment-585055831), [3](https://github.com/saltstack/salt/issues/56121#issuecomment-585063805), [4](https://github.com/saltstack/salt/issues/56131#issuecomment-585086113)
- I can help integrate the patch into your Salt bootstrapping process
- My availability is about ~8-12 hours per week

See the [home page footer](https://salt.tips) for my contact email.

## Unexplored areas

There are a couple of rabbit holes I wanted to explore but had no time to do so:

* How to override just one function without copying a whole module?
* Overriding modules using providers
* Why raising exceptions from patched core grains is bad (and what to do instead)?
* Different salt-cloud hooks that could be used to bootstrap patched minions
* How to prevent salt-minion from autostarting (even on Debian/Ubuntu systems) so you can patch it?
* Writing custom bootstrap scripts
* Distributing modules via setuptools entry points (example)
* Patching Salt minions on Windows
* Patching Salt master
* Loader internals (module search path, file loading algorithm)

This is not a commitment, but I hope to cover these topics in the future (if this post turns out to be useful).

### Want to learn more about SaltStack patching?

There are a couple of missing aspects I plan to cover in the future (and maybe release an expanded guide in the form of a small & focused ebook). [Subscribe to signal your interest in this topic](https://tips.us8.list-manage.com/subscribe?u=79863cb6907503346da4ed704&id=9de0347807).