[Sony Xperia XZ1 Compact ROMs, Kernels, Recoveries,] raphidae:
Looking at the source code, the keystore module (which on normal android is in /lib/keystore.default.so and /lib64/keystore.default.so) is missing, leading to my error. Other TWRP images (for instance potter) have this file in the recovery ramdisk at /vendor/lib64/hw/keystore.default.so.

So it seems this file is missing. Where do I get this file, is this normally built with TWRP? I'm going to try putting the /lib64/keystore.default.so from android in the TWRP ramdisk, or otherwise look whether there's such a file in stock recovery. But if someone knows where best to get this missing file... :)

Edit: Okay, so I went digging in TWRP source code, and the error is coming from one of two places:

@149
rc = hw_get_module_by_class(KEYSTORE_HARDWARE_MODULE_ID, NULL, &mod);
if (rc) {
printf("could not find any keystore module\n");
goto out;
}

or (based on a DEFINE):

@333
rc = hw_get_module_by_class(KEYSTORE_HARDWARE_MODULE_ID, NULL, &mod);
if (rc) {
printf("could not find any keystore module\n");
goto err;
}

So the DEFINE doesn't matter. KEYSTORE_HARDWARE_MODULE_ID is nowhere defined in TWRP source, so it probably comes from Android source at https://android.googlesource.com/pl.../+/master/include/hardware/keymaster_common.h (#define KEYSTORE_HARDWARE_MODULE_ID "keystore").

Function "hw_get_module_by_class" is also not in TWRP source, so that would need to come from libhardware.so at https://android.googlesource.com/platform/hardware/libhardware/+/master/hardware.c

This file has the following paths defined (I took a peek in libhardware.so with a hex editor, and these strings are in there):

#define HAL_LIBRARY_PATH1 "/system/lib64/hw"
#define HAL_LIBRARY_PATH2 "/vendor/lib64/hw"
#define HAL_LIBRARY_PATH3 "/odm/lib64/hw"

So hw_get_module_by_class("keystore", NULL, &mod) is called:

(var name is set to "keymaster)"

/* Nothing found, try the default */
if (hw_module_exists(path, sizeof(path), name, "default") == 0) {
goto found;
}
return -ENOENT;

hw_module_exists is in the same file and is called as hw_module_exists(path, sizeof(path), "keymaster", "default"):

static int hw_module_exists(char *path, size_t path_len, const char *name,
const char *subname)
{
snprintf(path, path_len, "%s/%s.%s.so",
HAL_LIBRARY_PATH3, name, subname);
if (access(path, R_OK) == 0)
return 0;

This should be checking existence and read access on /odm/lib64/hw/keymaster.default.so as far as I can tell.

snprintf(path, path_len, "%s/%s.%s.so",
HAL_LIBRARY_PATH2, name, subname);
if (access(path, R_OK) == 0)
return 0;

This should be checking existence and read access on /vendor/lib64/hw/keymaster.default.so as far as I can tell.

#ifndef __ANDROID_VNDK__
snprintf(path, path_len, "%s/%s.%s.so",
HAL_LIBRARY_PATH1, name, subname);
if (access(path, R_OK) == 0)
return 0;
#endif
return -ENOENT;
}

This should be checking existence and read access on /system/lib64/hw/keymaster.default.so as far as I can tell.

I have tried putting a 64bit keystore.default.so from android 8.1.0 in all three of those paths, but for some reason it doesn't work. It doesn't look like there's an error during load(), as that would cause an extra message in the log. So it should just be checking the existence and readability of this file and this is what fails, even though the file is definitely present. What am I missing here?