TIL: I Am Probably Going To Come To Hate `fapolicyd`
Tags: aws, fapolicyd, rhel8
Published: 2023-04-12 10:16:34

---
title: TIL: I Am Probably Going To Come To Hate `fapolicyd`
published: true
date: 2023-04-11 19:16:00 UTC
tags: AWS,fapolicyd,RHEL8
canonical_url: https://thjones2.blogspot.com/2023/04/til-i-am-probably-going-to-come-to-hate.html
---

One of the things I do in my role is write security automation. Part of that requires testing systems' hardening-compliance each time one of the security-benchmarks my customers use is updated.

In a recent update, the benchmarks for Red Hat Enterprise Linux 8 (and derivatives) added the requirement to enable and run the application-whitelisting service, [fapolicyd](https://github.com/linux-application-whitelisting/fapolicyd). I didn't immediately notice this change …until I went to offload the security-scans from my test EC2 to an S3 bucket. The AWS CLI was suddenly broken.

Worse, it was broken in an absolutely inscrutable way: if one executed _any_ AWS CLI command, even something as simple and basic as `aws help`, it would _immediately_ return, having neither performed the requested action nor emitting an error. As an initial debug attempt, I did:

```
echo $( aws help )$?
```

Which got me the oh-so-useful `255` for my troubles. But, it did allow me to start about to Googling. My first hit was [this guy](https://docs.aws.amazon.com/cli/latest/topic/return-codes.html#:~:text=255%20%2D%2D%20Command%20failed.,the%20request%20was%20made%20to.). It had the oh-so-helpful guidance:

> - 255 -- Command failed. There were errors from either the CLI or the service the request was made to.

Like I said: "oh-so-helpful guidance".

So, I opened a support request with Amazon. Initially, they were at least as in the dark as I was. 

Fortunately, another member of the team I work on noticed the support case when it came into his Inbox via our development account's auto-Cc configuration. Unlike me, he, apparently, hasn't deadmailed everything sent to that distro (which, given _how much_ is sent to that distro, deadmailing anything that arrived through the distro was the only way to preserve my sanity). He'd indicated that he had previously had similar issues and that he got around them by disabling the `fapolicyd` service. I quickly tested stopping the service …and the AWS CLI happily resumed functioning as it had before the hardening tasks had executed.

I knew that wholly disabling the service was _not_ going to be acceptable to our cyber-defense team. But, knowing _where_ the problem originated meant I at least now had a useful investigatory path.

The first (seeming) solution I found was to execute something like:

```
fapolicyd-cli --file add /usr/local/bin/aws
fapolicyd -u
```

This allowed both the AWS CLI to function and for the `fapolicyd` service to continue to be run.

For better or worse, though, I'm a curious type. I wanted to see what the rule looked like that the `fapolicyd-cli` utility had created. So, I dug around the documentation to find where I might be able to eyeball the exception.

```
# AUTOGENERATED FILE VERSION 2
# This file contains a list of trusted files
#
# FULL PATH SIZE SHA256
# /home/user/my-ls 157984 61a9960bf7d255a85811f4afcac51067b8f2e4c75e21cf4f2af95319d4ed1b87
/usr/local/aws-cli/v2/2.11.6/dist/aws 6658376 c48f667b861182c2785b5988c5041086e323cf2e29225da22bcd0f18e411e922
```

Which immediately rung alarm bells in my skull (strong "danger Will Robinson" vibes). By making the exception not only conditional on the binary's (real) path, but _also_ to its size and, particularly, its SHA256 signature, I knew that if anyone ever updated the installed binary, their exception-description was no longer going to match. This in turn would mean that the utility would stop working. Not wanting to deal with tickets that I could easily prevent, I continued my investigation.

Knowing that what I _actually_ wanted was to give a blanket exemption to everything under the `/usr/local/aws-cli/v2` directory, I started investigating how to do that. Ultimately, I came up with a exception-set that looked like:

```
allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-sharedlib trust 1
allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-executable trust 1
```

I saved the contents as `/etc/fapolicyd/rules.d/80-aws.rules` (the Red Hat documentation for `policyd` had given some general guidance around naming of rule-files that this seemed to aligne with) and reloaded the `fapolicyd` configuration. However, I was sadly disappointed to discover that the AWS CLI was still broken. On the plus side, it was broken _differently_. Now, instead of immediately and silently exiting (with a `255` error-code), it was giving me:

```
[2030] Error loading Python lib '/usr/local/aws-cli/v2/2.11.6/dist/libpython3.11.so.1.0':
dlopen: /usr/local/aws-cli/v2/2.11.6/dist/libpython3.11.so.1.0: cannot open shared object
file: Operation not permitted
```

Much more meat to chew on. Further searching later, I had two additional commands to help me in my digging:

```
ausearch --start today -m fanotify --raw | aureport --file --summary
```

Which gave me:

```
File Summary Report
===========================
total file
===========================
16 /usr/local/aws-cli/v2/2.11.6/dist/libz.so.1
16 /usr/local/aws-cli/v2/2.11.6/dist/libpython3.11.so.1.0
2 /usr/local/aws-cli/v2/2.11.6/dist/aws
```

And:

```
fapolicyd-cli --list
```

Which gave me:

```
-> %languages=application/x-bytecode.ocaml,application/x-bytecode.python,
application/java-archive,text/x-java,application/x-java-applet,application/javascript,
text/javascript,text/x-awk,text/x-gawk,text/x-lisp,application/x-elc,text/x-lua,
text/x-m4,text/x-nftables,text/x-perl,text/x-php,text/x-python,text/x-R,text/x-ruby,
text/x-script.guile,text/x-tcl,text/x-luatex,text/x-systemtap
 1. allow perm=any uid=0 : dir=/var/tmp/
 2. allow perm=any uid=0 trust=1 : all
 3. allow perm=open exe=/usr/bin/rpm : all
 4. allow perm=open exe=/usr/libexec/platform-python3.6 comm=dnf : all
 5. deny_audit perm=any pattern=ld_so : all
 6. deny_audit perm=any all : ftype=application/x-bad-elf
 7. allow perm=open all : ftype=application/x-sharedlib trust=1
 8. deny_audit perm=open all : ftype=application/x-sharedlib
 9. allow perm=execute all : trust=1
10. allow perm=open all : ftype=%languages trust=1
11. deny_audit perm=any all : ftype=%languages
12. allow perm=any all : ftype=text/x-shellscript
13. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-sharedlib trust 1
14. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-executable trust 1
15. deny_audit perm=execute all : all
16. allow perm=open all : all
```

Which told me why the binary was at least _trying_ to work, but was unable to load its shared-library. Since I'd named the file `/etc/fapolicyd/rules.d/80-aws.rules`, it was loading my exceptions later than a rule that was preventing access to shared libraries not in standard trust-paths. In the above, it was the file that created rule #8.

I `grep`ped through the `/etc/fapolicyd/rules.d/` directory looking for the file that created rule #8. Having found it, I moved my rule-file upwards with a quick `mv /etc/fapolicyd/rules.d/{8,3}0-aws.rules` and reloaded my rules. This time, my rules-list came up like:

```
-> %languages=application/x-bytecode.ocaml,application/x-bytecode.python,
application/java-archive,text/x-java,application/x-java-applet,application/javascript,
text/javascript,text/x-awk,text/x-gawk,text/x-lisp,application/x-elc,text/x-lua,
text/x-m4,text/x-nftables,text/x-perl,text/x-php,text/x-python,text/x-R,text/x-ruby,
text/x-script.guile,text/x-tcl,text/x-luatex,text/x-systemtap
 1. allow perm=any uid=0 : dir=/var/tmp/
 2. allow perm=any uid=0 trust=1 : all
 3. allow perm=open exe=/usr/bin/rpm : all
 4. allow perm=open exe=/usr/libexec/platform-python3.6 comm=dnf : all
 5. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-sharedlib trust 1
 6. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-executable trust 1
 7. deny_audit perm=any pattern=ld_so : all
 8. deny_audit perm=any all : ftype=application/x-bad-elf
 9. allow perm=open all : ftype=application/x-sharedlib trust=1
10. deny_audit perm=open all : ftype=application/x-sharedlib
11. allow perm=execute all : trust=1
12. allow perm=open all : ftype=%languages trust=1
13. deny_audit perm=any all : ftype=%languages
14. allow perm=any all : ftype=text/x-shellscript
15. deny_audit perm=execute all : all
16. allow perm=open all : all
```

With my `sharedlib` allow-rule now ahead of the default-deny `sharedlib` rule, I tested out the AWS CLI command again. Success!

Unfortuantely, while I solved the problem I set out to solve, my `ausearch` output was telling me that a few other standard tools were also likely having similar whitelisting issues. Ironically, those "other standard tools" are _all_ security-related.

Fun fact: a number of security-vendors write their products for Windows, first and foremost. Their Linux tooling is almost an afterthought. As such, it's often not well-delivered: if they deliver their software in RPMs _at all_, the RPMs are often poorly-constructed. I almost never see **signed** RPMS from security vendors. When I do actually get signed RPMs, they're very rarely signed in a way that's compatible with a Red Hat system that's configured to run in FIPS-mode. So, I guess I shouldn't be super surprised that these same security-tools aren't aware of the need or how to work with fapolicyd. Oh well, that's someone else's problem (realistically, probably "future me's").