If you expect that "domain users" will always do their first authn to the machine remotely via SSH, then you can skip making a custom PAM module, and instead set up the combination of domain SSH authentication, with dynamic provisioning of local POSIX accounts. It's honestly much less of a headache.

1. dynamic provisioning: in /etc/pam.d/sshd, add "pam_mkhomedir.so" as a session module. Easy as that.

2. domain SSH auth: in /etc/ssh/sshd_config, add an "AuthorizedKeysCommand". This is a program you'd write, that should accept a username on the command line (the username passed by the SSH client), and should emit a list of valid SSH pubkeys for that user. This slots in place of the default SSHd behavior of reading $USER_HOME/.ssh/authorized_keys — `cat ~/.ssh/authorized_keys` is essentially the "built-in" AuthorizedKeysCommand.

The key insight with step 2 is that you can make this authentication step serve double-duty as an authorization step. If you don't want a particular domain user on this system, you can just have your AuthorizedKeysCommand emit no acceptable SSH keys for that user, leaving them unable to authenticate.

If and when the AuthorizedKeysCommand authenticates+authorizes you (by whatever means), the pam_mkhomedir module will turn around and make you a local POSIX user and home directory to correspond with your domain account (with whatever username the client provided, so make sure to validate that), and you'll end up logged into said local POSIX account.

(You won't have a password configured / any way to convince PAM to re-authenticate you while logged in; but that's fine for most things — you can still fork out new processes, start new sessions of screen/tmux, etc. The only consideration is sudo access: just ensure that the sudo group has the NOPASSWD attribute set in /etc/sudoers.)

This is essentially how GCP's "IAM-based" OS Login (https://cloud.google.com/compute/docs/instances/managing-ins...) works: there's an AuthorizedKeysCommand (/usr/bin/google_authorized_keys) that fetches the SSH keys recorded in IAM for the SSH username [= escaped GCP IAM ID] you're claiming to be logging in as, and emits them if-and-only-if that GCP IAM ID has the OS-Login IAM role; and then PAM's dynamic provisioning kicks in and handles the rest.

Also, for a maybe-clearer example, I wrote my own little toy AuthorizedKeysCommand, that uses Github as an authentication domain (https://github.com/tsutsu/github-auth3). You tell it a Github org, and then anyone in that Github org can log in by with their Github username + an SSH key registered to that Github user. (Again, it checks the username for org membership, and only if the user is in the org, prints the SSH pubkeys registered to the user.)

=====

Of course, compared to doing everything in PAM, this approach isn't as elegant: every "domain user" that logs into the system ends up being able to leave junk there—at the very least, they end up with a UID permanently allocated to them in /etc/passwd.

You can ameliorate this slightly by NFS-mounting /home; or even individual homedirs as-needed using autofs.

(Intriguingly, macOS ships by default with autofs preconfigured to mount homedirs for OpenDirectory users, despite Server.app not exposing any such OpenDirectory capability — presumably there's some Apple-internal setup for their workstations that seeks parity with Windows' Roaming User Profiles.)