[Verizon Samsung Galaxy S 4] Saul Goodman:
It's a lot easier to audit a small exploit program & small superuser application than an entire Android system image / custom ROM. There are over 1,000 files containing executable code in a GS4 system image; that's a lot to check for malicious behavior.

Here are my thoughts on the matter - more or less why I prefer to use stock OS images and change them on the fly as needed, and don't like to unlock bootloaders unless necessary.

Chain of Trust

Assuming you trust that Samsung/Verizon follow proper security practices while building their images, and don't intentionally put in malware, and don't let others have their private key, their use of public-key cryptography to verify digital signatures ensures that updates you get and install through official methods are legitimate.

There is a chain of trust in place. Simplification: Samsung's public encryption key, d is known to everyone, and Samsung's private encryption key e is used to encrypt the hash of the packages they deliver; only the key d is able to decrypt messages encrypted with key e, so it can be verified that any package with a valid signature that can be decrypted using d is from Samsung.

All the ROMs / system images I've seen distributed around here aren't signed with a known public key. They may be signed, but with a random, untrusted key. Many, but not all, are offered with a MD5 hash. At best, a MD5 hash validates that there's a very good chance that the image downloaded is the same one the author intended. But:

An attacker can create a malicious image and offer to mirror it for free. No problem, if no MD5 hash is provided to users.

Many people won't bother validating MD5 hashes unless they have a reason to suspect the file is corrupt

MD5 is no longer considered cryptographically secure; an attacker could generate another file that has the same MD5 hash

An attacker could modify the page that lists the MD5 for a file.

Build System Attacks

An attacker could target a builder of a single package/project. They might compromise their build system somehow; if they can get into the NDK, for example, the might be able to do a lot to compromise a project built using the NDK. The NDK download seems to occur over plain HTTP, with md5 as the only validation, so it's possible to give someone a corrupt ndk. If an attacker can insert code into a kernel code, they can make it possible to hide all kinds of things from a running system. They may also replace source / binary packages included in the final product with malicious versions.

They could also try to target many developers by distributing compromised NDKs/SDKs.

Validating the provided binaries

If you trust the individual/group distributing the software you're using, and trust that their build system was not compromised, and you have a secure way to get the software from them, you have nothing to worry about. But if you don't have full trust in them, there are so many places to put malware and so many ways to do and hide . If you're able to verify that the binaries matches the expected sources (including any 3rd party binary closed source modules/resources that may be included), then you can probably trust the system, as much as you are able to trust the sources.