We tried something like that at a place I worked in the late '90s and early '00s. It was for Windows users.

It inserted itself between your email client and your SMTP and POP/IMAP servers.

When you installed it, it generated a public and private key pair for you, storing them both in your local key ring.

It would place your public key in the header of your outgoing mail, and add a header that identified itself.

If you had enabled signing, it would transform you outgoing mail into a signed outgoing mail.

If the mail was going to someone who was known to use this software and whose public key was in your key ring, it would then transform your outgoing mail into an encrypted outgoing mail.

(For mail to people who were not known to have it, it put a little blurb at the bottom of the message suggesting they get it).

For incoming mail it would check the header to see if the mail was from someone using this software. If it was, it would look for their public key in the headers and add it to your key ring.

If the mail was encrypted, it would transform it to plaintext. If it was signed, it would check the signature and put something in the message to tell you if the signature check had passed or not.

The marketing people had more sway than the engineering people, so if something came down to ease of use vs. security ease of use had a good chance of winning. The marketing people were really pushing the notion that it was "transparent", and so wanted minimal configuration and minimal interaction with the user. They wanted you to just install it, get your friends to install it, and have it then transparently encrypt the mail between you and your friends.

I don't know if such a system can be made reasonably secure, but I'm pretty sure that if it can be, it's going to have to be at least somewhat intrusive now and then. It needs to warn you if someone's key has changed, and provide some way for you to do an out of channel verification that the change is legitimate. There also should be some way when composing a message to indicate that it should not be sent unencrypted, and if you attempt to send it to someone whose key is not known it should be rejected.

(I'd even argue for an for flipping that last--a way to indicate that a message does not need to be encrypted, with it generating an error to try to send a message not so marked to someone whose key is not known. It should require a deliberate choice to send an insecure message rather than the other way around).

The way we inserted between the mail client and the SMTP/POP/IMAP server at first was to have our software run SMTP/POP/IMAP proxies on 127.0.0.1, and require the user to manually change their client configuration to use our proxy.

Not long after that was changed to automatically do that for various popular email clients. This was on Windows so for most of them that just meant some registry editing. I think there may have been one or two where it involved editing an actual configuration file.

I think we eventually replaced that with an LSP [1], which required no changes to email client configuration. (I may be misremembering that part, mixing it up with the LSP that I am sure we used for another product we had in the same general time frame, a pre-fetching caching web proxy).

Anyway, while technically interesting I'm not sure there is really much use for this kind of thing. You need users that need security but don't need it too badly, and are only worried about messages in transit. But it generally takes a pretty serious adversary to get your messages in transit--if you are facing that you probably need some more certain security, not something that is opportunistic.

For most people it is security at the ends that matters. It's your siblings/parents/spouses/roommates that you have to worry about, and this system doesn't really help against them.

[1] https://en.wikipedia.org/wiki/Layered_Service_Provider