Hmm, I can't seem to find the tutorial I followed. I thought it was on DigitalOcean or Linode but I can't seem to find it. It was one of the big vps providers though...

This one[1] is similar however, although it's a bit less detailed. Basically: I use postfix with mysql for a user database as my MTA (the postman so to speak) and dovecot for the IMAP client (a smart mailbox equivalent).

edit: it's slightly different from what perlgod wrote (rspamd+ldap vs spamassassin+mysql) but the idea is the same.

Now the tutorial will give you a basic set-up, with spamassassin as a spamfilter. Which already "just-worked(tm)" for me. In addition to what's listed I added the following steps over time:

- First check your ip address on mxtoolbox.com for any blacklists. If you're on any, you could get removed if you ask them or you could ask your hosting provider to give you a different ip.

- get a certificate from let's-encrypt and encrypt all outgoing mail. Rejecting unencrypted mail is not a good idea even if it would be in an ideal world.

- add a blacklist MySQL table and a regex addres to the users table. Postfix has an option for parsing regex IIRC, so you can just set the email-adress to be a regex in the table as you would any other email. Then set the MySQL query in postfix to something like "user in table users AND NOT in table blacklist". This way you can use a unique email for each website you sign up to (say: somesalt.domain.tld@yourdomain.tld) and if you ever get any spam, you will know what website got hacked/sold your info ;P. I have only one website on my blacklist so far, and that was because their unsubscribe link didn't work.

- Install Sieve, this let's you add a sorting-script to your imap, letting you automatically sort incoming mail into different folders using all kinds of regexp. I have for example "personal, work, work/personal (directly to me and not a list), anonymous (throwaway adresses for each website I sign up to), admin (postmaster, cron, etc.), purchases (regex match to anything containing order, shipment, etc. which gets put into a folder which is backed up for longer), Uni, git notifications, and Twitch (because they send a ton of short-lived notifications. Messages in this folder get purged after 2h).

- Set up a r-dns pointer (you said you wanted to try more obscure dns features :D). this is an ip->domain mapping. For me this meant sending a message to my vps provider asking them to do so. p.s. vpsdime has insanely good/fast support. Took them literally less than a minute.

and finally:

- Set up DMARC (DKIM+SPF). Spf is pretty simple. It's simply a dns record which says which ip-adresses are allowed to send mail on your behalf. DKIM is a bit more complicated: It use public-private key encryption (with the public-key in the dns records) to digitally sign various fields (to,from, content, cc, etc. can all be signed separately) of your email to make sure they haven't been tampered with. The daemon set-up is quite easy, but it's easy to mess up the settings.
If you're sending sensitive business emails I would set it up (my bank has it for example) but for personal email, I would only set it to sign the bare minimum such as the 'from' field, or nothing at all. Even if you don't sign any fields, having it set up will almost surely prevent you from being put into spam folders by the big providers.

I haven't had any issues so far, except for an overly strict DKIM set-up. Once marking email send to my work's mailing lists as spam when forwarded to gmail. (i.e. me->work list -> someone@work.tld -> someone@gmail.com) which in an ideal world wouldn't cause issues, but my work's mail server was misconfigured causing them to modify the email's envelope without respecting the DKIM signatures.

The other time was when my university email forwarded messages from @intel.com, which has strict security settings too. This was actually an issue when I forwarded from my uni to my gmail before too, but I never noticed because gmail was (as per Intel's configuration) silently discarding any emails I got. I only noticed the problem when I looked at my mail server logs for any rejections. I now have Intel.com whitelisted. (my uni said they'd fix it... 1.5 years ago...).

Having written all this out I noticed two things:

1. Okay, maybe setting up an email server is a bit of work after all... Mine grew organically over a few weekends so I never noticed.

2. When I finally start that blog I've been meaning to do, I should do a clean email-server install and write it up.

[1] https://www.digitalocean.com/community/tutorials/how-to-conf...