Sorry for the delayed response, I just noticed your reply...

> What I didn't explain yet is that I am doing this for a university project and I am fishing for extra points.

Okay. I was, for a few years, a "part-time" professor, so I'll respond with my professor hat on then...

> So I was trying to justify running my own DNS server. Is it reasonable?

That depends. Does running your own DNS server get you anything for this project? By that I mean, does it increase the security of your system any? If it does, then your professor may notice and/or appreciate the fact that you are "going the extra mile" to make it even more secure. It the overall security doesn't increase by adding a DNS server, it may be best to skip it. Additionally, if misconfiguration of the DNS server could decrease the security of the overall system, you may re-evaluate running your own.

> I really should have done more research on this, but I imagined I could encrypt the DNS queries themselves and forward them to a public recursive DNS server. Could I not use DNSCRYPT or DNS-over-TLS for this purpose?

Sure, you could route your queries through the public resolver. Again, what does get you? I'm assuming the goal is to protect against an attacker tampering with DNS responses, yes?

So, as we mentioned earlier, if DNSSEC is in place and all responses are signed, you're good. Nothing to worry about.

For now, however, let's assume that not all responses will be signed (I think this is a fairly safe assumption, for at least some public CAs; for your project, this may or may not be the case).

Without DNSSEC and even with DNSCRYPT or DNS-over-TLS to a public resolver, there is an opportunity for an attacker to tamper with DNS responses. Is tampering more likely to occur if you run your own DNS resolver or if you use a public one? Either way, the responses will be travelling over some portion of the Internet in the clear -- whether that is between you (if running your own server locally) and the root/authoritative nameservers or between the public resolver you're using and the root/authoritative nameservers. I'm not sure you can easily figure out the answer to that question.

If you use Google's public DNS service (8.8.8.8), for example, the DNS query/response will be "in the clear" (and subject to tampering) on the path between Google and the root/authoritative nameserver (depending on which one you're querying at the time). If you instead use a public DNSCRYPT server, the same will still be true. In between their server and the root/authoritative server, the query/response will still be in the clear. The only time this wouldn't be the case is when the root/authoritative nameservers themselves were available via DNSCRYPT/DNS-over-TLS.

Basically, unless everything is 100% DNSSEC, there's an opportunity for tampering. Fortunately, though, OCSP responses are signed, as we mentioned earlier. If the OCSP response comes with a valid signature, it can be trusted -- whether DNSSEC is in use or not. A response with an invalid signature (for whatever reason) obviously cannot be trusted. Let me ask you that first question again with this in mind: will running your own DNS server increase the security any? Will it increase the amount of trust you have in the validity of the certificates presented to you by clients?

> Assuming the server has at least downloaded an initial CRL, I could always fallback to that. I haven't played much with this yet, but I think that's the big advantage of a CRL versus an OCSP query, no?

Yes, and you probably should fallback on OCSP failure. One of the obvious advantages of CRLs (once they are downloaded) is that they can be cached. If the HTTP server goes down afterwards, the client can (and should!) use their cached version. Just make sure to refresh the CRL before it expires! CRLs (for public CAs) aren't really very feasible anymore simply because of their size. Some of them are 100 of MBs! Can you imagine having to download a 300 MB CRL on your mobile phone just because you visited some new web site for the first time!? This just isn't practical and is why we have OCSP.

(Another advantage of CRLs is that a client doesn't give away any information about what sites it is visiting because lookups are done locally (on the cached CRL). With OCSP, a passive listener can obtain the hostnames.)

> I guess I should "fail closed" to cover all holes but then I'm basically letting the attacker DoS the server. What is best?

That's another question with "that depends" as the answer.

All modern web browsers fail open (by default) if OCSP fails but is that acceptable for your project? What are the potential damages and/or losses if an attacker is able to prevent you from verifying a certificate is valid? You'll have to weigh the chances and risk of this happening against the security requirements of your project and determine whether fail open or fail closed is the right choice for you.

I hope this helps. Good luck!