SPF PermError: Too Many DNS Lookups - When SPF Record Exceeds 10-DNS-Lookup Limit
Tags: spf
Published: 2021-02-05 06:55:14

"**SPF PermError: too many DNS lookups**" is a common error seen in many SPF (Sender Policy Framework) implementations. When an often overlooked SPF 10-DNS-lookup limit is exceeded, an SPF PermError, aka SPF permanent error, is returned. SPF PermError's can affect your email deliverability.

This article explains what the SPF 10-DNS-lookup limit is, what the consequences are when an SPF record falls foul of it, and how to fix this issue using [DMARCLY](https://dmarcly.com)'s [Safe SPF](https://dmarcly.com/safe-spf) feature.

#### SPF PermError: too many DNS lookups

When you set up SPF on a domain, sometimes you run into some SPF permanent error along the lines of "SPF PermError: too many DNS lookups". This can be seen on an email server with compliant SPF support, or from an online [SPF record checker](https://dmarcly.com/tools/spf-record-checker).

#### How is "SPF PermError: too many DNS lookups" interpreted by DMARC?

When "SPF PermError: too many DNS lookups" is returned during an SPF check, DMARC treats that as fail since it's a permanent error, and all SPF permanent errors are interpreted as fail by DMARC.

#### What is SPF DNS lookup limit?

According to the official RFC specification document [RFC7208](https://tools.ietf.org/html/rfc7208):

> SPF implementations MUST limit the number of mechanisms and modifiers that do DNS lookups to at most 10 per SPF check, including any lookups caused by the use of the "include" mechanism or the "redirect" modifier. If this number is exceeded during a check, a PermError MUST be returned. The "include", "a", "mx", "ptr", and "exists" mechanisms as well as the "redirect" modifier do count against this limit. The "all", "ip4", and "ip6" mechanisms do not require DNS lookups and therefore do not count against this limit.

In other words, the SPF specification requires that the number of mechanisms and modifiers that do DNS lookups must not exceed 10 per SPF check, including any lookups caused by the use of the "include" mechanism or the "redirect" modifier. Otherwise, an SPF PermError, more specifically "SPF PermError: too many DNS lookups", is returned.

This limit is imposed on the receiving email server side. Here are a few popular SPF software packages that implement this limit:

* [libspf2](http://www.libspf2.org/)
* [Mail::SPF](http://search.cpan.org/dist/Mail-SPF/)
* [Mail::SPF::Query](http://search.cpan.org/dist/Mail-SPF-Query/)
* [pyspf](https://pypi.python.org/pypi/pyspf)

#### Why impose the SPF DNS lookup limit?

Why this seemingly artificial limit? Well, as it turns out, the 10-DNS-lookup limit is implemented to thwart Denial-of-Service (DoS) attacks. Consider such a scenario:

* a malicious user creates an SPF record on domain ```malicious.com```, with references to another domain ```victim.com```;
* he then sends a lot of emails from ```malicious.com``` to mailboxes hosted by different email service providers (ESP) with SPF implemented;
* upon receiving such an email, the ESP queries the DNS for ```victim.com```;
* since many ESP's are involved, they amplify this traffic; this effectively turns into a DoS attack at ```victim.com```;
* what's more, the true source of the attack is hidden.

As you can see, a pretty innocent email authentication mechanism can be exploited for malicious use, if no care has been taken! While the consequences can be severe, the solution to this problem is simple: putting a limit on the max number of DNS lookups per check on the ESP side can drastically mitigate it, since the amplification is limited to 10, instead of potentially much larger.

#### Does my SPF record exceed the SPF DNS lookup limit?

You can use our [SPF record lookup tool](https://dmarcly.com/tools/spf-record-checker) to check your SPF DNS lookup count. In addition to the basic information about your SPF setup on your domain, it also shows the number of DNS-querying mechanisms/modifiers. Here is the result of an SPF check on microsoft.com, which has exactly an SPF DNS lookup count of 10:

<img src="https://dmarcly.com/blog/user/pages/01.home/spf-permerror-too-many-dns-lookups-when-spf-record-exceeds-10-dns-lookup-limit/dmarcly-spf-record-dns-queries.png">

I suggest that you run a similar check on your domain, and see what the number looks like.

#### What happens if it exceeds the SPF DNS lookup limit?

When the SPF implementation on the receiving email server encounters more than 10 DNS-querying mechanisms/modifiers in the sender's domain's SPF record, it returns "SPF PermError: too many DNS lookups". As mentioned above, an SPF PermError is interpreted by DMARC as fail, and consequently, the email might not land in the inbox, depending on the email server's settings.

Therefore, your best bet is to keep the DNS-querying mechanisms/modifiers in your SPF record <= 10.

_But I can't. I have so much stuff in my SPF record!_

I understand - nowadays almost every company outsources essential services to 3rd-party service providers, like email delivery, marketing, and more. Putting an include for each of the services in the record counts 1 against the limit. And if they further contain DNS-querying mechanisms/modifiers, it reaches/exceeds the limit fairly quickly.

While many email service providers (ESPs) like Gmail send unauthenticated emails to spam by default, Microsoft Office 365 takes a step even further: they block email sender domains automatically if they fail email authentication, including SPF authentication.

> The Antispam policy allows administrators to “Allow” domains regardless of the reputation of the domain. We’re changing our policies to not honor Allow rules when the domain fails authentication.<br><br>— Microsoft Office 365, April 2020

Learn more about this on Microsoft Office 365's [roadmap](https://www.microsoft.com/en-us/microsoft-365/roadmap?filters=&searchterms=62398).

#### Enter SPF record flattening

There is one simple solution to this problem though. Through "flattening" an SPF record, one can reduce the number of DNS-querying mechanisms/modifiers so that it's smaller than 10.

This is how "SPF record flattening" works: for each of the DNS-querying mechanisms/modifiers, query the DNS to get the IP addresses, then replace the original mechanism/modifier with the IP addresses. Each time a mechanism or a modifier is replaced, the total DNS lookup count is decremented by 1.

Using this SPF record flattening technique, you can turn a very complex SPF record containing well over 10 DNS-querying mechanisms/modifiers into a "flat" IP address list, staying comfortably in the "safe zone".

Let's take a look at what a flattened SPF record looks like. Here are the IP addresses by flattening the SPF record on microsoft.com:

<img src="https://dmarcly.com/blog/user/pages/01.home/spf-permerror-too-many-dns-lookups-when-spf-record-exceeds-10-dns-lookup-limit/dmarcly-flattened-spf-record.png">

As you can see, this flattened SPF record contains the same IP addresses as those in the original SPF record on microsoft.com, and yet it has no DNS-querying mechanism/modifier in itself!

Problem solved? Well, not quite yet.

What if the IP addresses underlying one of the include mechanisms are changed? That means the flattened SPF record now goes out of synchronization on these IP addresses, which will produce incorrect results in SPF authentication.

Of course, you can manually flatten the SPF record again, and update it in the DNS. Needless to say, this is terribly tedious and error-prone, not to mention you will have to monitor it all the time.

Good news is, DMARCLY has a feature called "[Safe SPF](https://dmarcly.com/safe-spf)", which is exactly purpose-built to save your sanity.

#### Safe SPF to the rescue

Kill two birds with one stone with Safe SPF: always keep your SPF record's DNS-querying mechanisms/modifiers below 10, without having to worry about manually flattening the SPF record and updating it in the DNS at all!

Follow the steps here to set up Safe SPF on your domain:

**Step 1 Generate Safe SPF record**

  In dashboard->DNS Records->Safe SPF, choose the domain you want to set up Safe SPF on, then click the Generate Safe SPF Record button, as shown below:

  <img src="https://dmarcly.com/blog/user/pages/01.home/spf-permerror-too-many-dns-lookups-when-spf-record-exceeds-10-dns-lookup-limit/generate-safe-spf-record.png">

**Step 2 Publish the Safe SPF record**

  Now that the Safe SPF record is generated, you need to publish it on your domain in the DNS. Publish it as you would a regular SPF record. Keep in mind: a Safe SPF record is an SPF record.

  Here is [how to publish an SPF record](../how-to-implement-dmarc-dkim-spf-to-stop-email-spoofing-phishing-the-definitive-guide#publishing-an-spf-record).

**Step 3 Verify the Safe SPF record**

  Next you need to verify the Safe SPF record is published correctly and accessible to all. Click the Verify Safe SPF button now:

  <img src="https://dmarcly.com/blog/user/pages/01.home/spf-permerror-too-many-dns-lookups-when-spf-record-exceeds-10-dns-lookup-limit/verify-safe-spf.png">

**Step 4 Save Safe SPF**

  Finally, you need to save Safe SPF so that your SPF record is always up to date:

  <img src="https://dmarcly.com/blog/user/pages/01.home/spf-permerror-too-many-dns-lookups-when-spf-record-exceeds-10-dns-lookup-limit/save-safe-spf.png">

  Here is what it looks like:

  <img src="https://dmarcly.com/blog/user/pages/01.home/spf-permerror-too-many-dns-lookups-when-spf-record-exceeds-10-dns-lookup-limit/dmarcly-safe-spf.png">

  As can be seen from above, Safe SPF is activated on the specified domain.

Once Safe SPF is activated on a domain:

* the Safe SPF record contains the same IP addresses as those in the original SPF record;
* the Safe SPF record has no more DNS-querying mechanisms/modifiers than 10;
* it is always updated when the underlying IP addresses change;
* there is zero maintenance on your side.

#### How to update an existing Safe SPF record (Method 1)

After you generate/publish a Safe SPF record, you might want to update the original SPF record at a later time.

For example, you might want to:
1. add a new mechanism (include, ip4, mx, a, etc.) to the SPF record;
2. update an existing mechanism in the SPF record;
3. remove an existing mechanism from the SPF record.

You can use the same Safe SPF process in the last section for this purpose. One thing to note here is that you need to update your original SPF record with your intended changes before you plug it into the Safe SPF process.

Let's walk through a practical example to show exactly how to do this.

Say your domain is: yourdomain.com, and the original SPF record on the domain looked like this:
```
v=spf1 mx include:someservice.com -all
```
you created a Safe SPF record for it in the past:
```
v=spf1 include:_u.yourdomain.com._spf.dmarcly.com -all
```
and your organization plans to integrate a new email service called _anotherservice_. Now you need to include it in your SPF record, so that the emails sent from that service's hosts pass SPF authentication.

Let's assume anotherservice's SPF include is:
```
include:anotherservice.com
```
you need to update your original SPF record to include this service, so that it looks like:
```
v=spf1 mx include:someservice.com include:anotherservice.com -all
```

Next, you need to go through the whole Safe SPF process with the updated original SPF record:
```
v=spf1 mx include:someservice.com include:anotherservice.com -all
```

Specifically, use the above value for the Original SPF Record field in the "Generate Safe SPF record" step in the "Safe SPF to the rescue" section. Then go through all the remaining steps in the Safe SPF process.

Once you are done, your new Safe SPF record on yourdomain.com contains all the IP addresses in _include:anotherservice.com_, in addition to the existing IP addresses. And if there is any underlying change in _include:anotherservice.com_, your Safe SPF record will pick it up automatically.

This approach applies to all scenarios including adding, replacing, and removing.

For example, if you want to replace someservice.com with _anotherservice.com_ in your SPF record, just update it to:
```
v=spf1 mx include:anotherservice.com -all
```
then run it through the whole Safe SPF process.

In another example where you want to remove the _mx_ mechanism from the SPF record, simply update it to:
```
v=spf1 include:someservice.com -all
```
then run it through the whole Safe SPF process.

#### How to update an existing Safe SPF record (Method 2)

Another way to update your existing Safe SPF record is to add the new mechanism directly to your published Safe SPF record.

Note that this approach only applies to adding an additional mechanism, rather than replacing or removing an existing one. If you want to replace or remove an existing mechanism, please use Method 1 described above.

Let's say you've published a Safe SPF record on your domain:
```
v=spf1 include:_u.yourdomain.com._spf.dmarcly.com -all
```
This record contains all the IP addresses resulted from all the mechanisms in your original SPF record.

Then you need to add a new service _include:newservice.com_, you can simply update the SPF record on your domain to:
```
v=spf1 include:_u.yourdomain.com._spf.dmarcly.com include:newservice.com -all
```
Now the SPF record on your domain contains all the IP addresses resulted from all the mechanisms in your original SPF record, as well as those in _newservice.com_. In other words, the emails sent from newservice's hosts will pass SPF authentication.

#### Partial Safe SPF

If for some reason, you want to flatten only a part of your SPF record, while leaving the rest as is, you can use partial Safe SPF.

One good reason for this is that some email delivery services like HelpScout require that their SPF record be explicitly specified.

For example, in order to have HelpScout send emails on your behalf, you need to specify:
```
include:helpscoutemail.com
```
explicitly in your SPF record.

It's easy enough to use partial Safe SPF to solve this problem. In this particular case, you only need to leave out _include:helpscoutemail.com_ for the Safe SPF process, then add it back in after the Safe SPF record is generated.

Here is a concrete example. Your original SPF record looks like this: 
```
v=spf1 include:service1.com include:service2.com include:explicitservice.com -all
```
You only need to flatten _service1.com_ and _service2.com_, but not _explicitservice.com_. Partial Safe SPF can help here.

You need to run the following original SPF record:
```
v=spf1 include:service1.com include:service2.com -all
```
through Safe SPF. That is, flatten only _service1.com_ and _service2.com_, while leaving _explicitservice.com_ out of the record.

Now you've generated and published the Safe SPF record on your domain like this:
```
v=spf1 include:_u.yourdomain.com._spf.dmarcly.com -all
```
This record contains all the IP addresses from _service1.com_ and _service2.com_. We now need to add _explicitservice.com_, so that the record looks like:
```
v=spf1 include:_u.yourdomain.com._spf.dmarcly.com include:explicitservice.com -all
```

Once the above SPF record is published on your domain, it contains all the IP addresses from all the 3 services. What's more, _explicitservice.com_ is explicitly specified in the SPF record.

Original post: <a href="https://dmarcly.com/blog/spf-permerror-too-many-dns-lookups-when-spf-record-exceeds-10-dns-lookup-limit">https://dmarcly.com/blog/spf-permerror-too-many-dns-lookups-when-spf-record-exceeds-10-dns-lookup-limit</a>