spf and dkim illustrated

There are tons of documents on SPF and DKIM but I find none explains them clearly. All the articles say they are used to prevent email spoofing but none tells you how they can fight against email spoofing or forging or spamming.

Where to deploy spf policy?

SPF can be deployed on any mail server that acts as a SMTP server so it can be deployed on outgoing mail servers, transferring mail servers or incoming mail servers. For example, this article describes how to deploy SPF as a plugin on a postfix server. In other words, SPF can be deployed on mta and mda, but not mua, because mua only acts as a SMTP client when it talks with a SMTP server. SPF module in a mail server checks the domain in the “MAIL FROM:” SMTP command(the envelope from), retrieves the spf record of the domain using DNS query and see if the ip address of the smtp client(got from the tcp connection) is listed in the spf record. If listed, let the email pass(to the email box or another mail server), otherwise, reject the email or flag it according to the instruction in the spf record.

So how can SPF prevent email spoofing?

If you’ve read my post on how to send an email with openssl, you may think about using a fake “MAIL FROM:” domain when sending the envelope information to the email server. If the outgoing server uses SPF, you may find it difficult to do so as the server may reject your email in the first place if your machine’s (the machine you are running the openssl command) ip address is not listed in the SPF record of the fake domain. (But this situation rarely happens because the server’s configuration usually makes the rule of authenticated user take higher precedence over the SPF policy and let the email pass).  Even if the outgoing mail server does not deploy a SPF plugin(or let relay authenticated user’s email), your email may still be intercepted at the destination mail server if it uses SPF. This is because the “MAIL FROM” and “RCPT TO” (the envelope information) won’t change across MTAs. When your outgoing server talks with the remote mail server using SMTP protocol, the “MAIL FROM” still uses your forged domain. To pass SPF verification,  you must use a domain you own in the “MAIL FROM” command and at least list your outgoing mail server’s ip address in the domain’s SPF DNS record. But that does not guarantee the email is not going to be rejected by all the following servers. Consider the following situation: mua->outgoing mta->middle mta->incoming mta->mda->mua.  You list your outgoing server’s ip in the SPF record, but the incoming mta may still block the email if the middle mta’s ip is not listed in the SPF record. Although in usual cases, there is no middle mta, and it is enough to only include the ip of the outgoing mta in the SPF record.

Now you know that with SPF in effect, you can hardly use a faked email address such as admin@microsoft.com (envelope from) to cheat the recipient unless the faked domain has an open-relay SMTP server and you use your email client to use that open-relay server to send the email to the victim, or config your outgoing mail server to use the open-relay server as the next hop mta. Google has a free relay server but I’ve not tried that and do not know if it can be used to send forged emails. But we know the email addresses in the message and the envelope are different. We still can forge from: header,to: header in the message.

 

We can use openssl to send an email to check-auth@verifier.port25.com to verify that SPF can really prevent forged envelope from:

In the above example, we created a forged mail from:faked@outlook.com. Since we send this email using our mail server domainhostseotool.com instead of outlook.com, the receiving mail server verifies.port25.com which exerts SPF will detect this. The email address faked@outlook.com in “mail from” will be used to construct the Return-Path: email header in the returned email message at verifies.port25.com, and the returned email is sent to this email address. The following is part of the content of the email received in faked@outlook.com:

The Port25 Solutions, Inc. team

==========================================================
Summary of Results
==========================================================
SPF check:          softfail
“iprev” check:      pass
DKIM check:         none
SpamAssassin check: ham

 

Notice that SPF check is softfail because the ip of the sending mail server(domainhostseotool.com) is not listed in the SPF record of outlook.com.

To summarize, in order to prevent forged envelope email address, the owner of email domain and all email servers need to cooperate.  If the owner of email domain does not want others(not its authorized users) to use an email address belong to his domain, he should do all the following:

  • publish the SPF record in the DNS zone of the domain
  • configure his mail servers to avoid being open-relay
  • protect the host of his email servers so that hackers can not intrude to set up their own mail server on the same ip

If an email server on the internet does not want to receive a forged (mail from)email, it should:

  • exert  SPF to block, reject, or flag such emails

 

Well-known email service providers such as outlook.com exerts stricter policy on their customers. If you want to use outlook.com’s smtp server to send emails, you must be a register user of outlook.com, your email client must login(with the auth login smtp command) before sending email. You can use a forged envelope-from( mail from) but you cannot use a fraudulent message From header. The message From header must be your registered email account(username) at outlook.com. If your message From header is not genuine, the smtp server will reject your email. If the envelope from is faked, outlook’s smtp server will use the message From as the envelope from when talking to receiving smtp server. If the email client does not provide a message From but provides a genuine envelope from(the account email), outlook will generate a message from header using the envelope from. In this way, outlook makes it certain that its customers won’t be able to send emails with forged From address using its email service. But as I said before, malicious people who want to disguise as an outlook user won’t even bother to use outlook’s smtp server to send forged emails. They set up their private email servers or their email client directly delivers emails to receiving mail servers to deceive the recipient. That is why SPF comes into play. By the way, outlook SPF policy is unusual: it does not block emails with forged envelope from but block emails with forged message From header,i.e., if the domain in the message From has a SPF that does not list the ip of the tcp connection, block the email. This sounds more reasonable because the message From is what the end user sees while the envelope from is invisible to end users.

 

Before I go forward to DKIM and DMARC, I would like to make it clear that SPF,DKIM,and DMARC don’t prevent spammers, they are just not designed to fight against spammers. Today’s email marketers all know to buy their own domain, set up SPF,DKIM,and DMARC records in the DNS, and then send massive emails that can pass SPF,DKIM, and DMARC check. What SPF can do is protecting the brand (domain) of famous companies like PayPal. The ability of protection comes from the belief that the machine(SMTP server) using SPF, not the human,  can take action on those emails that break the SPF policy. Neither SPF nor DKIM can prevent a spammer from creating an email with message From set to “xxx@paypal.com” and sending it successfully to cheat the recipient.  This is called email phishing. You may say DKIM can sign the From header. Yes, DKIM often signs headers including message From. An example of DKIM signature is:

DKIM-Signature: v=1; a=rsa-sha256; d=domainhostseotool.com; s=news;
c=relaxed/relaxed; q=dns/txt; t=1126524832; x=1149015927;
h=from:to:subject:date:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=MHIzKDU2dfzf3FGyNzR1Njc5OTAyMjM0MUY3ODlqBLP=;
b=hyjCnOfAKDdLHJFc9G1q7LoDWlEniSbzc+yuU2zGrtruF00ldcF
VoG4WTHNiYwG

 

The message headers DKIM takes into account when calculating the signature(b)   are listed in the h parameter, which often(not necessarily) includes from, to, subject, date. DKIM plugin such as opendkim calculates the hash of these headers and signs the hash using the domain’s private key, which results in the signature. Note that the key is not necessarily pertaining to the domain in the From header. It can be the domain of the spammer. The key’s domain is manifested in the d parameter.  The value of the s parameter is called selector. The selector, together with the domain, is be used to obtain the public key. For example, if d=outlook.com and s=selector1, we can query DNS to get the TXT record of selector1._domainkey.outlook.com

nslookup -type=TXT selector1._domainkey.outlook.com

Non-authoritative answer:
selector1._domainkey.outlook.com        canonical name = selector1._domainkey.outbound.protection.outlook.com
selector1._domainkey.outbound.protection.outlook.com    text =

“v=DKIM1;k=rsa;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvWyktrIL8DO
/+UGvMbv7cPd/Xogpbs7pgVw8y9ldO6AAMmg8+ijENl/c7Fb1MfKM7uG3LMwAr0dDFRyM+mbkoX2k5L7
lsROQr0Z9gGSpu7xrnZOa58+/pIhd2Xk/DFPpa5+TKbWodbsSZPRN8z0RY5x59jdzSclXlEyN9mEZdmO
iKTsOP6A7vQxfSya9jg5″
“N81dfNNvP7HnWejMMFERFMrXptxOhIBuEYH67JDe98QgX14oHvGM2Uz53if/SW8MF09rYh9
sp4ZsaWLIg6T343JzlbtrsGRGCDJ9JPpxRWZimtz+Up/BlKzT6sCCrBihb/Bi3pZiEBB4Ui/vruL5RCQ
IDAQAB;n=2048,1452627113,3458351913″

 

Note that  querying the TXT record of the domain can not get the DKIM public key, you must query the subdomain:selector._domainkey.domain.  selector._domainkey would be the host name you fill in when publishing the DKIM TXT record in your DNS. This is different than publishing a SPF record, where you can leave the host name empty or input a @.

Since the DKIM signature domain can be difference from the From header domain, a spammer can claim the email to be from PayPal and sign the email with the private key of his own domain and pass the DKIM verification. You can’t even stop Paypal itself from forging the From header to admin@Miscroft.com. Of course, paypal won’t be that silly to ruine its business. But what on earth is DKIM used for? Excluding those bad things, a company like paypal normally keeps the signature domain consistent with the From domain. The email recipient or a mail server in the middle way cannot modify the content of the email while claiming the email is from Paypal. The modified email cannot pass DKIM verification thus the email integrity is guaranteed. ( The DKIM verification process is querying the signature domain for the public key, using the key to decode the signature to get the decoded hash, calculating the hash of signing fields, comparing the calculated hash with the decoded hash to see if they are the same.  )In short, DKIM can help a company to maintain its brand.

 

Now the question:should I use both SPF and DKIM considering they are independent of each other. The answer is yes if you want to protect your brand better. Why should I use DKIM if SPF is already used? SPF can only verify the ip/mail from of the last hop. If your email is forwarded, chances are your email will break SPF at the inbound mail server. In this case, you can still use DKIM to prove the email is from you. Why should I use SPF if DKIM is already used? Because anybody who receives email from you can forward the email untouched to anybody else. Without SPF, you can not guarantee the emails received by a recipient is really sent by you.

 

We’ve known that neither SPF or  DKIM can stop spoofed message from header. This is where DMARC comes into play. DMARC requires SPF or DKIM or both in use. Only when email passing SPF check and the domain in mail from/signature is equal to the one in message from can it pass the DMARC check. Additionally, DMARC allows you to specify a reporting email address in the DMARC DNS record so that the DMARC verification information can be sent to that email address.

 

 

https://serverfault.com/questions/523443/spf-vs-dkim-the-exact-use-cases-and-differences

https://security.stackexchange.com/questions/151241/security-of-spf-vs-spf-and-dkim-in-email

https://www.emailonacid.com/blog/article/email-development/what_is_dkim_everything_you_need_to_know_about_digital_signatures/

https://help.returnpath.com/hc/en-us/articles/222438487-DKIM-signature-header-detail

https://stackoverflow.com/questions/31239022/should-i-sign-x-headers-in-dkim

https://www.anchor.com.au/hosting/support/Setting_From_and_Bounce_Address_on_Email_From_Websites

https://serverfault.com/questions/400028/in-the-mail-header-what-does-the-received-from-ip-refer-to

https://www.pobox.com/helpspot/index.php?pg=kb.page&id=253

https://www.sparkpost.com/blog/understanding-spf-and-dkim/

https://support.rackspace.com/how-to/create-a-dkim-txt-record/

https://blog.returnpath.com/how-to-explain-spf-in-plain-english/

https://blog.returnpath.com/how-to-explain-dkim-in-plain-english-2/

https://www.endpoint.com/blog/2014/04/15/spf-dkim-and-dmarc-brief-explanation

Posted in tips of hosting