dmartel@blog
← dmartel.dev

Securing SMTP Transport with DNSSEC, DANE, and TLSA

· 6 min read
#dns#dnssec#dane#smtp#security

1. The problem these protocols solve

SMTP, the mail transport protocol, has no security built in at the protocol level (RFC 5321). One extension lets it encrypt the transport:

  • STARTTLS (RFC 3207): opportunistic TLS negotiation after the connection starts in the clear.
  • The problem: STARTTLS is optional and, by default, unauthenticated. An attacker in a man-in-the-middle position can:
    • strip the STARTTLS keyword from the server’s response (a downgrade attack, known as STRIPTLS);
    • present a self-signed or spoofed certificate, which the client often accepts anyway, since SMTP doesn’t normally validate the certificate name under opportunistic TLS.

DANE (DNS-based Authentication of Named Entities, RFC 6698) solves exactly this problem: it lets a domain publish, in the DNS itself, the fingerprint of the TLS certificate its servers must present — making that information verifiable independently of traditional certificate authorities (CAs). But for that DNS announcement to be trustworthy, it must itself be protected against tampering — that’s the role of DNSSEC.

2. DNSSEC: authenticating DNS data

2.1 Principle

Plain DNS offers no authenticity guarantee: a response can be forged via spoofing, cache poisoning, or an on-path attack. DNSSEC (RFC 4033, 4034, 4035) adds a layer of asymmetric cryptographic signatures on top of the DNS hierarchy, without changing the transport protocol itself.

Each signed zone publishes:

RecordRole
DNSKEYThe zone’s public key (KSK, Key Signing Key, and ZSK, Zone Signing Key)
RRSIGSignature of a record set (RRset) by the corresponding private key
DS (Delegation Signer)Fingerprint of the child zone’s public key, published in the parent zone — the link in the chain of trust
NSEC / NSEC3Authenticated proof that a record does not exist

2.2 The chain of trust

Trust flows down from the root zone (whose key is embedded in resolvers, managed by IANA) to the final domain, with each zone signing the public key of the zone delegated immediately below it.

DNSSEC chain of trust

Concretely, a validating DNS resolver:

  1. holds the root’s trust anchor;
  2. follows the delegation . → .com → example.com, checking at each step that the DS published by the parent matches the child’s key (DNSKEY), and that every RRset is correctly signed (RRSIG);
  3. if the whole chain validates, sets the AD (Authentic Data) bit in the DNS response header.

Without that AD bit, a record — including a TLSA record — must not be treated as trustworthy for DANE purposes.

2.3 What DNSSEC does not do

DNSSEC guarantees the authenticity and integrity of DNS data (it really comes from the legitimate zone owner and hasn’t been altered), but it does not encrypt DNS queries/responses (no confidentiality — that’s the role of DoT/DoH, out of scope here).

3. TLSA: publishing a certificate fingerprint in the DNS

3.1 Record name and structure

A TLSA record (RFC 6698) is published under a name built as:

TLSA record name pattern
_<port>._<protocol>.<hostname>

For a mail server mail.example.com listening for SMTP on port 25:

Example TLSA record
_25._tcp.mail.example.com.  IN TLSA  3 1 1 <SHA-256 fingerprint of the public key>

3.2 The four TLSA fields

FieldPossible valuesMeaning
Usage (0–3)0 PKIX-TA, 1 PKIX-EE, 2 DANE-TA, 3 DANE-EEWhat’s being constrained: a specific CA, or the end-entity certificate directly, with or without classic PKIX validation on top
Selector (0–1)0 = full certificate, 1 = public key (SubjectPublicKeyInfo) onlyWhich part of the certificate is hashed
Matching type (0–2)0 = raw data, 1 = SHA-256, 2 = SHA-512Hashing algorithm used
Association dataHex-encoded fingerprintThe value to compare against

For email, the combination recommended by RFC 7672 is 3 1 1 (DANE-EE, public key, SHA-256): the presented certificate must match exactly, independent of any certificate authority — which even allows self-signed certificates, as long as the DNS fingerprint itself is trustworthy.

3.3 Client-side verification

DANE/TLSA validation flow

Before sending a message, the sending SMTP server (more precisely, its outbound MTA):

  1. resolves the recipient domain’s MX records;
  2. queries the DNS for the TLSA record matching _25._tcp.<MX host>;
  3. requires the response to be DNSSEC-validated (AD bit present) — without it, DANE doesn’t apply, and the connection falls back to plain opportunistic TLS, or even plaintext, depending on local policy;
  4. opens the SMTP connection, negotiates STARTTLS, and retrieves the certificate presented by the remote server;
  5. compares that certificate’s (or its public key’s) fingerprint against the TLSA records obtained;
  6. accepts the connection only on a match.

4. DANE applied to SMTP: RFC 7672

RFC 6698 defines DANE generically (usable for HTTPS, XMPP, and more). RFC 7672 (SMTP Security via Opportunistic DNS-Based Authentication of Named Entities) specifies how it applies to outbound email (MTA-to-MTA), notably:

  • DANE for SMTP applies at the MX level, not the email domain: each MX record needs its own TLSA, since a domain can delegate mail to several different hosts.
  • The mode is called “opportunistic secure”: if no TLSA record exists (and that absence is itself proven by DNSSEC via NSEC/NSEC3), the client falls back to plain opportunistic STARTTLS without DANE. But if a TLSA record exists and validates under DNSSEC, DANE becomes mandatory: a certificate mismatch causes the connection to be refused, with no plaintext fallback.
  • This directly fixes STARTTLS’s weakness: an attacker can no longer strip the TLS negotiation or substitute a fake certificate without it being detected, since the security policy (“TLS + this exact certificate, mandatory”) is anchored in signed DNS, out of the attacker’s reach on the network path.

Full MTA-to-MTA sequence

Secure SMTP sequence with DANE

5. Security guarantees, summarized

ThreatWithout DANEWith DANE + DNSSEC
STARTTLS stripped (STRIPTLS)Mail goes out in the clear, no warningConnection refused if TLSA requires TLS
Spoofed/self-signed certificate wrongly acceptedAccepted, since SMTP doesn’t validate the hostname by defaultRejected if the fingerprint doesn’t match
MX or TLSA record forged in transitPossible via DNS poisoningPrevented: DNSSEC signature is verified
Dependency on a third-party certificate authorityYes (classic X.509 PKI)Optional: usage 3 1 1 does away with CAs

Limitations and prerequisites

  • The recipient domain must sign its zone with DNSSEC and publish TLSA records consistent with the certificate actually presented (a certificate rotation forgotten in the TLSA record breaks mail delivery).
  • The sending MTA must implement DANE (e.g. Postfix ≥ 2.11 with smtp_dns_support_level = dnssec, Exim, some large providers’ relays).
  • DANE for SMTP only protects the MTA-to-MTA hop; end-to-end content encryption needs S/MIME or PGP, which is out of scope here.
  • Don’t confuse it with MTA-STS (RFC 8461), an alternative/complement that relies on HTTPS and a policy published on an mta-sts subdomain, without depending on DNSSEC — useful where DNSSEC isn’t deployed, but with a different trust model (TOFU, Trust On First Use, rather than DNS-anchored cryptographic trust).

6. Normative references (RFCs) and further reading

Further documentation and tools