Email used to be easy: receive it, deliver it. Then it became harder: receive it, spamassassin it, bounce or deliver it. But now, with the firehose of spam targeting billions of mail accounts, mail services have had to adapt increasingly draconian measures to filter out spam and forgery but reliably deliver real email.

Technology background: email authentication technologies

SPF

Purpose: to verify that email for a domain is directly originating from an IP address approved by that domain.

Sender: lists names of outgoing mail servers in text records in DNS.

Receiver: verifies envelope "From " header during SMTP connection, to check IP address vs. SPF records.

DKIM

Purpose: to verify that portions of a message/body have not been altered by intermediaries.

Sender: posts DKIM public keys for domain in DNS. Cryptographically signs a subset or superset of headers+body, and adds that signature as new header.

Receiver: verifies each DKIM signature against received subsequent headers+body

ARC

Purpose: to verify that intermediate servers passed along authentication results.

Sender: adds ARC headers for authenticated emails traveling through system.

Receiver: checks ARC signatures.

SRS

Purpose: allow forwarded emails to trigger bounces that go back to originator

Sender: rewrites the message envelope with a temporarily valid address that's associated with the originator

Receiver: replies to temporary address if an email is rejected/bounced

DMARC

Purpose: to verify alignment of user-visible email headers & content with SPF and/or DKIM authentication.

Sender: publishes a policy record in DNS

Receiver: verifies that message was signed with DKIM and/or came directly from an approved sender for the user-visible ("From:") domain.

personal email cryptography

Additional cryptography layers added by the end-users via PGP or S/MIME but these do not really help signal their authenticity. Personal signatures are hard to verify (key distribution problem). Encryption frustrates content based analysis.

IP reputation

Mail originators are scored by multiple agencies about the quantity of traffic and spam they pass. This reputation is available via DNS queries to recipients as an additional signal. Sourceware has signed up with DNSWL, a commonly used whitelist.

spam filtering

Receiver side tools including spamassassin collect validation signals from the above mechanisms, email content analysis, crowdsourced indications of spamminess, to arrive at a scalar numeric score. Email hitting a spam threshold is rejected, generally during the SMTP transaction itself.

Policies of major mail servers

Yahoo, google, apple, and microsoft are increasingly demanding DMARC alignment from incoming mail.

Sourceware's postfix mail server includes "milter" plugins for all of the above mail server tech. Sourceware publishes DKIM/DMARC/ARC policies and keys in DNS. Our DMARC policy is transitioning to improve deliverability and further forwarding of our email.

Incoming mail

Incoming mail is checked against all the criteria above, and positive and negative signals mixed into local spamassassin calculations. Messages that pass the criteria are delivered to user, service, or mailing list accounts as appropriate.

Outgoing mail, locally originated

Outgoing mail originating from sourceware services such as bugzilla is fully signed, protected, sealed, and is bound to be delivered.

Outgoing mail, personally forwarded

Outgoing mail originating from forwarding by ordinary users, are subjected to SRS envelope rewriting, which are generally reliably received.

Mailing list traffic, forwarded through mailman2

Mailing list traffic necessitates changes to normal email. A single email to a single email alias turns into many outgoing emails to many recipients, with a number of customary changes such as added headers to track mailing list links. Most importantly, an unsubscribe link! Each mailing list has distinct lists of subscribers, and distinct policies with respect to content and header adjustment operations.

There is an inherent conflict between forwarding email through mailing lists and assuring authenticity of the original mail. The forwarded email will appear to come from a different sending machine and going to a different destination than original. Despite these changes, mailing list hosts must try to make the forwarded outgoing messages as deliverable as possible. Undeliverable mail is a double problem: not only will a subscriber miss a message, but enough undeliverable bounces will cause subscribers to be automatically removed from mailing lists.

The controversy: mailing list From: rewriting

The DMARC policies aim to prove to email recipients that the email either arrived directly from its From:-visible author (SPF), or may have been relayed but was unmolested on the way (DKIM). The first criterion clearly cannot be met by a mailing list relay, which leaves DKIM as the only way to assure receiving systems. However, a DKIM signature associated with a given sender needs to come from a server associated with that author (with access to the domain's private keys), which a third party mailing list host cannot.

Even if the author's originating mail server does sign the outgoing message with DKIM, and sourceware relays that signature, mailman could accidentally invalidate that signature. Even quite innocent changes can mess it up, like changing message content encoding or tweaking white space. Another example is by adding list-related headers that were not there before and were "oversigned". Some MTA software such as Exim defaults to oversigning several list-related fields. This results in invalid DKIM signatures, and thus a DMARC failure, and thus mail bouncing.

Consider the possibility that the originating mail server does not sign with DKIM at all. In that case, sourceware's relay breaks SPF From: alignment, and cannot sign with DKIM on behalf of the originating server (nobody can). Instant DMARC failure, mail bouncing.

In neither of these cases did the originator system do anything wrong. In both cases, the original messages were perfectly compliant. It is sourceware's relaying operation that broke what would otherwise have passed DMARC.

The policy cure to this problem is to rewrite the From: field of the email as it passes through the mailing list. This looks like:

From: Mark Wielaard via Overseers <overseers@sourceware.org>
Reply-To: Overseers mailing list <overseers@sourceware.org>
Cc: Mark Wielaard <mark@klomp.org>, [...]

The effect is that the user-visible "From:" address identifies the name of the person. But instead of his/her email address, the local mailing list name is used as email address, and the person's email is added to Cc: and/or Reply-To:. From a mail authentication perspective, this makes all the difference, as SPF, DKIM, etc., all treat this new message as though it was locally originated, and cryptographically authenticate it with DKIM etc. Receivers validate and accept it. People can read it. People can reply to it.

But it's ugly to look at.

Yes, it is.

And it's fundamentally wrong and evil.

It is a technical compromise with trade-offs, including breaking with some traditions.

And it breaks git patches.

Yes, if you just "git am" received emails, the Author field of the git commit may be screwed up.

However, you can

• hand-amend the author field of the commit ("git commit --amend --author=FOO")
• instruct your contributors to send git patches with a safe From: author field in the patch body by adding this into their .gitconfig:

:

[format]
    from = true
    forceInBodyFrom = true

But it works.

Yes, it does.

dmarc_moderation_action = Munge From
dmarc_quarantine_moderation_action = yes
dmarc_none_moderation_action = yes
from_is_list = Munge From