The From: rewriting breaks “git am”, which makes patch-based contribution workflows unnecessarily difficult. Other Mailman 2 mailing lists (e.g., at gnu.org or mail.openjdk.org) do not have this problem.
From: rewriting from mailman is done to ensure high reputation for our mail server, so other mailing systems continue to gladly receive email from us and reduce the likelihood of false spam/rejection. Raw messages are available via: https://inbox.sourceware.org/libc-alpha/ and are suitable for use by "git am".
I would also really like to see this for elfutils-devel. If we don't modify the message to invalidate the original DKIM signature we don't have to do From rewriting. But that does mean making sure we don't do things like To/CC rewriting (the no-duplicates setting), Subject rewriting, adding footers, HTML scrubbing, etc. lists.gnu.org also has a mode which allows this. Ian (added to CC) might have the full setup (they use exim instead of postfix, but should be mostly identical to our mailman setup otherwise). You can setup your list with "unmodified messages" (which has the above restrictions) or with From rewriting (if you use any of the message modification settings).
> If we don't modify the message to invalidate the original DKIM signature we > don't have to do From rewriting. [...] That still breaks SPF, which some mailing systems use to detect fraudulent senders.
(In reply to Frank Ch. Eigler from comment #3) > > If we don't modify the message to invalidate the original DKIM signature we > > don't have to do From rewriting. [...] > > That still breaks SPF, which some mailing systems use to detect fraudulent > senders. Mailman puts its bounce processor address into the SMTP envelope MAIL FROM: (sometimes also called return path), and that's what's being checked by SPF. There is no way around that, obviously. The request here is about the user-visible From: header, not the envelope sender.
> The request here is about the user-visible From: header, not the envelope > sender. OK, but DMARC then adds From: header alignment checking. Anyway, mailing list admins are already able to change the relevant mailman2 setting for their own lists. I would advise against that in isolation ("from_is_list" change "Munge From" to "No"), because other mailing list processing may well corrupt whatever remaining authentication info is in the messages. This would result in drops or rejections & DMARC reports. There may be recipes for additional mailman2 options to reliably minimize those risks, and will let you know when/if we find them. (Note that google-groups does the same From: munging for reflected traffic for DMARC participant domains.) In the mean time, the raw mailing list traffic may be processed via imap, nntp, or http via inbox.sourceware.org.
I admin the GNU lists. Here is some info about how to accomplish this: The mailman docs imply that it's From: rewriting is needed for DMARC compliance, but that isn't the case. Instead, you just ensure that dkim signatures are not broken for that list. The mailman options needed are to make the mailman list have no subject modification or footer addition, and then set dmarc_moderation_action to accept. In mailman web interface, that under privacy, sender filter. Also to fix where mailman breaks dkim signatures by removing CCs using no duplicates user setting, either use new mailman https://launchpad.net/mailman/+milestone/2.1.30 and set appropriate DEFAULT_DROP_CC described there, or a lightly patched older mailman. We use a patch, https://launchpadlibrarian.net/444501204/nodup-dkim.patch, which is linked from the relevant bug. After applying the patch, compile mailman with "pycompile -p mailman /usr/lib/mailman/Mailman -V 2.4-", then restart. There are a few other rare non-default settings that would break all dkim signatures. Just send a test message to make sure you don't have these. Then, there is the rare case someone sends from a domain with DMARC set to reject or quarantine and a missing or broken dkim signature. In that case, the sender has sent a bad message. These are quite rare, and rare enough that other big free software lists I've looked at ignore them and let them get rejected by any recipient which does dmarc validation. On GNU lists, I decided that wasn't good enough because if the sender just made a mistake, or just had a misconfiguration, those rejections just go into the list server mta log and no one notices them, and then only some subscribers will get their message, leading to confusion. It is better to either reject the message from the poster, telling them the problem in the reject message, or be even nicer and do from rewriting in the mta because their message may still be useful even with a rewritten from. Then, if the rewriting causes a problem, a human can tell them to fix the issue on their end. That is what we do, it is documented here, https://wiki.debian.org/Exim#For_running_a_mailing_list_and_ensuring_all_sent_mail_is_DMARC_compliant . Sourcware uses postfix, so that is not a drop in solution, and is probably much easier to setup rejecting of the message.
The other main setting is from_is_list in the main list settings, of course that needs to be set to no. Minor correction to my previous comment, I was talking about lists.gnu.org, of course some GNU lists exist elsewhere like here.
I can take a look at this in the next few days, if no one beats me to it.
Also note: the compile command I mentioned has a path specific to a debian based mailman packages.
Actually, except for the patching issue, it looks like mailman list administrators can make the changes themselves. Is anyone going to try this?
(In reply to Christopher G. Faylor from comment #10) > Actually, except for the patching issue, it looks like mailman list > administrators can make the changes themselves. Is anyone going to try this? Looking at https://launchpadlibrarian.net/444501204/nodup-dkim.patch, I think this patch is absolutely required: Like many Mailman 2 installations, sourceware defaults “Avoid duplicate copies of messages?” to Yes, and users can edit anyway. As a result, most subscribers will receive rewritten messages which fail DKIM checks. I believe this patch disables that.
I rebuilt the mailman 2.1.29 package from rhel8-modules with that patch, and installed as mailman-3:2.1.29-12.el8.sourceware.3.x86_64. We may have to do it again later, so left /root/rpmbuild with all the artifact to redo this work later, should later distro updates nuke this patch.
I tried setting up test-list@sourceware.org according to these settings. But the dmarc_moderation_action for privacy/sender cannot be set to "Accept" it will revert to "Munge From" and an error is generated: "Error: dmarc_moderation_action must be >= the configured default value."
Yes, I forgot about this. Mailman refuses to let list owners change the setting to what you are trying if you've set a site-wide/system setting. To be specific, on debian based systems there is /usr/lib/mailman/Mailman/Defaults.py which are the default mailman settings, it has DEFAULT_DMARC_MODERATION_ACTION = 0 And the system mailman config file overrides the defaults. On debian based systems, that file is /etc/mailman/mm_cfg.py I think the issue is that you've set DEFAULT_DMARC_MODERATION_ACTION to something other than 0 in your system config. So, remove it it to 0 to overcome this issue.
correction: remove it or set it to 0
(In reply to iank from comment #15) > correction: remove it or set it to 0 I set /etc/mailman/mm_cfg.py DEFAULT_DMARC_MODERATION_ACTION to 0.
I believe test-list@sourceware.org is now setup so that it keeps dkim signatures valid when sent through that list keeping the From header. But it would be good if people could test it by subscribing and/or sending with addresses that use a strict dmarc policy to see if we do get unexpected rewrites which cause bounces. https://sourceware.org/mailman/listinfo/test-list
(In reply to iank from comment #6) > There are a few other rare non-default settings that would > break all dkim signatures. One of these is stripping text/html attachments, which is done by default on our install under Content filtering, filter_mime_types. For test-list I set filter_content to No. It would be good if there was a way to reject emails containing text/html. Also public-inbox won't archive them (which is already a problem, because inbox sees the messages before the text/html part is stripped).
On Mon, 31 Oct 2022, mark at klomp dot org via Overseers wrote: > It would be good if there was a way to reject emails containing text/html. At least for some lists, accepting them is deliberate, to avoid losing contributions by placing requirement on email formatting that are problematic for many users nowadays. > Also public-inbox won't archive them (which is already a problem, > because inbox sees the messages before the text/html part is stripped). That's clearly a bug in our public-inbox configuration. Since we're deliberate accepting text/html we should set up our archiving to handle messages using it.
(In reply to joseph@codesourcery.com from comment #19) > On Mon, 31 Oct 2022, mark at klomp dot org via Overseers wrote: > > > It would be good if there was a way to reject emails containing text/html. > > At least for some lists, accepting them is deliberate, to avoid losing > contributions by placing requirement on email formatting that are > problematic for many users nowadays. Sad, but true. So if we want to accept them and keep DKIM signatures correct then we'll need to turn off content filtering, because stripping text/html parts or turning them into plain text will alter the message body. > > Also public-inbox won't archive them (which is already a problem, > > because inbox sees the messages before the text/html part is stripped). > > That's clearly a bug in our public-inbox configuration. Since we're > deliberate accepting text/html we should set up our archiving to handle > messages using it. Lets open a separate bug for that. This is a design choice of public-inbox: To discourage phishing, trackers, exploits and other nuisances, only plain-text emails are allowed and HTML is rejected by default. This improves accessibility, and saves bandwidth and storage as mail is archived forever. We have other archives of course, the messages are in the mailman archives, but it would be good to see what we can do for the public-inbox archives. Will discuss with the upstream project.
(In reply to Mark Wielaard from comment #17) > I believe test-list@sourceware.org is now setup so that it keeps dkim > signatures valid when sent through that list keeping the From header. But it > would be good if people could test it by subscribing and/or sending with > addresses that use a strict dmarc policy to see if we do get unexpected > rewrites which cause bounces. > > https://sourceware.org/mailman/listinfo/test-list So it almost works, but almost of course is not good enough because it means some messages do arrive with broken dkim verification. Specifically it seems that the Content-Transfer-Encoding of attachments are changed from 8bit to quoted-printable for the text/html and to base64 for text/plain parts.
I looked closer at list settings I did on lists.gnu.org and the I found one more: Content Filter, filter_content = No How is that set on the test list?
filter_content is set to No. But looking around it seems 8bit content transfer encodings are problematic with DKIM signing in general. The spec says before signing a message 8bit content should be transformed to a 7bit encoding so the signature survives if any transport doesn't support the 8bit encoding. So it could be said this is just an issue with the mimecast dkim signing in the above case, since it doesn't know that the full transport to the end user is 8bit clean, it should have transformed to a 7bit encoding itself before signing. There is a postfix setting that seems to try to keep the 8bit content transfer encoding even if a transport doesn't explicitly say it supports 8bit encodings: http://www.postfix.org/postconf.5.html#disable_mime_output_conversion Disable the conversion of 8BITMIME format to 7BIT format. Mime output conversion is needed when the destination does not advertise 8BITMIME support. This is recommended for postfix filters and milters to avoid breaking DKIM etc. signatures: https://www.postfix.org/MILTER_README.html#workarounds https://www.postfix.org/FILTER_README.html#advanced_filter So we probably just need to set disable_mime_output_conversion=yes and hope that any further transports do actually support 8BITMIME.
added that flag to /etc/pstfix/main.cf
> almost of course is not good enough because it means some messages do > arrive with broken dkim verification It is ok that some messages come in with broken dkim signatures. Those domains are not strict DMARC domains. > 8bit content transfer encodings are problematic with DKIM signing in general I don't think so. https://cr.yp.to/smtp/8bitmime.html was written in the 90s and suggests 7 bit only MTA's are not a thing that exist on the internet anymore, so the DKIM spec suggesting conversion is not based on reality, it is only there to conform to really old RFCs. Exim doesn't support any conversion to 7 bit. The setting you added to postfix sounds right.
(In reply to iank from comment #25) > > 8bit content transfer encodings are problematic with DKIM signing in general > > I don't think so. https://cr.yp.to/smtp/8bitmime.html was written in the > 90s and suggests 7 bit only MTA's are not a thing that exist on the > internet anymore, so the DKIM spec suggesting conversion is not based on > reality, it is only there to conform to really old RFCs. > > Exim doesn't support any conversion to 7 bit. The setting you added to > postfix sounds right. The setting does indeed help, no more conversions of the attachments. It is slightly odd that this isn't the default of postfix. By default postfix goes out of its way to "downgrade" things to 7bit encodings. But that does indeed not seem to be needed. Everything looks correct now, unless someone with the nodups flag is on CC (then the CC is altered and the DKIM signature fails). So the last remaining issue is the AvoidDuplicates. It looks like the mailman-nodup-dkim.patch from Comment 11 isn't yet installed. At least our install of /usr/lib/mailman/Mailman/Handlers/AvoidDuplicates.py isn't patched. I might misunderstand Comment 12. Do we have the package, but it isn't installed yet?
> It looks like the mailman-nodup-dkim.patch from Comment 11 isn't yet installed. It was installed. A later distro update package replaced it. This is the problem with de facto forking the package with a local patch.
(In reply to Frank Ch. Eigler from comment #27) > > It looks like the mailman-nodup-dkim.patch from Comment 11 isn't yet installed. > > It was installed. A later distro update package replaced it. This is the > problem with de facto forking the package with a local patch. Would it make sense to get this patch applied in the Red Hat Enterprise Linux mailman package?
(In reply to Florian Weimer from comment #28) > Would it make sense to get this patch applied in the Red Hat Enterprise > Linux mailman package? O, yes, that would be great. I assumed it was an orphaned EPEL package. But if it is a RHEL package then having this patch applied there would be very good. For now it has been fixed by using a "higher" version for the local build mailman-2.1.29-12.sourceware.el8.3.x86_64
I filed a bug: https://bugzilla.redhat.com/show_bug.cgi?id=2140631 It may make sense to open a Customer Portal ticket as well (and link it from the bug) if you have a supported subscription. Thanks.
We really need some people with strict dmarc policies and/or who use dkim signatures to subscribe to the test-list to see if all settings are correct now. One remaining problem is what I believe Ian described in comment #6, people who sign dkim messages in a way that we cannot avoid to break. e.g. when people include List-* or Sender headers in their dkim signature. I am unsure what to do about this. Should we reject such messages? It seems that is signaling they don't want their messages on a mailinglist or forwarded by a different sender because they "protected" those headers. But maybe it isn't done deliberately? Frank your domain, elastic.org, seems to be configured to sign (empty) list headers (Sender:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive) on all your messages. How would you like us to handle that?
(In reply to Mark Wielaard from comment #31) > We really need some people with strict dmarc policies and/or who use dkim > signatures to subscribe to the test-list to see if all settings are correct > now. Fangrui Song sent a couple of test messages: https://sourceware.org/pipermail/test-list/2022/thread.html They made it through correctly and back into Gmail, as far as I can see. I believe google.com has a rather restricted DMARC policy, and Gmail enforces it.
(In reply to Mark Wielaard from comment #31) > Frank your domain, elastic.org, seems to be configured to sign (empty) list > headers > (Sender:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent- > Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:List-Post:List- > Owner:List-Archive) on all your messages. How would you like us to handle > that? This must be the default for the exim MTA in use here. It goes to show how fragile this sort of forwarding is - not sure you or I can do anything about it.
(In reply to Frank Ch. Eigler from comment #33) > (In reply to Mark Wielaard from comment #31) > > Frank your domain, elastic.org, seems to be configured to sign (empty) list > > headers > > (Sender:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent- > > Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe:List-Post:List- > > Owner:List-Archive) on all your messages. How would you like us to handle > > that? > > This must be the default for the exim MTA in use here. It goes to show how > fragile this sort of forwarding is - not sure you or I can do anything about > it. This is somewhat disappointing since this seems against the spec. These headers shouldn't be included by default if they aren't set on the original outgoing message. Is there a page we can point people to that explains why these defaults are wrong and how to properly setup the exim MTA to do dkim signing? If I understand Ian's comment #6 these misconfigured dkim defaults are not common and we could just ignore them. People will just get periodically unsubscribed till they fix their dkim setup to not include empty Sender or List-* headers. But it would be nice to apply From rewriting to help such misconfigurations. There is a recipy for Exim to do that in comment #6. I don't know if that can easily translate to postfix.
> This is somewhat disappointing since this seems against the spec. These > headers shouldn't be included by default if they aren't set on the original > outgoing message. The RFC4871 standard allows more and non-existent headers to be signed. I suppose one argument is that signing non-existent headers protects email against later insertion of such headers. Characterizing this default as "wrong" or a "misconfiguration" in need of a "fix" is a little too far. They are inconvenient for the efforts associated with this PR, that's all.
(In reply to Frank Ch. Eigler from comment #35) > > This is somewhat disappointing since this seems against the spec. These > > headers shouldn't be included by default if they aren't set on the original > > outgoing message. > > The RFC4871 standard allows more and non-existent headers to be signed. I > suppose one argument is that signing non-existent headers protects email > against later insertion of such headers. Characterizing this default as > "wrong" or a "misconfiguration" in need of a "fix" is a little too far. > They are inconvenient for the efforts associated with this PR, that's all. Asserting the non-existence of these headers is what breaks DKIM signatures with mailing lists. I reported this upstream-upstream here: Signed header defaults break mailing lists <https://github.com/duncanthrax/pdkim/issues/4> If Exim (or pdkim really) are unwilling to fix this, Mailman (or something in front of Mailman) needs to be taught to strip such anti-mailing-list DKIM signatures.
(In reply to Florian Weimer from comment #36) > Asserting the non-existence of these headers is what breaks DKIM signatures > with mailing lists. I reported this upstream-upstream here: > > Signed header defaults break mailing lists > <https://github.com/duncanthrax/pdkim/issues/4> Thanks. There is also a Exim and Debian bug for this: https://bugs.exim.org/show_bug.cgi?id=2394 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=939808 It looks like a workaround is to not rely on the default exim setting of the dkim sign headers, but explicitly set it to: DKIM_SIGN_HEADERS = From:From:=Sender:Reply-To:Subject:Subject:Date:To:Cc:Message-ID:In-Reply-To:References:MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID:Content-Description:=Resent-Date:=Resent-From:=Resent-Sender:=Resent-To:=Resent-Cc:=Resent-Message-ID:=List-Id:=List-Help:=List-Unsubscribe:=List-Subscribe:=List-Post:=List-Owner:=List-Archive" > If Exim (or pdkim really) are unwilling to fix this, Mailman (or something > in front of Mailman) needs to be taught to strip such anti-mailing-list DKIM > signatures. The problem with getting this fixed seems to be that some people are convinced dmarc and dkim will just break mailinglists always, so they don't see a point in fixing the defaults. Stripping can only be done if the dmarc policy for the domain doesn't require DKIM signatures, otherwise we will either have to bounce them before they reach the list or do From munging anyway (as suggested in comment #6).
Just general fyi: My exim configuration is DKIM_SIGN_HEADERS = mime-version:in-reply-to:references:from:date:subject:to I got that from looking at the headers that gmail signs. What you posted seems excessive, but should still work. I think exim basically tries to follow the RFCs in it's defaults even when they don't make any practical sense, and the best change you might get exim upstream to accept is to add a note to the documentation.
(In reply to Mark Wielaard from comment #37) > (In reply to Florian Weimer from comment #36) > > Asserting the non-existence of these headers is what breaks DKIM signatures > > with mailing lists. I reported this upstream-upstream here: > > > > Signed header defaults break mailing lists > > <https://github.com/duncanthrax/pdkim/issues/4> > > Thanks. There is also a Exim and Debian bug for this: > https://bugs.exim.org/show_bug.cgi?id=2394 > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=939808 > > It looks like a workaround is to not rely on the default exim setting of the > dkim sign headers, but explicitly set it to: > > DKIM_SIGN_HEADERS = > From:From:=Sender:Reply-To:Subject:Subject:Date:To:Cc:Message-ID:In-Reply-To: > References:MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: > Content-Description:=Resent-Date:=Resent-From:=Resent-Sender:=Resent-To: > =Resent-Cc:=Resent-Message-ID:=List-Id:=List-Help:=List-Unsubscribe:=List- > Subscribe:=List-Post:=List-Owner:=List-Archive" “=” for the mailing list headers disables nested mailing lists (or forwarding mail to lists with “S D r” in Gnus, “b” in mutt). It should be much more compatible with the default. > > If Exim (or pdkim really) are unwilling to fix this, Mailman (or something > > in front of Mailman) needs to be taught to strip such anti-mailing-list DKIM > > signatures. > > The problem with getting this fixed seems to be that some people are > convinced dmarc and dkim will just break mailinglists always, so they don't > see a point in fixing the defaults. Some of them even have the power to change the defaults. It might be tricky to convince them to make the change that proves them wrong. > Stripping can only be done if the dmarc policy for the domain doesn't > require DKIM signatures, otherwise we will either have to bounce them before > they reach the list or do From munging anyway (as suggested in comment #6). Right, it's non-trivial logic, and not going to be implemented for Mailman 2.
Note that c8s seems to have been updated to mailman 2.1.30 https://gitlab.com/redhat/centos-stream/rpms/mailman/-/commits/stream-mailman-2.1-rhel-8.9.0 https://bugzilla.redhat.com/show_bug.cgi?id=2140631
(In reply to Mark Wielaard from comment #40) > Note that c8s seems to have been updated to mailman 2.1.30 > https://gitlab.com/redhat/centos-stream/rpms/mailman/-/commits/stream- > mailman-2.1-rhel-8.9.0 > https://bugzilla.redhat.com/show_bug.cgi?id=2140631 Package available at https://kojihub.stream.centos.org/koji/buildinfo?buildID=35377
(In reply to Mark Wielaard from comment #41) > (In reply to Mark Wielaard from comment #40) > > Note that c8s seems to have been updated to mailman 2.1.30 > > https://gitlab.com/redhat/centos-stream/rpms/mailman/-/commits/stream- > > mailman-2.1-rhel-8.9.0 > > https://bugzilla.redhat.com/show_bug.cgi?id=2140631 > > Package available at > https://kojihub.stream.centos.org/koji/buildinfo?buildID=35377 I went through the diff and it contains the patch we use in mailman-2.1.29-12.sourceware.el8.3.x86_64 It also contains a simpler way to prevent dropping CCs: +# The process which avoids sending a list copy of a message to a member who +# is also directly addressed in To: or Cc: can drop the address from Cc: to +# avoid growing a long Cc: list in long threads. This can be undesirable as +# it can break DKIM signatures and possibly cause confusion. To avoid changes +# to Cc: headers, set the list's drop_cc to No. +DEFAULT_DROP_CC = Yes We should set that to No. There is also a new dmarc_moderation_addresses: + ('dmarc_moderation_addresses', mm_cfg.EmailListEx, (10, WIDTH), 1, + _("""List of addresses (or regexps) whose posts should always apply + <a href="?VARHELP=privacy/sender/dmarc_moderation_action" + >dmarc_moderation_action</a> + regardless of any domain specific DMARC Policy."""), + + _("""Postings from any of these addresses will automatically + apply any DMARC action mitigation. This can be utilized to + automatically wrap or munge postings from known addresses or + domains that might have policies rejecting external mail From: + themselves. We could use this for addresses that add any list headers to their own DKIM signatures.
We now run 2.1.30 with DEFAULT_DROP_CC = No
I analyzed the ~7000 messages from this year to libc-alpha. Only one legitimate email had signed (empty) List headers (the other was spam). The email is from a one time contributor. patchwork did capture the patch: https://patchwork.sourceware.org/project/glibc/patch/20230119034258.15881-1-kilobyte@angband.pl/ But it was never reviewed (maybe because people had bounced that email?) Anyway, my conclusion is that in general libc-alpha contributors don't use dkim signing of list headers.
So to make this work we have to make sure none of the list settings change the email body, subject or any other header that is likely covered by a dkim signature. Which means the following settings (see also Comment #6, Comment #7, Comment #18 and Comment 22): - subject_prefix (general): (empty) - from_is_list (general): No - anonymous_list (general): No - first_strip_reply_to (general): No - reply_goes_to_list (general): Poster - reply_to_address (general): (empty) - include_sender_header (general): No - drop_cc (general): No - msg_header (nondigest): (empty) - msg_footer (nondigest): (empty) - scrub_nondigest (nondigest): No - dmarc_moderation_action (privacy): Accept - filter_content (contentfilter): No
This is now used by libc-alpha, gcc-patches, libstdc++, jit, fortran, gcc-rust, newlib, gdb-patches, binutils, elfutils, libabigail.
Thank you so much!