Only the Mail Ports Went Quiet

A morning without mail. Thunderbird collected nothing, and neither did the phone. Webmail kept working in the browser and showed the same mailboxes as the day before. The host was reachable.

The first suspicions pointed in the wrong direction. A provider outage, a DNS record, an expired firewall rule: all plausible, all wrong. The cause was a configuration line that no longer existed on the host at all.

One note about the addresses in this text. The actually whitelisted IP addresses have been replaced with documentation addresses from RFC 5737 and RFC 3849. The shape is real, the digits are not.

What the ports showed from outside

The first useful measurement was a port probe from the affected connection. Measured with nc and a short connect timeout, so that one hanging port would not block the whole probe.

 25  no response
143  no response
465  no response
587  no response
993  no response
995  no response

443  OPEN, HTTP 200
563  OPEN

Six mail ports do not answer. HTTPS on the same host serves a page, and SSH on port 563 accepts connections. That already rules out DNS and the host firewall: neither would distinguish between individual ports on the same machine, and name resolution returned the same address for every port.

What a timeout says about the cause

The important part of that table is not in the port numbers. It is in the manner of failure. A refused connection comes back immediately and means nobody is listening on that port. A timeout means somebody is discarding the packets in silence.

That is exactly the difference between REJECT and DROP in the kernel packet filter. A REJECT sends a refusal, a DROP stays quiet and lets the client run into its own timeout. Our mail server runs with FAIL2BAN_BLOCKTYPE=drop.

Then there is the scope, limited to individual ports. Fail2Ban installs its blocking rule inside the mail server container, in that container’s own netfilter table. Services outside the container, meaning the reverse proxy on 443 and the host’s SSH daemon, never see it.

A look inside the container confirmed it. Fail2Ban groups its rules into jails, one per monitored service. Our own address was in dovecot, postfix and postfix-sasl.

What fail2ban actually ignores

That only moved the question. This address has been whitelisted for a long time, through FAIL2BAN_IGNOREIP, the docker-mailserver environment variable made for exactly this purpose. A whitelisted address must never be banned in the first place.

So we stopped asking the configuration and asked the running process.

These IP addresses/networks are ignored:
| - 127.0.0.1/8
| - ::1
| - 10.0.0.0/8
`- 172.16.0.0/12

Those are the internal networks and nothing else. Not one of the public addresses from FAIL2BAN_IGNOREIP was there. ignoreip is the list of addresses a jail must never ban, and that list had never heard of our connection.

jail.local and jail.d in parse order

At startup, docker-mailserver writes the contents of FAIL2BAN_IGNOREIP into /etc/fail2ban/jail.local. That works, and it is the intended path.

Our Ansible role additionally ships a file of its own, which lands in the container as /etc/fail2ban/jail.d/user-jail.local. It arrived with a later extension of the role, for an unrelated reason: Roundcube announces every login from a single address on the Swarm overlay, the internal network between containers. After enough user typos, webmail locked itself out. So the file added the internal networks to ignoreip.

Both files are correct on their own. What matters is the order in which Fail2Ban reads them.

/etc/fail2ban/jail.conf
/etc/fail2ban/jail.d/*.conf
/etc/fail2ban/jail.local          <- FAIL2BAN_IGNOREIP lands here
/etc/fail2ban/jail.d/*.local      <- our role lands here, last

jail.d is the directory for supplementary configuration, and inside it .local beats .conf. Our file is therefore read last. It brings a [DEFAULT] section of its own, and that section carries its own ignoreip.

From that point on, FAIL2BAN_IGNOREIP had no effect. The variable was still read, still written to disk, and then overwritten. That is what makes this fault so persistent: anyone inspecting the configuration finds the addresses exactly where they expect them.

What the post-deploy check asked for

The extension that introduced the second file was not unverified. It ended with a verification step, and that step even ran the right command, fail2ban-client get dovecot ignoreip.

It reported success. The internal networks appeared in the output, because they had just been added. What was missing from the output, nobody checked.

That is the actual lesson, and it is less comfortable than the configuration fault. A check that can only confirm something was added will never notice that something disappeared. The command that would have exposed the defect was executed, and it was green.

There is a second, older weakness on top of that. We had already overwritten ourselves at this same spot once before, back then through user-patches.sh. The rule we took away from it was: no Fail2Ban setup via user-patches.sh. The later extension obeyed that rule and produced the same effect through a different path. We had written down the incident, but not its shape.

What the plan got wrong

While assembling the new list, two IPv6 entries were up for debate, 2001:db8:271:1889::2 and its /64. Neither mail.squibble.me nor the host itself has an AAAA record in DNS. The entries looked like leftovers from an old migration, and the first draft proposed dropping them.

On the host, ip -6 addr show scope global said otherwise. The address is configured and live, and a service running there uses it as the source for its own connections to the mail server. Without the entry, the next service would have locked itself out.

DNS describes how a host is found. It says nothing about the address that same host speaks from. We came close to deleting a working entry while repairing a deleted one.

Order of operations when clearing bans

The fix makes the shipped file the single source for ignoreip and puts the internal networks and the public whitelist into the same [DEFAULT] section. FAIL2BAN_IGNOREIP is now deliberately empty, with a comment next to it explaining why the variable must not come back. Eight entries have to be visible in every jail after a deploy.

When clearing the existing bans, order matters. ignoreip prevents new bans and does not release existing ones. The ban list lives as a SQLite database on the state volume and survives restarts and redeployments. Clear first and roll out the whitelist afterwards, and legitimate traffic is simply banned again.

So: roll out the whitelist, verify it in all three jails, and only then flush. Measured with fail2ban-client status <jail> immediately before and immediately after the flush:

dovecot         451  ->  0
postfix         468  ->  0
postfix-sasl    446  ->  0
custom            0  ->  0

Those numbers are filling up again, and that is as it should be. What arrives there are real brute-force attempts against a publicly reachable mail server. The difference is that the whitelisted addresses can no longer be among them.

Conclusion

The specific fault belongs to docker-mailserver and to a role that hands the container a file of its own. Anyone using only the environment variable will never hit it. Anyone placing their own file in jail.d should carry every line in it that they want to keep.

The general part is the shape behind it. Two mechanisms configure the same setting, each correct on its own, and the later one wins completely. As long as both values look the same, nobody notices. That shape exists everywhere an environment variable and a configuration file describe the same field.

We have now written this configuration line down several times, correctly each time, and every subsequent change removed the previous answer. Documentation is clearly not the effective control here. The effective control would be a post-deploy check that turns red when the expected entries are missing from the running jail. That check is still outstanding on our side.

A whitelist nobody reads back is a guess.