Distributed SSH and mail brute force: correlating usernames across addresses

The relay has been under a continuous SSH brute force for as long as I have had it. fail2ban is running. It was, at the point I sat down to look properly, holding 808 addresses in its sshd jail. The attack had not slowed down by any measurable amount.

fail2ban is not failing here. It is counting failed attempts per address and banning the ones that cross a threshold, which is what it is for, and the botnet is built so that no single address ever crosses it.

The question: how do you stop a distributed brute force when banning the source does nothing?

Short answer: stop counting the address. Count the username, across addresses, inside a time window. The botnet can rotate through a /16 all day and it still has to type something into the login prompt, and what it types is the same on every one of those addresses.

Why per-address counting loses#

Seven days of sshd logs off the relay, parsed into (timestamp, username, address):

6,404 failed attempts
  953 distinct source addresses
  597 distinct usernames

Two numbers out of that set decide everything:

Four out of five attackers never reach a maxretry of 5. They do not need to. The limit is on attempts per address and nothing limits how many addresses they use, so each one spends three or four guesses and is discarded.

Meanwhile the usernames repeat constantly. root came from 117 distinct addresses in a single day. ubuntu from 72. deploy from 58.

per-address counting                per-username counting

45.148.10.5   -> 3 attempts  OK     root      <- 117 addresses  BAN
45.148.10.9   -> 2 attempts  OK     ubuntu    <-  72 addresses  BAN
92.118.39.14  -> 4 attempts  OK     deploy    <-  58 addresses  BAN
...949 more, all under the limit
                                    same 953 addresses, seen
nothing ever trips                  through what they asked for

Same log lines, same seven days. Counted per address it is nine hundred separate visitors, none of them interesting. Counted per username it is a handful of operations.

What the usernames give away#

There are three tiers in that list of 597 and they want completely different treatment.

Generic spray. root, admin, ubuntu, test, oracle, postgres, git. Every scanner on the internet tries these. They tell you nothing about the target and they are also the least dangerous, because none of them exist.

Estate-specific names. archworks came from 40 addresses, zusammen-stark from 32. Neither is in a default wordlist - somebody read the domain, or a certificate, or this blog, and built a list. An address trying one of those is not scanning the internet, it is scanning me.

Near misses. A small number of attempts land within one or two characters of something real. I am not going to print them, because a near miss describes the account it is near. They are the reason you cannot simply ban on "username not found": a real user fat-fingering their own login looks identical to a scanner from the log's point of view, and the log is all you have.

And then there is root, which is none of those. It is attacked harder than anything else - 626 addresses in seven days, 4,186 attempts - and it is the one name the invalid-user filters can never see, because the account exists, so sshd logs Connection closed by authenticating user root rather than Invalid user. It needs its own rule and it turns out to deserve a blunt one. PermitRootLogin is prohibit-password on that box, so a password attempt cannot succeed no matter how many times it is made, and seven days of logs hold zero accepted root logins by key either. Nothing legitimate uses it, so one attempt is enough:

[sshd-root]
filter   = sshd-root
maxretry = 1
findtime = 1d
bantime  = 4w

Worth checking your own real accounts for the same shape. The interesting ones are not the accounts that get attacked, they are the accounts that get attacked and never legitimately used.

So the rule cannot be "unknown user". It has to be "unknown user, and far enough away from a known one that a human could not have got there by accident". I took every account that actually exists, measured Levenshtein distance against every attempted name, and kept only attempts at distance 3 or greater.

Of 608 distinct invalid usernames in seven days, 55 fall inside that radius and are spared. The other 553 are things nobody with an account here would ever type, and they get their own jail with maxretry = 1. One attempt at a name that cannot be a typo is not a mistake, so there is nothing to be forgiving about:

[sshd-tripwire]
enabled  = true
filter   = sshd-tripwire
backend  = systemd
maxretry = 1
findtime = 10m
bantime  = 4w

Everything else stays at maxretry = 3, because everything else might be a person.

I am not publishing the list. Not because it is sensitive on its own, but because the useful half of publishing it is telling an attacker which 55 names are not on it.

Validate the regex against the real journal before you arm anything, and check both directions:

$ fail2ban-regex systemd-journal sshd-tripwire
Lines: 3661 lines, 0 ignored, 976 matched, 2685 missed

976 matched and not one of them is an accepted login, which is structural rather than lucky: the pattern is anchored on Invalid user, and that line only exists for an account that does not.

The list is derived, not curated, so it goes stale. Mine sat at 83 names for long enough that regenerating it against the current seven days took it to 553 and picked up 28 more addresses that had been walking away clean. Regenerate it whenever you add or remove an account, and on a timer besides.

The correlation fail2ban cannot do#

Everything above is still a per-address counter, just pointed at a better field. None of it can see that 24 different addresses tried the same obscure username inside one minute, because fail2ban holds no state that spans sources.

So that part runs outside fail2ban, as an hourly collector that reads a 24-hour window back out of Loki and emits Prometheus metrics. The correlation itself is three lines of logic: group the attempts by username, sort each group by time, and slide a 600-second window looking for the widest set of distinct addresses inside it.

BURST_SEC     = 600   # one campaign, if the same name comes from several addresses inside this
BURST_MIN_IPS = 3

A username sprayed from three or more addresses inside ten minutes is one operation using many hosts, not three people independently guessing the same obscure string. Over the first 24 hours it found 13 coordinated campaigns, the widest spread being 24 addresses against a single username.

That number is the alert. Not "we are under attack" - that is always true and therefore worthless as a signal - but "somebody is running a coordinated operation right now, and here is the name they picked". It also feeds a /24 concentration metric, which is how the two networks in the next section made themselves obvious.

The collector emits attempts, distinct addresses, distinct usernames, the campaign count, the widest spread, the top sprayed names, the hostile /24s and a geo breakdown. It also emits its own success flag. A collector that fails and writes zero is indistinguishable from one that ran and found nothing, and zero gets read as clean.

One address, several unrelated names#

Everything above runs the correlation one way: one username, many addresses. Run it the other way round and it catches a different attacker entirely.

Same seven days, asking which sources the baseline jail never touches. maxretry = 3 over a one-day window catches 482 of 709 addresses; 227 walk away clean. 208 of those tried a single username. The tripwire takes any that picked a name off the 553, which is 159 of the 227, and of the 68 left over, 67 had only ever knocked on root - which is what sent me to the jail in the section above. It caught 104 addresses in its first minute, against the 152 the three-strike baseline had been managing across the whole week.

The other 19 tried two.

221.213.129.46   bitnami, ubuntu
113.249.121.140  airflow, arkserver
118.145.228.55   vendas, zte
123.122.36.191   md, ravi

Two attempts, two names, under every limit, nothing fires. Those pairs are not variants of each other and not near anything that exists on the box - they are two entries off a wordlist, tried from a host that is not coming back.

That is a sharper discriminator than the one in the last section. A person who fails a login fails on the same username, because it is theirs and what they got wrong was the password. A source failing on two unrelated names is not failing to authenticate, it is asking who exists. Counting distinct usernames splits those at two, where counting attempts needs three and still misses them.

It is also where I found the one rule in this exercise that would have locked me out. Checking the successful logins for any source presenting more than one username:

<my own address>   2 usernames: <two accounts I actually use>

One row, and it is mine - a single address that legitimately authenticates as two different accounts. So a flat "two distinct usernames is enumeration" rule bans the admin path on day one, and the only thing standing between that rule and a lockout is ignoreip. That is the argument for writing the whitelist before the first jail rather than after the first mistake. Check your own successful logins for this shape before picking a threshold - a jump host or a shared NAT will have it too, and so will anything with a service account beside a human one.

Mail runs the opposite way and needs nothing new. 32 of 41 attacking addresses try three or more distinct usernames and 21 try exactly four, because the mail wordlist is short and every host works the whole of it. All four are bare names, so each one is its own match against mail-tripwire, and a source reaches the second long before it reaches the fourth. Already covered.

SSH is not, because fail2ban counts matches per host and has no notion of distinct values within them. So this one lives with the campaign collector rather than in a jail - the same parsed (time, username, address) stream, grouped by address instead of by name, flagging anything that reaches three distinct usernames. It reports rather than bans, for the reason above. Over a 24-hour window it names 101 addresses.

The same attack against mail, one hop further out#

I went to apply the same thing to mail and found 386 authentication failures on mail01, from exactly one address.

imap-login: Login aborted: Connection closed (auth failed, 1 attempts in 2 secs)
  (auth_failed): user=<admin>, method=PLAIN, rip=10.200.200.1, lip=10.0.20.61

rip=10.200.200.1 is my own WireGuard endpoint. Mail comes in through a dovecot proxy on the relay and gets forwarded inside, so by the time the mailstore writes a log line the attacker's address has been replaced with the tunnel's. Every correlation you could do on that host is correlation against one number.

The real addresses only exist on the proxy, one hop earlier, in a completely different line format:

auth(admin,45.148.10.5,sasl:plain): Error: relay-plain: Authentication failed:
  [AUTHENTICATIONFAILED] Authentication failed.

Two things fell out of reading those properly. My filter for this jail matched Error: imap: and the daemon writes Error: relay-plain:, so the jail had been enabled, running, and matching nothing at all since the day I wrote it. And every failure is logged twice, once bare and once inside an imapc(...) frame, so a regex matching both halves your effective maxretry silently. Anchor on the bare form.

Same seven days, read from the right host:

386 failures, 40 addresses, 8 usernames

test    120 attempts from 30 addresses
admin   111 attempts from 31 addresses
sales   110 attempts from 30 addresses

341 of 386 is three names from about thirty addresses each. Textbook spray, and the geo-diverse kind.

Mail needs a softer hand than SSH, because a person can genuinely typo their own address and there is no Invalid user line to lean on. The discriminator was sitting in the success logs:

614  <a service account>@archworks.co
  2  <a person>@archworks.co

Seven days of successful logins. Every single one is a full user@domain address, and not one is a bare username. Of course it is not - a mail client is configured once and sends whatever is in its settings, domain included. A human typo lands on inof@archworks.co, never on admin.

So the rule for mail is not an alias list. It is the domain:

failregex  = ^.*auth\([^,]*,<HOST>,sasl:[^)]+\): Error: relay-plain: Authentication failed
             ^.*warning: \S*\[<HOST>\]: SASL \S+ authentication failed:.*\ssasl_username=\S
ignoreregex = (?i)auth\([^,]*@archworks\.co,
              (?i)sasl_username=\S*@archworks\.co(?:\s|$)

Anything carrying the real domain falls through to the forgiving jail at maxretry = 5. Anything else - bare names, foreign domains - hits the tripwire at 2 and gets two weeks. test, admin and sales are gone on the second try. postmaster@archworks.co, which is a real alias and gets probed constantly, is never touched by it.

An alias list would have worked too, and I would still be maintaining it. 7 mailboxes and 21 aliases, stale the first time somebody adds an address. The domain rule maintains itself.

What country is actually worth#

Everything I log in from is Austrian and the sprays overwhelmingly are not, so country is worth something. It is not worth a ban: "not Austrian" is also true of most of the internet, and a rule built on it breaks the first time I travel.

So it changes patience, never outcome. Three tiers, each a jail running the same filter with an ignorecommand that skips the addresses it is not meant to police:

The whole mechanism is fail2ban's ignorecommand, which skips an address when the command exits 0:

c=$(geoiplookup "$2" | sed -n 's/.*Country Edition: \([A-Z][A-Z]\),.*/\1/p' | head -1)
[ -z "$c" ] && exit 1              # unresolvable is never trusted
case "$1" in
  near) echo "$NEAR"     | grep -qw "$c" ;;
  eu)   echo "$NEAR $EU" | grep -qw "$c" ;;
esac

An address can only ever be counted by the tiers at or below its own trust, so a nearby address still gets the full baseline patience and nothing is banned on a single failure anywhere. Within minutes of arming, the two geo jails held 129 and 152 addresses that the baseline jail was still counting toward three.

One caveat worth stating plainly: the two GeoIP databases on this estate disagree. DB-IP places 45.148.10.5 in Andorra, GeoLite puts it in the Netherlands, and they split on other addresses too. This is a trust heuristic, not a fact, which is exactly why it adjusts patience rather than deciding anything. My own addresses sit in ignoreip, so a misclassification cannot cost me a login.

Why none of this catches a real person#

Each rule above has a specific reason it cannot catch a real person:

• Admin sources are in ignoreip. Nothing below can touch them. If you build any of this, do that line first.
• The general limit is 3, not 1. Everybody gets to be wrong twice. The instant-ban jail only fires on names no human would type.
• Edit distance gates the tripwire. A name two characters off a real account is a typo. zusammen-stark is reconnaissance. Three characters of separation is what tells them apart.
• Mail leans on the domain, not a list. A mistyped local part still carries @archworks.co and lands in the forgiving jail.
• Geo reduces patience, never bans outright. The lowest tier still gets two attempts.
• Distinct usernames is a report, not a ban. Two separates enumeration from someone having a bad evening, and two is also what my own admin address produces, so that one stays a metric.

Two networks did earn a blunt answer. 45.148.10.0/24 and 92.118.39.0/24 were the only ranges hitting SSH and mail - 532 and 268 events across roughly thirty hosts each - and no legitimate traffic to this box has ever come from either. Those are dropped at the firewall rather than re-banned every time a jail expires.

That turned up an ordering trap. My firewall compiler emits an unconditional "accept TCP 22" near the top of INPUT as a lockout guard, and a source-drop placed after it does nothing on the one port you care most about:

-A INPUT -p tcp --dport 22 -j ACCEPT        <- the lockout guard
-A INPUT -s 45.148.10.0/24 -j DROP          <- never reached for ssh

A blocklist is only a blocklist if it outranks every accept above it. Worth checking the rendered ruleset rather than the config that generated it.

What I got wrong#

I shipped a filter without running it against real input. dovecot-proxy was enabled, the service was running, the config parsed, and it had matched zero lines since the day I wrote it, because it looked for Error: imap: and the daemon writes Error: relay-plain:. fail2ban has no way to tell you that: a jail with nothing to catch and a jail with a broken regex look identical from the outside, both quietly reporting zero. fail2ban-regex against the journal takes thirty seconds and is now the last step before anything gets enabled.

I built the mail side on the mailstore. That is where the mailboxes are, so that is where I went for the failed logins, and it has plenty of them. They are also useless - everything arrives through the proxy, so each one carries the tunnel address rather than the client's, and correlating them means correlating one number with itself. The evidence only exists one hop earlier. Anything behind a proxy has this shape, and the host that feels authoritative is usually the one that has already lost it.

I treated the tripwire as a list. It is a derivation - every name on it came out of measuring attempts against the accounts that exist - but once written it sat in the config as a fixed artifact while the traffic moved on underneath. Regenerating it against the current week took it from 83 names to 553 and picked up 28 addresses that had been walking away clean. It needs regenerating whenever accounts change, and on a timer regardless.

Where I landed#

Nine jails on the relay now: four for SSH, two for mail, and three geo tiers across both. The username rules catch what the address counters cannot, the collector correlates across addresses, and the geo tiers set how many attempts a source gets rather than whether it is allowed at all.

The attack has not stopped and will not. The addresses are free and the attempts cost nothing, so there is no version of this where the log goes quiet.

What changed is the coverage. Attempt counting was catching 482 of 709 sources on its own. The expanded username list takes 159 of the remaining 227, the root jail takes 104 addresses that had never been touched by anything, and the collector names the operations behind them. If I had to keep one part it would be the tripwires, since they are the only rules whose cost to the attacker does not go down as they add addresses.