c0t0d0s0.org

After the ban

c0t0d0s0 — Mon, 27 Apr 2026 15:21:00 +0000

At some point it also became clear to me that evaluating the result codes from all IPs (not what they loaded, only what the HTTP server thought of it) had, on the one hand, turned into a fairly substantial PII collection (which is why I kept it only in main memory) — one that under certain circumstances would have allowed me to “depseudonymize” the data in the pseudonymized logfile within the first 24h. Stopgap solution … yes. Final architecture … not really.

On top of that, this table was pretty annoying. Reboot the machine? Data gone. New daemon version? In most cases, data gone. I did learn a technique along the way that let me swap code on the fly, but that didn’t apply to data structures. I need a new field? Data gone! And once the data was gone, the reaction component lost a key data element. At some point a function had been added that checked whether the service had even been running long enough to have collected enough data. Which, due to changes, was rarely the case.

Beyond that, the signal was mostly fairly weak. Essentially the signal was only strong enough when an IP was making regular feed requests. The feed requests were a kind of heartbeat that the system monitored. My hypothesis was: a scanner doesn’t simulate a feed reader for hours on end.

For everyone else, the desired signal “there’s more behind this IP than just the abuser” was rather weak. Or to put it more precisely: in one direction it was indeed quite informative. If between hour -24 and hour -1 there were several valid accesses from an IP, I could assume it was probably a CGNAT, NAT or VPN. The reverse, however, didn’t hold — the number of accesses to my blog is simply too low for that. I think you’d need several orders of magnitude¹ more traffic. The data was so thin that the statement “because I haven’t seen a 200 response, it’s not a NAT, CGNAT or VPN” would have been pure speculation. So this approach was of limited value to me.

Now, ideas usually don’t come to you in front of the computer, but on the toilet, while making tea, or in the shower. I had this one while reading: I’m looking at the wrong side of the ban. For me, the information about what happens after the ban would have considerably more value than the information about what happens before the ban.

In the previous solution I was hooked into the firewall log. So I knew that something was happening. A request had been rejected by the firewall. I knew from the rule what it was: HTTP or HTTPS. But I didn’t know what was inside it. My server didn’t either. It refused to have the conversation. I had no log of what the client would have liked to do.

Rebuild

With this thought in mind, I rebuilt the setup. I had originally come at this from the angle that all the noise in the logfiles annoyed me. So a new solution had to keep requests away from my web server just like an IP block does. What doesn’t reach the server doesn’t get logged.

But I don’t necessarily have to block the request entirely. I can also redirect it elsewhere. And that’s exactly what the new solution does. As before, the system hooks into the honeypot log. The detection component is Apache. As before, an IP address can only end up in this file by being associated with a request that violated a detection pattern, so that — also as before — every IP address in it is handed over to the reaction component.

However, there is now only one reason why an IP address gets blocked outright. If an IP comes from an ASN on Spamhaus’s “Filter-on-Sight” list, a reject ban is issued. Nothing good comes out of those gates. I don’t even want to know what comes out of there. And that’s straight away for seven days. For all practical purposes that’s a permanent ban, since on the next violation the ban gets extended by another 7d.

I didn’t want to make the ban permanent outright. If no traffic is coming from an IP — or in some cases a subnet — then I don’t need a filter rule for it. And the existence of the filter rule gives me some visibility that something is coming out of those networks, without me having to look more closely into the logfile. I can also see more quickly where a subnet ban is worthwhile and where it isn’t.

The decisive difference between this solution and the old one is that alongside the rejects there are also redirects. And a redirect is essentially the default response when hitting the honeypot. If you’ve triggered the detection rules, nftables redirects you internally from the original webserver² to another web server³. Instead of port 80 you land on port 10080. Instead of port 443 you get internally redirected to port 10043⁴. This means all further accesses from these IP addresses are no longer in the logfile of the primary server. So, my first requirement was met. Second, I now have an additional logfile that contains only the actions of banned users. I now know what happens after the ban.

What happened after the ban?

# cat c0d0s0.org-access.log-blocked | head -1
a:b:c:: - - [23/Apr/2026:05:25:30 +0200] "GET / HTTP/2.0" 301 854 "-" "curl/8.14.1"
# cat c0d0s0.org-access.log-blocked | tail -1
a.b.c.d - - [27/Apr/2026:05:14:05 +0200] "GET /setup.php HTTP/1.1" 404 14601 "-" "-"

The dataset covers about 4 days.

# wc -l c0d0s0.org-access.log-blocked 
3208 c0d0s0.org-access.log-blocked
# wc -l honeypot.log-blocked 
2098 honeypot.log-blocked

Based on this, 1100 requests landed on the ban web server that didn’t match the patterns in the detection component.

The vast majority of these accesses are, frankly, candidates for new filter patterns. There was only one IP address from which actual accesses to HTML files had taken place. There are clear indications, though, that this was just a camouflage pattern. But even if it were a genuine access, I’d consider a false-positive count of one IP acceptable. At least the first four days suggest a high specificity of the measures.

New detection logic

I took the opportunity to put the detection logic on a new footing as well.

Determining that something should become a 403, and actually making it a 403, are now separate functions.

A large set of SetEnvIfNoCase statements sets an environment variable. This contains a short identifier for the rule.

SetEnvIfNoCase Request_URI "/wp-[a-z-]+\.php"  honeypot=S2-WP001

If this environment variable is set, a central rewrite rule kicks in that actually produces the 403.

RewriteCond %{ENV:honeypot} !^$
RewriteRule .* - [F,L]

Improvement in monitoring

This rebuild was necessary primarily because I wanted to introduce one significant change. The environment variable honeypot now contains a short identifier that names the rule. This means I can carry that information through into the log.

The last matching rule wins, not the one that fits best. Unfortunately I couldn’t get the correct content to reliably end up there within my tangle of RewriteRules. Sometimes it would say -, sometimes 1. Rarely did it contain the correct rule identifier. Only after the rewrite to SetEnvIfNoCase was the variable honeypot reliably set. Or so I thought. Here too I had to deal with the issue of redirects and the resulting renaming of environment variables. Internally, displaying the ErrorDocument is a redirect, which means the environment variables are renamed accordingly. I log both the variable honeypot and REDIRECT_honeypot. A somewhat brute-force solution, but it makes sure I get the information I want into the files.

LogFormat "%t %a %{remote}p %A %{local}p \"%r\" %{honeypot}e %{REDIRECT_honeypot}e" connlog

The flyswatter then assembles the correct reason for filtering from both:

2026-04-27 19:28:54,469 INFO ipapi check a.b.c.d: triggered (datacenter) -> extending ban to 86400s (redirect)
2026-04-27 19:28:54,541 INFO Verdict on a.b.c.d: REASON:S2-WP008 NOSELF NOFASTTRACK NOBADASN:9999999/NARF-ASN NOABUSE:32 IPAPI:datacenter NOGROUP NOPREBAN DURATION:1d/redirect

I can see right away which rule led to the ban: Section 2 — Wordpress rule 8.

# grep "S2-WP008" .htaccess 
SetEnvIfNoCase Request_URI "(wp-conflg|setup-config|wp-setup|readme\.html|license\.txt)" honeypot=S2-WP008

Implementing the new logic on two servers

For this I have two .htaccess templates in my repository⁵. This version goes into the .htaccess of the primary server (reachable on 80, 443).

A slightly modified ruleset is meant for the .htaccess of the redirect server. Note in particular the second whitelist rul SetEnvIf Request_URI "^/notfound/". That’s where the always-served document I mentioned earlier lives. This second whitelist entry was needed to make sure the rewrite logic doesn’t fire on /notfound/index.html.

/notfound/index.html in turn is wired up via the ErrorDocument block.

ErrorDocument 404 /notfound/
ErrorDocument 403 /notfound/
ErrorDocument 410 /notfound/

Since there’s almost nothing else on the web server, practically every request becomes a 404 and the corresponding ErrorDocument is served. And because I’ve assigned the same ErrorDocument to 403 and 410, the redirect server only ever responds with this one document.

You could also just copy the .htaccess from the primary server into the redirect server and add the second whitelist rule. Since the redirect server is supposed to display exactly one page, please clear out all other rewrite rules and redirects in the process. Anything else leads to surprisingly hard-to-find problems (for example, the main web server’s site still being served).

Feeds

There is one exception: the feeds remain reachable even on the redirect server. So the server isn’t entirely contentless. My readership consumes this blog largely as an RSS feed. If for whatever reason a feed reader’s IP gets banned, they can still read the feeds. The collateral damage of a false-positive block is therefore considerably reduced.

I had previously considered building an exception for IPs that regularly poll the RSS feeds, until I noticed that it’s significantly simpler and more stable to just serve the feeds from the redirect server as well. That way I can also block an IP that scans and pulls feeds at the same time.

Additional information source

The previous version already queried an IP’s reputation (at AbuseIPDB) and issued a 1-day ban once a certain threshold was exceeded.

In this version another information source has been added. The system queries ipapi.is to learn more about those IP numbers that ended up in the honeypot (and only those). Part of the information this service provides is whether an IP is a VPN endpoint, a TOR exit node or a proxy. If that’s the case, the ban is shortened to 1 hour, since here the probability is highest that legitimate users may also be behind that IP. I built a caching daemon for the lookup. The free queries are limited and I want to stay within that budget.

That said: if the quota is exhausted, the worst that can happen is that all bans are issued at the default length of one day. The system is set up to fail conservatively at this point.

There are a number of additional conditions in my version of the script that shorten or extend a ban. I’ve removed those from the version in the Codeberg repository, because they work for me but not necessarily for anyone else. Feel free to go wild here.

Weakness

While I was at it, I found another weakness in this mechanism. A non-trivial number of honeypot requests still arrive at the primary server. This happens whenever the scanner works with high parallelism. For example, 16 requests come in at the same time, but the mechanism hasn’t fired yet because it’s hooked into the logfile and has to wait for something to appear there. The follow-up batches do then end up on the redirect server.

The response to these requests, however, is identical on the primary server and on the redirect server: 403 Forbidden for everything that’s known, 404 for everything that isn’t.

The access log of the redirect server is very tidy. Which is why a single command is also quite handy for distilling out new rules.

cat /var/log/apache2/c0d0s0.org-access.log-blocked | grep "404" | cut -d " " -f 7 | sort | uniq -c | sort -n

Anything that’s a 403 is already covered by rules. Anything that returns 200 is intentional. Anything that’s a 404 isn’t covered by the rules. It’s relatively easy to define rules from the rest.

Strictly speaking, this expansion isn’t necessary. From what I’ve observed, while individual requests may not be caught by the ruleset, scan runs almost always contain at least one pattern that is caught. And that’s enough to trigger the redirect. So not full coverage, but enough tripwires that one of them fires during typical scanner runs.

New version

I’ve uploaded a new version of the setup. You can find it with instructions here. The warning still applies that this is mostly vibecoded. So: caveat usor!⁶

Closing remark

I think this will be the last blog entry on this topic, because the experimenting — both with the idea itself and with vibecoding — has run its course. I’m still working through my conclusions on both fronts. But that will take a while before it makes it into its own post. That said, if a new idea on this topic hits me on the toilet or in the kitchen, I’ll definitely post about the implementation here.

In my opinion, “order of magnitude” is misused about as often as “quantum leap”. ↩
I will call this server “primary server”. ↩
I call it “redirect server” from now on. ↩
Yes, that was a typo. I just never bothered to fix it. ↩
I still need to restructure these so the ruleset isn’t duplicated. Later …. ↩
“Caveat utilitor” is probably more correct (my Latin is so far back that there were still native speakers around), but “caveat usor” sounds better … ↩

New release schedule for Solaris 11.4

c0t0d0s0 — Sun, 26 Apr 2026 12:30:00 +0000

I would like to draw your attention to a blog post by Joost Pronk van Hoogeveen: “Announcing a new simplified Support Repository Update (SRU) schedule for Oracle Solaris 11.4 and ZFS Storage Appliance software”.

Up to now, Solaris 11¹ had three SRU releases per quarter: one in the “Critical Patch Update (CPU)” timeframe containing mostly security fixes², one with all the new features², and one in between with regular bug fixes only. So a new SRU was released every 4 weeks.

In the future there will be two releases per quarter, one with the security fixes² and one with the new features². The cadence of new SRUs will be 6 weeks.

If you go from CPU to CPU³, nothing changes. There is one every quarter as before.

And the ZFS Storage Appliance Software (albeit under a diffent name) ↩
… and normal fixes ↩ ↩² ↩³ ↩⁴
Which is in many cases my practical recommendation for production systems. That said, i’m a big fan of “Patch early, Patch often”. ↩

Java 17 available on SPARC

c0t0d0s0 — Thu, 23 Apr 2026 20:00:00 +0000

A new Java release has been published by Oracle: the Java SE Development Kit 17.0.19 is available for download, giving you a more current version of the language. Please read the installation notes regarding certain features that are missing from this release.

Schroedingers Honeypot revisited

c0t0d0s0 — Thu, 23 Apr 2026 19:03:00 +0000

In Fly swatter, I wrote that the first implementation of ‘Schrödinger’s Honeypot’ carried an inherent potential for Denial of Service, since it blocked the IP address for a whole day on every single violation.

Denial-of-Service

Over the following evenings, I gave some thought to how this problem could be addressed: How do you solve the dilemma of keeping scanners off your back on one side, while keeping as few legitimate users away as possible on the other? And, taking it one step further: how do you prevent someone from making a sport out of abusing the detection component to block access for everyone else behind a NAT gateway? It’s relatively easy to figure out what the detection component is looking for — just try a few well-known scanner patterns. When the lights go out, you’ve found a pattern.

Why?

Before I go on, let me answer one question up front. Do I actually need all this? Probably not. I went down this path out of curiosity. My current suspicion is that, given the traffic volume my blog has, the mechanism I’m presenting here consumes more resources than the webserver itself. The complexity is probably grotesque. After all, it just serves static pages. Sendfile on incoming requests to port 80. Just about the simplest thing a server on the internet can do. But with dynamic sites, the picture might already look different. What actually bugged me was that my Apache log file was full of scan attempts. For me, all of this is primarily a hypothesis-tester. I had a few hypotheses about this scanner crowd before I implemented this, and I’ve picked up a few more from the process, which I’m in turn now testing.

I haven’t finished implementing everything yet. I’ll say up front: large parts of this were written for me by an LLM based on a mile-long specification. I couldn’t have gotten this prototype up and running this quickly on my own. I haven’t done the exact math, but in the time since the idea for Schrödinger’s Honeypot came to me, I wouldn’t have been able to type it all out by hand.¹ To put it mildly, the process of building this was at times like backing up a car with a trailer. Look away for a second and everything has veered off in the wrong direction.² This was exhausting.

The reason I let myself in for this is that without it I wouldn’t have had the time to build it, but above all because this mechanism has no interface of its own facing the outside. Everything that flows in from the outside goes through Apache. It consumes log files. Aside from that, it makes DNS and HTTP requests to a service. I think the risk is manageable, at least for now and at least for me.

Architecture

Essentially, the mechanism consists of a series of daemons that supply data and in some cases pre-process it, plus one daemon that takes action based on that data.

The daemons listen exclusively on 127.0.0.1. I didn’t want to bolt TLS onto this. It’s a hypothesis tester, after all. And all you can query through these daemons is information — “how often have you seen this IP in the honeypot over the last 24 hours?”, “is this IP in an autonomous system worth blocking?” If an attacker can reach these ports, I’ve got a very different problem on my hands.

There’s no detection logic in these daemons; that’s handled by Apache itself through the mod_rewrite rules. Those rules determine what ends up going through this mechanism. It writes a log file that contains only accesses matching those rules. I’ve put a selection of these rules in a file derived from what I was seeing in my logfiles. As long as I’m still actively tweaking them, I’m keeping them in .htaccess so I can change them easily. Eventually, for performance reasons, they should move into the Apache configuration itself to avoid constant re-parsing.

In the end it turned into quite a Rube Goldberg machine. If anyone wants to blame the LLM for the wonky architecture: nope… that was me.

Approach

Back to the underlying question: How high is the collateral damage when I hand out a ban? Maybe the IP is a VPN gateway, a NAT device, maybe even CGNAT. Behind that IP there might not be just one user, but many. Only one of them scanned, but a ban locks all of them out of the blog for a day.

My first reaction to this was to block offenders for only 120 seconds. That stopped the really coarse mass scans, but most scanners came right back, others after an hour or so.

Something in between was needed. Not the orbital anvil, but not the flyswatter either. Maybe an orbital socket wrench.³

Rule zero is: you have to misbehave to come into contact with any of this. A regular user making proper use of the system isn’t looking for WordPress artifacts or web shells. It’s impossible for a legitimate user to trigger this mechanism. Nobody types in a scanner pattern by accident.

The first rule is probably the simplest: if you scan again within the 24 hours following a block, you get another 24 hours to wait. Until you’ve learned. FlySwatter decides this on its own. It’s the shortest path from the honeypot into the firewall rules.

Okay, the second rule checks whether the IP is a known cloud IP. The hypothesis is: it’s considerably more likely that someone — legitimately or not — has obtained the credentials of a cloud OS instance and is scanning from there, than that this is a NAT exit node, when I’m seeing clear signs of a scan coming from it. The risk of collateral damage is much lower than the probability of a scanner. This was the first test I implemented, which is also why my implementation lives under /opt/cloudcheck. If you come from a cloud, it’s off to the quiet corner for a day. The first collateral-damage problem already shows up here: VPN providers put their endpoints in exactly such clouds.

The third check works on the basis of the IP’s ASN. First, the responsible ASN is determined for the IP. That happens via Team Cymru’s IP-to-ASN mapping service. The result is checked against the Spamhaus DROP list — which I refresh regularly in the background — and against a local list. That local list contains ASNs like those of Hetzner, OVH, or Contabo. Contabo servers have proven to be repeat offenders in my logfiles. If this check decides you’re trouble, that also means a one-day block. Only the specific IP that misbehaved gets blocked. The whole thing isn’t meant as collective punishment. With those providers, you get a server with an IP number, and if you use one of those servers to scan, it’s pretty unlikely that regular users would also be coming through it. With one exception: What does worry me here are egress points of VPN providers hosted with these kinds of providers.

The fourth rule queries an external resource for the first time: AbuseIPDB. If you’re flagged as “bad” there with a certain confidence, that results in a one-day ban. It’s deliberately placed at the end. The number of requests to AbuseIPDB is limited in the free tier, and later on depends on how much you pay. The mechanism is meant to keep as many requests away from AbuseIPDB as possible.

The rule ordering follows a simple logic: from cheap to expensive. Rule 0 is a by-product of the web server, Rule 1 queries an internal in-memory state table, Rule 2 queries daemons that access a local dataset, Rule 3 makes a DNS query against Cymru’s offering (if it’s not cached) and cross-references it with the regularly refreshed DROP list and a manually maintained dataset of my own, and with Rule 4 we finally reach a service that is resource-limited.

And even though the current implementation works the other way around, the whole mechanism is actually thought from the opposite direction. I started from “I see you in honeypot.log, you’re out!” In essence, I started to look for reasons not to do it. “You’re from a cloud, I’m blocking you for that” also means “you’re not from a cloud, I’m not blocking you for that. But maybe for other reasons.” Blocking everything with a reputation score of 80 or higher also means not blocking things on the basis of the reputation rule if the reputation is lower than 80%.

Massive Retaliation

There’s another feature too: Group Bans. I’ve observed that some scanners don’t come from a single IP; instead an entire subnet lights up, with each individual IP sending only one or two requests. When I spot that in my log files, an entry goes into a groupban list that blocks whole groups of IPs at once. The first request to the honeypot then prevents 254 others (or more) from scanning. I have a rule to close off several subnets at once, because they tend to come all at the same time. They live, for instance, at a bulletproof hoster that doesn’t care where traffic comes from. These are manually added bans where I’m reasonably sure that no normal users are moving behind them. I can also use this group feature to hand out long-term bans or permanent bans.

Why 403 and filter?

Why both, actually? Why either one at all? Since I don’t have the resources of a WordPress on my box, every scan would just hit a 404. That said, the 404 page on my site — as I explained elsewhere — is pretty heavyweight, because it also hosts a Lunr-based search function with a pre-generated index. If a user arrives at my blog via a dead link, they should be able to check whether comparable information exists elsewhere. But even that barely causes any damage. It just fills up the log file. So why didn’t I just leave it at a 404?

Schrödinger’s Honeypot cheats in my implementation. A 404 would give the attacker the information that something is not there. A 403 communicates that something is forbidden. The difference is significant, because a 403 doesn’t reveal whether the thing exists. So the superposition, as far as the attacker is concerned, doesn’t just always collapse into a ban — they never had any chance of collapsing it into the other position either, because over there a different superposition is posing a different question: “present and forbidden = WordPress” or “not present and forbidden = no WordPress”.

Why the IP filtering on top? My ruleset only blocks patterns I know about, and in the same scan run it would hand over the requested information (there or not there) for unknown patterns. By blocking at the firewall level, unknown requests don’t even get far enough to reveal any information. Unless, by chance, they use a pattern unknown to me first, they don’t even get to learn whether that pattern is familiar to me (they could, of course, infer that it’s unknown to me from the 404 I would send, if it’s not specifically blocked by the rules).

Data Protection

There are several points in this solution where data protection questions come into play. In my opinion, there’s no better way to develop a migraine than to ponder the implications of data protection rules as a layperson. It might be different for lawyers. They probably look forward to it with delight, given its potential for “billable hours.”

My system queries AbuseIPDB about IP addresses, asking what reputation the IP has. Now, an IP address is explicitly PII under GDPR. I take the position that I have a legitimate interest in finding out the reputation of an IP that is engaged in non-intended-use behaviour (any only if).

And one question that preoccupies me: can an automated system whose purpose is to collect information about me itself have rights under GDPR? Especially considering that GDPR talks about natural persons? I understand that both regulations protect the person behind the IP, but is the person behind the IP of a VM running an automated process a natural person — namely, the one who owns the VM — or is it the company (and thus not a natural person) whose infrastructure hosts the VM? I don’t know. But then again, I’m not a lawyer.

But I have a more direct problem. One of my ideas that hasn’t been implemented yet needs the IP address of every request that hits my blog in order to collect successful requests of the last 24 hours. I’ve solved this problem as follows. I have a log file on disk where the IP address is pseudonymized by replacing the last octet with a 0. There is a second log area that carries a non-anonymized set of data. It’s considerably reduced — time, IP address, and result code (that’s all) — and this is piped to a daemon that keeps its information exclusively in main memory. This introduced two intentional problems for me: a restart means data loss, which in turn means that the function’s usefulness is limited for the next few hours.

More importantly, I can’t query information with the anonymized IP address. In the current implementation I need the exact IP. I could of course try all 254 IP addresses and thus de-pseudonymize a pseudonymized address, but that only works if (first) only one IP in that /24 subnet is the single user in question. If, within the entire 1.2.3.0/24 subnet, only one IP address (say 1.2.3.4) has 200-result-code data recorded for it, then it’s clear from whom the retrieval in the access log came, even though it was pseudonymized to 1.2.3.0. On one hand, GDPR does permit storing PII on the basis of legitimate interest; on the other hand, you really have to go out of your way to pull this off, and it only works under specific conditions. Full pseudonymization kicks in after 24 hours, in a way I don’t have to do anything for. I haven’t yet found a way to achieve immediate anonymization.

And if you want a really migraine-worthy question: is creating a firewall rule for a single IP itself a storage of PII?

(Side note of 23.04.2026: I solved the problem by turning the problem on the head, but more on that in a future blog entry)

Building Your Own

You can build this mechanism yourself. It would be pointless, though, to write the instructions into this blog post. You’ll find all the components along with instructions in my Codeberg repository.

The Flyswatter for Apache is available at https://codeberg.org/c0t0d0s0/flyswatter/src/tag/v0.01/flyswatter. The daemons Flyswatter needs for its work are available at https://codeberg.org/c0t0d0s0/flyswatter/src/tag/v0.01/cloudcheck-suite.

Should you build it on your own? Probably not.

One comment: The ASN check still uses the term BADASN. That’s an artifact from the time when only the DROP list was being compared against and so such ASNs were bad. At some point I added a second list, which I can populate manually, to flag names or ASNs as worth a day’s ban. I didn’t remove the BADASN term. A cosmetic issue. BADASN can simply mean “is from a cloud provider”. Pushing a reason through here that could also stand for CLOUDASN or DROPASN is already on my list to integrate.

Ideas Not Yet Implemented

There are some statistics I’m already maintaining, but not yet using, because I want to gather numbers first.

Number of prebans in the last 24 hours. The idea is: if you’ve run into a preban, say, 19 times in the last 24 hours, you get a full day’s time-out on the twentieth.
There’s another daemon that counts successful access attempts over a longer period. The assumption is: if I see many 200s from one IP, then the probability is high that there is more than one user behind that IP. In that case, the time-out is only 120 seconds — maybe I’ll extend that to 5 minutes when I implement it.
It’s also being counted how often you stepped into the honeypot in the last 24 hours and how often you ran into the firewall block.

For some of these ideas there are already prototypes that haven’t yet made their way into the codebase, because they still need some fine-tuning before they’re even remotely usable. And I need much more data to know if those ideas are somewhat sensible.

Weakness

This is important if you really want to use this: honeypotd has a weakness. It also looks for requests containing a particular string in it. If that string is included, the IP of that request is considered your IP. The consequence in Flyswatter is that no firewall rule gets set. You can essentially use this to switch off the main function of this entire contraption. I therefore suggest treating the string like a password and keeping it equally complex. Important: it appears, of course, in log files, in the config file, and gets transmitted across the network. By the requests, but as a referrer as well. Maybe it’s not a bad idea, once everything works, to unset the string.

I’m aware that the magic-string hack I used to implement a half-dry-run of the logic is a security hole. If you know the string, you can bypass the whole mechanism. I built it this way because I don’t have a static IP and it made testing easier for me. I use a browser plugin that appends it to every request (and only those) for https://www.c0t0d0s0.org/. Actually, that plugin has a different job: I use it to filter my own requests out of the log files.

So why did I still build it this way? Because I locked myself out a few times. Because it was something that was already there.

There are some protections in place: First of all, the default is for the function to be switched off. There is no default. If the SELF_MARKER parameter isn’t set, then the “remember my own IP” function is disabled and there is no IP address that would only trigger a dry run. Second: What’s the possible damage? The requests in question aren’t just logged, they’re also answered with a 403. So more will be written to the log, and if other, unknown probes follow the initial known ones, those won’t be blocked. The damage is manageable, at least initially. There is no compromise of the security goals confidentiality and integrity, and with high probability — in my environment — also not of the goal “availability”. Second: in my environment there’s only one risk that the secret might leak to the outside, namely when someone follows an external link. In that case, the magic would appear in the Referer. Otherwise, the blog is HTTPS-encrypted, so the magic is invisible in-transit thanks to TLS encryption. And anyone with access to the log file can probably just switch off the whole kit and caboodle in my configuration anyway.

Otherwise, if you really want to use what I’ve built here (which please only do if you’ve examined the code yourself and found it fit for purpose), I would, after successful tests where you didn’t accidentally always block yourself, simply remove the magic completely. Then there’s no way left to switch off the test. That’s how I handle it now that I no longer need to test whether the reactive mechanisms work. If I need it again, I can switch it back on and restart the corresponding daemon.

Assessment

The collateral damage problem isn’t finally solved in the current version, but it seems solvable to a large extend. Empirically, I can say that an IP that gets blocked has almost never shown significant 200s in the last 23 hours. That’s only a weak signal given my blog, though. I don’t have millions of hits. On the other hand, a peculiarity of my readership helps me out. I have well over 1000 subscribers, which was a very surprising number for me given how little has been happening on the blog for so long. These subscribers query the blog regularly and automatically, producing a steady stream of successful requests. If an attacker and a feed reader are behind the same IP, the attack is accompanied by a certain number of 200 requests. That can be used to mitigate the VPN egress problem.
Here too, a signal you get for free — the feed-reader heartbeat — is being put to use. The feed readers, as the most loyal audience, produce their own heartbeat. Yes, sure, that could be exploited in turn.
All of this could still be built with a sufficiently well-implemented Fail2Ban script. But I’d have had to build the helper daemons there too, so that the scripts in Fail2Ban would have something to lean on.

Observations

I’ve been collecting data for a few days now with increasing levels of detail. Pseudonymized. But sufficient to draw some conclusions. However, this is all data for one day — the day before yesterday (21 April 2026). It’s a private blog with relatively little readership. The conclusions are therefore to be taken with a big grain of salt.

A few things I noticed:

Yesterday, 572 honey pot requests came through in 24 hours.
They came from 427 IP addresses.
- That fits with the observation that scan runs are terminated fairly quickly.
For me, scanning is primarily an IPv4 phenomenon.
- 11 IP addresses on the honeypot are IPv6.
- 416 IP addresses are IPv4.
- I still have to invest some time into improving the handling of IPv6.
This resulted in a total stock of 149 entries in the firewall, which are now being blocked.
In total, the firewall rules have rejected 6913 connection attempts in that time. Scanners rarely let themselves be deterred. Even when their connection attempts are slapped away with an RST, they keep trying again.
Almost never is a scan attempt preceded by significant amounts of traffic with result code 200. That allows two conclusions:
- The collateral damage may have been small so far.
- For the scanners encountered so far, they don’t seem to use shared IPs. But to be honest, the data is insufficient for this conclusion to be more than anecdotal.
- The idea of checking, on a ban, whether legitimate traffic preceded the scan attempt could prove useful to avoid collateral damage.
- I still need to come up with something to prevent a scanner from simply switching off the protection by, say, requesting / before the scan attempt. Perhaps a useful hypothesis here is that a mass scanner isn’t going to simulate the picture of a legitimate user for several hours in advance just to come back later and scan. At least not for a small target like my blog.
I see relatively many scanners from an AS in Vietnam. Those, however, are all one-shot scanners. One request is made and the IP doesn’t show up again. At least not during the short measurement interval this article is based on.
For the main goal of this project I need much more data. I think I will need a few weeks worth of data, to test the hypotheses I mentioned before.
First data from a new mechanism, not described in this post (it will be in a future blog entry) suggests a near-zero false positive rate of this mechanism. However this is currently just anecdotal based on one day of data and could prove totally wrong.
In 1200 runs of the mechanism with the fully implemented AbuseIP daemon in place, the mechanism was skipped in roughly a third of the cases. However this is not the full picture. The AbuseIP-Daemon caches and I didn’t put that information into my debug lines. Thus it’s entirely possible that further requests didn’t hit the service as it was answered by the cache. This could happen if a scanner isn’t blocked for a day, but just for two minutes, and appears multiple times of the day.

Outlook and Postscript

I’ve significantly expanded the concept and the script over the past few days. I may have found a solution for the PII problem that would allow breaking through pseudonymization for 24h. But all of that still needs more testing over the coming days. I will write about this in the next few days.

An idea that is so obvious that I’m sure it isn’t new. ↩
Without Trailer Assist. I simply despair at backing up with a trailer without this feature. It’s easier for me to unhitch and just push the silly thing. Since I only have a small trailer for the occasional trip to the garden waste disposal (vulgo: “Klaufix”), that works… ↩
And yes, I’m aware that this would probably burn up on the way to Earth. Let’s just assume a socket wrench made of unobtanium. ↩

Fly swatter

c0t0d0s0 — Mon, 20 Apr 2026 06:45:00 +0000

Sometimes you have a good idea, implement it, and only afterwards realize that this idea has a denial-of-service attack built into it. And you ask yourself: Hmm, really a good idea?

I then moved away from the old solution and thought about what I could continue to use and what I needed to redesign.

The detection side remained intact. Requests matching known questionable patterns still trigger the reaction component. However, the reaction component is now built quite differently.

For the first iteration, I kept the reaction component very simple¹.

The source IP is blocked in the firewall for 120 seconds. It’s a rule with reject with tcp reset and not drop.
I remove the connection from the connection tracking via conntrack.
I terminate the connection via ss.

The first step prevents new connections for 120 seconds, while the other two steps actively sever the connection. A false positive no longer leads to blocking access for an entire day. The scanner should know that I don’t want it here.

Why do I use reject with tcp reset? People often use drop to conceal that something is there. But the scanners already know the web server exists — I don’t need to hide it from them anymore. And I’ll probably lose the race for resources anyway. So I convey the message that there’s a mechanism at work here that’s keeping an eye on them. “Listen, I just told you that you’re not getting through here, because I know what you’re doing.”

I consider tarpitting to be out of date. It worked when there was a balance of resources. Resources in the cloud paid for with stolen credit card information or started with stolen cloud credentials mean that this balance no longer exists. The other side has more and no financial constraints, because they’re probably not paying for it. In some cases they even set themselves up “parasitically” on free services. I’ve found some scans originating from GitHub Actions.

Although I’m not sure whether there’s any intelligence at work here. When I look at the patterns, an attempt is made at 12:00 to find out whether I have a WordPress blog, and the same thing again at 15:00. The probability that this has changed in the last 3 hours and I’ve suddenly switched to WordPress is low. I can’t tell from my log files that this is being taken into account.²

Of course, I could nail down all access paths so that hardly any scanner gets through. But this is a public blog. As such, by definition, it has to be accessible. The price of accessibility is the massive influx of requests that my web server configuration sends into 403 territory. My goal is not to prevent this access entirely, just to thin it out without taking on too high a risk of false positives or collateral damage.

And that’s exactly what the mechanism does. I’ve named the tool accordingly: apache-flyswatter. I can’t keep the flies out entirely if I want someone to come into the room, but if I see one, I can swat it.

How does this work in practice?

Now I’ve talked a lot about what I did. I’d like to briefly explain how I set it up.

The detection side works as it did yesterday. I still use rewrite rules that “tag” certain requests with an environment variable.

RewriteCond %{REQUEST_URI} wlwmanifest\.xml$ [NC]
RewriteRule .* - [E=honeypot:1,F,L]

I use conditional logging here so that the tagged requests end up in the corresponding additional log files.

SetEnvIf REDIRECT_honeypot 1 honeypot
SetEnvIf Remote_Addr "^203\.0\.113\.1$" !honeypot
SetEnvIf Remote_Addr "^127\." !honeypot
SetEnvIf Remote_Addr "^::1$" !honeypot
LogFormat "%h %{%Y-%m-%dT%H:%M:%S%z}t \"%r\" %>s \"%{User-Agent}i\"" honeypot
CustomLog /var/log/apache2/honeypot.log honeypot env=honeypot
LogFormat "%t %a %{remote}p %A %{local}p" connlog
CustomLog /var/log/apache2/honeypot_connections.log connlog env=honeypot

I’m again working with the hypothesis that every access that lands in this log file is malicious, or at least questionable. There is no valid reason why these requests would be legitimate under intended use.

I have two log files. The first log file, /var/log/apache2/honeypot.log, was used by the old mechanism. It’s no longer used by any tool. I’ve kept it so I can manually check what kind of requests land there.

I would strongly recommend keeping it. You can also use the timestamp in the normal log to find out what kind of request it was. But if you have tail -f running on both /var/log/apache2/honeypot.log and /var/log/apache2/honeypot_connections.log in parallel, it’s significantly easier to quickly establish the correlation.

The log file I’m working with now contains different information:

Timestamp
Source IP
Source port
Destination IP
Destination port

The new mechanism doesn’t need more information than that. This is the data the script needs to carry out the three steps.

You need three components:

Download the three components to your system and install them:

install -m 0755 apache-flyswatter /usr/local/bin/apache-flyswatter
install -m 0644 apache-flyswatter.conf /etc/apache-flyswatter.conf
install -m 0644 apache-flyswatter.service /etc/systemd/system/apache-flyswatter.service
apt install nftables conntrack iproute2
systemctl daemon-reload
systemctl enable --now apache-flyswatter

If you already have the packages installed, that step isn’t necessary, of course. With journalctl -u apache-flyswatter -f you should now see the script doing its work.

root@v2202604350965450827:~# journalctl -u apache-flyswatter -f
Apr 18 20:00:26 v2202604350965450827 apache-flyswatter[2853]: 2026-04-18 20:00:26,868 INFO SWAT 203.0.113.254:23562 -> 159.195.145.249:443
Apr 18 20:05:31 v2202604350965450827 apache-flyswatter[2853]: 2026-04-18 20:05:31,861 INFO SWAT 203.0.113.254:42408 -> 159.195.145.249:443
Apr 18 20:16:52 v2202604350965450827 apache-flyswatter[2853]: 2026-04-18 20:16:52,293 INFO SWAT 203.0.113.254:50682 -> 159.195.145.249:443

If you run nft list ruleset, you’ll get the following output:

table inet flyswatter {
 set blocked4 {
  type ipv4_addr
  flags timeout
  elements = { aaa.bbb.ccc.ddd timeout 2m expires 1m30s900ms }
 }

 set blocked6 {
  type ipv6_addr
  flags timeout
 }

 chain input {
  type filter hook input priority filter - 10; policy accept;
  ip saddr @blocked4 tcp dport { 80, 443 } reject with tcp reset
  ip6 saddr @blocked6 tcp dport { 80, 443 } reject with tcp reset
 }
}

I’d like to draw your attention to the element in the blocked4 set. You can see that it’s an element with a timeout. The entries clean themselves up — nftables removes them automatically after the timeout expires. I don’t have to go around mopping up after the rules.

Observations

After running the script for a few hours now, the following became apparent:

They come back. The scanners won’t be deterred.
120 seconds is too short to largely relieve the log files of scanner noise. You still frequently find scanning activity, but it gets shut down quickly³. However, in the config file you can set a significantly longer ban time. Possibly even the 24h used in yesterday’s solution.
What’s missing are scan runs that fire off 200-300 requests within a short time. The mechanism seems to set limits on such activity.
With some bots I’ve noticed that they appear to start their scan run from scratch again. After a certain time, the same request is executed again from the same IP address. As if they were working through a list, and after I’ve cut the connection, they start over from the beginning.

Postscript

I probably could have implemented all of this in fail2ban with nearly equivalent functionality. But I’m currently trying to find ways to assess IP addresses in terms of how high their risk of collateral damage is, because multiple users sit behind them. Such a mechanism would allow me to more safely increase the ban time significantly. I already have a few ideas. It’s easier for me to implement such logic in my own script. I’ll be reporting on this in the coming days.

This blog entry is a work-in-progress report. By the time you read it, the reaction component is probably a little more sophisticated than what’s described here. ↩
Maybe you really do need LLMs to avoid doing mass vulnerability scanning in the dumbest possible way. That would still be a script kiddie, but a script kiddie on steroids and with a sugar rush. ↩
Provided they use patterns that are caught by the detection component. ↩

Schroedingers Honeypot

c0t0d0s0 — Fri, 17 Apr 2026 19:03:00 +0000

I’ve moved my website back to a self-managed server. c0t0d0s0.org now runs at netcup¹. I was just getting increasingly annoyed at not being able to do anything administratively about the various things I was observing in my logfile. For example, to cut down the background noise in the Apache logfile by blocking accesses on an IP basis.

One of the problems was the anonymisation of the logfile. I knew that I was getting thousands of requests hitting WordPress components at times. But I haven’t used WordPress at all for almost 20 years. I switched to s9y afterwards and used it for years. And with the migration to Jekyll in 2021, there wasn’t even a single php file necessary for the operation of the blog. So I knew for sure all those WordPress .php accesses weren’t kosher.

However, thanks to the anonymisation in the Strato logfiles there was little I could do about it, because I only had a very imprecise idea of where they were coming from. That made it impossible to implement firewall blocks² or aggressive Apache access controls. The collateral damage would have been too large. The bot operators have also become too clever by now to offer any other distinguishing feature by which you could reliably identify and block them.

A dilemma arose. I don’t want to collect any PII, but IP addresses count as PII. So my new configuration on my own webserver would also have to anonymise the IP addresses.

At the same time, though, I wanted to be able to surgically block a single IP address at the network level. But for that I need the complete, non-anonymised IP address.

I have a solution that addresses both requirements. I redirect all that nonsense traffic aimed at WordPress components³ to a 403 anyway. Let me show you the part of my .htaccess that handles all the WordPress scans. I could of course also let the requests run into a 404, but in my case that’s a page styled like the rest of the blog, which would generate a lot of follow-up traffic. Suboptimal.

RewriteCond %{REQUEST_URI} /wp-[a-z-]+\.php [NC]
RewriteRule .* - [E=honeypot:1,F,L]

RewriteCond %{REQUEST_URI} (wp-admin|wp-json|wp-signup|wp-cron) [NC]
RewriteRule .* - [E=honeypot:1,F,L]

RewriteCond %{REQUEST_URI} xmlrpc\.php$ [NC]
RewriteRule .* - [E=honeypot:1,F,L]

RewriteCond %{REQUEST_URI} wlwmanifest\.xml$ [NC]
RewriteRule .* - [E=honeypot:1,F,L]

RewriteCond %{REQUEST_URI} (wp-includes|wp-content) [NC]
RewriteRule .* - [E=honeypot:1,F,L]

RewriteCond %{REQUEST_URI} wp-config [NC]
RewriteRule .* - [E=honeypot:1,F,L]

I could surely fold the rules into a single one, but the attempt at doing so looked extremely messy and unmaintainable. If the multiple regexps ever cause problems down the road, I can still rebuild it then. Besides, I’m potentially saving a lot of requests from hitting the webserver daemon in the first place. I think I still come out ahead on net.

The decisive bit here is the [E=honeypot:1,F,L]. L to end the discussion with mod_rewrite, and F to throw a 403. The E sets the environment variable honeypot.

It’s Schroedinger’s honeypot, so to speak. Without checking, a scanner doesn’t know whether the file is there or not. The honeypot sits in a superposition. I might be running a WordPress blog — or I might not. The scanner has to look, and gets a 403.

The interesting information for me is the fact that it asked at all: by doing so, the scanner has told me it’s a scanner, because there’s no other reason to be looking for WordPress artefacts on my Jekyll site.

You’d actually expect the 403 to be taken as a signal: “I know what you’re doing. I’ve done something about it. Stop it. Now!” But the scanners keep trying over and over.

So with those rules I’ve marked the requests that may not necessarily be malicious, but are certainly questionable. That’s something I can work with.

In a second step, I use conditional logging.

    LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined_anon
    CustomLog "|/usr/local/bin/anonymise-log /var/log/apache2/c0d0s0.org-access.log" combined_anon
    ErrorLog "|/usr/local/bin/anonymise-errorlog /var/log/apache2/c0d0s0.org-error.log"

These first three lines set up the normal logging. I pipe the log through an AWK script that anonymises the IP addresses. For IPv4 addresses, that simply means replacing the last octet with a zero. For IPv6, I anonymise down to /48.

In the configuration fragments below you’ll find an IP address 203.0.113.1. That’s a placeholder for your own IP, the one you reach your own webserver from. In most cases this will be the external IP of your router.

Please add the following lines to the VHost configuration of your webserver. You have to repeat this in the TLS section of the VHost config.

    SetEnvIf REDIRECT_honeypot 1 honeypot
    SetEnvIf Remote_Addr "^127\." !honeypot
    SetEnvIf Remote_Addr "^::1$" !honeypot
    SetEnvIf Remote_Addr "^203\.0\.113\.1$" !honeypot
    

and

    LogFormat "%h %{%Y-%m-%dT%H:%M:%S%z}t \"%r\" %>s \"%{User-Agent}i\"" honeypot
    CustomLog /var/log/apache2/honeypot.log honeypot env=honeypot

The REDIRECT_ in the construct SetEnvIf REDIRECT_honeypot 1 honeypot cost me a moment and some debugging.

The F flag in the rewrite rule doesn’t produce a direct 403 internally; instead it produces an internal redirect to the error document. During that internal redirect, Apache renames all environment variables. That’s how honeypot turns into REDIRECT_honeypot. If you don’t notice this, you’ll spend hours trying SetEnvIf honeypot 1 honeypot and wonder with an increasingly furrowed brow and dwindling patience why nothing ever ends up in the log. This behaviour can be found in the documentation.

Once the environment variable is set, a second log kicks in. And that one doesn’t anonymise the IP numbers.

At this point I’d like to mention that you can protect yourself even further against locking yourself out.

To do so, add the following line to the .htaccess right at the very top, after enabling the rewrite:

RewriteCond %{REMOTE_ADDR} ^203\.0\.113\.1$ 
RewriteRule .* - [E=honeypot_whitelist:1]

Whatever else happens, this unsets the honeypot environment variable further down the line.

In the VHost configuration you can then insert the following instead of the SetEnvIf block from before.

SetEnvIf REDIRECT_honeypot 1 honeypot
SetEnvIf REDIRECT_honeypot_whitelist 1 !honeypot 
SetEnvIf honeypot_whitelist 1 !honeypot
SetEnvIf Remote_Addr "^127\." !honeypot
SetEnvIf Remote_Addr "^::1$" !honeypot
SetEnvIf Remote_Addr "^203\.0\.113\.1$" !honeypot

The charm of this optional configuration is that your own IP is also stored in the .htaccess. That file is often part of the deployment process. It’s easier to transfer along with the website via rsync than to modify the Apache configuration on the webserver via SSH.

Now I can take this log and drop it in front of Fail2Ban’s feet. Configuring it can be dead simple. One time and you are out. If an IP address shows up in this logfile, I don’t have to give it any benefit of the doubt about maybe being a legitimate user. I know it isn’t.

The Fail2Ban configuration therefore looks like this:

# cat /etc/fail2ban/jail.d/apache-honeypot.conf

[apache-honeypot]
enabled   = true
filter    = apache-honeypot
logpath   = /var/log/apache2/honeypot.log
maxretry  = 1
findtime  = 60
bantime   = 86399
banaction = nftables-multiport
port      = http,https
protocol  = tcp

ignoreip  = 127.0.0.1/8 ::1 203.0.113.1

The ignoreip line isn’t strictly necessary, but it’s a second⁴ safety net in case I mess something up in the Apache config file. fail2ban has sent me to the console too many times because I managed to lock myself out. I’m not taking any more chances with that.

And then a filter rule that simply looks for the IP address at the beginning.

# cat /etc/fail2ban/filter.d/apache-honeypot.conf
[Definition]
failregex = ^<HOST>\s
ignoreregex =

With that, every IP address caught red-handed in a scan gets blocked. It can still happen that a scanner gets quite a few requests through. It can take a second until fail2ban has set the filter rule. Depending on how fast the scanner fires its requests, a handful of requests get through before the door slams shut and the “You shall not pass” is pronounced.

There are ways to speed up the processing, but since the request itself is already stopped by the 403, I didn’t want to raise the complexity of the solution.

Since then, my logfile has been a lot quieter. In return, though, the firewall configuration grows over time. At the time I’m writing this article, I have 39 entries after running this configuration for 3 hours. Let’s check with nft list ruleset:

	set addr-set-apache-honeypot {
		type ipv4_addr
		elements = { aaa.bbb.ccc.ddd, eee.fff.ggg.hhh,
			     [...]
			     qqq.xxx.yyy.zzz }
	}

For anyone wondering about the missing ports in that structure: They are defined in the f2b-chain nftables chain.

	chain f2b-chain {
		type filter hook input priority filter - 1; policy accept;
		[...]
		tcp dport { 80, 443 } ip saddr @addr-set-apache-honeypot reject with icmp port-unreachable
	}

The reason I use multiport here and not allport is that on top of belt and braces I also brought in double-sided tape on the waistband.⁵ Even if this mechanism accidentally blocks my own IP, only the webserver is affected. ssh stays available. For locking myself out of SSH, I have another Fail2Ban configuration.

There are people whose technical judgement I trust, and they trust netcup. So I decided to trust them as well. ↩
Which I couldn’t have configured anyway, because I didn’t have admin access to the system. ↩
requesting a lot of files like .env or .git ↩
Or third … ↩
In case you were wondering how Kylie Minogue’s costume in “Can’t Get You Out of My Head” stayed in place … it was explained to me: “double-sided tape”. ↩