new blog post + a couple minor changes

3 files changed, 247 insertions, 1 deletions

diff --git a/blog/blog.atom b/blog/blog.atom
index ddb59dd..a6cc5b8 100644
--- a/blog/blog.atom
+++ b/blog/blog.atom
@@ -3,7 +3,7 @@
     <title>isopod.cool blog</title>
     <id>https://isopod.cool/blog/</id>
     <link rel="alternate" href="https://isopod.cool/blog/"></link>
-    <updated>2025-04-29T00:00:00+00:00</updated>
+    <updated>2025-05-12T00:00:00+00:00</updated>
     <author>
         <name>Hannah Isopod</name>
         <uri>https://isopod.cool/</uri>
@@ -23,6 +23,17 @@
         <category term="php"/>
     </entry-->
     <entry>
+        <title>Fighting LLM Scrapers With Nepenthes And Fail2Ban</title>
+        <id>grow_spikes</id>
+        <link rel="alternate" href="https://isopod.cool/blog/posts/grow_spikes/" type="html" title="Fighting LLM Scrapers With Nepenthes And Fail2Ban"></link>
+        <published>2025-05-12T00:00:00+00:00</published>
+        <updated>2025-05-12T00:00:00+00:00</updated>
+        <summary>A comprehensive explanation of (and guide to replicate) my new and improved anti-scraper defenses.</summary>
+        <category term="guide"/>
+        <category term="self-hosting"/>
+        <category term="ai"/>
+    </entry>
+    <entry>
         <title>I've Just Moved Hosting Providers</title>
         <id>renting_an_unremarkable_server</id>
         <link rel="alternate" href="https://isopod.cool/blog/posts/renting_an_unremarkable_server/" type="html" title="I've Just Moved Hosting Providers"></link>
diff --git a/blog/posts/grow_spikes/index.md b/blog/posts/grow_spikes/index.md
new file mode 100644
index 0000000..4bf7c61
--- /dev/null
+++ b/blog/posts/grow_spikes/index.md
@@ -0,0 +1,234 @@
+As long-time readers will certainly know, I don't much like the idea of LLM vendors scraping my website to train their ocean-boiling slop machines on my writing as they've been doing lately. Up 'til recently, my way of combatting this was to compile a big list of every known user agent string of a scraper whose motives I disagree with and have Nginx return a `418 I Am A Teapot` response code to any request from any of them. Funny, but problematic for a couple reasons.
+
+See, the thing about user agent strings is that they can be whatever you want, which means there's nothing stopping GPTBot or whatever from just lying and saying it's a legitimate web browser, and I'd have no way to tell the difference. This also means that I need to play a never-ending game of whack-a-mole, having to manually learn about and block new crawlers forever. This obviously won't do.
+
+When I switched reverse proxies to Caddy last month, I figured it was time to upgrade my setup in this department as well. As it stands, my new setup consists of a `robots.txt` file defined in Caddy and enforced with Nepenthes and Fail2Ban. If you're curious what all that is, how to set it up, and/or why, read on.
+
+This guide assumes you already have Caddy working and are familiar with the basics of its use. It also assumes you're running Ubuntu Server 24.04 LTS, since that's what I'm using. I hear some of the filepaths are different depending on whether you're on a Debian-based distro or not, so adjust accordingly if you're on, like, NixOS or whatever. Although, NixOS users probably don't need my help.
+
+## robots.txt
+
+`robots.txt`, if you're not familiar, is a text file you serve from the root of your website that tells crawlers and other bots whether they have your permission to access your site. I didn't bother with this last time, because like user agent strings it's basically reliant on the honor system to work, and most of the bots I'm interested in blocking are scummy enough that they'd probably ignore it anyway. This time around, though, I'm planning to give it some teeth (which we'll get to later) so it's actually important here. Here's the relevant excerpt from mine:
+
+<p class="codetitle">/robots.txt</p>
+
+```
+User-Agent: *
+Disallow: /
+
+User-agent: search.marginalia.nu
+Allow: /
+Disallow: /slop/
+```
+
+The first two lines tell all bots (`User-Agent: *`) not to look at anything (`Disallow: /`). You can stop there, but if you want, you can also allow specific user agents, for example your favourite indie search engines, by following that with a directive specific to them. `Allow: /` tells the bot it's allowed to crawl your whole site, and `Disallow: /slop/` lets it know to avoid a subdirectory you'd rather it not crawl. There are a few reasons you might want to do this, but I'm using it to spare the crawlers I like from the trap I'll be setting in that subdirectory.
+
+### Defining this universally in Caddy
+
+Serving a plain text file, like most things in Caddy, is pretty simple to do in your caddyfile:
+
+<p class="codetitle">/etc/caddy/Caddyfile</p>
+
+```
+(bans) {
+    respond /robots.txt 200 {
+        body "User-Agent: *
+Disallow: /
+
+User-agent: search.marginalia.nu
+Allow: /
+Disallow: /slop/"
+    }
+}
+```
+
+Here we're telling Caddy to `respond` to requests for `/robots.txt` with a `200` response (essentially the code for "Everything's good, here's the file you asked for") with a `body` consisting of the full text of our robots.txt file, line breaks and all, in between the two quotation marks. It also serves any whitespace you include, so if you indent your response body to match the surrounding configuration, it'll still be indented in the file when it's served. I don't know if that affects anything, but I'm erring on the side of caution.
+
+I'm also putting this in a snippet called `bans` that I can then import into my different sites to avoid writing the same code a bunch of times, like so:
+
+<p class="codetitle">/etc/caddy/Caddyfile</p>
+
+```
+(bans) {
+    # your configuration here
+}
+
+yoursite.example.com {
+    import bans
+    # more configuration here
+}
+```
+
+If you do all that and reload Caddy, you should be able to go to yoursite.example.com/robots.txt and see the `robots.txt` file you entered. Now on to the fun part.
+
+## Nepenthes
+
+[Nepenthes](https://zadzmo.org/code/nepenthes/) is a tarpit for misbehaving scrapers, named after a species of carnivorous pitcher plant. There's a few of these around now, but they all seem to work in pretty much the same way: generating, as the developer of [Iocane](https://iocaine.madhouse-project.org/) so eloquently puts it, "an infinite maze of garbage" for scrapers to get trapped in.
+
+In other words, it craps out an infinte series of procedurally generated pages that link to other procedurally generated pages with no way back to your real site, with optional Markov-chain generated babbling to try to poison the training data of any unfortunate LLMs that happen across it. This is how we give our `robots.txt` teeth &mdash; any misbehaving scraper that ignores it will find its way to your infinite maze of garbage and get trapped there forever, or until it gets bored and quits crawling your site.
+
+Obligatory warning here: Nepenthes and programs like it are intentionally malicious software designed to cause harm, and will also probably get your site delisted from any search engines whose crawlers get caught in it. Proceed only if you're comfortable with all that.
+
+### Setting up Nepenthes
+
+Setting up Nepenthes is pretty simple if you're comfortable self-hosting stuff in general, and well-documented on its website. I'm lazy so I used the provided Docker image. If you're not familiar with Docker, here's the series of commands to spin it up identically to how I did (adding `sudo` as needed):
+
+```
+apt install docker docker-compose wget
+mkdir /opt/nepenthes
+cd /opt/nepenthes
+wget https://svn.zadzmo.org/repo/nepenthes/head/docker/Dockerfile
+wget https://svn.zadzmo.org/repo/nepenthes/head/docker/bootstrap.lua
+wget https://svn.zadzmo.org/repo/nepenthes/head/docker/compose.yaml
+wget https://svn.zadzmo.org/repo/nepenthes/head/docker/config.yaml
+docker compose up -d
+```
+
+You shouldn't have to change any of those files if you don't want to, but if you do, some basic settings are available in `config.yaml`.
+
+You can now interact with Nepenthes by using `curl` to ping its API. You can use this to view its statistics in JSON format:
+
+```
+curl http://localhost:8893/stats
+```
+
+Or you can set up the Markov babbler by training it on the large text corpus of your choice:
+
+```
+curl -XPOST -d @./corpus.txt -H'Content-type: text/plain' http://localhost:8893/train
+```
+
+Where `corpus.txt` is the text file you want to train it on. The majority of my training data so far came from a plain text copy of the King James Bible, which is pretty sizeable in its own right at ~4MB of text, but I encourage you to source yours from wherever you see fit. If you use something really big, this might take a while, long enough that `curl` might time out. Luckily for us, the developer provides a script called `load.sh` to circumvent this:
+
+```
+wget https://svn.zadzmo.org/repo/nepenthes/head/load.sh
+chmod +x load.sh
+./load.sh beemoviescript.txt http://localhost:8893/train
+```
+
+### Pointing Caddy at it
+
+Simply add a `reverse_proxy` directive to the `bans` snippet you made before (you did make it, right?):
+
+<p class="codetitle">/etc/caddy/Caddyfile</p>
+
+```
+(bans) {
+    respond /robots.txt 200 {
+        # robots.txt stuff from before
+    }
+
+    reverse_proxy /slop* localhost:8893 {
+        header_up X-Prefix /slop
+    }
+}
+```
+
+That will create a subdirectory called /slop/ on every website you use that snippet on that points to your shiny new tarpit. The `header_up` directive is there to tell Nepenthes what sudirectory it's running under. It's important to have this, or it won't work.
+
+### Setting the trap
+
+Now all that's left to do is add a link to the tarpit on your website, ideally every page. You can make this invisible to the user using CSS (in fact, I'd recommend doing this) and the scrapers won't care &mdash; they don't look at that anyway. They'll just see a yummy juicy link in the HTML, follow it, and fall directly into your trap.
+
+If you've got processing power and bandwidth to burn and you're content to just let the AI scrapers, as the Nepenthes dev puts it, "suck down as much bullshit as they have diskspace for and choke on it", stop here. In fact, you could even skip the robots.txt bit if you want and target the well-behaved LLM scrapers too, if indeed any exist. I'm not your mom. I'm *especially* not your dad.
+
+I was initially content to do this, too, until some dipshit on Alibaba Cloud with like 300-odd IP addresses in the ranges `47.82.0.0/16` and `47.79.0.0/16` showed up and hammered my Nepenthes instance so hard it hit some built-in resource usage limit in Docker and couldn't even respond to queries to its statistics endpoint anymore and I decided it was time to intervene. Here's what I came up with.
+
+## Integrating this with Fail2Ban
+
+[Fail2Ban](https://github.com/fail2ban/fail2ban) is a program that scans log files for failed authentication attempts and bans the offending IP addresses. You should already have this set up. If you don't, go set it up right now. It's probably packaged for your distro. Go do it. Do it now. `sudo apt install` that bitch. I'll wait. It takes like two seconds. While you're at it, set up key-based auth for your SSH server and turn password login off.
+
+Fail2Ban is, strictly speaking, *for* blocking unwanted login attempts, but it's fairly easy to adapt to just about anything that emits a log file full of IPs. The Nepenthes dev has alluded to the possibility of Fail2Ban integration in the future, but it is unfortunately not yet the future, so I had to take this into my own hands. For the time being, the easiest way to do that is to point a custom Fail2Ban jail to your HTTP access logs that scans for requests to your tarpit.
+
+Turns out, Caddy in particular is kind of a nuisance to integrate with Fail2Ban because it dropped support for the standard access log format that Fail2Ban expects from Apache and Nginx a couple years ago. That didn't stop me, though, and now thanks to my suffering it won't stop you either.
+
+First, you need to configure Caddy to actually generate access logs, since it doesn't by default. You can do that by just adding this to all the sites you want to keep logs for:
+
+<p class="codetitle">/etc/caddy/Caddyfile</p>
+
+```
+yoursite.example.com {
+    log {
+        output file /var/log/caddy/access.log
+    }
+}
+```
+
+I put it in a separate snippet, though, because I have a lot of sites and I might want to make my logging configuration more elaborate in the future:
+
+<p class="codetitle">/etc/caddy/Caddyfile</p>
+
+```
+(logs) {
+    log {
+        output file /var/log/caddy/access.log
+    }
+}
+
+yoursite.example.com {
+    import logs
+    # your other site configuration here. idk your life
+}
+```
+
+Now we can configure Fail2Ban to read that by creating a custom jail. First, add the following to `/etc/fail2ban/jail.local` (create this file if it doesn't exist):
+
+<p class="codetitle">/etc/fail2ban/jail.local</p>
+
+```
+[nepenthes-caddy]
+enabled  = true
+backend  = auto
+port     = http,https
+filter   = nepenthes-caddy
+logpath  = /var/log/caddy/access.log
+maxretry = 5
+findtime = 36000
+bantime  = 604800
+```
+
+`jail.local` is a user override file for the configurations present in `jail.conf`, which you're not supposed to touch. Here's what all these settings mean:
+
+- `[nepenthes-caddy]` is the name of our jail. We'll refer to it by this in Fail2Ban's CLI.
+- `enabled`: Pretty self-explanatory, I should think
+- `backend`: Apparently this defaults to `systemd` on Debian-based distros and that breaks some things. Either way, it works with it set to `auto`.
+- `port`: The ports you want Fail2Ban to block traffic on. As you can see, several common ports have named aliases. Only the ports for HTTP and HTTPS are relevant here.
+- `filter`: The name of a filter file we'll be creating shortly.
+- `maxretry`: The amount of bad requests Fail2Ban has to see before banning an IP address.
+- `findtime`: The window, in seconds, that the `maxretry` threshold has to be hit within to trigger a ban.
+- `bantime`: The time, in seconds, to ban an IP for.
+
+Feel free to tune those last three parameters to taste to find a balance between punishing scrapers with Nepenthes and blocking them to save your resources that you personally find favourable.
+
+Now, create the file `/etc/fail2ban/filter.d/nepenthes-caddy.conf` and fill it with the following:
+
+<p class="codetitle">/etc/fail2ban/filter.d/nepenthes-caddy.conf</p>
+
+```
+[Definition]
+failregex = ^.*"remote_ip":"<HOST>",.*?"uri":"/slop.*$
+ignoreregex =
+datepattern = LongEpoch
+```
+
+This will flag for banning any line in the access log that matches the regular expression defined under `failregex`. The one in here looks for any requests whose URI starts with "/slop", and also tells Fail2Ban where to find the IP we want banned. `datepattern = LongEpoch` seems to refer to the date format used by Caddy, so that Fail2Ban can look for that on its own.
+
+If you know you already have some offending requests in your Caddy log file, you can now test your filter with this command:
+
+```
+sudo fail2ban-regex -v -l debug /var/log/caddy/access.log /etc/fail2ban/filter.d/nepenthes-caddy.conf
+```
+
+If your filter works, it should spit out a list of offending IPs with corresponding times, after possibly taking a couple seconds to process. Now just reload Fail2Ban:
+
+```
+sudo fail2ban-client reload
+```
+
+and you should be up and running. You can check the status of your new jail with this command:
+
+```
+sudo fail2ban-client status nepenthes-caddy
+```
+
+I was confused initially because the list of banned IPs didn't populate immediately. Rest assured, it will. I set this up and then took a nap and went to work and when I came back it had banned, like, 300 of those Alibaba Cloud IPs I mentioned before and my server had dropped back down from a sustained 15-20% CPU usage to its typical 0-1%, and my Nepenthes instance was responsive again! I'd say that worked a treat. I should have set this up years ago.
\ No newline at end of file
diff --git a/blog/posts/grow_spikes/index.php b/blog/posts/grow_spikes/index.php
new file mode 100644
index 0000000..f572f79
--- /dev/null
+++ b/blog/posts/grow_spikes/index.php
@@ -0,0 +1 @@
+<?php include("../post.php");
\ No newline at end of file