
The crawler that cost us over 9 GB a day in image 404s
Bandwidth started creeping up, caused by a crawler hitting old/deleted image URLs. Each 404 cost a full WordPress render. Here's the fix.
23 Oct 2026
Most of the traffic your website gets these days isn't human. Search engines, SEO tools, AI training crawlers, link previewers, and plain old scrapers all show up in your access logs, and most of them behave themselves. Below you'll find articles on telling the two apart: honest crawlers that respect robots.txt and pace themselves, versus disguised bots and outright botnets that hammer a site, steal content, or run card-testing fraud through your checkout.
Some of this is investigative (working out from raw log files what's actually requesting your pages), some of it is defensive (firewall rules, rate-limiting, blocking the traffic that costs you bandwidth without ever converting), and some of it is just useful context for anyone wondering why their analytics don't add up. The goal isn't to block every bot. It's knowing which ones are worth the fight.

Bandwidth started creeping up, caused by a crawler hitting old/deleted image URLs. Each 404 cost a full WordPress render. Here's the fix.
23 Oct 2026