A WooCommerce shop we host started using more bandwidth than usual. Not a spike, a steady climb: about 3 GB a day in early October, rising to over 9 GB a day two weeks later. At that rate, we’d normally be talking to the client about a bigger hosting package.
Before sending anyone a bill, I looked at the access logs.
Where the bandwidth was going
The client’s own traffic was flat, at roughly 1 GB a day for the whole fortnight. Almost all of the growth came from a single user agent asking for image URLs under /wp-content/uploads/.
Those images dated from 2021 to 2023 and had been deleted about a year earlier. None of the site’s current sitemaps mention them. Something was crawling an old list of image URLs, and every request missed. A missing image doesn’t get a small 404 from Apache. WordPress’s rewrite rules hand the request to index.php, and WordPress renders the full theme 404 page at about 440 KB. On one day in October, the nearly 23,000 requests came in at over 9 GB, with a PHP-FPM worker tied up for each uncached response.
A crawler that looks like a residential proxy network
Blocking by IP was never going to work. In four days, the crawler made 88,000 requests from over 70,000 different IP addresses. It sent no referrer and never asked for robots.txt. It looked exactly like the sort of rotating IP address botnets we see, cycling through residential proxies from all over the world.
Returning a 410 for missing images
I wanted to stop these requests hitting WordPress, which meant detecting and blocking them in Apache. If a request is for an image under uploads and the file doesn’t exist, Apache should respond with its own 410 Gone:
# Missing upload images get Apache's 410 instead of a WordPress 404 render.
# Set in the vhost config, not .htaccess.
# Don't use this if a plugin generates images on request, or on a reverse proxy
# where the files aren't on local disk: every image would return 410.
<If "%{REQUEST_URI} =~ m#^/wp-content/uploads/.+\.(?:jpe?g|png|gif|webp|avif)$# && ! -f '%{REQUEST_FILENAME}'">
Redirect 410 /
</If>Blocking the user agent too
The crawler identified itself as Mozilla/5.0 (compatible; crawler): no bot name, no contact URL. So I blocked that exact string too:
# Exact match only. Returns a 403 without touching WordPress.
<If "%{HTTP_USER_AGENT} == 'Mozilla/5.0 (compatible; crawler)'">
Require all denied
</If>This is a weaker fix, because the botnet can change its user agent whenever it likes. If it does, the 410 rule still catches it.
The result
With both rules in place, each of these requests now costs about 4.6 KB in response bytes, instead of 446 KB. That’s roughly 1% of what it was, and PHP-FPM isn’t involved at all.
Half an hour of log analysis and a few lines of Apache config, and we don’t have to increase the client’s hosting bill. If your bandwidth is creeping up, check what your 404s weigh before you pay for more.
If you’ve seen something similar on your own sites, I’d be interested to hear how you handled it.