Stop Meta’s crawler wasting WooCommerce hosting bandwidth

A WooCommerce client started haemorrhaging 20GB of bandwidth daily – up from around 2GB per day. On investigation, the access logs were full of calls by the Facebook/Meta crawler’s user agent. It was smashing through the client’s hosting bandwidth, so I needed to develop a fix.

Analyse the logs and assess the problem

With website hosting issues, I always go straight to the log files. In this case, we had thousands of log records hitting URLs like this:

57.141.20.28 - - [17/Aug/2026:00:00:36 +0100] "GET /product/amazing-thing/null HTTP/2.0" 404 401931 "https://example.com/product/amazing-thing/?add-to-cart=1234" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))"

We’d already set up robots.txt to prevent crawling ?add-to-cart= URLs. They have no SEO value because they either reload the current page, or redirect to the cart page. But Meta’s crawler clearly disregards that rule and crawls these anyway.

So there were two things we needed to address here:

• Prevent crawler access to all URLs with query parameters. It’s a bit heavy-handed, but on WordPress sites, there’s no good reason for a URL with a query parameter to be crawled.
• Reject access to URLs that match /product/<product-slug>/null and return a HTTP response such as 410 Gone.

The win here would be to do this in the web server (Apache) rather than in WordPress. Apache’s response body for error pages is tiny compared to pages served by WordPress:

• WordPress’s 404 page: 401,931 bytes
• Apache’s own 404 page: 236 bytes

A scan of yesterday’s access log for this client reveals just how hard Meta have been hitting the site:

The first change: stop crawlers hitting URLs with query parameters

Given that all requests to those “/null” product URLs came from add-to-cart referral URLs, it made sense to stop those ?add-to-cart=1234 URLs being crawled outright. In fact, there’s almost never a good reason for a crawler to hit a URL that has a query parameter.

##
# Block some crawlers hitting any URLs that have query parameters (e.g add-to-cart)
#
<If "%{QUERY_STRING} != '' && %{HTTP_USER_AGENT} =~ m#(meta-externalagent)#i">
	Require all denied
</If>

Dropping that into the site’s config had an immediate positive impact on bandwidth, with requests to things like GET /product/amazing-thing/?add-to-cart=1234 returning a 403 response at only 420 bytes each. And… each request was no longer followed by a request to GET /product/amazing-thing/null.

If you want to extend this to cover more user agents, you can swap meta-externalagent for the following. Notice that GoogleBot and BingBot are not in there, as they properly respect robots.txt.

meta-externalagent|Bytespider|GPTBot|ClaudeBot|Amazonbot|PetalBot|DataForSeoBot|Barkrowler|MJ12bot|DotBot|ImagesiftBot|SemrushBot|AhrefsBot

Looking at the log file now, we can see a huge improvement in efficiency. We can see the response is now 403 Forbidden and the response size is only 420 bytes.

Change #2: Catch requests to “/null” in the web server

Requests to GET /product/amazing-thing/null still return 404 Not Found from WordPress/WooCommerce. That’s technically correct, but those response bodies were still a performance hit because they’re served by WordPress/WooCommerce. There’s no page caching on 404s, and the response body is a huge HTML document containing the site’s layout & branding.

Blocking access to these URLs with a 410 Gone was a simple addition to the site’s vhost config.

##
# Block access to any URL ending in "/null" or "/null/"
#
RedirectMatch 410 "/null/?$"

With both those modifications in place, the next thing to do was check the logs again. The effects were immediate. The crawler was being blocked by Apache (instead of WordPress), leading to a huge reduction in CPU drag and bandwidth 👍

Wrapping up

The above actions slashed the resource problems caused by this crawler, but the site is still being crawled… even though ?add-to-cart=1234 URLs are marked as rel=nofollow and robots.txt asks crawlers to leave them alone.

The only way to really block the crawlers is to add those IP addresses in our firewall. But they’re legitimate Facebook/Meta IP addresses – blocking those could be detrimental to legitimate activity (although I did consider it).

Overall, this was an interesting analysis. All we really did was move the response out of WordPress and into the web server itself. Rejections are now served quickly and with a tiny response body. It was a better situation than reaching out to the client and handing them a bill for excessive bandwidth egress this month, and it reduces the load on my servers.