These days the vast majority of website page views on my clients’ sites are from bots and crawlers. Is this actually a problem, or is this the new normal? I examined three WooCommerce sites on my network to see what the breakdown of page-view traffic actually looks like.
I wanted a bit of a spread, so I chose a fashion accessories website, an office stationery website, and a website selling mobility aids. The mobility aids website sells internationally, while the other two sell mostly within the UK. All three sites see a moderate-to-high level of website traffic.
I spend a lot of time sifting through website server logs. Error logs are great when chasing down bugs and issues with websites, of course. But access logs can be equally interesting. A big site generates thousands of records every day, so it took a bit of creative shenanigans to extract the data in a meaningful form.
Technical dive into the bot traffic investigationGo on then, what are the numbers like?
The actual page-view stats over the course of a working week were… mind-blowing!
| Fashion accessories | Office stationery | Mobility aids (international) | |
|---|---|---|---|
| Page requests / day | 17,650 | 11,789 | 15,763 |
| Human | 1.6% | 15.9% | 23.9% |
| Search engines (verified) | 12.7% | 12.1% | 16.9% |
| Declared crawlers | 16.6% | 11.0% | 7.1% |
| Disguised bots (data centre) | 50.7% | 37.7% | 40.7% |
| Fake Googlebot/bingbot | 0.2% | 0.2% | 0.8% |
| Unknown (residential, page only) | 18.1% | 23.0% | 10.7% |
The stand out thing for me is that the big crawlers (GoogleBot, BingBot, etc) were perfectly well behaved (apart from the meta crawler – that’s terrible). Most importantly, they don’t overwhelm the sites. Instead, they crawl at a steady pace throughout the day.
The disguised bots and unknown crawlers usually show up, batter the site with requests in something resembling drive-by violence, then disappear. Unless they trip an alarm that puts them onto my firewall’s blocklist, of course.
What should we do about it?
E-commerce store owners pay for website hosting to be their digital shop front. So they need page-views to convert into sales – that’s how it works. If a website is generating the right level of revenue to support the business, the skewed human-to-bot ratio is more of a curiosity than a problem.
GoogleBot and BingBot can keep doing what they’re doing. Google is still the dominant source of organic traffic, no matter how many people say “everyone uses AI for everything these days”. Not according to website analytics referrer metrics. But… AI Agents, OpenAI+Anthropic crawlers and other modern bots are growing, so blocking all bots would be ridiculous.
OpenAI’s training crawler is a tiny fraction of traffic. It’s not worth putting in effort to block it.
It’s the drive-by events that I try to catch and block. The bots that show up and “steal” a ton of content while trying to disguise themselves as something legitimate… I hate those things. But they usually stand out in the logs, so we can block those IP addresses in our firewall. It stops them hitting other sites on our network, which is a win.
So, if you’re a bot/crawler, you can grab pages from my site if…
- you don’t hit the site so hard that humans are blocked from making purchases.
- you don’t crawl all my site’s images, as that would dent my hosting bandwidth allowance and cost me money.
There’s also some good standard practice you should follow:
- Check your site’s page caching is enabled and working. This can turn a page-load request from 1 second to just milliseconds. That means you can serve more pages in parallel.
- If your images & videos are getting hit by bots, consider a CDN. It can be a cheaper way to server media than regular hosting.
Any good managed hosting provider should be able to fight enough of the bad bots for you, so you can focus on promoting your business and fulfilling orders.