Table of Contents
Introduction
I’ve had numerous clients contact me about their servers having memory and CPU issues. Upon further investigation, the issue seems to be bot’s scraping the Event’s Calendar pages, and following all of the links within the page.
Now, this wouldn’t be a problem for a site that has caching and regular pages, as the pages would be cached and the data returned without breaking a sweat as PHP is never involved.
Unfortunately with The Events Calendar, the links being scraped contain query strings and these aren’t natively cached. Furthermore, the bot’s are going through pagination and any links they can find on the page which might be the links to the iCal URL.
Blocking Scraping Bots accessing Events Calendar Pages
Block with Cloudflare
The solutions is pretty simple, we just need to block know bots from accessing the Events Calendar URL’s. Here’s a Cloudflare rule that does just this.
(http.request.uri.query contains "ical" and cf.client.bot) or (http.request.uri.query contains "eventDisplay" and cf.client.bot) or (http.request.uri.path contains "/events" and cf.client.bot)
The above will block requests containing the following
- ical
- eventDisplay
- /events
This seems to encapsulate most of the requests that the bots are scraping.
Bots Blocking List
This is the master list of bot useragents.
facebookexternalhit meta-externalagent googlebot semrushbot bingbot mj12bot ahrefsbot barkrowler amazonbot gptbot bytespider petalbot slurp duckduckbot baiduspider yandexbot sogou exabot twitterbot linkedinbot pinterest dotbot crawler4j dataforseo scrapy wget curl claudebot
Blocking Scraping Bots with Nginx
If you don’t use Cloudflare for your site, you can block the requests with Nginx. Here’s an example. You will have to adapt it.
Step 1 – Identify the Bots via Maps in http section
Add this to your http section of your Nginx Config.
GridPane: Create and add to new file /etc/nginx/conf.d/block-scraping-events.conf
http { variables_hash_max_size 2048; # Define bot user agents if needed map $http_user_agent $is_scraping_bot { default 0; ~*facebookexternalhit 1; ~*meta-externalagent 1; ~*googlebot 1; ~*semrushbot 1; ~*bingbot 1; ~*mj12bot 1; ~*ahrefsbot 1; ~*barkrowler 1; ~*amazonbot 1; ~*gptbot 1; ~*bytespider 1; ~*petalbot 1; ~*slurp 1; ~*duckduckbot 1; ~*baiduspider 1; ~*yandexbot 1; ~*sogou 1; ~*exabot 1; ~*twitterbot 1; ~*linkedinbot 1; ~*pinterest 1; ~*dotbot 1; ~*crawler4j 1; ~*dataforseo 1; ~*scrapy 1; ~*wget 1; ~*curl 1; ~*claudebot 1; } # Map to handle request URIs map $request_uri $block_scraping_uri { default 0; "~*ical" 1; "~*eventDisplay" 1; "~*\/events" 1; } # Combine both conditions into one variable map "$is_scraping_bot$block_scraping_uri" $block_scraping_request{ default 0; 11 1; # Both $is_scraping_bot and $block_scraping_uri are true } }
Step 2 – Apply blocking to your site
Add the following to your location /
section.
GridPane: Create a file called /etc/nginx/extra.d/block-scraping-root-context.conf
which will affect all sites. For a specific site set create a file called /var/www/domain.com/nginx/block-scraping-events-root-context.conf
if ($block_scraping_request) { return 403; # Forbidden }
Blocking Scraping Bots with Litespeed/Openlitespeed
Here is the appropriate .htaccess rules for blocking bots via Litespeed/Openlitespeed
# Define partial matches for known bots RewriteCond %{HTTP_USER_AGENT} facebookexternalhit [NC,OR] RewriteCond %{HTTP_USER_AGENT} meta-externalagent [NC,OR] RewriteCond %{HTTP_USER_AGENT} googlebot [NC,OR] RewriteCond %{HTTP_USER_AGENT} semrushbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} bingbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} mj12bot [NC,OR] RewriteCond %{HTTP_USER_AGENT} ahrefsbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} barkrowler [NC,OR] RewriteCond %{HTTP_USER_AGENT} amazonbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} gptbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} bytespider [NC,OR] RewriteCond %{HTTP_USER_AGENT} petalbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} slurp [NC,OR] RewriteCond %{HTTP_USER_AGENT} duckduckbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} baiduspider [NC,OR] RewriteCond %{HTTP_USER_AGENT} yandexbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} sogou [NC,OR] RewriteCond %{HTTP_USER_AGENT} exabot [NC,OR] RewriteCond %{HTTP_USER_AGENT} twitterbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} linkedinbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} pinterest [NC,OR] RewriteCond %{HTTP_USER_AGENT} dotbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} crawler4j [NC,OR] RewriteCond %{HTTP_USER_AGENT} dataforseo [NC,OR] RewriteCond %{HTTP_USER_AGENT} scrapy [NC,OR] RewriteCond %{HTTP_USER_AGENT} wget [NC,OR] RewriteCond %{HTTP_USER_AGENT} curl [NC,OR] RewriteCond %{HTTP_USER_AGENT} claudebot [NC] # Block URLs with specific query strings or paths for these bots RewriteCond %{QUERY_STRING} (ical|eventDisplay) [NC,OR] RewriteCond %{REQUEST_URI} ^/events/ [NC] RewriteRule ^ - [F,L]
Confirming Blocks
You can check your access log to see if you’ve missed any user agents
tail -f /var/www/domain.com/logs/domain.com.access.log | grep -v 403 | grep event
User Agents
Collecting User Agents
If you’re on GridPane using the Openlitespeed logs, you can use the following command to get a list of unique User Agents accessing eventDisplay
cat /var/www/domain.com/logs/domain.com.access.log | grep "eventDisplay" |awk -F'"' '{print $7}' | sort | uniq
Valid User Agents
I’ve listed some common bot User Agents above, however there are also some to be aware of as actual users.
- iOS/17.5.1 (21F90) dataaccessd/1.0 = iOS Calendar App
Changelog
- 08-12-2024 – Updated Nginx to be correct syntax, also added more user agents and GridPane files.
- 08-18-2024 – Updated rules to include PetalBot