Introduction

I’ve had numerous clients contact me about their servers having memory and CPU issues. Upon further investigation, the issue seems to be bot’s scraping the Event’s Calendar pages, and following all of the links within the page.

Now, this wouldn’t be a problem for a site that has caching and regular pages, as the pages would be cached and the data returned without breaking a sweat as PHP is never involved.

Unfortunately with The Events Calendar, the links being scraped contain query strings and these aren’t natively cached. Furthermore, the bot’s are going through pagination and any links they can find on the page which might be the links to the iCal URL.

Blocking Scraping Bots accessing Events Calendar Pages

Block with Cloudflare

The solutions is pretty simple, we just need to block know bots from accessing the Events Calendar URL’s. Here’s a Cloudflare rule that does just this.

(http.request.uri.query contains "ical" and cf.client.bot) or (http.request.uri.query contains "eventDisplay" and cf.client.bot) or (http.request.uri.path contains "/events" and cf.client.bot)

The above will block requests containing the following

• ical
• eventDisplay
• /events

This seems to encapsulate most of the requests that the bots are scraping.

Bots Blocking List

This is the master list of bot useragents.

facebookexternalhit
meta-externalagent
googlebot
semrushbot
bingbot
mj12bot
ahrefsbot
barkrowler
amazonbot
gptbot
bytespider
petalbot
slurp
duckduckbot
baiduspider
yandexbot
sogou
exabot
twitterbot
linkedinbot
pinterest
dotbot
crawler4j
dataforseo
scrapy
wget
curl

Blocking Scraping Bots with Nginx

If you don’t use Cloudflare for your site, you can block the requests with Nginx. Here’s an example. You will have to adapt it.

Step 1 – Identify the Bots via Maps in http section

Add this to your http section of your Nginx Config.

GridPane: Create and add to new file /etc/nginx/conf.d/block-scraping-events.conf

http {
    variables_hash_max_size 2048;
    # Define bot user agents if needed
    map $http_user_agent $is_scraping_bot {
        default 0;
        ~*facebookexternalhit 1;
        ~*meta-externalagent 1;
        ~*googlebot 1;
        ~*semrushbot 1;
        ~*bingbot 1;
        ~*mj12bot 1;
        ~*ahrefsbot 1;
        ~*barkrowler 1;
        ~*amazonbot 1;
        ~*gptbot 1;
        ~*bytespider 1;
        ~*petalbot 1;
        ~*slurp 1;
        ~*duckduckbot 1;
        ~*baiduspider 1;
        ~*yandexbot 1;
        ~*sogou 1;
        ~*exabot 1;
        ~*twitterbot 1;
        ~*linkedinbot 1;
        ~*pinterest 1;
        ~*dotbot 1;
        ~*crawler4j 1;
        ~*dataforseo 1;
        ~*scrapy 1;
        ~*wget 1;
        ~*curl 1;
    }

    # Map to handle request URIs
    map $request_uri $block_scraping_uri {
        default 0;
        "~*ical" 1;
        "~*eventDisplay" 1;
        "~*\/events" 1;
    }

    # Combine both conditions into one variable
    map "$is_scraping_bot$block_scraping_uri" $block_scraping_request{
        default 0;
        11 1; # Both $is_scraping_bot and $block_scraping_uri are true
    }
}

Step 2 – Apply blocking to your site

Add the following to your location / section.

GridPane: Create a file called /etc/nginx/extra.d/block-scraping-root-context.conf which will affect all sites. For a specific site set create a file called /var/www/domain.com/nginx/block-scraping-events-root-context.conf

if ($block_scraping_request) {
    return 403; # Forbidden
}

Blocking Scraping Bots with Litespeed/Openlitespeed

Here is the appropriate .htaccess rules for blocking bots via Litespeed/Openlitespeed

# Define partial matches for known bots
RewriteCond %{HTTP_USER_AGENT} facebookexternalhit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} meta-externalagent [NC,OR]
RewriteCond %{HTTP_USER_AGENT} googlebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} semrushbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} bingbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} mj12bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ahrefsbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} barkrowler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} amazonbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} gptbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} bytespider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} petalbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} slurp [NC,OR]
RewriteCond %{HTTP_USER_AGENT} duckduckbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} baiduspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} yandexbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} sogou [NC,OR]
RewriteCond %{HTTP_USER_AGENT} exabot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} twitterbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} linkedinbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} pinterest [NC,OR]
RewriteCond %{HTTP_USER_AGENT} dotbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} crawler4j [NC,OR]
RewriteCond %{HTTP_USER_AGENT} dataforseo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} scrapy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} wget [NC,OR]
RewriteCond %{HTTP_USER_AGENT} curl [NC]

# Block URLs with specific query strings or paths for these bots
RewriteCond %{QUERY_STRING} (ical|eventDisplay) [NC,OR]
RewriteCond %{REQUEST_URI} ^/events/ [NC]
RewriteRule ^ - [F,L]

Confirming Blocks

You can check your access log to see if you’ve missed any user agents

tail -f /var/www/domain.com/logs/domain.com.access.log | grep -v 403 | grep event  

User Agents

Collecting User Agents

If you’re on GridPane using the Openlitespeed logs, you can use the following command to get a list of unique User Agents accessing eventDisplay

cat /var/www/domain.com/logs/domain.com.access.log | grep "eventDisplay" |awk -F'"' '{print $7}' | sort | uniq

Valid User Agents

I’ve listed some common bot User Agents above, however there are also some to be aware of as actual users.

• iOS/17.5.1 (21F90) dataaccessd/1.0 = iOS Calendar App

Changelog

• 08-12-2024 – Updated Nginx to be correct syntax, also added more user agents and GridPane files.
• 08-18-2024 – Updated rules to include PetalBot