Question Too many CPU Seconds

Greg,
is this a simpel fix which can be added to the actual 2.2.1 release? If so, we could test this as well.

HELP! My site has been shut down again. MANY calendar queries from China, Hong Kong, Vietnam, Brazil, etc.
These IP's and countries have already been blocked using the security tools for SiteGround, but are still getting through. We need a fix soonest, please!

Very quick and dirty fix: make your trees private, so logging in is required.
I did post another fix last week that stops access to the calendar for non-registered users (and bots)

I'm just playing whack-a-mole once a day.
I'm with SiteGround as well.
I download todays access log - see what is the main country/counties of choice and block those.
Today it's Iran.
A good log viewer helps.
I've managed to keep under the CPU limit so far this month. 
None of the fixes so far are fully satisfactory but a stop gap until something can be done.  

This was sent by the SiteGround host support tech, but I don't know what to do with it, or if it is of any value:
"By removing this header and ensuring the the cookies are only set on pages that need them, the site will be cached by our dynamic caching, further reducing the CPU usage."

Ivan-Maznev:~ ivan.m$ curl -ILXGET adkins.ws
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 19 Apr 2025 15:21:45 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 226
Connection: keep-alive
Location: adkins.ws/
Host-Header: 6b7412fb82ca5edfd0917e3957f05d89
X-Proxy-Cache: MISS
X-Proxy-Cache-Info: 0301 NC:000000 UP:

HTTP/2 302
server: nginx
date: Sat, 19 Apr 2025 15:21:47 GMT
content-type: text/html; charset=UTF-8
content-length: 0
location: adkins.ws/tree/Adkins.GED
set-cookie: __Secure-WT-ID=09e4f4301acae94e7b2333e3346a1078; path=/; domain=adkins.ws; secure; HttpOnly; SameSite=Lax
set-cookie: __Secure-WT-ID=6ce3a71a579ebb63dd9b3f2f0d7867ed; path=/; domain=adkins.ws; secure; HttpOnly; SameSite=Lax
permissions-policy: browsing-topics=()
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000
cache-control: no-store
x-httpd-modphp: 1
host-header: 6b7412fb82ca5edfd0917e3957f05d89
x-proxy-cache: MISS
x-proxy-cache-info: 0302 NC:000000 UP:SKIP_CACHE_SET_COOKIE

HTTP/2 200
server: nginx
date: Sat, 19 Apr 2025 15:21:48 GMT
content-type: text/html; charset=UTF-8
content-length: 435483
vary: Accept-Encoding
set-cookie: __Secure-WT-ID=c5110be11994c2cc1248c943497ab387; path=/; domain=adkins.ws; secure; HttpOnly; SameSite=Lax
set-cookie: __Secure-WT-ID=887dd389c7035167ecfbdd582910adae; path=/; domain=adkins.ws; secure; HttpOnly; SameSite=Lax
permissions-policy: browsing-topics=()
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000
cache-control: no-store
x-httpd-modphp: 1
host-header: 6b7412fb82ca5edfd0917e3957f05d89
x-proxy-cache: MISS
x-proxy-cache-info: 0 NC:000000 UP:SKIP_CACHE_SET_COOKIE

This was sent by the SiteGround host support tech, but I don't know what to do with it, or if it is of any value:
"By removing this header and ensuring the the cookies are only set on pages that need them, the site will be cached by our dynamic caching, further reducing the CPU usage."

Ivan-Maznev:~ ivan.m$ curl -ILXGET adkins.ws
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 19 Apr 2025 15:21:45 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 226
Connection: keep-alive
Location: adkins.ws/
Host-Header: 6b7412fb82ca5edfd0917e3957f05d89
X-Proxy-Cache: MISS
X-Proxy-Cache-Info: 0301 NC:000000 UP:

HTTP/2 302
server: nginx
date: Sat, 19 Apr 2025 15:21:47 GMT
content-type: text/html; charset=UTF-8
content-length: 0
location: adkins.ws/tree/Adkins.GED
set-cookie: __Secure-WT-ID=09e4f4301acae94e7b2333e3346a1078; path=/; domain=adkins.ws; secure; HttpOnly; SameSite=Lax
set-cookie: __Secure-WT-ID=6ce3a71a579ebb63dd9b3f2f0d7867ed; path=/; domain=adkins.ws; secure; HttpOnly; SameSite=Lax
permissions-policy: browsing-topics=()
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000
cache-control: no-store
x-httpd-modphp: 1
host-header: 6b7412fb82ca5edfd0917e3957f05d89
x-proxy-cache: MISS
x-proxy-cache-info: 0302 NC:000000 UP:SKIP_CACHE_SET_COOKIE

HTTP/2 200
server: nginx
date: Sat, 19 Apr 2025 15:21:48 GMT
content-type: text/html; charset=UTF-8
content-length: 435483
vary: Accept-Encoding
set-cookie: __Secure-WT-ID=c5110be11994c2cc1248c943497ab387; path=/; domain=adkins.ws; secure; HttpOnly; SameSite=Lax
set-cookie: __Secure-WT-ID=887dd389c7035167ecfbdd582910adae; path=/; domain=adkins.ws; secure; HttpOnly; SameSite=Lax
permissions-policy: browsing-topics=()
referrer-policy: same-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000
cache-control: no-store
x-httpd-modphp: 1
host-header: 6b7412fb82ca5edfd0917e3957f05d89
x-proxy-cache: MISS
x-proxy-cache-info: 0 NC:000000 UP:SKIP_CACHE_SET_COOKIE

Mandatory read with background info:
jan.wildeboer.net/2025/02/Blocking-Stealthy-Botnets/
jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-2/
(no good news, nor any solutions, alas)

fisharebest wrote:

The crawlers were using random version numbers - but within a fixed range.

So on my server, I have add blocks for these

* Chrome versions 1-99.
* Firefox versions 1-69
* Opera versions 8-9
* All "Trident" based browsers

These are all very old/unuspported browsers, so I'm 99% confident that I'm not blocking any legitimate users.

It has been 100% successful in blocking these crawlers.

Until I can be 100% certain that this does not affect any valid users/browsers, I am hesitant to add the same logic to webtrees.
 

@Greg,

How did you actually block these?? Whilst I can understand your reluctance to add to webtrees until there is some certainty about effectiveness and such, this may be of help to ALL who are having this sort of issue AND those who may not even realise they are..

Hopefully something simple like in .htaccess or robots.txt <smile>

You also mentioned a 'fix' for the calendar bot issue.. Would you advise what file that fix should be added to!!

The four webtrees sites Martin and I have on the same server are getting hit at differing rates, but getting hit we are.. So any help to cut that down would really be appreciated!!
 

The 'fix' I made nearly two weeks ago simply showed an error message when any url from the calendar module was requested. This helps in reducing CPU load a bit, since the calendar events of your tree aren't calculated all the time. But the bots still get urls from all the pages to the calendar module.

After thinking a bit more about it, a better fix is to not give links to dates to visitors & bots. That should reduce traffic somewhat more.

I have attached this 'patched' version for webtrees 2.2 to this message. It's supposed to go in the /app folder and replace the Date.php which is already there.
 

fisharebest wrote:

For the calendar page, we set "meta robots=noindex".

this should be "meta index=noindex,nofollow".

Changed with Commit 204f8a9, it worked for me, Bingbot stopped scanning the calendar.

Where/what is the calendar page?

Do you mean, what page has to be updated?
github.com/fisharebest/webtrees/commit/2...c2bbc4d1b5f589f6da57

Got it, thanks:
/resources/views/layouts/default.phtml