Web based family history software

file Question Google bot

  • vbueschken
  • vbueschken's Avatar Topic Author
  • Offline
  • New Member
  • New Member
More
1 month 1 week ago - 1 month 1 week ago #116706 by vbueschken
Replied by vbueschken on topic Google bot
The hope was that by switching to "Pretty URLs," the website would be found and indexed by Google Search. Unfortunately, that hope did not come true. As a result, this problem still persists. The HTTP Status is still 406 for the Google Bot
Last edit: 1 month 1 week ago by vbueschken.

Please Log in or Create an account to join the conversation.

  • bertkoor
  • bertkoor's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Greetings from Utrecht, Holland
More
1 month 1 week ago - 1 month 1 week ago #116709 by bertkoor
Replied by bertkoor on topic Google bot
That is a bit strange. This response comes from the BadBot checker in webtrees, which should allow GoogleBot.

Yesterday I have submitted my sitemap again (had not done that in years) and some hours later the Search Console reported me it was crawling happily.

Just checking: you are running a recent version (I recall but cannot check at this moment: v2.2.6) of webtrees without any external / 3rd party modules and no modifications or patches?

stamboom.BertKoor.nl runs on webtrees v2.2.6
Last edit: 1 month 1 week ago by bertkoor.

Please Log in or Create an account to join the conversation.

  • vbueschken
  • vbueschken's Avatar Topic Author
  • Offline
  • New Member
  • New Member
More
1 month 1 week ago #116710 by vbueschken
Replied by vbueschken on topic Google bot
Yes, I'm running the newest version v2.2.6 - fresh downloaded an fresh installed without any 3rd part module.

Is it possible to generally disable the Bad Bot Blocker - just for testing?

Please Log in or Create an account to join the conversation.

  • vbueschken
  • vbueschken's Avatar Topic Author
  • Offline
  • New Member
  • New Member
More
1 month 1 week ago #116711 by vbueschken
Replied by vbueschken on topic Google bot
@BertKoor
I have removed all lines with "Google" in the BadBotBlocker.php
Than I have checked the Page in Google Search Console again. now I was able to request the indexing.

I assume I dont need to delete ALL Google lines in the BadBotBocker.php....

Please Log in or Create an account to join the conversation.

  • bertkoor
  • bertkoor's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Greetings from Utrecht, Holland
More
1 month 1 week ago #116712 by bertkoor
Replied by bertkoor on topic Google bot
Removing all that is generally not a good idea. That will result in that any bot claiming to be google will be served, even when it is just pretending or using a user agent string vaguely resembling the user agent used by GoogleBot.

I think it should suffice to remove the 'google' line within the array ROBOT_REV_FWD_DNS (line 1480)

Can you go to the webtrees Control Panel - click link Server Information - find "disable_functions".
If the value of that contains either 'gethostbyaddr' or 'gethostbyname' then that could explain why google is not recognised. Although I'm not sure... if those functions were disabled then I expect a concrete error about that.

stamboom.BertKoor.nl runs on webtrees v2.2.6

Please Log in or Create an account to join the conversation.

  • vbueschken
  • vbueschken's Avatar Topic Author
  • Offline
  • New Member
  • New Member
More
1 month 1 week ago - 1 month 1 week ago #116713 by vbueschken
Replied by vbueschken on topic Google bot
Only  removing the 'google' line within the array ROBOT_REV_FWD_DNS (line 1480) doesn't help - errorFehler: Wegen eines anderen 4xx-Problems blockiert

In the fiels "disable_functions" is 'no value'
Last edit: 1 month 1 week ago by vbueschken.

Please Log in or Create an account to join the conversation.

  • bertkoor
  • bertkoor's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Greetings from Utrecht, Holland
More
1 month 1 week ago #116714 by bertkoor
Replied by bertkoor on topic Google bot
Sorry, I'm out of ideas to debug and trouble-shoot this.
I guess something on your server is not acting as expected, but what...

stamboom.BertKoor.nl runs on webtrees v2.2.6

Please Log in or Create an account to join the conversation.

  • vbueschken
  • vbueschken's Avatar Topic Author
  • Offline
  • New Member
  • New Member
More
1 month 1 week ago #116716 by vbueschken
Replied by vbueschken on topic Google bot
@bertkoor
I have diskussed it withe the support of my hosting provider - they pointet me to the BadBotBlocker.php based on their logfiles.
After deleting all Google bots from BadBotBlocker.php the issue seems to be resolved. Now I have to wait vor Google to index my page and display it in Google Search

Please Log in or Create an account to join the conversation.

  • jbyvosges
  • jbyvosges's Avatar
  • Offline
  • New Member
  • New Member
  • Français mais moi je comprendre l'anglois.
More
3 weeks 5 days ago - 3 weeks 5 days ago #116848 by jbyvosges
Replied by jbyvosges on topic Google bot
Bonjour.
Je suis intéressé par ce sujet, obtenant les mêmes erreurs.
La commande

curl -A "Googlebot" -I lesbarbry.fr/tree/lesbarbry.ged/individu...-Anne-Josephe-BARBRY


me renvoye toujours HTTP/2 406. Je n'ai pas ce problème avec Slurp ou Bingbot , mais je l'ai avec DuckDuckBot .
J'ai essayé sans succès d'ajouter 

RewriteCond %{HTTP_USER_AGENT} ^(Googlebot|Bingbot|Slurp|DuckDuckBot|Baiduspider|YandexBot|facebot|ia_archiver|Twitterbot|Pinterest) [NC]


dans mon fichier .htaccess ainsi que supprimer toutes les lignes Google dans  BadBotBlocker.php
Je ne trouve pas de solution.
 

PHP 8.4 MariaDB 11.8.3
webtrees 2.2.6
O.S. : Linux Ubuntu 26.04 64 bits

[url] dolman.fr/ [/url] [url] lesbarbry.fr/ [/url]
Hégergés par hostinger.com
Last edit: 3 weeks 5 days ago by jbyvosges.

Please Log in or Create an account to join the conversation.

  • bertkoor
  • bertkoor's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Greetings from Utrecht, Holland
More
3 weeks 5 days ago - 3 weeks 5 days ago #116849 by bertkoor
Replied by bertkoor on topic Google bot
You spoofed a fake request by google. webtrees checks whether the ip address really belongs to google.
github.com/fisharebest/webtrees/blob/23b...BotBlocker.php#L1480
github.com/fisharebest/webtrees/blob/23b...BotBlocker.php#L1647

I bet you did not do the request from a server in a google datacentre, so these requests get blocked.

May I ask why you went investigating this? Are you having specific errors with google?

stamboom.BertKoor.nl runs on webtrees v2.2.6
Last edit: 3 weeks 5 days ago by bertkoor.

Please Log in or Create an account to join the conversation.

Powered by Kunena Forum