Question Google bot

Je pose la question parce que je reçois fréquemment des message de Google Search Console qui me disent qu mes pages ne sont pas indexées. Mes sites webrees sont hébergées par Hostinger.
J'ai posé la question à plusieurs moteurs d'Intelligence Artificielle qui m'ont proposé plusieurs solutions, mais aucune ne résout le problème.

Could you share the content of such an email? Do you perhaps have fragments of a server access log file which can show detail of why requests were blocked?
The few emails from google I received myself, after analysis I concluded they could all be safely ignored.

If you used their Search Console it is possible you had visits from an odd google bot "Google-InspectionTool" which is currently blocked. You can lift the block by either removing it from the BAD_ROBOTS list yourself, or add some code with configuration:
github.com/fisharebest/webtrees/pull/5386

BOnjour et veuillez m'excuser pour ce retard.
lesbarbry.fr/tree/lesbarbry.ged/individu...Antoine-BARBRYRépons de google search console :

Découverte
Plans du site
lesbarbry.fr/sitemap.xml
Page d'origine
lesbarbry.fr/tree/lesbarbry.ged/family/F...arie-Francoise-LEROY
lesbarbry.fr/tree/lesbarbry.ged/source/S4/Geneanet
lesbarbry.fr/tree/lesbarbry.ged/individu...256/Francoise-BARBRY
Exploration
Dernière exploration
Sans objet
Explorée avec
Sans objet
Exploration autorisée ?
Sans objet
Récupération de page
Sans objet
Indexation autorisée ?
Sans objet
Indexation
URL canonique déclarée par l'utilisateur
Sans objet
URL canonique sélectionnée par Google

Sans objet
 

Demande d'indexation :

Lors du test en direct, des problèmes d'indexation ont été détectés au niveau de l'URL

Dans la page concernée : Sur 7740 individus  recensés sur mon site, seulement 76 pages sont indexées.
 

Search term:


My favourite search engine DuckDuckGo has no problem finding the page. It seems indeed that Google has not yet indexed that page.

I have a bit of a problem interpreting what you quoted from the email. For all properties it stated "Sans objet", in English "not apliccable", in other words: they don't know. So we don't know.

Have you tried to force a page crawl with that user agent string 'Google-InspectionTool' removed from the bad robots list?

It is possible your hosting provider is blocking google for whatever reason. The (in memory generated) robots.txt contains a line "Crawl-delay: 10". Maybe for your provider that is still a bit too fast?

Anyway, thanks for sharing, but there's not much detail here for a proper analysis of what is wrong for google.

I was reading this thread and it dawned on me that I should check my google console.  Turns out that I had over 400 HTTP 406 errors reported.  I have a valid sitemap.xml and am using pretty URLs. I did a search on several unique names in the site map both on Bing and Google search, only BIng worked.
I deleted the google entries in  BadBotBlocker.php and requested that google run my site again.  Will let your know.

Bonjour.
Le nombre de mes pages indexées par Google est passé de 76 à 113 en 2 jours.
Comme j'ai modifié plusieurs choses, je ne sais pas quelle est la modification qui a induit ce résultat.
J'ai modifié .htaccess, config.ini.php ( ), BadBotBlocker.php ...
Je vais essayer de reprendre une par une ces modifications sur un autre site (dolman.fr) pour essayer de trouver ce qui a amélioré l'indexation.
 

makitso wrote:

I deleted the google entries in BadBotBlocker.php and requested that google run my site again.  Will let your know.

Deleting everything Google-related in BadBotBlocker.php is a bit drastic.
You should at least leave it in array ROBOT_REV_FWD_DNS. This will let webtrees check whether requests are genuinely from Google or not. Although if your server is unable to execute PHP functions gethostbyaddr or gethostbyname for whatever reason, then webtrees is unable to verify it really is Google doing the requests and thus will block it.

May I ask, do you still have some access log lines from your server where Google got 406 responses? These might help us a bit, eg what the user agent string and ip address exactly were.

jbyvosges wrote:

The number of my pages indexed by Google jumped from 76 to 113 in two days.

Indexing by google is a slow process indeed. You need to be patient.
At least this shows that basically the indexing process is not completely broken.

Google console now shows that these pages are able to be indexed.

Google always used to be well-behaved.

But at the time of the last release, Google was not behaving.

It had started crawling many pages a second, ignoring robots.txt, etc.

So I added it to the bad-bot list.

I don't care how big/important Google is. If it doesn't play nicely, then we don't play.

If it is now behaving, then I may allow it again.

fisharebest wrote:

Google always used to be well-behaved.

But at the time of the last release, Google was not behaving.

It had started crawling many pages a second, ignoring robots.txt, etc.

So I added it to the bad-bot list.

I don't care how big/important Google is. If it doesn't play nicely, then we don't play.

If it is now behaving, then I may allow it again.
 

AND the bigger they are the harder they fall..

Good for you Greg.. Some of these big companies are just getting too damned big! And us mere peasants are nothing more than fodder for their fun!

It is the price of 'progress' and 'progress' has its price!