Web based family history software

file Question Google bot

More
4 days 1 hour ago #117016 by makitso
Replied by makitso on topic Google bot
Well, things haven't gone as I expected.  When I started, google was not indexing my site and Bing was. I fiddled around on the google console and Bing workbench, refined my robots.txt file and resubmitted my sitemap.xml file and gave google another shotl.  Now, google search works, yea, but Bing doesn't --ugh.

Rob
www.skatekey.net ( webtrees 2.2.6
webtrees forum admin
Author custom css module
PHP 8.3.30
Hosted at tigertech.net

Please Log in or Create an account to join the conversation.

  • bertkoor
  • bertkoor's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Greetings from Utrecht, Holland
More
3 days 8 hours ago - 3 days 6 hours ago #117021 by bertkoor
Replied by bertkoor on topic Google bot
You opened this bug report some weeks ago. If that issue is resolved or explained, could you please click on the "close" button?

I do not agree with your alternative robots.txt, specifically this might put off Bing:
Code:
User-agent: * Disallow: /

Bing might interpret that as "all user agents (including me) may not access anything"

Advice: revert to the robots.txt generated by webtrees and try again. I personally have no indication it is faulty.

stamboom.BertKoor.nl runs on webtrees v2.2.6
Last edit: 3 days 6 hours ago by bertkoor.

Please Log in or Create an account to join the conversation.

More
3 days 5 hours ago #117030 by makitso
Replied by makitso on topic Google bot
My understanding is that the two Allow statements let Googlebot and Bingbot to index the site. Is my understand wrong?
User-agent: GooglebotAllow: /User-agent: BingbotAllow: /User-agent: DuckDuckBotAllow: /User-agent: *Disallow: /


 

Rob
www.skatekey.net ( webtrees 2.2.6
webtrees forum admin
Author custom css module
PHP 8.3.30
Hosted at tigertech.net

Please Log in or Create an account to join the conversation.

  • bertkoor
  • bertkoor's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Greetings from Utrecht, Holland
More
3 days 54 minutes ago - 3 days 53 minutes ago #117031 by bertkoor
Replied by bertkoor on topic Google bot
My understanding is that parsing of robots.txt does not need to stop once a matching rule for a user agent string is found. A set of conflicting rules like yours can be interpreted in a number of ways, and that is exactly what you have observed.

Aside of that, you should not allow all urls to be fetched by these three crawlers. They can know in advance that charts, reports etc are off limits.

Again, I have not seen any evidence that the robots.txt file as generated by webtrees is/was stopping google or bing.
If it ain't broken, don't fix it.

stamboom.BertKoor.nl runs on webtrees v2.2.6
Last edit: 3 days 53 minutes ago by bertkoor.

Please Log in or Create an account to join the conversation.

Powered by Kunena Forum