Question Google bot

Hi, my webtrees page worked for several years without any problem and I could find it with Google search.
Since some days I can open the website, but I can't find it anymore in Google.
The Google Search Console said it is not available - with " httpstatus.io " I can verify, that the pages sends the Status "406" instead of "200".... so it is not aveilable for Google Serch .

Any Idea how to fix it?
 

I thought we had written some documentation but alas it appears "under construction". The German manual seems also to have nothing on this subject, so I guess I'll have to give you a mini version of what ought to be there.

Short version of webtrees SEO principles:

• {your.domain}}/robots.txt is used by friendly web crawlers to determine what pages may be indexed. webtrees will generate one (in memory on demand) but only if you have not stored one yourself and you have enabled 'rewrite_urls' in data/config.ini.php aka the pretty urls feature.
• {your.domain}/sitemap.xml is also used by friendly web crawlers. To let webtrees generate it (in memory on demand) you need to enable the module called Sitemaps and configure for which trees the sitemap should be created. The config page will also tell you the proper url of your main sitemap.xml so you can submit that to search engines.
• Some specifics regarding google: developers.google.com/search/docs/fundam...ls/seo-starter-guide (sorry if that is not new to you)

It might well be that your site is already indexed, but your Google searches just don't get it back as a result.

To assist you in what could be wrong it really helps if you share the url of your website with us. Then I can take a look at eg the robots.txt and sitemap.xml.

The website httpstatus.io is right that your webtrees site responds with a http status 406. Your own browser will then show "cookie check" and a bit later will show the home page.
There is a dropdown on that page to select the user agent, which defaults to that of your own browser. If you change that to the user agent used by google, then webtrees will check what ip address the request comes from. If that ip address is not owned by google (which in this case it probably is not) then webtrees will respond with status 406 (not acceptable) and a body with text 'bad-dns'.

Thanks Bert for your response.

My Webpage is: familie.bueschken.com
Thats the response from the google search Console:
 
Hope you can help with this issue
 

OK, thank you.

I'm in the process of checking the google search console myself, it is currently pending verification...

Your robots.txt can be fetched. It contains the text "(C) 2019" and a list of separate .php files, so that is a left-over from the old 1.7.x branch. You should delete it, just like most content you find in the directory where webtrees is installed.

Your sitemap.xml cannot be fetched, I get the http status '500 - Internal Server Error'.

When I request your website in a browser, then the url becomes index.php with query parameters, so you don't have the "pretty urls" (rewrite_urls) feature enabled. That means you won't profit from auto-generated robots.txt. Also in the webtrees control panel is a module Legacy URLs. When enabled (and with "pretty urls" enabled) the old urls with query parameters will be redirected to the new ones. So that ensures that using old URLs will still work, so nobody (also no bots) looses access with the new form of URLs.

When I request your website on the command line with curl, then I get the response '404 - Not Found' while I expected a '302 - Redirect'. This response comes from Apache. Maybe you have something in the Apache configuration or .htaccess which blocks these requests?

When I enter the Sitemap URL into the browser I get:
 

Also Google Search Console accepted the sitemap

I tried also to change to "Pretty URLs) by adding:
rewrite_urls="1" to my data/config.ini.php

I got:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

Ah yes, I see. Since you don't have the "pretty" url's enabled, your sitemap has an ugly url

and that works!
I expected / tested:


It seems you own your domain a decade longer than mine exists, and indeed a query very specific to your website (eg "Catharina Wieschmann" site:bueschken.com) gives no results.

When I query one of these URLs

then I get a certificate error (connection is not private) and when I accept the risk then I get http response '403 - Forbidden'

Maybe you are being penalised for having only a subdomain and no main "www" domain? You could do a redirect to the "familie" subdomain via .htaccess:

a general rerouting via htaccess from the main domain makes no sense for me, because I run also 3 other subdomains like feigen.bueschken.com.
With this sub-domain I have a similar issue, that it works for several years, but now I can't find it anymore in Google Search, but in contrast to webtrees I see no problem in Google Search Console.

I checked it - there is no issue with Google and sub-domains. they are handled as normal webpages

Something is definitely wrong, but I'm struggling what exactly. Google refuses to cough up something about your website. Google search console should be able to tell you what is wrong.

 

Regarding the redirection, a simple static page with a list with links to your subdomains I'd consider just visitor friendly. And it could host one copy of your mandatory impressum.

No problem with duckduckgo, which under the hood uses data from Bing.