Web based family history software

Question Increase Access denied

  • braulio_holtz
  • Topic Author
  • Visitor
  • Visitor
10 years 5 months ago #1 by braulio_holtz
Increase Access denied was created by braulio_holtz
Increase Access denied by Google Webmaster Tools would check the user_agent?
Example, holtzgen.com/famlist.php?alpha=F goes normally, but in Webmaster Tools shows as access denied.
Robots.txt
Code:
User-Agent: * Allow: /family.php Allow: /index.php Allow: /indilist.php Allow: /individual.php Allow: /mediafirewall.php Allow: /mediaviewer.php Allow: /module.php?mod=sitemap Allow: /note.php Allow: /repo.php Allow: /source.php Disallow: /site-offline.php Disallow: /site-unavailable.php Disallow: /login.php Disallow: /module.php?mod=*&mod_action=ajax&sb_action=* Disallow: /reportengine.php Disallow: /index.php/install/* Disallow: /login.php

The table Wt_site_access_rule has over 1000 records.
user_agent as SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) is unknown

Please Log in or Create an account to join the conversation.

  • fisharebest
  • Away
  • Administrator
  • Administrator
More
10 years 5 months ago #2 by fisharebest
Replied by fisharebest on topic Increase Access denied
We do not show famlist.php to search engines because
(a) these pages can consume a lot of CPU/memory
(b) it is possible for a search engine to find all the individuals from indilist.php

If you enable the sitemap module (you should!), then you should also block indilist.php in robots.txt

Greg Roach - greg@subaqua.co.uk - @fisharebest@phpc.social - fisharebest.webtrees.net

Please Log in or Create an account to join the conversation.

  • braulio_holtz
  • Topic Author
  • Visitor
  • Visitor
10 years 5 months ago #3 by braulio_holtz
Replied by braulio_holtz on topic Increase Access denied
Understand, funny that the problem occurs with a constant number of days during tracking
I will create a sitemap and I will put as allow: famlist.php the robots.txt to see if it improves

Please Log in or Create an account to join the conversation.

  • ToyGuy
  • Offline
  • Moderator
  • Moderator
  • Live like it's Christmas every day - Santa Stephen
More
10 years 5 months ago #4 by ToyGuy
Replied by ToyGuy on topic Increase Access denied
Think you may have misunderstood. We do NOT recommend giving the bots access to famlist.php. On any site with a substantial family tree, the burden of bandwidth and processing time can be exorbitant. A sitemap will direct the bots to all INDI records and that is much more efficient. Little can be gained from adding the FAM records. Of course, the choice is yours, but don't complain if you suffer performance issues from the abuse, or complaints on being a resource hog from your ISP .

Santa Stephen the Fabled Santa
Latest webtrees at MyArnolds.com
Hosted by webtreesonline.com , a division of GeneHosts LLC
MacOS 10.6.8, Apache 2.2+, PHP 5.4.16, MySQL 5.5.28

Please Log in or Create an account to join the conversation.

Powered by Kunena Forum
}