Item2427: http://foswiki.org/robots.txt does not exist, though it should

On second thought, apart from the robots.txt file, maybe we could also add a sitemap.xml file (should be placed in the root directory as well and can also be referenced by the former).

• The Disallow directives should also help to weed out 'redundant' (i.e., merely reformatted) content
• Using a sitemap, we could give WebHome topics a higher priority (>0.5)

Note that we need one robots.txt[ / sitemap.xml pair] per subdomain!

-- MarkusUeberall - 28 Nov 2009

Added an Alias /robots.txt "/path/to/robots.txt" to the apache config file for foswiki.org as suggested in ApacheConfigGenerator and restarted the daemon.

-- OliverKrueger - 29 Nov 2009