This question about Not sure...: Asked

Prevent Crawling Certain Webs

I have the recommended robots.txt file at the top of my wiki so that robots can only access view links.

Is there a best practice for preventing indexing/crawling of one or more webs?

Is that as simple as, e.g., adding
/bin/view/System

to the top level robots.txt file, or is there something else I need to do?

If it makes a difference, I've got short URLs enabled via Apache.

-- VickiBrown - 27 Feb 2014

 

QuestionForm edit

Subject Not sure...
Extension
Version Foswiki 1.1.9
Status Asked
Related Topics
Topic revision: r1 - 27 Feb 2014, VickiBrown
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy