changeset 726:04671b858641

add a crawl-delay for all robots
author Thomas Waldmann <tw AT waldmann-edv DOT de>
date Sat, 27 May 2006 21:54:51 +0200
parents 6b433c2d324e
children 005d8cee8f32
files wiki/htdocs/robots.txt
diffstat 1 files changed, 4 insertions(+), 1 deletions(-) [+]
line wrap: on
line diff
--- a/wiki/htdocs/robots.txt	Sat May 27 14:07:32 2006 +0200
+++ b/wiki/htdocs/robots.txt	Sat May 27 21:54:51 2006 +0200
@@ -1,1 +1,4 @@
-# You can define rules for nice robots here
\ No newline at end of file
+User-agent: *
+Crawl-delay: 20
+Disallow:
+