Sitemap: http://www.centos.org/sitemap.xml User-Agent: * Crawl-delay: 1.0 Disallow: /scripts/ Disallow: /themes/ Disallow: /docs/2/ Disallow: /docs/3/ Disallow: /docs/4/