Sitemap: http://www.centos.org/sitemap.xml User-Agent: *Crawl-delay: 1.0Disallow: /scripts/Disallow: /themes/Disallow: /docs/2/Disallow: /docs/3/Disallow: /docs/4/