arrfab / centos / centos.org

Forked from centos/centos.org 3 years ago
Clone

Blame static/robots.txt

e8ec7a
Sitemap: http://www.centos.org/sitemap.xml
e8ec7a
e8ec7a
User-Agent: *
e8ec7a
Crawl-delay: 1.0
e8ec7a
Disallow: /scripts/
e8ec7a
Disallow: /themes/
c2ad4e
Disallow: /docs/2/
c2ad4e
Disallow: /docs/3/
c2ad4e
Disallow: /docs/4/