User-agent: * #Several crawlers support a Crawl-delay parameter, set to the number of seconds to wait between successive requests to the same server: #Crawl-delay: 10 #Request-rate: 1/10 # maximum rate is one page every 5 seconds #Visit-time: 0100-0745 # only visit between 6:00 AM and 8:45 AM UT (GMT) Disallow: /about Disallow: /difdocs Disallow: /dis Disallow: /disweb Disallow: /contenido/icdp/upload Disallow: /icdp Disallow: /images Disallow: /news #Disallow: /scientific-drilling #Disallow: /sites