# # robots.txt for http://www.efs-consultants.com/ and friends # The crawlers listed below are allowed on the Gymso site. # please crawl minimum one second. # Gymso allows other crawlers on a case by case basis. # Send requests to mafs.grp@gmail.com. # User-agent: gulliver Disallow: /cgi-bin User-agent: slurp Disallow: /cgi-bin User-agent: fast Disallow: /cgi-bin User-agent: scooter Disallow: /cgi-bin User-agent: vscooter Disallow: /cgi-bin User-agent: ia_archiver Disallow: /cgi-bin User-agent: Nutch Disallow: /cgi-bin User-agent: FAST-WebCrawler Disallow: /cgi-bin User-agent: teoma_agent1 Disallow: /cgi-bin User-agent: Lycos_Spider_(T-Rex) Disallow: /cgi-bin User-agent: MSNBOT/0.1 Disallow: /cgi-bin User-agent: msnbot-media/1.0 Disallow: /cgi-bin User-agent: ArchitextSpider Disallow: /cgi-bin User-agent: ZyBorg Disallow: /cgi-bin User-agent: SurveyBot Disallow: /cgi-bin # Please note: there are # some misbehaved spiders out there that go _way_ too fast. If you're # irresponsible, your access to the site may be blocked. # # Some bots are known to be trouble, particularly those designed to copy # entire sites. Please obey robots.txt. # Please note: there are # some misbehaved spiders out there that go _way_ too fast. If you're # irresponsible, your access to the site may be blocked. # # Some bots are known to be trouble, particularly those designed to copy # entire sites. Please obey robots.txt. # Please note: there are # some misbehaved spiders out there that go _way_ too fast. If you're # irresponsible, your access to the site may be blocked. # # Some bots are known to be trouble, particularly those designed to copy # entire sites. Please obey robots.txt. # # Friendly, low-speed bots are welcome # Inktomi's "Slurp" can read a minimum delay between hits; if your # bot supports such a thing using the 'Crawl-delay' or another # instruction, please let us know. #