User-agent: * Disallow: /cgi-bin/ Disallow: /admin/ Disallow: /webalizer/ Disallow: /analog/ Disallow: /printfriend.jsp Disallow: /tdca/printfriend.jsp Disallow: /edcl/printfriend.jsp Disallow: /cca/printfriend.jsp Disallow: /ccat/printfriend.jsp Disallow: /sca/printfriend.jsp Disallow: /test/printfriend.jsp Disallow: /bcmcl/printfriend.jsp #This is an unwanted crawler User-agent: TurnitinBot Disallow: / # Another one to reject #User-agent: FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) User-agent: fast Disallow: /