robots.txt
Upload User: seven77cht
Upload Date: 2007-01-04
Package Size: 486k
Code Size: 0k
Category:

Browser Client

Development Platform:

Unix_Linux

  1. #
  2. # WWWOFFLE Version 2.5 - robots exclusion file.
  3. #
  4. # See: http://info.webcrawler.com/mak/projects/robots/norobots.html
  5. #
  6. User-Agent: *
  7. # The control pages
  8. Disallow: /control
  9. # The refresh forms
  10. Disallow: /refresh
  11. # The monitor forms
  12. Disallow: /monitor
  13. # The indexes
  14. Disallow: /index