Home > Coding Syntax > Robots Txt Codes > Disallow

Disallow

The code below instructs robots to stay away files or directories that you must keep secure or private,

Some plausible reasons for choosing to disallow crawling are:

  • To lock away credit card and personal information of customers
  • To secure copyrighted digital content
  • To secure passwords of users
  • To restrict access to database files
  • To not confuse robots with duplicate content

 

 

Robots.txt Disallow Directives Generator