Robots.txt analysis

The robots.txt file contains strict instructions for the bot about how to index the site. For more information about this file, see the Help section.