Webmaster. Feedback

Possible issues

Robots.txt file not found

The robots.txt file contains strict instructions for the bot about how to index the site. For more information about this file, see the Help.

If the robots.txt file is missing on the site, the bot may begin accessing any known pages on the site. So it is important for the robots.txt file to prohibit crawling any duplicated or technical pages on the site, as well as pages with confidential information.

Create a robots.txt file and put it on the server. The bot will begin using the file when crawling the site, and the error message will disappear within a few days.

To check whether the file is formed correctly, use our tool.

Host directive not given in robots.txt

The robots.txt file contains strict instructions for the bot about how to index the site. For more information about this file, see the Help.

If the main mirror is selected by the bot automatically, you can use the Host directive to point the bot to the desired main mirror address. The robots.txt file on your site should specify the Host directive with the correct main mirror to index and show in search results.

Add the Host directive to the file and update the file on the server. After this, the bot will begin using the file when crawling the site, and the error message will disappear within a few days. If you can't modify robots.txt yourself, contact the support service for the site's hosting provider.

To check whether the file is formed correctly, use our tool.

No sitemap files are currently used by the indexing bot

The sitemap file helps with site indexing by regularly telling the indexing bot about new pages on the site. This warning appears if the bot isn't using any sitemap files for the site.

To make the bot start using a created file, add it in the Site information section and wait for the file to be processed by the bot. This may take up to two weeks. Then the warning will disappear.

Contradictory Host directive entered in robots.txt file

If the site can be accessed at multiple addresses that are all mirrors, the Host directive tells the bot which site is the main mirror to index and show in search results.

The main mirror can be detected automatically if the robots.txt files on different site mirrors contain different values in the Host directive. To avoid having the wrong address chosen as the main mirror, fix the Host directive in the robots.txt files so it is the same on all the mirrors.

The "title" and "description" meta tags are missing

These meta tags help to form an accurate site description in search results. For more information, see the Help.

If one or both of these meta tags are missing on your site's pages, add them to the HTML code and save your changes. If you have already added the tags, wait for the bot to crawl the page again. Then the error message will disappear.

Note. Open Graph markup is not supported in page descriptions.
Other questions about possible issues
Rate this article
Thank you for your feedback!