Pages in search results and the site structure

What information is provided in the archive of loaded pages?

The archive of loaded pages is a file listing all the pages on the site that the robot has visited. It contains:

  • URLs of pages known to the robot.
  • The HTTP code returned to the robot during the last visit.
  • The time of the robot's last visit in Unix-time format.
  • Information about whether the page was shown in search results: 1 — yes, 0 — no.
  • Redirects.