Skip to main content
  • Crawling infrastructure
/
  • English
  • Deutsch
  • Español
  • Español – América Latina
  • Français
  • Indonesia
  • Italiano
  • Polski
  • Português – Brasil
  • Tiếng Việt
  • Türkçe
  • Русский
  • العربيّة
  • हिंदी
  • ภาษาไทย
  • 中文 – 简体
  • 中文 – 繁體
  • 日本語
  • 한국어
Sign in
Home Docs
  • Crawling infrastructure
  • Home
  • Docs
  • Intro
  • About Google's web crawling
  • How to...
  • Verify requests from Google
  • Authenticate requests with Web Bot Auth (experimental)
  • Reduce Google's crawl rate
  • Use robots.txt to manage crawling
    • Create and submit a robots.txt file
    • How Google interprets the robots.txt specification
    • Update your robots.txt file
    • List of useful robots.txt rules
  • Optimize crawling performance
    • Optimize your crawl budget
    • Myths about crawling
    • Improve crawling of faceted navigation URLs
  • Reference
  • Common crawlers
  • Special case crawlers
  • User-triggered fetchers
  • Specific crawlers and user-triggered fetchers
    • APIs-Google
    • Feedfetcher
    • Googlebot
    • Google Read Aloud
  • Troubleshooting
  • HTTP status codes
  • Network and DNS errors
  • What's new
  • Changelog