Robots.txt Generator

Create a Robots.txt File for Crawler Rules

Use the generator above to create a starting robots.txt file for your website. A robots.txt file can tell supported web crawlers which paths they are allowed or not allowed to crawl.

Robots.txt Controls Crawling, Not Access Security

Do not use robots.txt to protect private, confidential or sensitive content. The file is publicly accessible, and a disallowed URL can still be discovered in other ways. Protect private content with proper authentication and access controls.

Basic Parts of a Robots.txt File

User-agent: Identifies the crawler the following rules apply to.

Disallow: Requests that the selected crawler not crawl a particular path.

Allow: Can permit a more specific path within a broader disallowed area for crawlers that support the directive.

Sitemap: Can point crawlers to the location of an XML sitemap.

Before You Upload the File

Review every rule carefully. A broad disallow rule can accidentally block important sections of a site from crawling. Place robots.txt at the root of the hostname it controls, for example https://example.com/robots.txt.

Robots.txt and Noindex

If you need to prevent a page from being indexed, use an appropriate supported noindex method on a page that crawlers can access. Blocking a page in robots.txt is not a reliable substitute for an index-control directive.