robots.txt is a file placed in your website root directory that tells search engine crawlers which pages they can and cannot crawl. It is part of the Robots Exclusion Protocol — a standard that most major search engines respect.

Does robots.txt prevent pages from being indexed?

No — robots.txt only blocks crawling. If other sites link to a disallowed page, Google can still index it without crawling the content. To prevent indexing, use a noindex meta tag or X-Robots-Tag header instead.

Should I disallow my admin directory?

Yes, it is good practice to disallow /admin/, /wp-admin/, and any other backend paths from crawlers. This reduces unnecessary crawl budget usage and keeps sensitive URLs out of search engine caches.

Where do I put the robots.txt file?

It must be placed in the root of your domain at exactly https://yourdomain.com/robots.txt. In Next.js, put it in /public/robots.txt. In static sites, place it in the root folder alongside index.html.

🤖

Robots.txt Generator

Build a robots.txt file visually. Block specific bots, disallow directories, set crawl delays, and add your sitemap URL. Free robots.txt generator — no sign-up.

User Agent #1

User-agent

Allow (one per line)

Disallow (one per line)

Sitemap URL

Crawl Delay (optional)

Generated robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/


Sitemap: https://example.com/sitemap.xml

About This Tool

Build a valid robots.txt file visually without memorizing the syntax. Add rules for specific bots or all crawlers, disallow directories and files, set crawl delays, and include your sitemap URL.