robots.txt Generator

Draft rules for bots: multiple Allow/Disallow lines (one path per line), optional crawl-delay, and sitemap URLs.

User-agent

Allow (one path per line)

Disallow (one path per line)

Crawl-delay (optional, seconds, some bots only)

Sitemap URLs (one per line)

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://it3.site/sitemap.xml

FAQ

Does robots.txt block hackers?

No—it is a polite suggestion to crawlers. Sensitive dirs need auth, network rules, and proper session handling. Attackers ignore robots.

Allow before Disallow precedence?

Some bots use first-match wins; Google documents longest rule precedence—read current vendor docs when mixing patterns.

Wildcard support varies?

* and $ endings behave differently across crawlers—test in Search Console for Google-specific behavior.

Sitemap placement?

Multiple `Sitemap:` lines are valid; ensure URLs are absolute https and return 200.

Crawl-delay ignored?

Google ignores non-standard crawl-delay; Bing may honor it—know your bot audience.

Staging sites?

Use Disallow all during QA but remember to remove before launch or you will de-index production accidentally.

🆓100% Free

🔒Secure & Private

⚡No Signup Needed

❓

A robots.txt file tells crawlers which paths are allowed or disallowed for crawling.

No. It blocks crawling, not guaranteed indexing removal. Use noindex for removal intent.

Yes. Admin, dashboard, and account pages usually have no search value and should be excluded from crawling.

Crawl budget is how many pages search engines crawl in a period. Blocking low-value pages helps focus crawling on important content.

Still have questions?

We are here to help you get the most out of this tool.

Tap a tag to open a tool. The current page is highlighted.