Robots.txt Generator
Build a valid robots.txt that tells search engines which parts of your site to crawl.
How it works
Robots.txt is the first file well-behaved crawlers fetch from your domain — Googlebot reads it before crawling a single page. Each block starts with a User-agent line naming which crawler the rules apply to (* means everyone), followed by Allow and Disallow lines that are matched as path prefixes, so Disallow: /admin blocks /admin, /admin/users, and everything beneath it. Crawl-delay asks polite crawlers to pause between requests, and the Sitemap line points them at your XML sitemap for discovery. Two caveats: robots.txt is a request, not access control — malicious bots ignore it, so never use it to hide secrets — and the file must live at your domain root (e.g. example.com/robots.txt) to be found.