Robots.txt & XML Sitemap Generator Online Free
Compose a valid robots.txt with user-agent, allow/disallow paths, optional crawl delay, and a sitemap directive — and build a matching sitemap.xml listing your URLs with priority and change frequency — then copy to clipboard or download each file directly from your browser.
Robots.txt & XML Sitemap Generator In-Browser Workspace
Input data or select files to begin. 100% private, client-side execution with zero server transmission.
How to Use Robots.txt & XML Sitemap Generator
- Step 1: On the robots.txt tab, set the user-agent line (* for all bots or a specific crawler such as Googlebot).
- Step 2: Add Disallow paths to block (for example /admin/ or /api/) and Allow paths to permit, using the add/remove buttons.
- Step 3: Paste your absolute sitemap URL so crawlers can discover the sitemap reference from robots.txt.
- Step 4: Switch to the sitemap.xml tab and enter each page URL with its priority (0.0–1.0) and change frequency.
- Step 5: Use Copy or Download to save robots.txt and sitemap.xml into your site's root directory.
About Robots.txt & XML Sitemap Generator
robots.txt is a plain-text protocol file (the Robots Exclusion Protocol, originally proposed in 1994) served from a website's root that tells compliant crawlers which paths they may or may not request. Each rule pairs a user-agent pattern with Allow or Disallow directives; more specific rules override broader ones, and a trailing slash or asterisk in a path signals directory or wildcard matching. The Crawl-delay directive, supported by some crawlers, requests a minimum interval between requests to reduce server load — though Googlebot ignores it in favor of its own crawl budget logic. The Sitemap: line at the bottom points crawlers to your XML sitemap, which is the canonical discovery mechanism for URLs you want indexed. A sitemap.xml uses the sitemaps.org 0.9 namespace with one <url> element per page containing <loc> (an absolute, XML-escaped URL), optional <lastmod>, <changefreq> (always, hourly, daily, weekly, monthly, yearly, never), and <priority> (0.0 to 1.0, a hint rather than a ranking factor). Together the two files form the contract between your site and search engines: robots.txt governs access, sitemap.xml advertises inventory. This generator emits both with correct line breaks and XML headers, validates URLs client-side by filtering empties, and never transmits your data anywhere.
Frequently Asked Questions
Where do these files go on my server?
Both robots.txt and sitemap.xml must be served from the site root (e.g. https://example.com/robots.txt) because crawlers request them at that fixed path by convention.
Does Disallow prevent a page from being indexed?
Not reliably. Disallow blocks crawling, but a URL can still appear in search results if other sites link to it. To remove an indexed page, combine crawl controls with noindex or removal tools in Search Console.
Does the priority tag influence rankings?
Google ignores <priority> for ranking purposes; it is a hint other engines may use. Setting sensible values (1.0 for home, lower for utility pages) is still good practice for interoperability.
Is Crawl-delay supported by Google?
No. Googlebot does not honor Crawl-delay; Bing and some other crawlers do. The field here is optional and emitted only when you fill it in.