ROBOTS.TXT GENERATOR

Robots.txt Generator

Create a custom robots.txt file to control search engine crawling

Crawling Policy:

Select how search engine bots should interact with your site

Disallowed Paths:

Enter one path per line. These directories/files will be blocked from crawling.

Sitemap URL (optional):

If you have a sitemap, provide its full URL here

Advanced Options

Additional Directives:

Add Crawl-Delay

User-Agent Specific Rules

Crawl Delay (seconds):

Time in seconds between requests from the same crawler

Specific User-Agent:

Generated robots.txt

Copied to clipboard!