Robots.txt
Robots.txt Generator
Preset Templates
Basic Settings
Add User-agent Rule
Usage Guide
Features
- Multi User-agent — Set different rules for different search engine crawlers
- Preset Templates — Apply common robots.txt configs with one click
- Batch Add — Quickly import multiple rules
- Syntax Validation — Auto-detect path format issues
- One-Click Export — Copy content or download robots.txt file
Common Directives
- User-agent — Name of the crawler the rule applies to
- Allow — Paths the crawler is allowed to access
- Disallow — Paths the crawler is not allowed to access
- Sitemap — Location of the site map
- Crawl-delay — Delay between crawler requests (seconds)