Robots.txt Validator
Crawl Rules Syntax Checker & Simulator
Usage Guide
Features
- Rule Parsing — Identifies Allow / Disallow / Sitemap / Crawl-delay
- Syntax Highlighting — Color-coded display of directive types
- URL Test — Simulate whether a URL is allowed by the crawler
- Error Detection — Finds common syntax issues
Common Directives
- User-agent — Crawler name (* for all)
- Allow — Allowed paths
- Disallow — Blocked paths
- Sitemap — Sitemap file location
- Crawl-delay — Crawl interval (seconds)