Crawl Directive Reference
Complete syntax documentation, search engine extensions, and AI-optimized directives for precision content control.
Specifies which crawler the following rules apply to. Use * for all bots.
User-agent: *
Blocks crawlers from accessing specified paths or files. Paths are relative to the root.
Disallow: /api/v1/
# Blocks entire path & subdirectories
Explicitly permits crawling of a path that would otherwise be blocked. Overrides Disallow.
Allow: /docs/public/
Requests bots wait between requests (in seconds). Primarily respected by Yandex and friendly bots.
Crawl-delay: 2
Points to the absolute URL of your XML sitemap to accelerate indexing of important pages.
Prevents all images on the page from appearing in Google Image Search results.
Noimageindex: /gallery/
Controls how much of a page is shown as a preview when hovering over search results.
# Options: none, standard, large
Blocks Google from indexing pages matching the path. Alternative to meta noindex tags.
Noindex: /checkout/
Instructs Bing to ignore tracking parameters and consolidate duplicate URL variants.
Clean-param: utm_source|utm_medium
Controls access for ad-targeting crawlers. Separate from main web bots.
Disallow:
# Allow full access for ad data
Enables dynamic rule generation based on content freshness, traffic patterns, and server load.
Priority: high-value-content
Propagates rule updates to all CDN edge nodes instantly with zero-downtime deployment.
Rollback: auto
Advanced challenge-response for unverified crawlers. Routes them through a lightweight verification handshake.
Timeout: 5s
Best Practices & Compliance
Order Matters
Place specific directives before broad ones. Group by User-agent to avoid rule bleeding.
Test Before Pushing
Use Google Search Console or our Validator to verify path matching and bot targeting accuracy.
Keep It Lean
Large files slow down parsing. Combine overlapping paths and use wildcards strategically.
Bot Expectations
Malicious bots ignore robots.txt. Use it for SEO control, not security. Pair with WAF rules.
Directive Syntax Validator
Paste your robots.txt content to instantly verify syntax, detect conflicts, and ensure platform compatibility.