How can we help?

To block a specific bot, add a User-agent directive followed by the bot's name, then use Disallow: /. For example:

User-agent: BadBot
Disallow: /

You can find the user-agent string for most crawlers in your server logs or through our Crawl Analytics dashboard.

Disallow tells crawlers they should not access specific paths. Allow explicitly permits access to a path, which can override a broader disallow rule.

For example:
Disallow: /private/
Allow: /public/

This blocks everything under /private/ but ensures /public/ remains accessible.

Changes to your robots.txt file are typically picked up by search engines within 24-48 hours. However, this varies by crawler. Googlebot often checks robots.txt before each crawl, so changes can take effect almost immediately for Google.

You can force a re-check in our dashboard by clicking "Notify Crawlers" under the Actions menu.

No. Robots.txt controls crawling, not indexing. If a page has already been indexed, disallowing it in robots.txt will prevent future crawling but may not remove it from search results.

To remove indexed content, use the noindex meta tag or HTTP header, or use the Search Console removal tool. Robots.txt is not a security feature and should not be used to hide sensitive data.

You can regenerate your API key from the Dashboard → Settings → API Keys. Click the "Regenerate" button next to the key you wish to replace. Note that this will immediately invalidate the old key.

If you've lost access to your account, please contact support at support@robotstxt.io.