Frequently Asked Questions

Robots.txt is an intelligent content curation platform that helps businesses manage, optimize, and secure their web crawling rules. Unlike static text files, our platform provides AI-driven insights, real-time analytics, and centralized control over how search engines and bots interact with your digital assets.

We offer multiple integration methods:

1. DNS CNAME: Point your root zone to our edge network.
2. API/CLI: Push rules programmatically using our SDK.
3. CI/CD Plugins: Automate deployments via GitHub Actions or GitLab CI.
4. WordPress Plugin: One-click setup for WP sites.

Detailed guides are available in our Documentation.

On the contrary, Robots.txt is designed to improve your SEO. By optimizing crawl budgets, preventing indexing of low-quality or duplicate content, and ensuring search engines prioritize your important pages, you help search bots index your site more efficiently. Our AI suggestions are aligned with current Google and Bing best practices.

Yes! Our Pro and Enterprise plans support multi-domain management. You can create projects for each domain, apply templates across subdomains, and view unified analytics for your entire portfolio.

Yes, we offer a Starter Plan for free. It supports up to 1 domain, basic rule management, and standard analytics. For advanced features like AI optimization, real-time alerts, and multi-domain support, check our Pricing Page.

If you cancel, your account downgrades to the Starter tier. You retain access to your basic configuration, but premium features like AI suggestions and advanced analytics will pause. Your data is retained for 30 days in case you wish to reactivate.

Absolutely. You can add custom lines to your generated file, including directives like Max-image-preview, Crawl-delay, and vendor-specific instructions. Our editor validates syntax in real-time to prevent errors.

Our AI analyzes your site structure, content hierarchy, and traffic patterns to suggest optimal crawl rules. It identifies pages that are wasting crawl budget, detects potential indexing conflicts, and recommends allow/disallow patterns based on your SEO goals.