v3.2.0 Released — GraphQL Support Added

Build at Scale with
Robots.txt Developer Tools

Production-ready APIs, SDKs, CLI, and webhooks designed for modern engineering teams. Integrate intelligent crawl management and content orchestration into your workflow in minutes.

Try the Playground → Read the Docs 📖

Toolkit

Everything for Your Stack

From REST to GraphQL, serverless to edge functions. We've got you covered.

⚡

REST & GraphQL API

Fully typed endpoints for managing crawl directives, sitemap generation, and bot analytics. Idempotent, versioned, and rate-limited.

/api/v3

📦

Official SDKs

Native libraries for Node.js, Python, Go, Ruby, and PHP. Automatic retries, type hints, and first-class CI/CD integration.

npm install @robots.txt/sdk

💻

CLI & Terraform

Infrastructure-as-Code support. Deploy robots.txt configurations alongside your apps using our CLI or Terraform provider.

terraform init

🔔

Webhooks & Events

Subscribe to crawl events, rule changes, and indexing status updates. Reliable delivery with signature verification.

POST /webhooks

🧪

Interactive Playground

Test endpoints, validate JSON schemas, and preview generated rules without leaving your browser. No auth required for sandbox.

sandbox.robots.txt

🛡️

Rate Limits & Quotas

Transparent usage metrics, burst handling, and custom quota negotiation for enterprise workloads. Pay-per-request options.

X-RateLimit-Reset

Playground

See It in Action

Copy-paste ready examples. Switch languages and test live.

Connected to Sandbox

// Initialize the SDK import { RobotsClient } from '@robots.txt/sdk'; const client = new RobotsClient({ apiKey: process.env.ROBOTS_API_KEY, environment: 'production' }); // Generate optimized crawl rules const config = await client.rules.generate({ domain: 'example.com', strategy: 'seo-optimized', excludePaths: ['/api/v2', '/admin'] }); console.log(config.preview);

# Initialize the SDK from robots_txt import RobotsClient client = RobotsClient( api_key=os.getenv("ROBOTS_API_KEY"), environment="production" ) # Generate optimized crawl rules config = client.rules.generate( domain="example.com", strategy="seo-optimized", exclude_paths=["/api/v2", "/admin"] ) print(config.preview)

curl -X POST https://api.robots.txt/v3/rules/generate \n -H "Authorization: Bearer $ROBOTS_API_KEY" \n -H "Content-Type: application/json" \n -d '{ "domain": "example.com", "strategy": "seo-optimized", "excludePaths": ["/api/v2", "/admin"] }'

Quick Start

Get Started in 4 Minutes

From zero to production-ready crawl configuration.

Get Your API Key

Create a free account and generate a scoped key from the dashboard.

GET /api/v3/keys

Install the SDK

Add the package to your project using your preferred package manager.

npm i @robots.txt/sdk

Initialize & Configure

Point to your domain and let AI suggest optimal rules automatically.

client.rules.generate()

Deploy & Monitor

Push to production and watch real-time crawl analytics in the dashboard.

client.deploy({force: true})

Build at Scale with Robots.txt Developer Tools