Welcome to Robots.txt Documentation

Overview

Robots.txt is an enterprise-grade platform that automates and optimizes how web crawlers interact with your digital assets. Instead of manually maintaining static robots.txt files across dozens of environments, our AI-driven engine dynamically generates, validates, and deploys crawl directives while protecting sensitive routes and maximizing indexing efficiency.

Quick Start

Get up and running in three simple steps:

1. Install the CLI

# Install via npm
npm install -g @robots-txt/cli

# Or via Homebrew (macOS/Linux)
brew tap robots-txt/cli
brew install robots-txt

2. Initialize your project

robots init my-project
# Creates robots.config.js with best-practice defaults

3. Deploy & Monitor

robots deploy --env production
# Live dashboard: https://app.robots.txt/dashboard/my-project

Core Directives

The platform supports all standard Robots Exclusion Protocol directives, plus several proprietary extensions for advanced control:

• User-agent - Target specific crawlers or use * for all agents
• Allow / Disallow - Path-based access control with regex support
• Crawl-delay - Throttle request rates (proprietary extension)
• Max-image-preview - Control thumbnail generation size
• Sitemap - XML sitemap discovery endpoints

Configuration Schema

module.exports = {
  userAgent: ["Googlebot", "Bingbot", "*"],
  rules: {
    "/api/*": "deny",
    "/private/*": "deny",
    "/public/*": "allow",
    "/blog/*": { allow: true, maxAge: 86400 }
  },
  crawlDelay: 2,
  sitemaps: ["https://example.com/sitemap.xml"],
  aiOptimization: true // Auto-tunes rules weekly
};

API Reference

Manage crawl policies programmatically via our REST API. All endpoints require Bearer token authentication.